Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation

In plain words

A single photo does not tell you how far away things are. A telephoto shot of a small object and a wide-angle shot of a large one can look the same, so the camera's focal length is the missing piece. This paper's models estimate both the distance to every pixel and the direction each surface faces, in real units, from one image.

The trick is to make the training data look as if every photo came from one standard camera. The photos are resized or the distance labels are rescaled to that camera. The network then learns one consistent notion of scale. At use time the model's output is scaled back to the real camera.

Surface direction is harder to label at scale, because good labels need dense 3D reconstructions. The authors let the direction estimate learn from the distance estimate: the two are refined together in a loop, and a consistency term ties them. That lets the direction part use the much larger pool of distance-only data.

TL;DR

• Predicts metric depth (in metres) and per-pixel surface normals from one RGB image, assuming the focal length is known §3.1.
• Canonical camera transform: training images or depth labels are rescaled to one canonical focal length, so one model handles many camera models §3.2.
• Depth and normal are refined jointly with recurrent ConvGRU updates, and a depth-normal consistency loss lets normals learn from depth-only data §3.3 §3.4.
• Author-reported: strong zero-shot results on NYUv2 and KITTI, and state of the art on many depth and normal benchmarks §1 tab. 1 tab. 2 tab. 4.

Problem

Metric depth models trained on one dataset's camera do not transfer to other cameras. Affine-invariant depth models generalize better but cannot give real distances §1. Normal estimators have little zero-shot ability because outdoor normal labels are scarce; the authors report under 20K outdoor normal-labelled images against about 9488K depth-labelled ones §3.3. Because the focal length is unknown, the same image can correspond to different distances §3.1 fig. 6 fig. 7. The authors assume known focal lengths for training and test images §3.1. Sensor size does not change metric depth, but focal length does §3.1.

Method

Pipeline overview fig. 4 fig. 9:

1. Canonical camera transform §3.2. Two variants. In CSTM_label, the ground-truth depth is multiplied by \omega_d = f^c / f and the image is unchanged §3.2. In CSTM_image, the image is resized by \omega_r = f^c / f and the depth label is resized spatially without value scaling §3.2. The canonical focal length is f^c = 1000 px §4 [supp §1.2]. A random patch crop follows; it changes only the field of view and principal point §3.2.
2. Encoder and decoder §4 [supp §1]. Two families are described: a ConvNeXt-L encoder (ImageNet-22K) with an AdaBins-style decoder and hourglass head, and DINOv2-with-registers ViTs with DPT decoders. The decoder outputs a low-resolution depth \hat{D}_c and an unnormalized normal \hat{N}_u at H/4 \times W/4, plus hidden features §3.3.
3. Joint recurrent refinement §3.3 fig. 10. Starting from the decoder's outputs, T+1 steps update hidden features with ConvGRU sub-blocks eq. 2 eq. 4. Two projection heads predict additive updates for depth and normal from the hidden state eq. 3 eq. 4. The paper gives 4 steps for ViT-S and 8 for ViT-L and ViT-g [supp tab. 1].
4. Output §3.3 eq. 5. After upsampling, depth passes through a ReLU to keep it non-negative, and normals are normalized to unit length.
5. De-canonical transform §3.2. In the label variant the predicted depth is divided by \omega_d. In the image variant the prediction is resized back to the original size without scaling.

Losses §3.4:

• Depth: L_d = L_{PWN} + L_{VNL} + L_{silog} + L_{RPNL} eq. 9. PWN is the pair-wise normal regression loss, VNL the virtual normal loss, and silog the scale-invariant log loss §3.4.
• Random proposal normalization loss (RPNL), the paper's main depth addition eq. 8. It takes M random patches, each with a side between 0.125 and 0.5 of the image, normalizes the ground-truth and predicted patch values by their median and mean absolute deviation, and averages the absolute difference. The paper sets M = 32 §3.4.
• Normal: when ground-truth normals exist, an aleatoric-uncertainty-aware loss is used §3.4. Otherwise a depth-normal consistency loss compares the predicted normal with a pseudo-normal from the predicted depth, computed by least squares §3.4. The consistency term uses real-world depth, not the canonical depth §3.4.
• Total: L = w_d L_d + w_n L_n + w_{d-n} L_{d-n} with w_d = 0.5, w_n = 1, w_{d-n} = 0.01 eq. 10.

# Canonical camera transform, label variant  [§3.2]
def to_canonical_label(depth_gt, fx, canonical_focal=1000.0):
    ratio = canonical_focal / fx     # omega_d = f^c / f  [§3.2]
    depth_c = ratio * depth_gt       # supervise in canonical space  [§3.2]
    return depth_c                   # image is unchanged  [§3.2]

# Canonical camera transform, image variant  [§3.2]
def to_canonical_image(image, depth_gt, fx, canonical_focal=1000.0):
    ratio = canonical_focal / fx     # omega_r = f^c / f  [§3.2]
    image_c = resize(image, ratio)   # principal point scales too  [§3.2]
    depth_c = resize(depth_gt, ratio)  # resize only, no value scaling  [§3.2]
    return image_c, depth_c

# Joint depth-normal refinement, T+1 steps  [§3.3] [eq. 2] [eq. 3] [eq. 4]
# Input:  depth0 (H/4, W/4), normal0 (H/4, W/4, 3), hidden0 (decoder)
# Output: depth (H/4, W/4), normal (H/4, W/4, 3), unit-length after upsample

def refine_depth_and_normal(depth0, normal0, hidden0, steps):
    depth = depth0
    normal = normal0
    hidden = hidden0
    for step in range(steps):
        # ConvGRU updates the hidden state from the current predictions [eq. 4]
        hidden = conv_gru(depth, normal, hidden0, hidden)
        # Two projection heads predict additive updates from hidden [eq. 4]
        delta_depth = head_depth(hidden)
        delta_normal = head_normal(hidden)
        # Apply the updates additively  [eq. 3]
        depth = depth + delta_depth
        normal = normal + delta_normal
    return depth, normal

# Random proposal normalization loss  [§3.4] [eq. 8]
# M = 32 proposals; crop side 0.125 to 0.5 of the image  [§3.4]

def rpnl(pred, gt, num_proposals=32):
    total = 0.0
    for patch in random_patches(pred, gt, num_proposals):
        # Median-centre and divide by mean absolute deviation  [eq. 8]
        gt_norm = normalize_by_mad(gt[patch])
        pred_norm = normalize_by_mad(pred[patch])
        # Mean absolute difference over the patch's pixels  [eq. 8]
        total = total + mean(abs(gt_norm - pred_norm))
    return total / num_proposals

# Total training loss  [eq. 10]
# Normal term: aleatoric NLL with GT normals, else consistency  [§3.4]

def training_loss(depth_c, normal, gt_depth_c, gt_normal, depth_metric):
    loss_depth = depth_loss(depth_c, gt_depth_c)     # eq. 9  [§3.4]
    if gt_normal is not None:
        loss_normal = normal_nll(normal, gt_normal)  # [§3.4]
    else:
        loss_normal = 0.0
    loss_consistency = depth_normal_consistency(depth_metric, normal)
    # Weights w_d, w_n, w_{d-n} as stated in the paper  [eq. 10]
    return 0.5 * loss_depth + 1.0 * loss_normal + 0.01 * loss_consistency

Hyperparameters

Values in the code are from the released training configs at code_commit. Paper values come from the main text (§4) and supplement ([supp §1]).

Results

Author-reported. Setup: models trained on about 16M images §4. The zero-shot rows use one checkpoint for all benchmarks §4. Table 1 is metric depth with no scale alignment §4. Table 2 is normals. Table 4 is affine-invariant depth, where the paper aligns scale and shift per image before scoring §4. The affine-invariant results are not comparable with the metric results.

Metric depth, zero-shot (ZS) and fine-tuned (FT), subset tab. 1:

Surface normals on NYUv2, zero-shot (ZS) and fine-tuned (FT), subset tab. 2:

Affine-invariant depth, five zero-shot sets, AbsRel / δ1, subset tab. 4:

ScanNet is not zero-shot for the ViT models, so those entries are struck out tab. 4.

Ablations on the training set, zero-shot AbsRel, subset:

KITTI dense SLAM (Droid-SLAM with predicted depth), translational RMS drift on sequence 00, lower is better: Droid 33.9% and Droid+Ours 1.44% tab. 7. Scene reconstruction on 9 NYUv2 scenes is in tab. 6. The full tables are in the paper; this card lists the headline rows.

Paper vs code

Searched: the paper's source (summary_of_changes.tex and rebuttal.tex) links github.com/YvanYin/Metric3D. The owner matches the lead author. The clone is at code_commit. Weights are not in the clone, so checkpoints were not checked. The configs read are under training/mono/configs/RAFTDecoder/ (training) and mono/ (inference).

• Learning rate. Paper: "initial learning rate 0.0001 for all layers" §4. Code: encoder 1e-5 (ViT-S and ViT-L) or 8e-6 (ViT-g), decoder 1e-4 vit.raft5.large.py vit.raft5.giant2.py.
• Batch size. Paper: 192 on 48 A100 GPUs §4. Code: per-GPU batch 4 (ViT-L) vit.raft5.large.py, 3 (ViT-g) vit.raft5.giant2.py, 6 (ViT-S) vit.raft5.small.py. The GPU count is not in the configs, so the total batch cannot be checked. 4 × 48 = 192 fits the ViT-L config.
• Horizontal flip. Paper: 50% §4. Code: prob 0.4 in each dataset pipeline, for example vit.raft5.large.py.
• Loss weights. Paper: w_d = 0.5, w_n = 1, w_{d-n} = 0.01 eq. 10. Released ViT-L config: VNL 1.0, GRU sequence 0.5, normal NLL 1.0, consistency 0.01 with depth_detach=True vit.raft5.large.py. ViT-g NYU config: VNL 1.0, GRU sequence 1.0, normal NLL 1.5, consistency 0.001 vit.raft5.giant2.nyu.py vit.raft5.giant2.nyu.py. The ViT-g general config has normal NLL 1.5 and consistency 0.01 vit.raft5.giant2.py. The configs also add a GRU sequence loss and a sky regularizer that eq. 9 does not list.
• Silog loss. Paper: L_{silog} is part of L_d eq. 9. None of the RAFTDecoder configs list a silog loss in their decoder or GRU loss lists vit.raft5.large.py. The supplement says silog is turned off for MapillaryPSD only [supp §1.1].
• RPNL. Paper: random spatial patches, side 0.125 to 0.5 of the image, M = 32 §3.4. Code: the loss named HDNRandomLoss samples depth-value intervals instead of image patches. The minimum is drawn from [0, 0.75] of the depth range and the maximum at least 0.1 above it HDNL_random.py. It normalizes by the median and by the mean absolute deviation, which matches the denominator of eq. 8. The class default is 32 HDNL_random.py. The configs set 10 (ViT-L) vit.raft5.large.py and 10 (ViT-g NYU) vit.raft5.giant2.nyu.py. A second loss, HDSNRandomLoss, uses 20 vit.raft5.large.py.
• Depth-normal consistency. Paper: uses real-world depth, not canonical depth §3.4. Code: depth_detach=True in the ViT-L and ViT-g configs vit.raft5.large.py vit.raft5.giant2.py, so the consistency gradient does not reach depth. The paper does not mention this.
• Canonical ratio. Training uses f_x only transform.py. Inference uses the mean of f_x and f_y do_test.py. The paper assumes f^c_x = f^c_y §3.2.
• Depth at inference. Output is divided by the label scale factor do_test.py and clipped to (0, 300) m do_test.py. The paper does not mention the clip.
• Fine-tuning. Paper: 20K steps on 8 GPUs §4; supplement: Adam, LR from 1e-6 to 1e-7 over 6K steps, converges in about 2K [supp §1.2]. Code: AdamW, 20,010 iterations, encoder LR 5e-7, decoder LR 1e-5, 2 per GPU vit.raft5.giant2.nyu.py.
• ConvNeXt decoder. Main text: UNet §4. Supplement: AdaBins-style decoder with 256 bins plus an hourglass head [supp §1]. Code: HourglassDecoder config with depth range (0.3, 150) and crop 544 × 1216 convlarge.0.3_150.py.
• Warmup. Not stated in the paper. Code gives 500 or 1000 iterations vit.raft5.large.py.

The RPNL sampler draws from numpy's global RNG (np.random.uniform) with no seed set in the loss HDNL_random.py, so runs are not reproducible unless the caller seeds numpy.

Pitfalls

• Scale is only as good as the focal length. Metric output needs the true f §3.1. The paper's zero-shot claims use dataset intrinsics. Image-only inputs with guessed focal lengths are a different setting, and the paper uses guessed values only for qualitative in-the-wild results §4.
• Eq. 10 vs §3.4. Eq. 10 always adds w_{d-n} L_{d-n}. §3.4 introduces the consistency loss as the replacement for GT-normal supervision. Pick one reading and say which you used.
• RPNL is ambiguous. The paper defines patch crops; the code uses depth intervals. The two give different gradients. Implement the one you can justify and note which.
• Learning rate. Paper says 1e-4 for all layers; code uses 10× lower encoder LR. Using 1e-4 everywhere is not the released setting.
• Loss weights differ across configs. The normal NLL weight is 1.0 in some configs and 1.5 in others; the consistency weight is 0.001 or 0.01. The paper's single set of weights does not match any one config.
• Canonical focal is 1000 px in every config, but the set of tested values differs (2500 in the main text, 2000 in the supplement).
• Canonical transform changes the numerical range. With the label variant the network's depth is in canonical units, so its raw output is not metres. Use the de-canonical step on every output path.
• Clamp and bins. The ViT head clamps depth to the normalization bounds RAFTDepthNormalDPTDecoder5.py. The bounds and bin counts for ViT and ConvNeXt differ, so do not reuse one setting for both.
• Datasets. The paper gives 16 sets §4 in one place and 18 in another [supp §1.1]. The number of images also varies by model (8M for ConvNeXt, 16M for ViT). Treat the exact dataset list as unresolved.
• Hidden costs. ViT-g runs at 5.0 fps and uses 15.6 GB GPU memory on an A100 40 GB, with no TensorRT or ONNX optimization [supp tab. 1]. The ViT models use bfloat16 for speed [supp tab. 1].
• Affine-invariant results are not metric. The affine benchmarks align scale and shift per image, so the numbers in tab. 4 are not comparable with the metric numbers in tab. 1.
• Fine-tuning needs pseudo-labels. The paper's fine-tune loss uses the pre-fine-tuned model's outputs as extra labels [supp §1.2]. Without them, the annotation-absent regions get no supervision.

Lineage

• Builds on: canonical pose-space normalization in human reconstruction §1; RAFT-style iterative refinement, which the joint depth-normal module adapts to normals §3.3; scale-shift invariant losses from MiDaS and LeReS §3.4; DINOv2 with registers and DPT decoders §4; ConvNeXt-L §4; AdaBins decoder [supp §1].
• Compared against: CamConvs, which encodes the camera in the network, and SSIL, the scale-shift invariant loss, in the ablations tab. 9 tab. 10. Metric baselines include ZoeDepth and ZeroDepth tab. 1. Normal baselines include Bae et al. and Polymax tab. 2.
• Superseded by: not known from content already in this wiki.