Surface normal estimation

Overview

Surface normal estimation predicts, for each pixel, the unit vector that points out of the visible surface there 2404.15506 §3.3 2404.15506 eq. 5. Labelled normals are hard to get at scale. Good labels need dense 3D reconstructions, and the authors report under 20K outdoor normal-labelled images against about 9488K depth-labelled ones 2404.15506 §3.3. The invariant the image-based methods below share is that a normal is normalized to unit length after the network predicts it 2404.15506 eq. 5.

A second family does not predict normals from a single image. It renders normals from a 3D Gaussian scene and supervises them with a normal model that is already trained 2508.15376 §3.3 2508.15376 §3.4. This avoids normal labels altogether.

This page covers two papers that solve different problems: per-pixel normals from one image, and rendered normals in a reconstructed driving scene. Their numbers are not comparable, so read the tables with that in mind.

Variants

Direct supervision with normal labels

When ground-truth normals exist, the normal branch is trained with an aleatoric-uncertainty-aware loss 2404.15506 §3.4. In-domain results are reported for ScanNet with a ViT-g model: mean angular error 9.2° and median 5.0° 2404.15506 tab. 2. Card: Metric3D v2.

Depth-normal consistency for depth-only data

When a sample has no normal label, the normal is compared with a pseudo-normal fitted by least squares to the predicted depth 2404.15506 §3.4. The authors state that this term uses real-world depth, not the canonical depth used in training 2404.15506 §3.4. This is how normals learn from the larger depth-only pool 2404.15506 §3.3. Card: Metric3D v2.

Joint depth-normal refinement

Depth and normal are refined together over several recurrent ConvGRU steps, and each step predicts additive updates for both 2404.15506 §3.3 2404.15506 eq. 2 2404.15506 eq. 3 2404.15506 eq. 4. On DIODE outdoor, the normal median error is 14.91° with the joint model, 16.25° without the depth branch and 16.45° without the consistency term 2404.15506 tab. 11. Card: Metric3D v2.

Pretrained normal prior on rendered normals

Here the normal is an output of a Gaussian splatting scene, not of a per-pixel head. The training target is a normal map from a pretrained normal model, and the loss is an L1 term plus a cosine-similarity term 2508.15376 §3.4 2508.15376 eq. 18. The weight is 0.01 2508.15376 §4.2 2508.15376 eq. 15. The paper's own ablation drops the normal term and reports the full model at PSNR 33.83 against 33.31 without it, with normal MAE 0.84 against 1.21 2508.15376 tab. 8. The card does not state which dataset that table uses. Card: DriveSplat.

When to use which

Implementation sketch

Only the normal-specific parts are shown. The depth branch and the joint refinement loop are on the depth page.

# Normal head output: unit length per pixel  [arxiv:2404.15506 §3.3] [eq. 5]
def normal_output(raw_normal):
    return raw_normal / norm(raw_normal, axis=-1)  # unit vectors  [arxiv:2404.15506 eq. 5]

# Normal loss, chosen by label availability  [arxiv:2404.15506 §3.4]
def normal_loss(pred_normal, gt_normal, pred_depth_metric):
    if gt_normal is not None:
        # Aleatoric-uncertainty-aware loss with ground-truth normals  [arxiv:2404.15506 §3.4]
        return aleatoric_normal_loss(pred_normal, gt_normal)
    # No normal labels: fit normals to the predicted real-world depth  [arxiv:2404.15506 §3.4]
    pseudo_normal = normal_from_depth_least_squares(pred_depth_metric)
    # Compare the predicted normal with the pseudo-normal  [arxiv:2404.15506 §3.4]
    return normal_consistency(pred_normal, pseudo_normal)

# Normal prior loss on rendered normals, weight lambda_n = 0.01  [arxiv:2508.15376 eq. 15]
def rendered_normal_loss(render_normal, prior_normal):
    # L1 term between the splatted normal and the pretrained prior  [arxiv:2508.15376 §3.4]
    l1_term = abs(render_normal - prior_normal).mean()
    # Cosine term: rendered and prior normal should point the same way  [arxiv:2508.15376 eq. 18]
    cos_term = 1 - (render_normal * prior_normal).sum(axis=-1).mean()
    return l1_term + cos_term

Contradictions

No direct cross-paper disagreement: the two papers do not measure the same thing. Inside the papers, the released code and the text differ on normal-specific values.

• Normal NLL weight in Metric3D v2: 1.0 in the ViT-L config, 1.5 in the ViT-g general config 2404.15506 eq. 10 vit.raft5.large.py vit.raft5.giant2.py.
• Consistency weight in Metric3D v2: 0.01 in the paper and the ViT-L config, 0.001 in the ViT-g NYU config 2404.15506 eq. 10 vit.raft5.giant2.nyu.py.
• Depth gradient in Metric3D v2: the paper says the consistency term uses real-world depth 2404.15506 §3.4, and the ViT-L and ViT-g configs set depth_detach=True, so the consistency gradient does not reach depth vit.raft5.large.py. The paper does not mention the detach.
• Eq. 10 in Metric3D v2 always adds the consistency term, while §3.4 presents it as the replacement for ground-truth normal supervision 2404.15506 eq. 10 2404.15506 §3.4. Pick one reading and say which you used.
• Cosine term in DriveSplat: the paper says "cosine similarity loss" and gives no separate formula 2508.15376 eq. 18. The code uses 1 minus the dot product, after rescaling the prior from [0, 1] to [-1, 1] and rotating it by the camera rotation train.py train.py.

Papers

• Metric3D v2: joint metric depth and normal model with a depth-normal consistency loss for depth-only data.
• DriveSplat: Gaussian scene reconstruction for driving that supervises rendered normals with a pretrained normal prior.