---
kind: concept
slug: surface-normal-estimation
title: Surface normal estimation
summary: >-
  Predicting a unit surface normal for every pixel of an image, or a normal rendered from a 3D model. Normal labels are scarce, so the main choices are whether you have normal labels or only depth labels, whether normals are predicted jointly with depth, and whether a pretrained normal model can supervise them. Two papers are covered.
tags: [surface-normals, monocular, zero-shot, depth-normal-consistency, normal-prior, gaussian-splatting]
papers: ["2404.15506", "2508.15376"]
related: [monocular-depth-estimation, foundation-model, vision-transformer, gaussian-splatting, novel-view-synthesis]
status: draft
updated_at: 2026-10-24
---

## Overview

Surface normal estimation predicts, for each pixel, the unit vector that points
out of the visible surface there [arxiv:2404.15506 §3.3] [arxiv:2404.15506 eq. 5].
Labelled normals are hard to get at scale. Good labels need dense 3D
reconstructions, and the authors report under 20K outdoor normal-labelled
images against about 9488K depth-labelled ones [arxiv:2404.15506 §3.3]. The
invariant the image-based methods below share is that a normal is normalized
to unit length after the network predicts it [arxiv:2404.15506 eq. 5].

A second family does not predict normals from a single image. It renders
normals from a 3D Gaussian scene and supervises them with a normal model that
is already trained [arxiv:2508.15376 §3.3] [arxiv:2508.15376 §3.4]. This
avoids normal labels altogether.

This page covers two papers that solve different problems: per-pixel normals
from one image, and rendered normals in a reconstructed driving scene. Their
numbers are not comparable, so read the tables with that in mind.

## Variants

### Direct supervision with normal labels

When ground-truth normals exist, the normal branch is trained with an
aleatoric-uncertainty-aware loss [arxiv:2404.15506 §3.4]. In-domain results are
reported for ScanNet with a ViT-g model: mean angular error 9.2° and median
5.0° [arxiv:2404.15506 tab. 2]. Card: [Metric3D v2](/papers/2404.15506).

### Depth-normal consistency for depth-only data

When a sample has no normal label, the normal is compared with a pseudo-normal
fitted by least squares to the predicted depth [arxiv:2404.15506 §3.4]. The
authors state that this term uses real-world depth, not the canonical depth
used in training [arxiv:2404.15506 §3.4]. This is how normals learn from the
larger depth-only pool [arxiv:2404.15506 §3.3]. Card: [Metric3D v2](/papers/2404.15506).

### Joint depth-normal refinement

Depth and normal are refined together over several recurrent ConvGRU steps,
and each step predicts additive updates for both [arxiv:2404.15506 §3.3]
[arxiv:2404.15506 eq. 2] [arxiv:2404.15506 eq. 3] [arxiv:2404.15506 eq. 4].
On DIODE outdoor, the normal median error is 14.91° with the joint model,
16.25° without the depth branch and 16.45° without the consistency term
[arxiv:2404.15506 tab. 11]. Card: [Metric3D v2](/papers/2404.15506).

### Pretrained normal prior on rendered normals

Here the normal is an output of a Gaussian splatting scene, not of a per-pixel
head. The training target is a normal map from a pretrained normal model, and
the loss is an L1 term plus a cosine-similarity term [arxiv:2508.15376 §3.4]
[arxiv:2508.15376 eq. 18]. The weight is 0.01 [arxiv:2508.15376 §4.2]
[arxiv:2508.15376 eq. 15]. The paper's own ablation drops the normal term and
reports the full model at PSNR 33.83 against 33.31 without it, with normal
MAE 0.84 against 1.21 [arxiv:2508.15376 tab. 8]. The card does not state
which dataset that table uses. Card: [DriveSplat](/papers/2508.15376).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Dense normal labels in your domain, e.g. indoor scenes | Direct supervision with an uncertainty-aware normal loss [arxiv:2404.15506 §3.4] | The paper's in-domain ScanNet result for ViT-g is a mean of 9.2° [arxiv:2404.15506 tab. 2]. It does not report an out-of-domain comparison for this setting. |
| Depth labels only, no normal labels, e.g. outdoor data | Depth-normal consistency loss, at weight 0.01 in the paper's total loss [arxiv:2404.15506 eq. 10] | Normal labels are scarce outdoors [arxiv:2404.15506 §3.3]. Removing the consistency term raised the DIODE outdoor median error from 14.91° to 16.45° [arxiv:2404.15506 tab. 11]. One benchmark only. |
| Zero-shot indoor use, no fine-tuning | Expect parity on the mean, not a clear win | On NYUv2 zero-shot, ViT-L has mean 13.1° and median 7.1°, the same mean as the Polymax baseline. Its gains are at the 11.25°, 22.5° and 30° thresholds [arxiv:2404.15506 tab. 2]. |
| Tight compute or memory budget | Check the cost of your model size first | The paper reports ViT-g at 5.0 fps and 15.6 GB GPU memory on an A100 40 GB, without TensorRT or ONNX optimization [arxiv:2404.15506 supp tab. 1]. It gives no separate numbers for ViT-S here. |
| Focal length unknown | This paper's method does not apply directly | The authors assume known focal lengths for training and test images [arxiv:2404.15506 §3.1]. The consistency term uses metric depth, which needs the true focal length [arxiv:2404.15506 §3.1]. |
| Normals needed inside a 3D reconstruction, no normal labels | Pretrained normal prior with an L1 plus cosine loss, weight 0.01 [arxiv:2508.15376 §3.4] [arxiv:2508.15376 eq. 15] | The paper's ablation keeps this term in its final model, and it lowers normal MAE and raises PSNR in its own table [arxiv:2508.15376 tab. 8]. A separate loss ablation gives PSNR 30.51 without the normal term and 30.96 with it [arxiv:2508.15376 tab. 10]. The two ablations use different numbers, so compare each within its own table. |
| Normals only from a single image, no 3D model | Image-based variants above | DriveSplat's normals exist only in its Gaussian scene [arxiv:2508.15376 §3.3]. |

## Implementation sketch

Only the normal-specific parts are shown. The depth branch and the joint
refinement loop are on [the depth page](/concepts/monocular-depth-estimation).

```pseudocode
# Normal head output: unit length per pixel  [arxiv:2404.15506 §3.3] [eq. 5]
def normal_output(raw_normal):
    return raw_normal / norm(raw_normal, axis=-1)  # unit vectors  [arxiv:2404.15506 eq. 5]
```

```pseudocode
# Normal loss, chosen by label availability  [arxiv:2404.15506 §3.4]
def normal_loss(pred_normal, gt_normal, pred_depth_metric):
    if gt_normal is not None:
        # Aleatoric-uncertainty-aware loss with ground-truth normals  [arxiv:2404.15506 §3.4]
        return aleatoric_normal_loss(pred_normal, gt_normal)
    # No normal labels: fit normals to the predicted real-world depth  [arxiv:2404.15506 §3.4]
    pseudo_normal = normal_from_depth_least_squares(pred_depth_metric)
    # Compare the predicted normal with the pseudo-normal  [arxiv:2404.15506 §3.4]
    return normal_consistency(pred_normal, pseudo_normal)
```

```pseudocode
# Normal prior loss on rendered normals, weight lambda_n = 0.01  [arxiv:2508.15376 eq. 15]
def rendered_normal_loss(render_normal, prior_normal):
    # L1 term between the splatted normal and the pretrained prior  [arxiv:2508.15376 §3.4]
    l1_term = abs(render_normal - prior_normal).mean()
    # Cosine term: rendered and prior normal should point the same way  [arxiv:2508.15376 eq. 18]
    cos_term = 1 - (render_normal * prior_normal).sum(axis=-1).mean()
    return l1_term + cos_term
```

## Contradictions

No direct cross-paper disagreement: the two papers do not measure the same
thing. Inside the papers, the released code and the text differ on
normal-specific values.

- Normal NLL weight in [Metric3D v2](/papers/2404.15506): 1.0 in the ViT-L
  config, 1.5 in the ViT-g general config [arxiv:2404.15506 eq. 10]
  [code: training/mono/configs/RAFTDecoder/vit.raft5.large.py#L40-L47]
  [code: training/mono/configs/RAFTDecoder/vit.raft5.giant2.py#L46-L47].
- Consistency weight in [Metric3D v2](/papers/2404.15506): 0.01 in the paper
  and the ViT-L config, 0.001 in the ViT-g NYU config [arxiv:2404.15506 eq. 10]
  [code: training/mono/configs/RAFTDecoder/vit.raft5.giant2.nyu.py#L23].
- Depth gradient in [Metric3D v2](/papers/2404.15506): the paper says the
  consistency term uses real-world depth [arxiv:2404.15506 §3.4], and the ViT-L
  and ViT-g configs set `depth_detach=True`, so the consistency gradient does
  not reach depth [code: training/mono/configs/RAFTDecoder/vit.raft5.large.py#L47].
  The paper does not mention the detach.
- Eq. 10 in [Metric3D v2](/papers/2404.15506) always adds the consistency term,
  while §3.4 presents it as the replacement for ground-truth normal supervision
  [arxiv:2404.15506 eq. 10] [arxiv:2404.15506 §3.4]. Pick one reading and say
  which you used.
- Cosine term in [DriveSplat](/papers/2508.15376): the paper says "cosine
  similarity loss" and gives no separate formula [arxiv:2508.15376 eq. 18]. The
  code uses 1 minus the dot product, after rescaling the prior from [0, 1] to
  [-1, 1] and rotating it by the camera rotation [code: train.py#L393-L394]
  [code: train.py#L403-L411].

## Papers

- [Metric3D v2](/papers/2404.15506): joint metric depth and normal model with a depth-normal consistency loss for depth-only data.
- [DriveSplat](/papers/2508.15376): Gaussian scene reconstruction for driving that supervises rendered normals with a pretrained normal prior.