---
kind: concept
slug: monocular-depth-estimation
title: Monocular depth estimation
summary: >-
  Predicting a value for every pixel of one RGB image: scene depth, or height in aerial imagery. Variants differ in training data, output head, loss and scoring. Omnidata, Marigold and HR-Depth report aligned depth; AdaBins, BinsFormer, NeW CRFs and Metric3D v2 report absolute errors on NYU and KITTI. The best-known Omnidata data and weights are non-commercial only.
tags: [computer-vision, depth-estimation, dense-prediction, transformer, synthetic-data, remote-sensing, diffusion, self-supervised, transparent-surfaces]
papers: ["1912.08804", "2110.04994", "2011.14141", "2204.00987", "2203.01502", "2304.07213", "2012.09856", "2312.00206", "2312.02145", "2106.10823", "2404.15506", "2103.01100", "2508.15376", "2104.00681", "2312.00451", "2406.01584", "2012.07356", "2408.06083"]
related: [self-supervised-learning, vision-transformer, object-queries, conditional-random-field, self-attention, differentiable-rendering, dataset-curation, multi-task-learning, data-augmentation, semantic-segmentation, gaussian-splatting, novel-view-synthesis, object-detection, bev-representation, foundation-model, focal-loss, surface-normal-estimation, latent-diffusion, noise-schedule, ensembles, generative-adversarial-network, multi-view-reconstruction, neural-implicit-surface, recurrent-neural-network, multimodal-language-model, visual-grounding, 3d-scene-graph, channel-attention, knowledge-distillation, variational-autoencoder]
status: draft
updated_at: 2026-11-01
---

## Overview

Monocular depth estimation takes one RGB image and predicts a value for each
pixel, most often the distance from the camera. Nine cards on this slug predict
a dense per-pixel map from one image [arxiv:2110.04994 §4.3]
[arxiv:2011.14141 §3.3] [arxiv:2204.00987 §3.1] [arxiv:2203.01502 §3.4]
[arxiv:2404.15506 §3.3] [arxiv:2304.07213 §3.2] [arxiv:2312.02145 §3.1]
[arxiv:2012.07356 §3.1] [arxiv:2408.06083 §3.1]. They differ in target and in how
the per-pixel value is produced. Eight predict scene depth: a DPT-Hybrid, a CNN
encoder with a small transformer head, a transformer decoder, a windowed CRF
decoder, a ViT with recurrent refinement that also predicts surface normals, a
diffusion model that denoises a depth latent, a self-supervised U-Net-style
decoder trained from video [arxiv:2012.07356 §3.3], and a fine-tuned Depth
Anything V2 Large [arxiv:2408.06083 §4.2]. One predicts tree height from
satellite photos, with a frozen transformer and a DPT decoder. Nine more cards use
depth inside a larger pipeline, or replace per-frame depth with something else: a
single-image view synthesis model whose depth network is trained only through
rendering [arxiv:1912.08804 §3.1], a hand-object fit [arxiv:2012.09856 §3.2], two
few-view Gaussian splatting methods that compare rendered depth with a monocular
prior [arxiv:2312.00206 §3.2] [arxiv:2312.00451 §3.3], a driving-scene Gaussian
method that takes depth and normal priors from pretrained models
[arxiv:2508.15376 §3.4], a vision-language model that takes a relative-depth map as
an extra input [arxiv:2406.01584 §3.3], a real-time reconstruction network that
predicts a TSDF per fragment instead of a depth map per frame [arxiv:2104.00681 §1],
a 3D detection survey whose image-based methods estimate depth and lift it to a
point cloud [arxiv:2106.10823 §3.1.2], and a monocular 3D detector that learns a
depth distribution per pixel [arxiv:2103.01100 §3.1].

[Omnidata](/papers/2110.04994) builds a rendering pipeline for labels and trains
a depth model on its output. [AdaBins](/papers/2011.14141) also predicts bins for
each image, but from a small transformer that reads the decoder's features rather
than from learned queries. [BinsFormer](/papers/2204.00987) predicts depth as
a probability-weighted mix of bin centres that are generated for each image.
[NeW CRFs](/papers/2203.01502) refines a coarse depth guess with
fully connected CRF energies computed inside small windows of patches.
[Canopy height](/papers/2304.07213) borrows the depth literature's log-scale
loss for a height map. [Metric3D v2](/papers/2404.15506) instead rescales its
training images to one focal length, so that its output is in metres.
[CaDDN](/papers/2103.01100) does not output a depth map for evaluation; it turns
a per-pixel depth distribution into features for a detector. AdaBins, BinsFormer
and NeW CRFs use the same scale-invariant loss family [arxiv:2011.14141 §3.4]
[arxiv:2204.00987 §3.2.3] [arxiv:2203.01502 §3.4]. Metric3D v2 includes a
scale-invariant log term among its depth losses [arxiv:2404.15506 eq. 9]. The
canopy numbers are not depth numbers, so it is compared only in the table below
and not directly against the others.

Omnidata and BinsFormer both report depth on NYU Depth v2, but with different
metrics. Omnidata's NYU error is aligned in scale and shift before scoring
[arxiv:2110.04994 §4.3]. Marigold's zero-shot errors are also aligned, by least
squares [arxiv:2312.02145 §4.2]. BinsFormer's NYU error is absolute
[arxiv:2204.00987 §4.1.2]. NeW CRFs also reports absolute error on NYU
[arxiv:2203.01502 tab. 3]. AdaBins reports absolute error on NYU, up to 10 m
[arxiv:2011.14141 §4.1]. Do not rank Omnidata or Marigold against the
absolute-error papers.

## Variants

### Depth from a DPT-Hybrid trained on mixed-source data

The depth model is a DPT-Hybrid with a setup similar to MiDaS v3.0. It is
trained on a five-dataset part of the starter set with scale- and
shift-invariant losses [arxiv:2110.04994 §4.3]. Evaluation is zero-shot, with
each prediction first aligned to the ground truth by one scale and one shift,
in inverse-depth space [arxiv:2110.04994 §4.3]. On the authors' table, the
starter-trained model has a lower L1 error than MiDaS v3 on NYU (0.2878 against
0.3838) and on the OASIS validation split (0.7901 against 0.8057). These are
author-reported [arxiv:2110.04994 tab. 2]. The card: [Omnidata](/papers/2110.04994).

### Labels from rendered 3D scans

Omnidata places cameras and points on a textured mesh, renders a view for each
clear camera-point pair, and labels each view with 21 mid-level cues [arxiv:2110.04994 §3.1].
By default, a point must be seen by at least three cameras to be kept
[arxiv:2110.04994 §3.1]. The starter set has 14,601,449 images from 2,414
spaces, built from seven mesh-based sources [arxiv:2110.04994 §4.1] [arxiv:2110.04994 §4.2].
A label takes 1 to 4 seconds on a CPU at 512×512 and can be run in parallel
[arxiv:2110.04994 §3.1]. Label accuracy is capped by the scan: the authors
cite up to 2 cm error for one source [arxiv:2110.04994 §6]. The cue-generation
chain is described only in supplementary material [arxiv:2110.04994 §3.1].

### Adaptive bins from a mini-ViT head on an EfficientNet encoder

AdaBins uses an EfficientNet-B5 encoder and a standard upsampling decoder, which
produce features at half the input resolution [arxiv:2011.14141 §3.2]
[arxiv:2011.14141 §3.3]. A small vision transformer, the mini-ViT, reads those
features in 16×16 patches [arxiv:2011.14141 tab. 1]. It outputs the widths of N
bins for the image and C = 128 range-attention maps [arxiv:2011.14141 §3.3]. The
widths are normalised and spread over the depth range, and each bin's centre is
its midpoint [arxiv:2011.14141 eq. 1] [arxiv:2011.14141 eq. 2]. Each pixel gets a
softmax over the bins from the range-attention maps, and its depth is the
probability-weighted sum of the centres [arxiv:2011.14141 eq. 3]. The paper says
this hybrid regression avoids the stepped output of picking the most likely bin
[arxiv:2011.14141 §3.3] [fig. 5]. The final model uses N = 256 bins
[arxiv:2011.14141 §4.4].

Training combines a scaled scale-invariant loss on log depth with a Chamfer term
that pulls the bin centres towards ground-truth depths, weighted 0.1
[arxiv:2011.14141 §3.4] [arxiv:2011.14141 eq. 4] [arxiv:2011.14141 eq. 5]
[arxiv:2011.14141 eq. 6].

On the paper's NYU ablation with the same base model, per-image adaptive bins
give δ1 0.903 and RMS 0.364. Fixed uniform, log-scale and trained bin widths give
δ1 0.892, 0.896 and 0.893, with RMS 0.383, 0.379 and 0.381. Standard regression
with no bins gives δ1 0.881 [arxiv:2011.14141 tab. 6]. Author-reported: on NYU,
Abs Rel 0.103 and δ1 0.903 [arxiv:2011.14141 tab. 2]; on the KITTI Eigen split,
Abs Rel 0.058 and δ1 0.964 [arxiv:2011.14141 tab. 3]. Trained on NYU and tested on
SUN RGB-D with no fine-tuning, δ1 is 0.771, above BTS at 0.740
[arxiv:2011.14141 tab. 5]. The mini-ViT is about 5.8 M parameters of a model of
about 78 M [arxiv:2011.14141 tab. 1] [arxiv:2011.14141 §4.2]. The paper gives no
runtime or memory figure for the head [arxiv:2011.14141 §4.2]. The card:
[AdaBins](/papers/2011.14141). Its official code is GPL-3.0 (see the card's
licence fields), so implement the method from the paper's equations.

### Adaptive bins from a Transformer decoder

BinsFormer generates N bins for each image from learned queries in a
Transformer decoder, with N = 64 by default [arxiv:2204.00987 §3.2.2]
[arxiv:2204.00987 tab. 6]. Each pixel gets a softmax over the bins, and its
depth is the sum of bin centre times probability [arxiv:2204.00987 §3.2.3]
[arxiv:2204.00987 eq. 2]. The bin bounds are the dataset's minimum and maximum
valid depth [arxiv:2204.00987 §3.2.3]. An extra scene query classifies the
scene, and the bins are refined at three feature scales [arxiv:2204.00987 §3.2.4]
[arxiv:2204.00987 §3.2.5]. On NYU Depth v2 with a Swin-Large backbone
pretrained on ImageNet-22K, δ1 is 0.925 and RMS is 0.330, against 0.903 and
0.364 for AdaBins [arxiv:2204.00987 tab. 3]. On the KITTI Eigen split with the
same backbone, δ1 is 0.974 and RMS is 2.098, against 0.964 and 2.360 for AdaBins
[arxiv:2204.00987 tab. 1]. These are author-reported. The ResNet-18 variant is
below AdaBins on NYU, with δ1 0.866 [arxiv:2204.00987 tab. 3]. The bin-count
ablation shows the gain saturating above 64 bins [arxiv:2204.00987 tab. 6]. The
card: [BinsFormer](/papers/2204.00987).

### Windowed fully connected CRF refinement on a Swin encoder

A Swin transformer encodes the image at four scales. The encoder size is not
stated in the paper [arxiv:2203.01502 §3.4]. A pyramid pooling head makes a
coarse depth guess from the top feature [arxiv:2203.01502 §3.4]. Four decoder
levels then refine it. At each level, the image is cut into non-overlapping
7×7 windows of patches, and every patch is a node that can exchange messages
with the others in its window [arxiv:2203.01502 §3.2]. The pairwise weights
are learned with multi-head attention, not set by hand as in classic CRFs
[arxiv:2203.01502 §3.3] [arxiv:2203.01502 eq. 9]. A second step uses windows
shifted by half a window, so that neighbours across a border also connect
[arxiv:2203.01502 §3.2] [arxiv:2203.01502 §3.4]. Head counts run 32, 16, 8, 4
from coarse to fine [arxiv:2203.01502 §3.4]. The loss is the scale-invariant
log loss with variance factor 0.85 and scale 10 [arxiv:2203.01502 §3.4].

Author-reported, on the KITTI Eigen split: Abs Rel 0.052 and RMSE 2.129,
against 0.058 and 2.360 for AdaBins [arxiv:2203.01502 tab. 1]. On NYU Depth v2:
Abs Rel 0.095 and RMSE 0.334, against 0.103 and 0.364 for AdaBins
[arxiv:2203.01502 tab. 3]. The paper's ablation on the same KITTI split moves
Abs Rel from 0.069 for a UNet decoder to 0.055 with the neural CRF and 0.052 with
the full model [arxiv:2203.01502 tab. 5]. The same perspective network is run
directly on 1024×512 equirectangular panoramas, with Abs Rel 0.0906 on
Matterport3D [arxiv:2203.01502 §4.2] [arxiv:2203.01502 tab. 4]. A variant with
50K extra pretraining images, whose source the paper does not state, reaches
0.0793 [arxiv:2203.01502 tab. 4]. The decoder timing is about 124 ms per
352×1216 image on an RTX 2080 Ti, for a model retrained on 352×352 crops
[arxiv:2203.01502 tab. 6]. The card: [NeW CRFs](/papers/2203.01502).

### Metric depth and normals from a canonical focal length

Metric3D v2 predicts depth in metres and a unit surface normal per pixel from one
RGB image, given the focal length [arxiv:2404.15506 §3.1]. Training data is mapped
to one canonical focal length of 1000 px [arxiv:2404.15506 §3.2]
[arxiv:2404.15506 §4]. The label variant multiplies the ground-truth depth by the
ratio of canonical to true focal length; the image variant resizes the image
instead, and leaves the depth values alone [arxiv:2404.15506 §3.2]. At inference
the prediction is mapped back by the inverse ratio [arxiv:2404.15506 §3.2]. The
decoder runs a few recurrent steps that update depth and normals together, with
the step count set by model size [arxiv:2404.15506 §3.3] [arxiv:2404.15506 supp
tab. 1]. A depth-normal consistency term lets the normal head learn from
depth-only data [arxiv:2404.15506 §3.4].

Author-reported, zero-shot, as labelled in the paper: on NYUv2 the ViT-g model
has Abs Rel 0.067 and δ1 0.980, and the ViT-L model has Abs Rel 0.063 and δ1 0.975
[arxiv:2404.15506 tab. 1]. On KITTI the ViT-g model has Abs Rel 0.051 and the
ViT-L model 0.052 [arxiv:2404.15506 tab. 1]. The ConvNeXt-L variant reaches Abs Rel
0.083 on NYU zero-shot [arxiv:2404.15506 tab. 1]. Fine-tuned on NYU, ViT-g reaches
Abs Rel 0.045 [arxiv:2404.15506 tab. 1]. On the training-set ablation, the
canonical transform lowers NYU zero-shot Abs Rel from 0.584 without it to 0.210
with the image variant [arxiv:2404.15506 tab. 9]. The ViT-g model runs at 5.0 fps
and uses 15.6 GB of GPU memory on an A100 40 GB, without TensorRT or ONNX
[arxiv:2404.15506 supp tab. 1]. The card: [Metric3D v2](/papers/2404.15506).

### Affine-invariant depth from a fine-tuned diffusion model

Marigold fine-tunes only the denoising U-Net of Stable Diffusion v2. The frozen
VAE encodes the image and the depth map, and the image latent is concatenated
with the noisy depth latent at the U-Net input [arxiv:2312.02145 §3.2]. Each
training depth map is mapped so that its 2nd and 98th percentiles fall at -1 and
1, which makes the target correct only up to one scale and one shift
[arxiv:2312.02145 §3.3] [arxiv:2312.02145 eq. 3]. Training uses synthetic Hypersim
and Virtual KITTI data only, with annealed multi-resolution noise
[arxiv:2312.02145 §3.3] [arxiv:2312.02145 app. A.2].

At inference, each run starts from Gaussian noise and takes 50 DDIM steps. The
runs are aligned by one scale and one shift each, then merged by pixel-wise
median [arxiv:2312.02145 §3.4] [arxiv:2312.02145 §4.1] [arxiv:2312.02145 eq. 4].
Author-reported, zero-shot, aligned by least squares: the 10-run ensemble has
Abs Rel 5.5 on NYUv2, 9.9 on KITTI, 6.5 on ETH3D, 6.4 on ScanNet and 30.8 on
DIODE, with an average rank of 1.4 [arxiv:2312.02145 tab. 1]. On DIODE the paper's
DPT row is 18.2 [arxiv:2312.02145 tab. 1]. A single run is slightly worse, and
the gain from ensembling saturates beyond 10 runs [arxiv:2312.02145 §4.3]. On the
paper's NYU validation split, annealed multi-resolution noise gives Abs Rel 5.6,
against 7.7 for plain Gaussian noise [arxiv:2312.02145 tab. 2]. The backbone was
pretrained on LAION-5B, so "no real depth" does not mean "no real data"
[arxiv:2312.02145 §3.2] [arxiv:2312.02145 tab. 1]. The authors call the approach
slower than feed-forward estimators [arxiv:2312.02145 app. B.5]. The card:
[Marigold](/papers/2312.02145). Its code is Apache-2.0, but its weights carry a
separate licence, so the card covers the method only (see its licence fields).

### Self-supervised depth from video with a high-resolution decoder

No depth labels are used. The depth network and a pose network are trained on
three consecutive video frames, and the loss asks that the target frame, warped
into a neighbouring frame using the predicted depth and pose, match that frame.
The photometric error mixes SSIM and L1 with weight 0.85, is taken as the minimum
over the two source frames, and is combined with an edge-aware smoothness term
of weight 1e-3 [arxiv:2012.07356 §3.1] [arxiv:2012.07356 §4.2]. The
argument for this design is that bilinear up-sampling of a low-resolution
prediction is most wrong where depth changes fast, so higher input resolution
alone does not help much. On KITTI at 640×192 and 1024×320, Monodepth2 scores
Abs Rel 0.115 at both sizes [arxiv:2012.07356 tab. 1]. The decoder is a U-Net
with dense skip connections, and its fusion nodes use a squeeze-and-excitation
style block, called fSE, with reduction ratio r = 4 in the paper
[arxiv:2012.07356 §3.3] [arxiv:2012.07356 §3.4].

Author-reported on the KITTI Eigen split, with per-image median scaling against
LiDAR at test time and depths capped at 80 m [arxiv:2012.07356 tab. 2]. The
monocular 640×192 model reaches Abs Rel 0.109, against 0.115 for Monodepth2.
The best self-supervised row is the stereo-plus-monocular model at 1024×320, at
0.101. The dense-skip and fSE ablation at 640×192 moves Abs Rel from 0.115 to
0.109 [arxiv:2012.07356 tab. 4]. A distilled MobileNetV3 variant, Lite-HR-Depth,
has 3.1M parameters against 14.84M for Monodepth2. With a teacher, it reaches
Abs Rel 0.105 at 1024×320 [arxiv:2012.07356 tab. 3]. Without a teacher at
640×192 it is slightly worse than Monodepth2, at 0.116 against 0.115. The teacher
is not named in the paper [arxiv:2012.07356 §3.5]. The card:
[HR-Depth](/papers/2012.07356). The released code is evaluation-only, and its
training code has not been released (see the card's Paper vs code notes).

### Zero-shot depth on glass and mirror surfaces by fine-tuning Depth Anything V2

Monocular models often predict the depth of what a transparent or mirror surface
reflects, not of the surface itself. This variant fine-tunes Depth Anything V2
Large on the synthetic Hypersim dataset [arxiv:2408.06083 §4.2], with three
additions. First, random tone-mapping during training, with γ = 1/2.2 and a
brightness scale that maps a random percentile between 70 and 99 to 0.8
[arxiv:2408.06083 §3.1] [arxiv:2408.06083 eq. 1]. Second, a guidance loss on the
transparent and mirror (ToM) mask. It compares normalised gradient magnitudes of
the predicted and true depth, and discards the largest 20% of residuals
[arxiv:2408.06083 §3.2] [arxiv:2408.06083 eq. 3] [arxiv:2408.06083 eq. 5]. The
total loss adds this to a scale-shift-invariant term with no weights
[arxiv:2408.06083 §4.2]. Third, an optional test-time step that averages the
Stable Diffusion VAE latents of several exposures of one scene
[arxiv:2408.06083 §3.3].

Author-reported, on the 456 Booster samples with ground truth, the ToM δ1.05 is
0.859, against 0.644 for Depth Anything V2 [arxiv:2408.06083 tab. 2]. On the
NYU Depth V2 subset with Mirror3D masks, 125 samples, the ToM δ1.05 is 0.606,
against 0.576 [arxiv:2408.06083 tab. 1]. In the Booster ablation, the
augmentation alone gives ToM δ1.05 0.800 and the guidance loss alone 0.815; with
both, it is 0.859. Test-time fusion moves it to 0.860 [arxiv:2408.06083 tab. 3].
The paper's evaluation does not say whether predictions are aligned before
scoring, so these numbers cannot be placed in the aligned or absolute groups
above. The card: [Towards Robust Monocular Depth Estimation in Non-Lambertian
Surfaces](/papers/2408.06083). No official code was found (see its Paper vs code
notes).

### Monocular depth as a prior in hand-object reconstruction

This is a user of monocular depth, not a depth estimator. The pipeline gets a
depth map from Ranftl et al. 2020 and compares it with the rendered depth of a
known object model, as one term in the object-pose fit [arxiv:2012.09856 §3.2]
[arxiv:2012.09856 eq. 4]. The paper does not say how the map's scale and shift
are aligned before this comparison [arxiv:2012.09856 §3.2]. The card:
[Hand-object reconstruction](/papers/2012.09856).

### Relative depth as a prior for few-view Gaussian splatting

This is also a user of monocular depth. [SparseGS](/papers/2312.00206) fits 3D
Gaussian Splatting from 12 views of unbounded scenes, or 3 forward-facing views.
Its prior comes from a pre-trained monocular model, which the paper names as
Marigold [arxiv:2312.00206 §3.2]. The loss is one minus the Pearson correlation
between the rendered depth and the prior, on random square patches plus a
whole-image term [arxiv:2312.00206 §3.2] [arxiv:2312.00206 eq. 8]. The
correlation ignores scale and shift, which suits a model that predicts relative
depth [arxiv:2312.00206 §3.2]. Rendered depth is read off the Gaussians with a
softmax weighting at β = 5, instead of plain alpha blending [arxiv:2312.00206 §3.1]
[arxiv:2312.00206 eq. 5] [arxiv:2312.00206 §4.1].

In the 12-view Mip-NeRF360 ablation, the 3DGS baseline reaches 17.49 PSNR. The
correlation loss with alpha-blended depth reaches 18.45, and with softmax depth
18.86. Adding the unseen-view terms gives 19.09, and floater pruning 19.37
[arxiv:2312.00206 tab. 3]. On this split the full model scores above FSGS at
19.31 and 3DGS at 17.49 [arxiv:2312.00206 tab. 1]. The margin over FSGS is 0.06 dB,
from single runs with no reported spread [arxiv:2312.00206 §4]. On DTU it is
slightly below DNGaussian, at 18.89 PSNR against 18.91, and below SparseNeRF at
19.55 [arxiv:2312.00206 tab. 2]. The paper gives no hardware or training time
[arxiv:2312.00206 §4.1], and it does not test other monocular models
[arxiv:2312.00206 §3.2]. The card: [SparseGS](/papers/2312.00206).

### Depth and normal priors for dynamic driving-scene Gaussians

This is another user of monocular models. [DriveSplat](/papers/2508.15376)
fits neural Gaussians to recorded driving sequences from Waymo and KITTI. Its
background is a hashed multi-scale table, and tracked actors get a rigid pose
plus an anchor-level deformation [arxiv:2508.15376 §3.2] [arxiv:2508.15376 §3.3].
The priors are named by the paper: DepthAnything-V2 for relative depth,
ZoeDepth for absolute depth, and Omnidata for normals [arxiv:2508.15376 §3.4].
The paper's chosen depth prior is the relative one, and its depth term is a
correlation with that prior [arxiv:2508.15376 §3.4] [arxiv:2508.15376 eq. 17].
Priors are used only during the supervised stage [arxiv:2508.15376 §3]. The
paper's ablation shows that dropping the depth loss lowers PSNR, and that the
absolute prior gives a lower Abs Rel than the relative one, while the relative
prior gives the best PSNR [arxiv:2508.15376 tab. 8]. The card: [DriveSplat](/papers/2508.15376).

### Depth as a hidden step in single-image view synthesis

This is a view-synthesis model, not a depth benchmark. A depth network and a
feature network run on one RGB image. Each pixel's depth lifts its feature to a
3D point, and a differentiable point-cloud renderer splats the points into the
new view [arxiv:1912.08804 §3.1] [arxiv:1912.08804 §3.2]. The depth output is a
sigmoid rescaled to per-dataset minimum and maximum depths [arxiv:1912.08804 §3.1].
No depth labels are used. The depth is learned only through the image losses
[arxiv:1912.08804 §1] [arxiv:1912.08804 §3.4]. Table 1 reports image metrics
only, with no depth error [arxiv:1912.08804 tab. 1]. On Matterport3D, a variant
trained with ground-truth depth reaches PSNR 22.65 on all pixels, against 20.91
for the full model, but its PSNR on regions not visible in the input is lower,
19.64 against 19.80 [arxiv:1912.08804 tab. 1]. The paper says true depth does not
necessarily help [arxiv:1912.08804 §4.3]. The card: [SynSin](/papers/1912.08804).

### Depth lifted to a point cloud for 3D detection

This is a survey of 3D object detection, not a depth paper. It says depth is
missing from a single image, and that recovering it is ill-posed
[arxiv:2106.10823 §3.1]. In the pseudo-LiDAR family, a depth map is estimated
from the image, back-projected to a point cloud, and passed to a point-cloud
detector [arxiv:2106.10823 §3.1.2]. Pseudo-LiDAR++ uses sparse real LiDAR points
as landmarks to correct depth bias. Pseudo-LiDAR E2E makes the chain trainable
end to end [arxiv:2106.10823 §3.1.2]. The survey reports that image-based methods
trail point-cloud methods on KITTI because of depth ambiguity [arxiv:2106.10823 §4.1].
The card: [3D Object Detection survey](/papers/2106.10823).

### Depth distribution lifted to bird's-eye view for 3D detection

This is a monocular 3D detector, not a depth benchmark. Each pixel gets a softmax
over D depth bins, and its image features are multiplied by these probabilities
to form a frustum volume [arxiv:2103.01100 §3.1] [arxiv:2103.01100 eq. 1]. Each
voxel centre is projected into the frustum and sampled at its depth, with bins
spaced by linear increasing discretisation [arxiv:2103.01100 §3.3]
[arxiv:2103.01100 eq. 2]. The height axis is folded into the channels, reduced to
a bird's-eye-view map, and passed to a PointPillars-style head [arxiv:2103.01100 §3.1]
[arxiv:2103.01100 §3.2]. The depth distribution is supervised with a focal loss on
depth bins derived from LiDAR, and the detector and depth network train together
[arxiv:2103.01100 §3.4] [arxiv:2103.01100 §3.5] [arxiv:2103.01100 eq. 3]
[arxiv:2103.01100 eq. 4].

On KITTI validation, Car moderate AP R40 is 5.66 without the depth distribution
or its supervision, and 16.31 with the full method [arxiv:2103.01100 tab. 3]. With
the depth network trained separately, BTS gives 10.18 and DORN 11.04 moderate AP;
the jointly trained depth network gives 13.71, and 16.31 once the full
distributions build the frustum [arxiv:2103.01100 tab. 4]. On the KITTI test set,
the authors report Car moderate AP R40 of 13.41, which they describe as first among
published monocular methods at submission [arxiv:2103.01100 tab. 1]. On the Waymo
front camera, the authors' extended M3D-RPN baseline has LEVEL_1 mAP 3.79 at IoU
0.5, against 17.54 for CaDDN [arxiv:2103.01100 tab. 2]. The paper reports no
depth-error metric for its depth network; its depth analysis is qualitative
[arxiv:2103.01100 §4.4]. The card: [CaDDN](/papers/2103.01100).

### Depth correlation loss with pseudo views in few-view Gaussian splatting

This is another user of monocular depth. [FSGS](/papers/2312.00451) fits 3D
Gaussian Splatting from three posed photos. Its growth step adds Gaussians between
sparse neighbours, and from iteration 2,000 it also renders pseudo views, placed
between pairs of training cameras with added noise [arxiv:2312.00451 §3.2]
[arxiv:2312.00451 §3.3] [arxiv:2312.00451 §4.1]. A pretrained monocular model
predicts depth for each training image and each pseudo view. The loss is a Pearson
correlation between that prior and the rendered depth, which ignores the prior's
unknown absolute scale [arxiv:2312.00451 §3.3] [arxiv:2312.00451 eq. 6]. The
appendix says the estimator outputs inverse depth, while the rendered depth is
z-depth, and no conversion is stated [arxiv:2312.00451 app. Training].

On the 3-view LLFF ablation at 1/8 resolution, 3D-GS reaches 17.43 PSNR. Adding
unpooling gives 18.64, depth regularisation 19.83, and pseudo views 20.31
[arxiv:2312.00451 tab. 4]. With the depth model varied, the full method scores
20.17 with MiDaS small, 20.29 with DPT Hybrid, 20.31 with DPT Large and 20.37 with
DepthAnything [arxiv:2312.00451 tab. 5]. The gaps are small and come from single
runs, so they are not a ranking of the depth models. The paper's equation for the
correlation term is written as an L1 norm, which as written would drive the
correlation towards zero; the sign is not stated [arxiv:2312.00451 eq. 8]. The
code is licensed for non-commercial research only, so this card uses the paper
alone (see the FSGS card's licence fields). The card: [FSGS](/papers/2312.00451).

### Fragment TSDF instead of per-frame depth

This is a reconstruction network, not a depth estimator, and it is named here as
the alternative to per-frame depth. [NeuralRecon](/papers/2104.00681) takes a
monocular video with camera poses that are assumed to be known [arxiv:2104.00681 §1].
It predicts a sparse signed-distance (TSDF) volume for each fragment of 9 key
frames. Image features are back-projected into a 3D grid, decoded coarse-to-fine in
three levels with sparse 3D convolutions, and fused with earlier fragments by a
ConvGRU [arxiv:2104.00681 §3.2] [arxiv:2104.00681 §3.3]. The authors argue that
per-frame depth gives scale that drifts between views, and that the same surface
is estimated again in overlapping key frames [arxiv:2104.00681 §1].

Author-reported on ScanNet, Atlas split: F-score 0.562 for the single-layer mesh
[arxiv:2104.00681 tab. 1], and 30 ms per key frame on an RTX 2080 Ti [arxiv:2104.00681 §4.2].
Its 2D depth row on the Atlas split has Abs Rel 0.065, equal to Atlas and below
MVDepthNet at 0.098. On the BA-Net split it has 0.047, against 0.057 for DeepV2D
[arxiv:2104.00681 tab. 2]. The card does not say how 2D depth is read from the
volume, and the supplement with the metric definitions was not in the source it
read [arxiv:2104.00681 §4.1]. The timing excludes TSDF fusion for the
depth-based baselines [arxiv:2104.00681 §4.2], so the speed comparison is not
like for like. The official code is Apache-2.0, and its training configs differ
from the paper (see the card's Paper vs code notes). The card:
[NeuralRecon](/papers/2104.00681).

### Relative depth as an input to a vision-language model

This is a vision-language model, not a depth estimator. [SpatialRGPT](/papers/2406.01584)
adds region prompts, boxes or masks, to a LLaVA-style model. An optional depth map
goes through the same visual encoder, is averaged inside each region, and is
projected into the language model's input [arxiv:2406.01584 §3.3]. The evaluation
code takes relative depth from Depth Anything ViT-L [code: llava/eval/eval_spatial.py#L37].
Training data is built with Metric3Dv2, given intrinsics estimated by WildCamera
[arxiv:2406.01584 §3.1] [arxiv:2406.01584 app. F.2]. Depth is metric only when
the intrinsics are right. In the second training stage the language model is
frozen and only the depth connector is trained [arxiv:2406.01584 §3.4].

Author-reported on SpatialRGPT-Bench, the 7B model scores 91.78 qualitative accuracy,
against 58.14 for GPT-4V. Its RGB-only ablation scores 89.80 [arxiv:2406.01584 tab. 1].
On BLINK relative depth it scores 82.3, against 66.9 for GPT-4V-Turbo
[arxiv:2406.01584 tab. 4]. These are accuracies and success rates, not depth
errors, so they do not measure depth accuracy. The code is Apache-2.0; the card
flags that the dataset licences were not checked. The card:
[SpatialRGPT](/papers/2406.01584).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need metric depth in metres from one photo, indoors or on roads | AdaBins, BinsFormer, NeW CRFs or Metric3D v2 (which needs the focal length), if their absolute-error results fit your depth range; measure on your data | Their reported errors are absolute, on NYU (0–10 m) and KITTI (0–80 m) [arxiv:2011.14141 §4.1] [arxiv:2204.00987 §1] [arxiv:2204.00987 tab. 1] [arxiv:2204.00987 tab. 3] [arxiv:2203.01502 tab. 1] [arxiv:2203.01502 tab. 3]. Omnidata's depth numbers are aligned and do not measure absolute error [arxiv:2110.04994 §4.3]. Marigold's are aligned too [arxiv:2312.02145 §4.2]. Metric3D v2's zero-shot rows are absolute, given the focal length [arxiv:2404.15506 tab. 1]. |
| You need relative depth or layout from one photo, with no metric scale, and can afford ten diffusion runs per image | Marigold, with its ensemble and 50-step schedule; measure on your data | Author-reported zero-shot Abs Rel is 5.5 on NYUv2 and 9.9 on KITTI for the 10-run ensemble, and 6.0 and 10.5 for a single run [arxiv:2312.02145 tab. 1]. The numbers are aligned, so they are not metric [arxiv:2312.02145 §4.2]. The authors call the approach slower than feed-forward estimators [arxiv:2312.02145 app. B.5]. On DIODE its Abs Rel is 30.8, against 18.2 for DPT [arxiv:2312.02145 tab. 1]. |
| Your depth range is far from the benchmark ranges | Measure on your data before using any card's numbers | The bin bounds are set per dataset [arxiv:2204.00987 §3.2.3] [arxiv:2011.14141 §3.2]. The paper's evaluations cover only NYU, KITTI and SUN RGB-D [arxiv:2204.00987 §4.1.1]. |
| You know the camera's focal length and need metric depth across camera models | Metric3D v2, ViT-L or ViT-g; measure on your cameras | The canonical-focal transform lets one model take many camera models [arxiv:2404.15506 §3.2]. Zero-shot Abs Rel is 0.063 (ViT-L) and 0.067 (ViT-g) on NYU, and 0.052 and 0.051 on KITTI [arxiv:2404.15506 tab. 1]. The model needs the true focal length, and the paper uses guessed focal lengths only for qualitative results [arxiv:2404.15506 §4]. ViT-g runs at 5.0 fps on an A100 40 GB [arxiv:2404.15506 supp tab. 1]. |
| You want the best KITTI or NYU absolute error and can afford a Swin-Large-class model | Either BinsFormer or NeW CRFs; the reported numbers are close | On KITTI Eigen both report Abs Rel 0.052, and BinsFormer's Swin-Large row has the lower RMS (2.098 against 2.129) [arxiv:2204.00987 tab. 1] [arxiv:2203.01502 tab. 1]. On NYU, BinsFormer's Swin-Large row has Abs Rel 0.094 against 0.095 [arxiv:2204.00987 tab. 3] [arxiv:2203.01502 tab. 3]. AdaBins has the higher Abs Rel on both, 0.058 on KITTI [arxiv:2011.14141 tab. 3] and 0.103 on NYU [arxiv:2011.14141 tab. 2], but it uses a different encoder, so the gap mixes head and backbone [arxiv:2011.14141 §3.2]. All are author-reported. Metric3D v2's KITTI and NYU rows are labelled zero-shot [arxiv:2404.15506 tab. 1], so they are not the same comparison as these rows. |
| You have equirectangular 360-degree images | NeW CRFs, with no projection changes; measure on your data | The authors report results on Matterport3D panoramas without panorama-specific projections [arxiv:2203.01502 §4.2] [arxiv:2203.01502 tab. 4]. The panorama evaluation is on Matterport3D only. |
| You want a small backbone | Expect a drop; check the table row for your backbone | The ResNet-18 BinsFormer trails AdaBins on NYU [arxiv:2204.00987 tab. 3]. The strong numbers use Swin-Large with ImageNet-22K pretraining [arxiv:2204.00987 §4.2.2]. NeW CRFs does not state its encoder size [arxiv:2203.01502 §3.4]. |
| You need to train your own BinsFormer | Plan for the cost and the gaps in the recipe | The paper reports about 5 hours on eight V100s for the NYU Swin-Large model [arxiv:2204.00987 §4.2.1]. Crop, colour augmentation and gradient clipping are not stated in the paper. |
| You need to train your own NeW CRFs | Choose the epoch count and learning rate yourself | The paper gives 20 epochs and a learning rate from 1e-4 to 1e-5 [arxiv:2203.01502 §4.1]. The released configs differ (see Contradictions). The paper does not state the GPU count. |
| You want Metric3D v2 code or weights in a product | Check the licence of each separately | The code is BSD-2-Clause (see the Metric3D card's licence fields). The weights are not in the code repository, and the card records no licence for them. |
| You want Marigold code or weights in a product | Check the weight licence separately | The code is Apache-2.0, but the weights are under OpenRAIL++-M, which the card does not cover (see its licence fields) [arxiv:2312.02145 §4.1]. |
| You need novel views from one photo, with no depth labels for training | A SynSin-style learned depth, point-cloud splatting and a refinement network; measure on your scenes | Author-reported PSNR is 22.31 on RealEstate10K and 20.91 on Matterport3D [arxiv:1912.08804 tab. 1]. Table 1 reports image metrics only, so the depth itself is not scored [arxiv:1912.08804 tab. 1]. The released code includes non-commercial components (see the SynSin card's licence fields), so build from the paper. |
| You want BinsFormer or NeW CRFs code in a product | Usable under its licence; check any weights separately | The BinsFormer card records Apache-2.0 for the code and no licence for pretrained weights (see its licence fields). The NeW CRFs card records MIT for its code (see its licence fields). |
| You have posed 3D scans or meshes you may render | Rendered labels, as in Omnidata | One mesh gives labels for any cue at any camera pose [arxiv:2110.04994 §1] [arxiv:2110.04994 §3.1] |
| You want the released starter data or weights in a product | Do not, per the license | The starter set and pretrained weights are for non-commercial research and education only (see the Omnidata card's license fields) |
| You need a depth prior inside a larger pipeline | A monocular model as one input term; check how its scale is aligned | The hand-object pipeline uses one as a depth target without stating the alignment [arxiv:2012.09856 §3.2] [arxiv:2012.09856 eq. 4] |
| You need a relative depth prior to supervise a 3D model | A scale-and-shift-invariant comparison, such as the Pearson correlation SparseGS uses | SparseGS's correlation loss ignores scale and shift [arxiv:2312.00206 §3.2]. The hand-object pipeline compares depth directly and does not say how it aligns it [arxiv:2012.09856 eq. 4]. |
| You reconstruct recorded driving scenes with moving vehicles or pedestrians, and can run pretrained depth and normal models offline | A DriveSplat-style hashed background with near, mid and far regions, plus tracked actors, supervised by relative depth and normal priors | Author-reported best novel-view PSNR on Waymo, 33.83, against 33.08 for AD-GS and 32.35 for Desire-GS [arxiv:2508.15376 tab. 1]. On KITTI the paper reports 24.53 novel-view PSNR, with the nearest baseline at 22.01 [arxiv:2508.15376 tab. 2]. Training takes 68 minutes on one L20 GPU for 30K iterations [arxiv:2508.15376 §4.2], and the cost table lists 69.6 minutes [arxiv:2508.15376 tab. 5]. The code has no licence, so build from the paper [arxiv:2508.15376 §4.2]. |
| You fit a 3D scene from a few photos and accept an extra diffusion model in training | SparseGS-style depth correlation with floater pruning, measured on your scenes | On 12-view Mip-NeRF360 the full model scores 19.37 PSNR, against 19.31 for FSGS and 17.49 for 3DGS [arxiv:2312.00206 tab. 1]. It uses Stable Diffusion for one loss [arxiv:2312.00206 §3.3.1]. On DTU it is slightly below DNGaussian [arxiv:2312.00206 tab. 2]. The code is non-commercial (see the SparseGS card's licence fields). |
| You fit a 3D scene from three photos and want the prior model to be swappable | FSGS-style unpooling and pseudo views with a depth correlation loss, using any pretrained monocular model; measure on your scenes | On 3-view LLFF at 1/8 resolution, depth regularisation adds 1.19 dB over unpooling alone, and pseudo views add 0.48 dB [arxiv:2312.00451 tab. 4]. The prior model moves the score by at most 0.2 dB across the four models tested [arxiv:2312.00451 tab. 5]. The code is non-commercial, so build from the paper (see the FSGS card's licence fields). |
| You need 3D boxes for driving from camera images, trained end to end, with LiDAR available for training | A CaDDN-style depth distribution lifted to bird's-eye view, with a PointPillars-style head | Author-reported Car moderate AP R40 of 13.41 on KITTI test [arxiv:2103.01100 tab. 1]. Training needs a LiDAR-derived depth label for each image [arxiv:2103.01100 §3.4]. The code is Apache-2.0 (see the CaDDN card's licence fields), but the depth maps it reads are not in the repository. |
| You need 3D boxes for driving from camera images | Read the survey for the taxonomy; for a depth-based route, see pseudo-LiDAR-style lifting [arxiv:2106.10823 §3.1.2] | The survey reports image-based methods trail point-cloud methods on KITTI because of depth ambiguity [arxiv:2106.10823 §4.1]. Its numbers are author-reported and mostly from the original papers [arxiv:2106.10823 §4.2.2]. |
| You need aerial or satellite height, with lidar over the area | Frozen self-supervised encoder and a decoder trained on lidar tiles [arxiv:2304.07213 §3.2] | The decoder takes 8 to 9 hours on 8 GPUs; the encoder pretraining is the costly step, at about 16,896 GPU-hours for the huge model [arxiv:2304.07213 app. D] |
| Canopy taller than about 30 m | Expect saturation; use lidar or GEDI there | Predictions saturate above about 30 m [arxiv:2304.07213 §5] |
| No lidar in the target region | Expect a transfer gap | The lidar training sites are all in the USA [arxiv:2304.07213 §3] |
| You need a real-time mesh or TSDF from a monocular video with known camera poses | A NeuralRecon-style fragment network with ConvGRU fusion, instead of per-frame depth; measure on your scenes | Author-reported: 30 ms per key frame on an RTX 2080 Ti, and F-score 0.562 on the ScanNet Atlas split [arxiv:2104.00681 tab. 1] [arxiv:2104.00681 §4.2]. The depth-based baselines are timed without their fusion step, so the speed gap is not like for like [arxiv:2104.00681 §4.2]. The model is trained on ScanNet, and the 7-Scenes results use it without further training [arxiv:2104.00681 §4.1]. |
| You want a vision-language model to answer distance or relative-depth questions about regions in a photo | A region-prompted VLM with an optional relative-depth input, measured on your questions | Author-reported: 91.78 on SpatialRGPT-Bench for the 7B model, against 58.14 for GPT-4V, and 89.80 for the RGB-only variant [arxiv:2406.01584 tab. 1]. On BLINK relative depth, 82.3 against 66.9 for GPT-4V-Turbo [arxiv:2406.01584 tab. 4]. Those are answer accuracies, not depth errors. Metric answers need camera intrinsics [arxiv:2406.01584 §3.1]. |
| You have only monocular video with no depth labels, on driving-like scenes, and need relative depth | An HR-Depth-style self-supervised decoder, with median scaling if you need units; measure on your data | Author-reported Abs Rel on KITTI Eigen is 0.109 at 640×192 (against 0.115 for Monodepth2), and 0.101 with stereo supervision at 1024×320 [arxiv:2012.07356 tab. 2]. The scores use per-image median scaling against LiDAR, so they do not give metric depth [arxiv:2012.07356 tab. 2 caption]. Training code is not released, so the loss and schedule cannot be reproduced from the release [arxiv:2012.07356 §4.2]. |
| You need a small self-supervised model at high resolution | Lite-HR-Depth-style distillation, with a teacher you train yourself; measure on your data | With a teacher, 3.1M parameters reach Abs Rel 0.105 at 1024×320, against 0.115 for Monodepth2 with 14.84M [arxiv:2012.07356 tab. 3]. Without a teacher, at 640×192, it is slightly worse than Monodepth2 (0.116 against 0.115) [arxiv:2012.07356 tab. 3]. The paper does not name the teacher [arxiv:2012.07356 §3.5]. |
| Your scenes have glass or mirrors, and you can fine-tune a large model on synthetic data | A Depth Anything V2-style fine-tune with the ToM guidance loss and tone-mapping augmentation; measure on your scenes | Author-reported ToM δ1.05 is 0.859 on Booster against 0.644 for Depth Anything V2 [arxiv:2408.06083 tab. 2]. The NYU Mirror3D gain is smaller, 0.606 against 0.576 [arxiv:2408.06083 tab. 1]. The method has no official code, and its training recipe leaves the loss weights and convergence test unstated [arxiv:2408.06083 §4.2]. |

## Implementation sketch

The variants share a core: an encoder, a per-pixel head, and a loss over
pixels that have ground truth. BinsFormer's head differs from the others in
that it builds the depth value from bin centres, with the bins generated by a
decoder. AdaBins builds the same kind of bin mix, with widths from a mini-ViT
on the decoder features. NeW CRFs replaces the head with windowed message
passing between patches. Scoring also differs. Omnidata scores after
scale-and-shift alignment. AdaBins, BinsFormer, NeW CRFs and canopy height score
absolute values. Marigold replaces the head with a denoiser over depth latents,
and scores after least-squares alignment, as Omnidata does. HR-Depth scores
after a per-image median scale, a weaker alignment than scale and shift. The
glass-and-mirror fine-tune does not state its protocol. SynSin's depth
is trained through a renderer, and the paper does not score it directly. The
two prior-based cards add a loss that compares a model's depth with an external
depth map, without fitting a model to it. HR-Depth and the video-trained
family replace depth labels with a photometric loss between neighbouring frames,
and the glass-and-mirror fine-tune adds a gradient-domain loss on its ToM mask.

```pseudocode
# Scale-and-shift alignment before scoring  [arxiv:2110.04994 §4.3]
# pred, truth: (H, W) maps in inverse-depth space; valid: (H, W) mask
def align_and_score(pred, truth, valid):
    # 1. Fit one scale s and one shift t that map pred onto truth.
    #    The card does not say how the fit is done.
    s, t = fit_scale_shift(pred[valid], truth[valid])
    # 2. Apply the same fit to the whole prediction
    aligned = s * pred + t
    # 3. Score only pixels with ground truth, e.g. L1 [arxiv:2110.04994 tab. 2]
    return mean(abs(aligned[valid] - truth[valid]))
```

```pseudocode
# Depth as a per-pixel mix of per-image bin centres  [arxiv:2204.00987 §3.2.3]
# bin_lengths: (N,) softmax over bins; probs: (N, H, W) softmax over bins
def mix_bins(bin_lengths, probs, d_min, d_max):
    # 1. Cumulative bin ends, scaled to the depth range [arxiv:2204.00987 eq. 1]
    ends = d_min + (d_max - d_min) * cumsum(bin_lengths)
    # 2. Each bin's centre is the middle of its span [arxiv:2204.00987 eq. 1]
    centres = ends - (d_max - d_min) * bin_lengths / 2
    # 3. Depth: probability-weighted sum of centres [arxiv:2204.00987 eq. 2]
    return sum(centres[:, None, None] * probs, axis=0)   # (H, W)
```

```pseudocode
# Windowed message passing inside one CRF step  [arxiv:2203.01502 §3.2] [arxiv:2203.01502 eq. 9]
# feats: (H, W, C) image stream; prev: (H, W, C) previous depth prediction
def window_crf_step(feats, prev, shift=0, n=7):
    # 1. Optionally shift the grid, by half a window in the second step
    feats = roll(feats, shift, axes=(0, 1))
    prev = roll(prev, shift, axes=(0, 1))
    # 2. Split into non-overlapping n x n windows; each patch is one node
    win_feats = partition(feats, n)          # (num_windows, n*n, C)
    win_prev = partition(prev, n)
    # 3. Attention weights between patches in the same window [arxiv:2203.01502 eq. 9]
    weights = softmax(queries(win_feats) @ transpose(keys(win_feats)))
    # 4. Send messages from the previous prediction along these weights
    return unpartition(weights @ win_prev, n, feats.shape)
```

```pseudocode
# Log-scale loss over valid pixels  [arxiv:2304.07213 §3.2] [arxiv:2304.07213 eq. 1]
# lam = 0.85 and alpha = 10 as stated in the card
# The same scale-invariant form appears in BinsFormer  [arxiv:2204.00987 eq. 3]
def log_scale_loss(pred, truth, valid, lam=0.85, alpha=10.0):
    # 1. Use only pixels with valid ground truth [arxiv:2304.07213 §3.2]
    d = log(pred[valid]) - log(truth[valid])
    # 2. Mean squared log error minus lam times the squared mean error.
    #    A constant log offset c is weighted by (1 - lam) only.
    spread = mean(d ** 2) - lam * mean(d) ** 2
    # 3. Scale and take the square root [arxiv:2304.07213 eq. 1]
    return alpha * sqrt(spread)
```

```pseudocode
# Correlation loss against a relative depth prior  [arxiv:2312.00206 §3.2] [arxiv:2312.00206 eq. 8]
# pred: (H, W) rendered depth; prior: (H, W) relative depth from a monocular model
def correlation_depth_loss(pred, prior, patches, w_local, w_global):
    # 1. Patch-wise Pearson correlation: one minus the correlation per patch.
    #    Correlation ignores scale and shift, so the prior needs no matching units
    local = mean(1 - pearson(pred[p], prior[p]) for p in patches)
    # 2. Whole-image term, so that patch borders do not show seams [arxiv:2312.00206 §3.2]
    global_term = 1 - pearson(pred, prior)
    # The paper's weights are 0.15 for both terms [arxiv:2312.00206 §4.1]
    return w_local * local + w_global * global_term
```

## Contradictions

- Different targets: Omnidata, AdaBins, BinsFormer and NeW CRFs predict depth,
  canopy height predicts height, and the hand-object pipeline, SparseGS and
  DriveSplat take depth as input. Their numbers are not comparable across these targets. Omnidata
  and BinsFormer both report NYU results, but one is aligned and the other
  absolute, so they are not comparable either.
- KITTI Eigen and NYU, absolute-error papers: on KITTI, BinsFormer's
  Swin-Large row reports Abs Rel 0.052, RMS 2.098 and δ1 0.974
  [arxiv:2204.00987 tab. 1]. NeW CRFs reports Abs Rel 0.052, RMS 2.129 and δ1
  0.974 [arxiv:2203.01502 tab. 1]. AdaBins reports Abs Rel 0.058, RMS 2.360 and
  δ1 0.964 [arxiv:2011.14141 tab. 3]. On NYU, BinsFormer's Swin-Large row reports
  Abs Rel 0.094 [arxiv:2204.00987 tab. 3], NeW CRFs reports 0.095
  [arxiv:2203.01502 tab. 3], and AdaBins reports 0.103 [arxiv:2011.14141 tab. 2].
  All are author-reported on the same splits. The cards do not explain these gaps,
  and the training settings are not compared here.
- Omnidata, paper against released code: the paper describes a DPT-Hybrid depth
  model [arxiv:2110.04994 §4.3]. The repository holds a second training tree
  that uses a different backbone. The paper does not say which setup produced
  its table 2 [arxiv:2110.04994 tab. 2]. Not resolved here.
- AdaBins, paper against released code: the paper gives weight decay 1e-2, a
  maximum learning rate of 3.5e-4 and a final rate of max rate / 75
  [arxiv:2011.14141 §4.2]. The released configs use other values for all three.
  The card's paper-versus-code notes record them. Not resolved here.
- BinsFormer, paper against released code: the paper runs the three layers of
  each scale one after another [arxiv:2204.00987 §3.2.5]. The released default
  cycles through the scales at every layer. The card does not say which the
  reported results used. Not resolved here.
- BinsFormer, scene loss weight: the paper gives μ = 1e-3 [arxiv:2204.00987 §3.2.5].
  The released Swin config uses 1e-2. The card does not say which produced the
  reported results.
- BinsFormer, depth bounds: the paper uses each dataset's minimum and maximum
  valid depth [arxiv:2204.00987 §3.2.3]. The released configs fix the minimum at
  1e-3 m and the maximum at 10 m for NYU or 80 m for KITTI. Both are stated in
  the card's notes.
- NeW CRFs, training schedule: the paper gives 20 epochs and a learning rate
  from 1e-4 to 1e-5 [arxiv:2203.01502 §4.1]. The released configs use 50 epochs
  and a start rate of 2e-5. The card does not say which produced the reported
  results. Not resolved here.
- NeW CRFs, window shift: the paper's shift is N/2 patches [arxiv:2203.01502 §3.2].
  For N = 7 that is 3.5, and the released code uses 3.
- NeW CRFs, test-time flip: the released test script averages the prediction
  with the one for the flipped image. The paper does not say whether its tables
  use this [arxiv:2203.01502 tab. 1].
- SparseGS, paper against released code: the paper gives loss weights of 0.15,
  0.15, 0.05 and 5 × 10⁻⁴, and 10k iterations for 12 views [arxiv:2312.00206 §4.1].
  The repository's example run uses other weights and more iterations, and it
  prunes midway through training. The paper's prior is Marigold, while the
  repository's depth preparation uses another model. The card's paper-versus-code
  notes record these. Not resolved here.
- Metric3D v2, zero-shot labels: the NYUv2 and KITTI rows are labelled zero-shot
  [arxiv:2404.15506 tab. 1]. The paper gives 16 training sets in one place and
  18 in another [arxiv:2404.15506 supp §1.1], and the sets are not named in the
  card. Whether NYUv2 or KITTI are among them is not checked here, so the label
  is not verified. Not resolved here.
- Metric3D v2, paper against released code: the paper gives an initial learning
  rate of 1e-4 for all layers [arxiv:2404.15506 §4]. The released ViT-L and ViT-g
  configs use 1e-5 or 8e-6 for the encoder and 1e-4 for the decoder. The released
  depth-normal consistency term stops gradients through depth, which the paper
  does not mention [arxiv:2404.15506 §3.4]. Not resolved here.
- CaDDN, depth labels: the paper projects LiDAR into the image and fills gaps
  with depth completion [arxiv:2103.01100 §3.4]. The released code loads
  precomputed depth maps that the repository does not include, so the completion
  method cannot be checked from it. Not resolved here.
- CaDDN, focal weighting: the paper gives a focal alpha of 3.25 for foreground
  and 0.25 for background [arxiv:2103.01100 §3.5]. The released code uses alpha
  0.25 with weights of 13 for foreground and 1 for background. The card reads the
  effective per-pixel factors as the same, but by a different mechanism. The paper
  does not say which one its reported numbers used. Not resolved here.
- Canopy height, paper against released code: the paper describes bins over
  [0, B] with B = 256 [arxiv:2304.07213 §3.2]. The released code uses a
  different bin range and multiplies the output by a fixed factor. The card's
  paper-versus-code notes record both, and the difference is not resolved here.
- DriveSplat, depth-loss form: the paper writes the depth term as the L1 norm
  of a normalised covariance between rendered depth and the relative prior
  [arxiv:2508.15376 eq. 17]. Minimising that magnitude would push the
  correlation toward zero, which does not fit a prior that should agree with
  the render. The released code minimises one minus a Pearson correlation,
  applied to a reciprocal of the prior with an offset the paper does not
  give [arxiv:2508.15376 §3.4]. Which form the reported numbers used is not
  resolved here.
- Canopy height, aerial results: the card records that the repository's
  aerial table and the paper's table 6 disagree on bias sign and size, and that
  the cause is not established [arxiv:2304.07213 tab. 6].
- Marigold, inference steps: the paper uses 50 DDIM steps per run [arxiv:2312.02145 §4.1].
  The card records that the released pipeline defaults to 4 steps, and its
  current evaluation script uses 1 [arxiv:2312.02145 §4.1]. The paper's numbers
  were not reproduced, so the cost and accuracy trade-off here is not resolved.
- Marigold, ensemble alignment: the paper's Eq. 4 takes a square root of the mean
  squared pairwise distance and does not give its penalty weight [arxiv:2312.02145 eq. 4].
  The card records that the released code sums root-mean-square distances over
  pairs, with a penalty of 0.02 [arxiv:2312.02145 eq. 4]. Not resolved here.
- SynSin, learning rates: the text gives 1e-4 for the generator and 1e-2 for the
  discriminator [arxiv:1912.08804 §3.4]. The appendix says "ours" uses 1e-3
  [arxiv:1912.08804 app. C]. The card records that the released defaults differ
  from both. Not resolved here.
- FSGS, depth model: the main text names DPT [arxiv:2312.00451 §3.3] [arxiv:2312.00451 §4.1].
  The appendix links PyTorch Hub MiDaS v2 and says the estimator outputs inverse
  depth [arxiv:2312.00451 app. Training]. The DPT Large row of table 5 matches the
  main-table numbers, which suggests DPT Large was used, but the paper does not
  say so [arxiv:2312.00451 tab. 5]. Not resolved here.
- FSGS, correlation term sign: the paper aims to align depth structure, but eq. 8
  adds an L1 norm of the correlation with no stated sign, which as written would
  push the correlation towards zero [arxiv:2312.00451 eq. 8] [arxiv:2312.00451 §3.3].
  Not resolved here.
- SpatialRGPT, depth connector start: the paper says the depth connector is
  initialised from the RGB connector [arxiv:2406.01584 §3.3]. The card records that
  the released code starts it from random weights (see its Paper vs code notes).
  Not resolved here.
- NeuralRecon, feature volume: the code adds one channel, the normalised mean depth
  of each voxel over its views, to the image features [code: ops/back_project.py#L75-L80]
  [code: models/neucon_network.py#L26]. The paper does not mention it
  [arxiv:2104.00681 §3.2]. Not resolved here.
- KITTI Eigen, median-scaled against absolute error: HR-Depth's KITTI rows use a
  per-image median scale against LiDAR [arxiv:2012.07356 tab. 2 caption]. The
  absolute-error papers above report on the same split without that scale
  [arxiv:2011.14141 tab. 3] [arxiv:2204.00987 tab. 1] [arxiv:2203.01502 tab. 1].
  The two sets of numbers are not comparable.
- HR-Depth, paper against released code: the paper gives fSE reduction r = 4
  [arxiv:2012.07356 §3.4], and the code uses 16 [code: layers.py#L309]. The paper
  gives a learning rate of 1e-3, and the repository's README says 1e-4
  [arxiv:2012.07356 §4.2]. The paper does not say which setting produced its
  table. Not resolved here.
- Glass and mirror fine-tune, internal copy error: the full method's "Other" row
  in the Booster ablation is identical to the "Ours Other" row of the NYU table,
  which cannot both be right [arxiv:2408.06083 tab. 3] [arxiv:2408.06083 tab. 1].
  This
  page uses only the ToM rows. Not resolved here.

## Papers

- [Omnidata](/papers/2110.04994) (Eftekhar et al., 2021): renders labelled views from 3D scans and trains a DPT-Hybrid depth model on the result.
- [AdaBins](/papers/2011.14141) (Bhat et al., 2020): per-image adaptive bins from a mini-ViT head, with hybrid regression and a Chamfer term on the bin centres.
- [BinsFormer](/papers/2204.00987) (Li et al., 2022): depth from per-image adaptive bins generated by a Transformer decoder, with multi-scale refinement.
- [NeW CRFs](/papers/2203.01502) (Yuan et al., 2022): depth from a Swin encoder and a decoder that refines the prediction with windowed fully connected CRF energies.
- [Canopy height maps](/papers/2304.07213) (Tolan et al., 2023): canopy height from satellite RGB with a frozen self-supervised encoder and a log-scale loss borrowed from depth estimation.
- [Hand-object reconstruction](/papers/2012.09856) (Cao et al., 2020): reconstructs a hand and a known object from one image, using a monocular depth map as one fitting term.
- [SparseGS](/papers/2312.00206) (Xiong et al., 2023): few-view Gaussian splatting that compares rendered depth with a relative monocular depth prior through a Pearson correlation loss.
- [3D Object Detection survey](/papers/2106.10823) (Qian et al., 2021): survey of 3D detection from images, LiDAR and fusion; its image-based pseudo-LiDAR methods estimate depth before lifting it to a point cloud.
- [Metric3D v2](/papers/2404.15506) (Hu et al., 2024): metric depth and surface normals from one image, given the focal length, via a canonical focal-length transform and joint recurrent refinement.
- [DriveSplat](/papers/2508.15376) (Wang et al., 2025): dynamic driving-scene Gaussian splatting that uses relative depth and normal priors from pretrained models during training.
- [CaDDN](/papers/2103.01100) (Reading et al., 2021): monocular 3D detector that learns a per-pixel depth distribution, lifts image features into a frustum and collapses it to bird's-eye view.
- [Marigold](/papers/2312.02145) (Ke et al., 2023): affine-invariant depth from a Stable Diffusion U-Net fine-tuned on synthetic data, with a 10-run ensemble.
- [SynSin](/papers/1912.08804) (Wiles et al., 2019): single-image view synthesis that learns depth through a differentiable point-cloud renderer, with no depth labels.
- [FSGS](/papers/2312.00451) (Zhu et al., 2023): few-view Gaussian splatting that adds Gaussians by proximity-guided unpooling and compares rendered depth with a pretrained monocular prior through a correlation loss.
- [NeuralRecon](/papers/2104.00681) (Sun et al., 2021): real-time TSDF reconstruction from monocular video, predicting a volume per fragment with a ConvGRU that fuses fragments, instead of per-frame depth maps.
- [SpatialRGPT](/papers/2406.01584) (Cheng et al., 2024): region-prompted vision-language model with an optional relative-depth input, trained on synthetic 3D scene-graph question answers.
- [HR-Depth](/papers/2012.07356) (Lyu et al., 2020): self-supervised depth from video with a dense-skip U-Net decoder and an fSE fusion block, plus a distilled MobileNetV3 variant.
- [Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces](/papers/2408.06083) (Zhang et al., 2024): fine-tunes Depth Anything V2 Large on Hypersim with a gradient-domain loss on transparent and mirror regions and tone-mapping augmentation.