Monocular depth estimation

Overview

Monocular depth estimation takes one RGB image and predicts a value for each pixel, most often the distance from the camera. Nine cards on this slug predict a dense per-pixel map from one image 2110.04994 §4.3 2011.14141 §3.3 2204.00987 §3.1 2203.01502 §3.4 2404.15506 §3.3 2304.07213 §3.2 2312.02145 §3.1 2012.07356 §3.1 2408.06083 §3.1. They differ in target and in how the per-pixel value is produced. Eight predict scene depth: a DPT-Hybrid, a CNN encoder with a small transformer head, a transformer decoder, a windowed CRF decoder, a ViT with recurrent refinement that also predicts surface normals, a diffusion model that denoises a depth latent, a self-supervised U-Net-style decoder trained from video 2012.07356 §3.3, and a fine-tuned Depth Anything V2 Large 2408.06083 §4.2. One predicts tree height from satellite photos, with a frozen transformer and a DPT decoder. Nine more cards use depth inside a larger pipeline, or replace per-frame depth with something else: a single-image view synthesis model whose depth network is trained only through rendering 1912.08804 §3.1, a hand-object fit 2012.09856 §3.2, two few-view Gaussian splatting methods that compare rendered depth with a monocular prior 2312.00206 §3.2 2312.00451 §3.3, a driving-scene Gaussian method that takes depth and normal priors from pretrained models 2508.15376 §3.4, a vision-language model that takes a relative-depth map as an extra input 2406.01584 §3.3, a real-time reconstruction network that predicts a TSDF per fragment instead of a depth map per frame 2104.00681 §1, a 3D detection survey whose image-based methods estimate depth and lift it to a point cloud 2106.10823 §3.1.2, and a monocular 3D detector that learns a depth distribution per pixel 2103.01100 §3.1.

Omnidata builds a rendering pipeline for labels and trains a depth model on its output. AdaBins also predicts bins for each image, but from a small transformer that reads the decoder's features rather than from learned queries. BinsFormer predicts depth as a probability-weighted mix of bin centres that are generated for each image. NeW CRFs refines a coarse depth guess with fully connected CRF energies computed inside small windows of patches. Canopy height borrows the depth literature's log-scale loss for a height map. Metric3D v2 instead rescales its training images to one focal length, so that its output is in metres. CaDDN does not output a depth map for evaluation; it turns a per-pixel depth distribution into features for a detector. AdaBins, BinsFormer and NeW CRFs use the same scale-invariant loss family 2011.14141 §3.4 2204.00987 §3.2.3 2203.01502 §3.4. Metric3D v2 includes a scale-invariant log term among its depth losses 2404.15506 eq. 9. The canopy numbers are not depth numbers, so it is compared only in the table below and not directly against the others.

Omnidata and BinsFormer both report depth on NYU Depth v2, but with different metrics. Omnidata's NYU error is aligned in scale and shift before scoring 2110.04994 §4.3. Marigold's zero-shot errors are also aligned, by least squares 2312.02145 §4.2. BinsFormer's NYU error is absolute 2204.00987 §4.1.2. NeW CRFs also reports absolute error on NYU 2203.01502 tab. 3. AdaBins reports absolute error on NYU, up to 10 m 2011.14141 §4.1. Do not rank Omnidata or Marigold against the absolute-error papers.

Variants

Depth from a DPT-Hybrid trained on mixed-source data

The depth model is a DPT-Hybrid with a setup similar to MiDaS v3.0. It is trained on a five-dataset part of the starter set with scale- and shift-invariant losses 2110.04994 §4.3. Evaluation is zero-shot, with each prediction first aligned to the ground truth by one scale and one shift, in inverse-depth space 2110.04994 §4.3. On the authors' table, the starter-trained model has a lower L1 error than MiDaS v3 on NYU (0.2878 against 0.3838) and on the OASIS validation split (0.7901 against 0.8057). These are author-reported 2110.04994 tab. 2. The card: Omnidata.

Labels from rendered 3D scans

Omnidata places cameras and points on a textured mesh, renders a view for each clear camera-point pair, and labels each view with 21 mid-level cues 2110.04994 §3.1. By default, a point must be seen by at least three cameras to be kept 2110.04994 §3.1. The starter set has 14,601,449 images from 2,414 spaces, built from seven mesh-based sources 2110.04994 §4.1 2110.04994 §4.2. A label takes 1 to 4 seconds on a CPU at 512×512 and can be run in parallel 2110.04994 §3.1. Label accuracy is capped by the scan: the authors cite up to 2 cm error for one source 2110.04994 §6. The cue-generation chain is described only in supplementary material 2110.04994 §3.1.

Adaptive bins from a mini-ViT head on an EfficientNet encoder

AdaBins uses an EfficientNet-B5 encoder and a standard upsampling decoder, which produce features at half the input resolution 2011.14141 §3.2 2011.14141 §3.3. A small vision transformer, the mini-ViT, reads those features in 16×16 patches 2011.14141 tab. 1. It outputs the widths of N bins for the image and C = 128 range-attention maps 2011.14141 §3.3. The widths are normalised and spread over the depth range, and each bin's centre is its midpoint 2011.14141 eq. 1 2011.14141 eq. 2. Each pixel gets a softmax over the bins from the range-attention maps, and its depth is the probability-weighted sum of the centres 2011.14141 eq. 3. The paper says this hybrid regression avoids the stepped output of picking the most likely bin 2011.14141 §3.3 fig. 5. The final model uses N = 256 bins 2011.14141 §4.4.

Training combines a scaled scale-invariant loss on log depth with a Chamfer term that pulls the bin centres towards ground-truth depths, weighted 0.1 2011.14141 §3.4 2011.14141 eq. 4 2011.14141 eq. 5 2011.14141 eq. 6.

On the paper's NYU ablation with the same base model, per-image adaptive bins give δ1 0.903 and RMS 0.364. Fixed uniform, log-scale and trained bin widths give δ1 0.892, 0.896 and 0.893, with RMS 0.383, 0.379 and 0.381. Standard regression with no bins gives δ1 0.881 2011.14141 tab. 6. Author-reported: on NYU, Abs Rel 0.103 and δ1 0.903 2011.14141 tab. 2; on the KITTI Eigen split, Abs Rel 0.058 and δ1 0.964 2011.14141 tab. 3. Trained on NYU and tested on SUN RGB-D with no fine-tuning, δ1 is 0.771, above BTS at 0.740 2011.14141 tab. 5. The mini-ViT is about 5.8 M parameters of a model of about 78 M 2011.14141 tab. 1 2011.14141 §4.2. The paper gives no runtime or memory figure for the head 2011.14141 §4.2. The card: AdaBins. Its official code is GPL-3.0 (see the card's licence fields), so implement the method from the paper's equations.

Adaptive bins from a Transformer decoder

BinsFormer generates N bins for each image from learned queries in a Transformer decoder, with N = 64 by default 2204.00987 §3.2.2 2204.00987 tab. 6. Each pixel gets a softmax over the bins, and its depth is the sum of bin centre times probability 2204.00987 §3.2.3 2204.00987 eq. 2. The bin bounds are the dataset's minimum and maximum valid depth 2204.00987 §3.2.3. An extra scene query classifies the scene, and the bins are refined at three feature scales 2204.00987 §3.2.4 2204.00987 §3.2.5. On NYU Depth v2 with a Swin-Large backbone pretrained on ImageNet-22K, δ1 is 0.925 and RMS is 0.330, against 0.903 and 0.364 for AdaBins 2204.00987 tab. 3. On the KITTI Eigen split with the same backbone, δ1 is 0.974 and RMS is 2.098, against 0.964 and 2.360 for AdaBins 2204.00987 tab. 1. These are author-reported. The ResNet-18 variant is below AdaBins on NYU, with δ1 0.866 2204.00987 tab. 3. The bin-count ablation shows the gain saturating above 64 bins 2204.00987 tab. 6. The card: BinsFormer.

Windowed fully connected CRF refinement on a Swin encoder

A Swin transformer encodes the image at four scales. The encoder size is not stated in the paper 2203.01502 §3.4. A pyramid pooling head makes a coarse depth guess from the top feature 2203.01502 §3.4. Four decoder levels then refine it. At each level, the image is cut into non-overlapping 7×7 windows of patches, and every patch is a node that can exchange messages with the others in its window 2203.01502 §3.2. The pairwise weights are learned with multi-head attention, not set by hand as in classic CRFs 2203.01502 §3.3 2203.01502 eq. 9. A second step uses windows shifted by half a window, so that neighbours across a border also connect 2203.01502 §3.2 2203.01502 §3.4. Head counts run 32, 16, 8, 4 from coarse to fine 2203.01502 §3.4. The loss is the scale-invariant log loss with variance factor 0.85 and scale 10 2203.01502 §3.4.

Author-reported, on the KITTI Eigen split: Abs Rel 0.052 and RMSE 2.129, against 0.058 and 2.360 for AdaBins 2203.01502 tab. 1. On NYU Depth v2: Abs Rel 0.095 and RMSE 0.334, against 0.103 and 0.364 for AdaBins 2203.01502 tab. 3. The paper's ablation on the same KITTI split moves Abs Rel from 0.069 for a UNet decoder to 0.055 with the neural CRF and 0.052 with the full model 2203.01502 tab. 5. The same perspective network is run directly on 1024×512 equirectangular panoramas, with Abs Rel 0.0906 on Matterport3D 2203.01502 §4.2 2203.01502 tab. 4. A variant with 50K extra pretraining images, whose source the paper does not state, reaches 0.0793 2203.01502 tab. 4. The decoder timing is about 124 ms per 352×1216 image on an RTX 2080 Ti, for a model retrained on 352×352 crops 2203.01502 tab. 6. The card: NeW CRFs.

Metric depth and normals from a canonical focal length

Metric3D v2 predicts depth in metres and a unit surface normal per pixel from one RGB image, given the focal length 2404.15506 §3.1. Training data is mapped to one canonical focal length of 1000 px 2404.15506 §3.2 2404.15506 §4. The label variant multiplies the ground-truth depth by the ratio of canonical to true focal length; the image variant resizes the image instead, and leaves the depth values alone 2404.15506 §3.2. At inference the prediction is mapped back by the inverse ratio 2404.15506 §3.2. The decoder runs a few recurrent steps that update depth and normals together, with the step count set by model size 2404.15506 §3.3 [arxiv:2404.15506 supp tab. 1]. A depth-normal consistency term lets the normal head learn from depth-only data 2404.15506 §3.4.

Author-reported, zero-shot, as labelled in the paper: on NYUv2 the ViT-g model has Abs Rel 0.067 and δ1 0.980, and the ViT-L model has Abs Rel 0.063 and δ1 0.975 2404.15506 tab. 1. On KITTI the ViT-g model has Abs Rel 0.051 and the ViT-L model 0.052 2404.15506 tab. 1. The ConvNeXt-L variant reaches Abs Rel 0.083 on NYU zero-shot 2404.15506 tab. 1. Fine-tuned on NYU, ViT-g reaches Abs Rel 0.045 2404.15506 tab. 1. On the training-set ablation, the canonical transform lowers NYU zero-shot Abs Rel from 0.584 without it to 0.210 with the image variant 2404.15506 tab. 9. The ViT-g model runs at 5.0 fps and uses 15.6 GB of GPU memory on an A100 40 GB, without TensorRT or ONNX 2404.15506 supp tab. 1. The card: Metric3D v2.

Affine-invariant depth from a fine-tuned diffusion model

Marigold fine-tunes only the denoising U-Net of Stable Diffusion v2. The frozen VAE encodes the image and the depth map, and the image latent is concatenated with the noisy depth latent at the U-Net input 2312.02145 §3.2. Each training depth map is mapped so that its 2nd and 98th percentiles fall at -1 and 1, which makes the target correct only up to one scale and one shift 2312.02145 §3.3 2312.02145 eq. 3. Training uses synthetic Hypersim and Virtual KITTI data only, with annealed multi-resolution noise 2312.02145 §3.3 2312.02145 app. A.2.

At inference, each run starts from Gaussian noise and takes 50 DDIM steps. The runs are aligned by one scale and one shift each, then merged by pixel-wise median 2312.02145 §3.4 2312.02145 §4.1 2312.02145 eq. 4. Author-reported, zero-shot, aligned by least squares: the 10-run ensemble has Abs Rel 5.5 on NYUv2, 9.9 on KITTI, 6.5 on ETH3D, 6.4 on ScanNet and 30.8 on DIODE, with an average rank of 1.4 2312.02145 tab. 1. On DIODE the paper's DPT row is 18.2 2312.02145 tab. 1. A single run is slightly worse, and the gain from ensembling saturates beyond 10 runs 2312.02145 §4.3. On the paper's NYU validation split, annealed multi-resolution noise gives Abs Rel 5.6, against 7.7 for plain Gaussian noise 2312.02145 tab. 2. The backbone was pretrained on LAION-5B, so "no real depth" does not mean "no real data" 2312.02145 §3.2 2312.02145 tab. 1. The authors call the approach slower than feed-forward estimators 2312.02145 app. B.5. The card: Marigold. Its code is Apache-2.0, but its weights carry a separate licence, so the card covers the method only (see its licence fields).

Self-supervised depth from video with a high-resolution decoder

No depth labels are used. The depth network and a pose network are trained on three consecutive video frames, and the loss asks that the target frame, warped into a neighbouring frame using the predicted depth and pose, match that frame. The photometric error mixes SSIM and L1 with weight 0.85, is taken as the minimum over the two source frames, and is combined with an edge-aware smoothness term of weight 1e-3 2012.07356 §3.1 2012.07356 §4.2. The argument for this design is that bilinear up-sampling of a low-resolution prediction is most wrong where depth changes fast, so higher input resolution alone does not help much. On KITTI at 640×192 and 1024×320, Monodepth2 scores Abs Rel 0.115 at both sizes 2012.07356 tab. 1. The decoder is a U-Net with dense skip connections, and its fusion nodes use a squeeze-and-excitation style block, called fSE, with reduction ratio r = 4 in the paper 2012.07356 §3.3 2012.07356 §3.4.

Author-reported on the KITTI Eigen split, with per-image median scaling against LiDAR at test time and depths capped at 80 m 2012.07356 tab. 2. The monocular 640×192 model reaches Abs Rel 0.109, against 0.115 for Monodepth2. The best self-supervised row is the stereo-plus-monocular model at 1024×320, at 0.101. The dense-skip and fSE ablation at 640×192 moves Abs Rel from 0.115 to 0.109 2012.07356 tab. 4. A distilled MobileNetV3 variant, Lite-HR-Depth, has 3.1M parameters against 14.84M for Monodepth2. With a teacher, it reaches Abs Rel 0.105 at 1024×320 2012.07356 tab. 3. Without a teacher at 640×192 it is slightly worse than Monodepth2, at 0.116 against 0.115. The teacher is not named in the paper 2012.07356 §3.5. The card: HR-Depth. The released code is evaluation-only, and its training code has not been released (see the card's Paper vs code notes).

Zero-shot depth on glass and mirror surfaces by fine-tuning Depth Anything V2

Monocular models often predict the depth of what a transparent or mirror surface reflects, not of the surface itself. This variant fine-tunes Depth Anything V2 Large on the synthetic Hypersim dataset 2408.06083 §4.2, with three additions. First, random tone-mapping during training, with γ = 1/2.2 and a brightness scale that maps a random percentile between 70 and 99 to 0.8 2408.06083 §3.1 2408.06083 eq. 1. Second, a guidance loss on the transparent and mirror (ToM) mask. It compares normalised gradient magnitudes of the predicted and true depth, and discards the largest 20% of residuals 2408.06083 §3.2 2408.06083 eq. 3 2408.06083 eq. 5. The total loss adds this to a scale-shift-invariant term with no weights 2408.06083 §4.2. Third, an optional test-time step that averages the Stable Diffusion VAE latents of several exposures of one scene 2408.06083 §3.3.

Author-reported, on the 456 Booster samples with ground truth, the ToM δ1.05 is 0.859, against 0.644 for Depth Anything V2 2408.06083 tab. 2. On the NYU Depth V2 subset with Mirror3D masks, 125 samples, the ToM δ1.05 is 0.606, against 0.576 2408.06083 tab. 1. In the Booster ablation, the augmentation alone gives ToM δ1.05 0.800 and the guidance loss alone 0.815; with both, it is 0.859. Test-time fusion moves it to 0.860 2408.06083 tab. 3. The paper's evaluation does not say whether predictions are aligned before scoring, so these numbers cannot be placed in the aligned or absolute groups above. The card: Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces. No official code was found (see its Paper vs code notes).

Monocular depth as a prior in hand-object reconstruction

This is a user of monocular depth, not a depth estimator. The pipeline gets a depth map from Ranftl et al. 2020 and compares it with the rendered depth of a known object model, as one term in the object-pose fit 2012.09856 §3.2 2012.09856 eq. 4. The paper does not say how the map's scale and shift are aligned before this comparison 2012.09856 §3.2. The card: Hand-object reconstruction.

Relative depth as a prior for few-view Gaussian splatting

This is also a user of monocular depth. SparseGS fits 3D Gaussian Splatting from 12 views of unbounded scenes, or 3 forward-facing views. Its prior comes from a pre-trained monocular model, which the paper names as Marigold 2312.00206 §3.2. The loss is one minus the Pearson correlation between the rendered depth and the prior, on random square patches plus a whole-image term 2312.00206 §3.2 2312.00206 eq. 8. The correlation ignores scale and shift, which suits a model that predicts relative depth 2312.00206 §3.2. Rendered depth is read off the Gaussians with a softmax weighting at β = 5, instead of plain alpha blending 2312.00206 §3.1 2312.00206 eq. 5 2312.00206 §4.1.

In the 12-view Mip-NeRF360 ablation, the 3DGS baseline reaches 17.49 PSNR. The correlation loss with alpha-blended depth reaches 18.45, and with softmax depth 18.86. Adding the unseen-view terms gives 19.09, and floater pruning 19.37 2312.00206 tab. 3. On this split the full model scores above FSGS at 19.31 and 3DGS at 17.49 2312.00206 tab. 1. The margin over FSGS is 0.06 dB, from single runs with no reported spread 2312.00206 §4. On DTU it is slightly below DNGaussian, at 18.89 PSNR against 18.91, and below SparseNeRF at 19.55 2312.00206 tab. 2. The paper gives no hardware or training time 2312.00206 §4.1, and it does not test other monocular models 2312.00206 §3.2. The card: SparseGS.

Depth and normal priors for dynamic driving-scene Gaussians

This is another user of monocular models. DriveSplat fits neural Gaussians to recorded driving sequences from Waymo and KITTI. Its background is a hashed multi-scale table, and tracked actors get a rigid pose plus an anchor-level deformation 2508.15376 §3.2 2508.15376 §3.3. The priors are named by the paper: DepthAnything-V2 for relative depth, ZoeDepth for absolute depth, and Omnidata for normals 2508.15376 §3.4. The paper's chosen depth prior is the relative one, and its depth term is a correlation with that prior 2508.15376 §3.4 2508.15376 eq. 17. Priors are used only during the supervised stage 2508.15376 §3. The paper's ablation shows that dropping the depth loss lowers PSNR, and that the absolute prior gives a lower Abs Rel than the relative one, while the relative prior gives the best PSNR 2508.15376 tab. 8. The card: DriveSplat.

Depth as a hidden step in single-image view synthesis

This is a view-synthesis model, not a depth benchmark. A depth network and a feature network run on one RGB image. Each pixel's depth lifts its feature to a 3D point, and a differentiable point-cloud renderer splats the points into the new view 1912.08804 §3.1 1912.08804 §3.2. The depth output is a sigmoid rescaled to per-dataset minimum and maximum depths 1912.08804 §3.1. No depth labels are used. The depth is learned only through the image losses 1912.08804 §1 1912.08804 §3.4. Table 1 reports image metrics only, with no depth error 1912.08804 tab. 1. On Matterport3D, a variant trained with ground-truth depth reaches PSNR 22.65 on all pixels, against 20.91 for the full model, but its PSNR on regions not visible in the input is lower, 19.64 against 19.80 1912.08804 tab. 1. The paper says true depth does not necessarily help 1912.08804 §4.3. The card: SynSin.

Depth lifted to a point cloud for 3D detection

This is a survey of 3D object detection, not a depth paper. It says depth is missing from a single image, and that recovering it is ill-posed 2106.10823 §3.1. In the pseudo-LiDAR family, a depth map is estimated from the image, back-projected to a point cloud, and passed to a point-cloud detector 2106.10823 §3.1.2. Pseudo-LiDAR++ uses sparse real LiDAR points as landmarks to correct depth bias. Pseudo-LiDAR E2E makes the chain trainable end to end 2106.10823 §3.1.2. The survey reports that image-based methods trail point-cloud methods on KITTI because of depth ambiguity 2106.10823 §4.1. The card: 3D Object Detection survey.

Depth distribution lifted to bird's-eye view for 3D detection

This is a monocular 3D detector, not a depth benchmark. Each pixel gets a softmax over D depth bins, and its image features are multiplied by these probabilities to form a frustum volume 2103.01100 §3.1 2103.01100 eq. 1. Each voxel centre is projected into the frustum and sampled at its depth, with bins spaced by linear increasing discretisation 2103.01100 §3.3 2103.01100 eq. 2. The height axis is folded into the channels, reduced to a bird's-eye-view map, and passed to a PointPillars-style head 2103.01100 §3.1 2103.01100 §3.2. The depth distribution is supervised with a focal loss on depth bins derived from LiDAR, and the detector and depth network train together 2103.01100 §3.4 2103.01100 §3.5 2103.01100 eq. 3 2103.01100 eq. 4.

On KITTI validation, Car moderate AP R40 is 5.66 without the depth distribution or its supervision, and 16.31 with the full method 2103.01100 tab. 3. With the depth network trained separately, BTS gives 10.18 and DORN 11.04 moderate AP; the jointly trained depth network gives 13.71, and 16.31 once the full distributions build the frustum 2103.01100 tab. 4. On the KITTI test set, the authors report Car moderate AP R40 of 13.41, which they describe as first among published monocular methods at submission 2103.01100 tab. 1. On the Waymo front camera, the authors' extended M3D-RPN baseline has LEVEL_1 mAP 3.79 at IoU 0.5, against 17.54 for CaDDN 2103.01100 tab. 2. The paper reports no depth-error metric for its depth network; its depth analysis is qualitative 2103.01100 §4.4. The card: CaDDN.

Depth correlation loss with pseudo views in few-view Gaussian splatting

This is another user of monocular depth. FSGS fits 3D Gaussian Splatting from three posed photos. Its growth step adds Gaussians between sparse neighbours, and from iteration 2,000 it also renders pseudo views, placed between pairs of training cameras with added noise 2312.00451 §3.2 2312.00451 §3.3 2312.00451 §4.1. A pretrained monocular model predicts depth for each training image and each pseudo view. The loss is a Pearson correlation between that prior and the rendered depth, which ignores the prior's unknown absolute scale 2312.00451 §3.3 2312.00451 eq. 6. The appendix says the estimator outputs inverse depth, while the rendered depth is z-depth, and no conversion is stated 2312.00451 app. Training.

On the 3-view LLFF ablation at 1/8 resolution, 3D-GS reaches 17.43 PSNR. Adding unpooling gives 18.64, depth regularisation 19.83, and pseudo views 20.31 2312.00451 tab. 4. With the depth model varied, the full method scores 20.17 with MiDaS small, 20.29 with DPT Hybrid, 20.31 with DPT Large and 20.37 with DepthAnything 2312.00451 tab. 5. The gaps are small and come from single runs, so they are not a ranking of the depth models. The paper's equation for the correlation term is written as an L1 norm, which as written would drive the correlation towards zero; the sign is not stated 2312.00451 eq. 8. The code is licensed for non-commercial research only, so this card uses the paper alone (see the FSGS card's licence fields). The card: FSGS.

Fragment TSDF instead of per-frame depth

This is a reconstruction network, not a depth estimator, and it is named here as the alternative to per-frame depth. NeuralRecon takes a monocular video with camera poses that are assumed to be known 2104.00681 §1. It predicts a sparse signed-distance (TSDF) volume for each fragment of 9 key frames. Image features are back-projected into a 3D grid, decoded coarse-to-fine in three levels with sparse 3D convolutions, and fused with earlier fragments by a ConvGRU 2104.00681 §3.2 2104.00681 §3.3. The authors argue that per-frame depth gives scale that drifts between views, and that the same surface is estimated again in overlapping key frames 2104.00681 §1.

Author-reported on ScanNet, Atlas split: F-score 0.562 for the single-layer mesh 2104.00681 tab. 1, and 30 ms per key frame on an RTX 2080 Ti 2104.00681 §4.2. Its 2D depth row on the Atlas split has Abs Rel 0.065, equal to Atlas and below MVDepthNet at 0.098. On the BA-Net split it has 0.047, against 0.057 for DeepV2D 2104.00681 tab. 2. The card does not say how 2D depth is read from the volume, and the supplement with the metric definitions was not in the source it read 2104.00681 §4.1. The timing excludes TSDF fusion for the depth-based baselines 2104.00681 §4.2, so the speed comparison is not like for like. The official code is Apache-2.0, and its training configs differ from the paper (see the card's Paper vs code notes). The card: NeuralRecon.

Relative depth as an input to a vision-language model

This is a vision-language model, not a depth estimator. SpatialRGPT adds region prompts, boxes or masks, to a LLaVA-style model. An optional depth map goes through the same visual encoder, is averaged inside each region, and is projected into the language model's input 2406.01584 §3.3. The evaluation code takes relative depth from Depth Anything ViT-L eval_spatial.py. Training data is built with Metric3Dv2, given intrinsics estimated by WildCamera 2406.01584 §3.1 2406.01584 app. F.2. Depth is metric only when the intrinsics are right. In the second training stage the language model is frozen and only the depth connector is trained 2406.01584 §3.4.

Author-reported on SpatialRGPT-Bench, the 7B model scores 91.78 qualitative accuracy, against 58.14 for GPT-4V. Its RGB-only ablation scores 89.80 2406.01584 tab. 1. On BLINK relative depth it scores 82.3, against 66.9 for GPT-4V-Turbo 2406.01584 tab. 4. These are accuracies and success rates, not depth errors, so they do not measure depth accuracy. The code is Apache-2.0; the card flags that the dataset licences were not checked. The card: SpatialRGPT.

When to use which

Implementation sketch

The variants share a core: an encoder, a per-pixel head, and a loss over pixels that have ground truth. BinsFormer's head differs from the others in that it builds the depth value from bin centres, with the bins generated by a decoder. AdaBins builds the same kind of bin mix, with widths from a mini-ViT on the decoder features. NeW CRFs replaces the head with windowed message passing between patches. Scoring also differs. Omnidata scores after scale-and-shift alignment. AdaBins, BinsFormer, NeW CRFs and canopy height score absolute values. Marigold replaces the head with a denoiser over depth latents, and scores after least-squares alignment, as Omnidata does. HR-Depth scores after a per-image median scale, a weaker alignment than scale and shift. The glass-and-mirror fine-tune does not state its protocol. SynSin's depth is trained through a renderer, and the paper does not score it directly. The two prior-based cards add a loss that compares a model's depth with an external depth map, without fitting a model to it. HR-Depth and the video-trained family replace depth labels with a photometric loss between neighbouring frames, and the glass-and-mirror fine-tune adds a gradient-domain loss on its ToM mask.

# Scale-and-shift alignment before scoring  [arxiv:2110.04994 §4.3]
# pred, truth: (H, W) maps in inverse-depth space; valid: (H, W) mask
def align_and_score(pred, truth, valid):
    # 1. Fit one scale s and one shift t that map pred onto truth.
    #    The card does not say how the fit is done.
    s, t = fit_scale_shift(pred[valid], truth[valid])
    # 2. Apply the same fit to the whole prediction
    aligned = s * pred + t
    # 3. Score only pixels with ground truth, e.g. L1 [arxiv:2110.04994 tab. 2]
    return mean(abs(aligned[valid] - truth[valid]))

# Depth as a per-pixel mix of per-image bin centres  [arxiv:2204.00987 §3.2.3]
# bin_lengths: (N,) softmax over bins; probs: (N, H, W) softmax over bins
def mix_bins(bin_lengths, probs, d_min, d_max):
    # 1. Cumulative bin ends, scaled to the depth range [arxiv:2204.00987 eq. 1]
    ends = d_min + (d_max - d_min) * cumsum(bin_lengths)
    # 2. Each bin's centre is the middle of its span [arxiv:2204.00987 eq. 1]
    centres = ends - (d_max - d_min) * bin_lengths / 2
    # 3. Depth: probability-weighted sum of centres [arxiv:2204.00987 eq. 2]
    return sum(centres[:, None, None] * probs, axis=0)   # (H, W)

# Windowed message passing inside one CRF step  [arxiv:2203.01502 §3.2] [arxiv:2203.01502 eq. 9]
# feats: (H, W, C) image stream; prev: (H, W, C) previous depth prediction
def window_crf_step(feats, prev, shift=0, n=7):
    # 1. Optionally shift the grid, by half a window in the second step
    feats = roll(feats, shift, axes=(0, 1))
    prev = roll(prev, shift, axes=(0, 1))
    # 2. Split into non-overlapping n x n windows; each patch is one node
    win_feats = partition(feats, n)          # (num_windows, n*n, C)
    win_prev = partition(prev, n)
    # 3. Attention weights between patches in the same window [arxiv:2203.01502 eq. 9]
    weights = softmax(queries(win_feats) @ transpose(keys(win_feats)))
    # 4. Send messages from the previous prediction along these weights
    return unpartition(weights @ win_prev, n, feats.shape)

# Log-scale loss over valid pixels  [arxiv:2304.07213 §3.2] [arxiv:2304.07213 eq. 1]
# lam = 0.85 and alpha = 10 as stated in the card
# The same scale-invariant form appears in BinsFormer  [arxiv:2204.00987 eq. 3]
def log_scale_loss(pred, truth, valid, lam=0.85, alpha=10.0):
    # 1. Use only pixels with valid ground truth [arxiv:2304.07213 §3.2]
    d = log(pred[valid]) - log(truth[valid])
    # 2. Mean squared log error minus lam times the squared mean error.
    #    A constant log offset c is weighted by (1 - lam) only.
    spread = mean(d ** 2) - lam * mean(d) ** 2
    # 3. Scale and take the square root [arxiv:2304.07213 eq. 1]
    return alpha * sqrt(spread)

# Correlation loss against a relative depth prior  [arxiv:2312.00206 §3.2] [arxiv:2312.00206 eq. 8]
# pred: (H, W) rendered depth; prior: (H, W) relative depth from a monocular model
def correlation_depth_loss(pred, prior, patches, w_local, w_global):
    # 1. Patch-wise Pearson correlation: one minus the correlation per patch.
    #    Correlation ignores scale and shift, so the prior needs no matching units
    local = mean(1 - pearson(pred[p], prior[p]) for p in patches)
    # 2. Whole-image term, so that patch borders do not show seams [arxiv:2312.00206 §3.2]
    global_term = 1 - pearson(pred, prior)
    # The paper's weights are 0.15 for both terms [arxiv:2312.00206 §4.1]
    return w_local * local + w_global * global_term

Contradictions

• Different targets: Omnidata, AdaBins, BinsFormer and NeW CRFs predict depth, canopy height predicts height, and the hand-object pipeline, SparseGS and DriveSplat take depth as input. Their numbers are not comparable across these targets. Omnidata and BinsFormer both report NYU results, but one is aligned and the other absolute, so they are not comparable either.
• KITTI Eigen and NYU, absolute-error papers: on KITTI, BinsFormer's Swin-Large row reports Abs Rel 0.052, RMS 2.098 and δ1 0.974 2204.00987 tab. 1. NeW CRFs reports Abs Rel 0.052, RMS 2.129 and δ1 0.974 2203.01502 tab. 1. AdaBins reports Abs Rel 0.058, RMS 2.360 and δ1 0.964 2011.14141 tab. 3. On NYU, BinsFormer's Swin-Large row reports Abs Rel 0.094 2204.00987 tab. 3, NeW CRFs reports 0.095 2203.01502 tab. 3, and AdaBins reports 0.103 2011.14141 tab. 2. All are author-reported on the same splits. The cards do not explain these gaps, and the training settings are not compared here.
• Omnidata, paper against released code: the paper describes a DPT-Hybrid depth model 2110.04994 §4.3. The repository holds a second training tree that uses a different backbone. The paper does not say which setup produced its table 2 2110.04994 tab. 2. Not resolved here.
• AdaBins, paper against released code: the paper gives weight decay 1e-2, a maximum learning rate of 3.5e-4 and a final rate of max rate / 75 2011.14141 §4.2. The released configs use other values for all three. The card's paper-versus-code notes record them. Not resolved here.
• BinsFormer, paper against released code: the paper runs the three layers of each scale one after another 2204.00987 §3.2.5. The released default cycles through the scales at every layer. The card does not say which the reported results used. Not resolved here.
• BinsFormer, scene loss weight: the paper gives μ = 1e-3 2204.00987 §3.2.5. The released Swin config uses 1e-2. The card does not say which produced the reported results.
• BinsFormer, depth bounds: the paper uses each dataset's minimum and maximum valid depth 2204.00987 §3.2.3. The released configs fix the minimum at 1e-3 m and the maximum at 10 m for NYU or 80 m for KITTI. Both are stated in the card's notes.
• NeW CRFs, training schedule: the paper gives 20 epochs and a learning rate from 1e-4 to 1e-5 2203.01502 §4.1. The released configs use 50 epochs and a start rate of 2e-5. The card does not say which produced the reported results. Not resolved here.
• NeW CRFs, window shift: the paper's shift is N/2 patches 2203.01502 §3.2. For N = 7 that is 3.5, and the released code uses 3.
• NeW CRFs, test-time flip: the released test script averages the prediction with the one for the flipped image. The paper does not say whether its tables use this 2203.01502 tab. 1.
• SparseGS, paper against released code: the paper gives loss weights of 0.15, 0.15, 0.05 and 5 × 10⁻⁴, and 10k iterations for 12 views 2312.00206 §4.1. The repository's example run uses other weights and more iterations, and it prunes midway through training. The paper's prior is Marigold, while the repository's depth preparation uses another model. The card's paper-versus-code notes record these. Not resolved here.
• Metric3D v2, zero-shot labels: the NYUv2 and KITTI rows are labelled zero-shot 2404.15506 tab. 1. The paper gives 16 training sets in one place and 18 in another 2404.15506 supp §1.1, and the sets are not named in the card. Whether NYUv2 or KITTI are among them is not checked here, so the label is not verified. Not resolved here.
• Metric3D v2, paper against released code: the paper gives an initial learning rate of 1e-4 for all layers 2404.15506 §4. The released ViT-L and ViT-g configs use 1e-5 or 8e-6 for the encoder and 1e-4 for the decoder. The released depth-normal consistency term stops gradients through depth, which the paper does not mention 2404.15506 §3.4. Not resolved here.
• CaDDN, depth labels: the paper projects LiDAR into the image and fills gaps with depth completion 2103.01100 §3.4. The released code loads precomputed depth maps that the repository does not include, so the completion method cannot be checked from it. Not resolved here.
• CaDDN, focal weighting: the paper gives a focal alpha of 3.25 for foreground and 0.25 for background 2103.01100 §3.5. The released code uses alpha 0.25 with weights of 13 for foreground and 1 for background. The card reads the effective per-pixel factors as the same, but by a different mechanism. The paper does not say which one its reported numbers used. Not resolved here.
• Canopy height, paper against released code: the paper describes bins over [0, B] with B = 256 2304.07213 §3.2. The released code uses a different bin range and multiplies the output by a fixed factor. The card's paper-versus-code notes record both, and the difference is not resolved here.
• DriveSplat, depth-loss form: the paper writes the depth term as the L1 norm of a normalised covariance between rendered depth and the relative prior 2508.15376 eq. 17. Minimising that magnitude would push the correlation toward zero, which does not fit a prior that should agree with the render. The released code minimises one minus a Pearson correlation, applied to a reciprocal of the prior with an offset the paper does not give 2508.15376 §3.4. Which form the reported numbers used is not resolved here.
• Canopy height, aerial results: the card records that the repository's aerial table and the paper's table 6 disagree on bias sign and size, and that the cause is not established 2304.07213 tab. 6.
• Marigold, inference steps: the paper uses 50 DDIM steps per run 2312.02145 §4.1. The card records that the released pipeline defaults to 4 steps, and its current evaluation script uses 1 2312.02145 §4.1. The paper's numbers were not reproduced, so the cost and accuracy trade-off here is not resolved.
• Marigold, ensemble alignment: the paper's Eq. 4 takes a square root of the mean squared pairwise distance and does not give its penalty weight 2312.02145 eq. 4. The card records that the released code sums root-mean-square distances over pairs, with a penalty of 0.02 2312.02145 eq. 4. Not resolved here.
• SynSin, learning rates: the text gives 1e-4 for the generator and 1e-2 for the discriminator 1912.08804 §3.4. The appendix says "ours" uses 1e-3 1912.08804 app. C. The card records that the released defaults differ from both. Not resolved here.
• FSGS, depth model: the main text names DPT 2312.00451 §3.3 2312.00451 §4.1. The appendix links PyTorch Hub MiDaS v2 and says the estimator outputs inverse depth 2312.00451 app. Training. The DPT Large row of table 5 matches the main-table numbers, which suggests DPT Large was used, but the paper does not say so 2312.00451 tab. 5. Not resolved here.
• FSGS, correlation term sign: the paper aims to align depth structure, but eq. 8 adds an L1 norm of the correlation with no stated sign, which as written would push the correlation towards zero 2312.00451 eq. 8 2312.00451 §3.3. Not resolved here.
• SpatialRGPT, depth connector start: the paper says the depth connector is initialised from the RGB connector 2406.01584 §3.3. The card records that the released code starts it from random weights (see its Paper vs code notes). Not resolved here.
• NeuralRecon, feature volume: the code adds one channel, the normalised mean depth of each voxel over its views, to the image features back_project.py neucon_network.py. The paper does not mention it 2104.00681 §3.2. Not resolved here.
• KITTI Eigen, median-scaled against absolute error: HR-Depth's KITTI rows use a per-image median scale against LiDAR 2012.07356 tab. 2 caption. The absolute-error papers above report on the same split without that scale 2011.14141 tab. 3 2204.00987 tab. 1 2203.01502 tab. 1. The two sets of numbers are not comparable.
• HR-Depth, paper against released code: the paper gives fSE reduction r = 4 2012.07356 §3.4, and the code uses 16 layers.py. The paper gives a learning rate of 1e-3, and the repository's README says 1e-4 2012.07356 §4.2. The paper does not say which setting produced its table. Not resolved here.
• Glass and mirror fine-tune, internal copy error: the full method's "Other" row in the Booster ablation is identical to the "Ours Other" row of the NYU table, which cannot both be right 2408.06083 tab. 3 2408.06083 tab. 1. This page uses only the ToM rows. Not resolved here.

Papers

• Omnidata (Eftekhar et al., 2021): renders labelled views from 3D scans and trains a DPT-Hybrid depth model on the result.
• AdaBins (Bhat et al., 2020): per-image adaptive bins from a mini-ViT head, with hybrid regression and a Chamfer term on the bin centres.
• BinsFormer (Li et al., 2022): depth from per-image adaptive bins generated by a Transformer decoder, with multi-scale refinement.
• NeW CRFs (Yuan et al., 2022): depth from a Swin encoder and a decoder that refines the prediction with windowed fully connected CRF energies.
• Canopy height maps (Tolan et al., 2023): canopy height from satellite RGB with a frozen self-supervised encoder and a log-scale loss borrowed from depth estimation.
• Hand-object reconstruction (Cao et al., 2020): reconstructs a hand and a known object from one image, using a monocular depth map as one fitting term.
• SparseGS (Xiong et al., 2023): few-view Gaussian splatting that compares rendered depth with a relative monocular depth prior through a Pearson correlation loss.
• 3D Object Detection survey (Qian et al., 2021): survey of 3D detection from images, LiDAR and fusion; its image-based pseudo-LiDAR methods estimate depth before lifting it to a point cloud.
• Metric3D v2 (Hu et al., 2024): metric depth and surface normals from one image, given the focal length, via a canonical focal-length transform and joint recurrent refinement.
• DriveSplat (Wang et al., 2025): dynamic driving-scene Gaussian splatting that uses relative depth and normal priors from pretrained models during training.
• CaDDN (Reading et al., 2021): monocular 3D detector that learns a per-pixel depth distribution, lifts image features into a frustum and collapses it to bird's-eye view.
• Marigold (Ke et al., 2023): affine-invariant depth from a Stable Diffusion U-Net fine-tuned on synthetic data, with a 10-run ensemble.
• SynSin (Wiles et al., 2019): single-image view synthesis that learns depth through a differentiable point-cloud renderer, with no depth labels.
• FSGS (Zhu et al., 2023): few-view Gaussian splatting that adds Gaussians by proximity-guided unpooling and compares rendered depth with a pretrained monocular prior through a correlation loss.
• NeuralRecon (Sun et al., 2021): real-time TSDF reconstruction from monocular video, predicting a volume per fragment with a ConvGRU that fuses fragments, instead of per-frame depth maps.
• SpatialRGPT (Cheng et al., 2024): region-prompted vision-language model with an optional relative-depth input, trained on synthetic 3D scene-graph question answers.
• HR-Depth (Lyu et al., 2020): self-supervised depth from video with a dense-skip U-Net decoder and an fSE fusion block, plus a distilled MobileNetV3 variant.
• Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces (Zhang et al., 2024): fine-tunes Depth Anything V2 Large on Hypersim with a gradient-domain loss on transparent and mirror regions and tone-mapping augmentation.