Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces

In plain words

Modern depth estimators work well on ordinary scenes, but they go wrong on glass and mirrors. A window or a mirror looks like a flat surface, yet the picture inside it shows a room that is not there, and the model often reports the depth of that reflected room. Strong lighting makes this worse, and so does dim light.

The authors fix this by changing how the model is trained rather than adding a separate mask input at test time. First, they generate extra training pictures by changing the brightness curve of synthetic indoor images, so the model sees the same scene under many lighting conditions. Second, they add a training penalty that only looks at the glass and mirror areas, and only compares the edges of the predicted depth with the edges of the true depth. The idea is that a glass pane should be a smooth plane, so its depth should not show the sharp jumps that a reflection causes. Third, when several photos of the same scene taken at different exposures are available, they blend the photos inside a pretrained image autoencoder and run depth estimation on the blended result.

The authors report that the method beats the model it starts from on the glass and mirror regions of two test sets. Their numbers come from their own evaluation.

TL;DR

The method fine-tunes Depth Anything V2 Large on the synthetic Hypersim dataset §3, §4.2. It adds a gradient-domain loss on transparent and mirror (ToM) regions, computed with robust per-region normalization and trimming of the largest 20% of residuals §3.2, eq. 2–5. It also applies random tone-mapping during training §3.1, eq. 1. An optional test-time step averages the Stable Diffusion VAE latents of several exposures and decodes one image §3.3. The paper reports author-computed gains over Depth Anything V2 on ToM regions, not a new benchmark protocol tab. 2, tab. 1.

Problem

Monocular depth models trained on general data often predict the depth of the reflected content, not of the transparent or mirror surface in front of it §1. Earlier work for this problem, Depth4ToM, needs a ToM segmentation mask at inference, inpaints the RGB image with random colours, and aggregates the predictions into pseudo-labels §1, §2.2. The authors argue these pseudo-labels are not reliable and that the random inpainting needs a lot of hyperparameter tuning §1. Lighting has a strong effect on these surfaces: strong light can wash out glass texture, and weak light adds noise and colour drift §1.

The paper's scope is zero-shot evaluation on the Booster dataset and on the small NYU Depth Dataset V2 subset annotated by Mirror3D §4.1.

Method

The base model is Depth Anything V2 Large §4.2. Training uses Hypersim with random tone-mapping §3.1, §4.2. The loss is the sum of the ToM guidance loss and the scale-shift-invariant loss from MiDaS §4.2. The test-time fusion is optional and needs several exposures of the same scene §3.3.

Random tone-mapping augmentation. The augmented image is a gamma-curved, scaled copy of the original, with γ = 1/2.2 and a scale α chosen so that a chosen percentile intensity maps to 0.8 §3.1, eq. 1. The percentile is drawn at random between 70 and 99 §3.1. The paper motivates this by lighting sensitivity of ToM surfaces §3.1, §1. Applying it requires the original Hypersim assets, which the paper says are available for this synthetic dataset §3.1.

I_{aug}=\alpha I_{ori}^{\gamma} eq. 1

Non-Lambertian surface regional guidance (NSRG). The guidance loss compares the gradient magnitude of the predicted depth D and ground-truth depth D★ inside the ToM mask M §3.2. The paper first gives a least-squares-aligned L1 form §3.2, eq. 2, then replaces it with a robust version §3.2, eq. 3–5.

\mathcal{L}_{ToM}=\frac{1}{M}\sum_{i=1}^M\Vert s \nabla D_i +t -\nabla D_i^\star \Vert_1 \,, \quad (s,t)=\mathop{\arg\min}\limits_{s,t}\sum_{i=1}^M\Vert (s \nabla D_i +t) -\nabla D_i^\star \Vert_2^2 eq. 2

The robust version centres each gradient map on its median and scales it by its mean absolute deviation, both computed over M §3.2, eq. 3. Each map is then normalised §3.2, eq. 4. The final loss takes the L1 residuals, discards the largest 20% of them, and normalises by 2M §3.2, eq. 5.

t(\nabla D)= median (\nabla D), \quad s(\nabla D)=\frac{1}{M}\sum_{i=1}^M\vert \nabla D - t(\nabla D) \vert eq. 3

\hat{\nabla D}=\frac{\nabla D - t(\nabla D)}{s(\nabla D)},\quad \hat{\nabla D^\star}=\frac{\nabla D^\star - t(\nabla D^\star)}{s(\nabla D^\star)} eq. 4

\mathcal{L}_{ToM}=\frac{1}{2M}\sum_{i=1}^{0.8M}\Vert \hat{\nabla D_i} - \hat{\nabla D_i^\star} \Vert_1 eq. 5

The paper's reasoning is that most ToM objects are continuous planes, so the predicted gradients inside the mask should match the ground truth after normalization §3.2.

# ToM regional guidance loss  [§3.2] [eq. 3-5]
# Input:  pred_depth, gt_depth (H, W); mask (H, W) bool for ToM pixels
# Output: scalar loss
# Gradient operator not stated; the text says "gradient magnitude".
def tom_guidance_loss(pred_depth, gt_depth, mask):
    # Gradient magnitude of each depth map  [§3.2]
    grad_pred = gradient_magnitude(pred_depth)    # (H, W)
    grad_gt = gradient_magnitude(gt_depth)        # (H, W)
    # Centre each map on its median inside the mask  [§3.2] [eq. 3]
    pred_shift = masked_median(grad_pred, mask)
    gt_shift = masked_median(grad_gt, mask)
    # Scale each map by its mean absolute deviation inside the mask  [eq. 3]
    pred_scale = masked_mean_abs_dev(grad_pred, mask, pred_shift)
    gt_scale = masked_mean_abs_dev(grad_gt, mask, gt_shift)
    # Normalise both maps  [§3.2] [eq. 4]
    pred_norm = (grad_pred - pred_shift) / pred_scale
    gt_norm = (grad_gt - gt_shift) / gt_scale
    # Per-pixel L1 residuals inside the mask  [§3.2] [eq. 5]
    residuals = abs(pred_norm - gt_norm)[mask]
    count = number_of_pixels(mask)                # M in eq. 5
    # Sort ascending, then keep the smallest 80%  [§3.2] [eq. 5]
    # Sorting is implied by "0.8M" in the sum, not stated in the text.
    kept = sorted(residuals)[: int(0.8 * count)]
    return sum(kept) / (2 * count)

Total training loss. The total loss is written as L_ToM + L_ssi, with no weights §4.2.

# Training objective  [§4.2]
# The paper writes a plain sum; no weights are stated.
def training_loss(pred, gt, tom_mask):
    loss_tom = tom_guidance_loss(pred, gt, tom_mask)  # [§3.2] [eq. 5]
    loss_ssi = scale_shift_invariant_loss(pred, gt)   # MiDaS form  [§4.2]
    return loss_tom + loss_ssi

Diverse lighting images fusion (optional, test time). The paper stacks the images of one scene, encodes each with the pretrained Stable Diffusion VAE, averages the latent features, and decodes the average to one image §3.3. The fused image is then passed to the depth model §3.3, fig. 6. The paper says the VAE is trained with a KL-divergence reconstruction loss on Open Images §3.3.

# Diverse lighting images fusion at test time  [§3.3]
# Input:  exposures, a list of N images of one scene, each (H, W, 3)
# Output: one fused image (H, W, 3) to feed the depth model
def fuse_exposures(exposures, vae):
    # Encode each exposure with the pretrained Stable Diffusion VAE  [§3.3]
    latents = [vae.encode(image) for image in exposures]
    # Average the latent features, not the pixels  [§3.3]
    mean_latent = sum(latents) / len(latents)
    # Decode the averaged latent to one image  [§3.3]
    return vae.decode(mean_latent)

Hyperparameters

Values are as stated in the paper (LaTeX source; no appendix exists in the source). Where a value is missing, the table says so.

Results

Author-reported. Setup: zero-shot evaluation after training on Hypersim; baselines are the public models named in §4.2 §4.1.

Booster, training split (456 samples with GT) tab. 2.

NYU Depth V2 subset with Mirror3D masks, 125 samples tab. 1.

Ablation on Booster, ToM region only tab. 3. The ablation columns are RTA (tone-mapping augmentation), NSRG (regional guidance) and DLIF (lighting fusion at test time).

Headline numbers, author-reported:

• Booster, ToM δ1.05: 0.859 vs 0.644 for Depth Anything V2, +33.39% tab. 2. The abstract attributes this to Depth Anything V2 and it matches the table values.
• NYU Mirror3D, ToM δ1.05: 0.606 vs 0.576 for Depth Anything V2, +5.21% tab. 1.
• Booster ToM relative to Depth Anything V1 (δ1.05 0.266): +222.93%; NYU ToM relative to V1 (0.207): +192.75% §4.3, tab. 1, tab. 2.
• Ablation, Booster ToM δ1.05 vs no-module baseline: RTA +24.22% (0.800); NSRG +26.55% (0.815) §4.4.1, §4.4.2, tab. 3.
• TRICKY2024 competition test set: ToM δ1.05 of 90.75 [abstract]. The paper gives no table or setup for this result.

Paper vs code

No official code.

No code link is in the paper, its abstract or its LaTeX source. Searches for the method name, the title words and the authors' lab found only third-party repositories (for example HatemHosam/Hypersimify_DA2 and IkeFwd/Icdepth), which are not used [search: github, 2026-11-01].

Pitfalls

• Which baseline the gains refer to. The abstract's 33.39% and 5.21% are against Depth Anything V2. The text's larger "maximum" gains of 222.93% and 192.75% are against Depth Anything V1, which is the weakest baseline in the tables §4.3, tab. 1, tab. 2. Compare against the same baseline when reporting.
• Inconsistent "Other" rows in tab. 3. For the full method (RTA+NSRG+DLIF) the "Other" row (δ1.05 0.655, Abs Rel 0.059, RMSE 0.209) is identical to the NYU "Ours Other" row in tab. 1 tab. 3, tab. 1. It cannot be right: the All value (0.885) is above the ToM value (0.860), so, if All pools the ToM and Other pixels, the Other value must also be above 0.885. The row is likely a copy error. Also the RTA+NSRG row in tab. 3 gives Other δ1.05 = 0.880, while tab. 2 "This paper" gives Other = 0.899, with identical All and ToM values. Do not use the Other rows without rechecking.
• DLIF's effect is marginal on ToM. Adding lighting fusion changes ToM δ1.05 from 0.859 to 0.860 tab. 3. The text calls this an improvement §4.4.3. Because of the copy error above, the Other-region effect of DLIF cannot be read from tab. 3.
• Loss definitions. The paper defines the residuals with the gradient magnitude, so the loss compares scalars, not gradient vectors §3.2. This discards gradient direction (inference from the text). The gradient operator (finite differences, stride, padding) is not stated.
• Two versions of the loss. Eq. 2 (least squares) is described, then replaced by the robust version in eq. 3–5 §3.2. Only eq. 3–5 are used in training. The text does not say whether eq. 2 is ever computed.
• Overloaded symbol M. M is both the ToM mask and the pixel count in the sums and normalizers §3.2, eq. 2–5. Implement M as the mask and take its count for the normalizer. The paper's 1/(2M) factor is unexplained.
• Trimming order and scope. Eq. 5 sums the first 0.8M terms, which implies sorting the residuals. The text does not say whether trimming happens per image or per batch, or whether sorting is by residual value §3.2.
• Missing gradient-scale details. Normalization in eq. 3–4 uses a median and a mean absolute deviation over the mask. The behaviour when a mask is empty, or when the scale is zero (flat depth), is not stated.
• Hypersim ToM mask threshold. The mask comes from thresholding a reflectance coefficient map §3.2. The threshold value and the reflectance map source are not stated.
• Augmentation formula is under-specified. Eq. 1 gives I_aug = α I^γ, and the text says α makes a percentile intensity map to 0.8 §3.1. The text does not write α in closed form. The natural reading, α = 0.8 / r_p^γ, is our inference from the words. It is not stated whether r_p is taken before or after the gamma step, what the intensity range of I_ori is, or whether the percentile is uniform on [70, 99]. Clip behaviour above 1 is not stated.
• Per-sample or per-batch augmentation. Not stated whether a new percentile is drawn per sample, per iteration or once.
• Loss weights. The total loss is written as a plain sum §4.2. Implement weight 1 for each term, or check the sum, since no weight is given.
• MiDaS scale-shift loss. L_ssi is cited to MiDaS §4.2. The variant (gradient-matching term, scale and shift estimator) is not restated in this paper.
• Training length and convergence. "Max 5 epochs" and "70K samples" are given §4.2, §4.3, but the convergence test is not stated. Hypersim has about 77,400 images §2.2, so the 70K subset is not identified.
• Ablation baseline is ambiguous. The first row of tab. 3 equals the Depth Anything V2 row of tab. 2. §4.4.1 says RTA was applied by re-training DA V2 on Hypersim with the MiDaS loss §4.4.1. It is not clear whether the "baseline" row is the public checkpoint or a Hypersim fine-tune with L_ssi only. Its numbers equal the Depth Anything V2 row of tab. 2, which does not settle the question.
• Evaluation protocol is not stated. The paper does not say whether predictions are scale-and-shift aligned before scoring, or evaluated in metric units, nor how the Booster and NYU masks were applied to predictions. The baselines mix affine-invariant and metric outputs §2.1, so this matters for comparison.
• TRICKY2024 result. The abstract's 90.75 has no unit in the paper. Other δ values in the tables are fractions, so the number is probably a percentage, but the paper does not say so [abstract]. No setup, table or training details are given §1.
• Fusion details missing. The VAE version, latent scaling factor, the number of exposures used on Booster §4.4.3 and the resolution fed to the depth model after decoding are not stated. Booster images are 4112 × 3008 §4.1. Downscaling before training or inference is not stated.
• Booster train split used for evaluation. The quantitative Booster results use the split that has ground truth, 456 samples §4.1. The paper does not discuss whether any of these scenes overlap with its training data, which is Hypersim only.
• Failure mode from the paper. Extremely overexposed images lose the glass texture, and the method can fail there §5.
• Cost. Fine-tuning of Depth Anything V2 Large at 518 × 518 with batch size 4 for 5 epochs §4.2. The paper gives no hardware, time or memory figures.

Lineage

• Builds on: Depth Anything V2 (the base model, §4.2); MiDaS scale-shift-invariant loss §3.2, §4.2; Hypersim training data §4.2; Stable Diffusion VAE for fusion §3.3.
• Compared against: Depth Anything V1, Metric3D, DepthFM §4.2.
• Related prior work: Depth4ToM (Learning Depth Estimation for Transparent and Mirror Surfaces, ICCV) uses inpainting with a mask at inference §1, §2.2. Mirror3D refines depth with a predicted mirror mask §2.2.
• Superseded by: not known from this card's sources.