DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation

In plain words

A single photo does not say how far away each thing is. Monocular depth estimation is the task of guessing a distance for every pixel from that one picture. The authors observed that good guessing needs two kinds of information. The first is the whole scene: which objects are far from which. The second is fine local detail, such as the texture of a nearby object. Transformer networks are good at the first, and convolutional networks are good at the second.

DepthFormer keeps both. A Transformer (a Swin model) reads the whole image, and a small convolutional branch keeps local detail. Those two branches only meet at the end, so the paper adds a module, HAHI, that lets their features exchange information at several scales. Attention over every position of a high-resolution map would need too much memory. HAHI therefore attends only to a few learned sample points around each location.

The authors report that this design beats earlier methods on the NYU Depth v2 and KITTI datasets. On the KITTI online benchmark it was the most competitive result at the time of submission.

TL;DR

• Supervised depth estimation: one RGB image in, one depth value per pixel out §1 §4.1.
• Encoder: Swin Transformer stages for global context, plus a convolution stem for local detail §3.2.
• HAHI, a deformable-attention neck, enhances the Transformer features and models affinity with the convolution features, then feeds a DenseDepth-style decoder §3.3.
• Author-reported: NYU Depth v2 delta1 0.921 vs AdaBins 0.903 tab. 2; KITTI Eigen delta1 0.975 vs 0.964 tab. 3.
• The official Apache-2.0 code differs from the paper in several places (loss weight, reference-point source, fusion kernel, window size, warm-up, and the convolution branch). Read "Paper vs code" before implementing.

Problem

Monocular depth needs global context, because the distance of an object depends on the objects around it, and it needs local detail, because object boundaries and textures carry depth cues §1. The authors argue that a pure CNN encoder has a limited receptive field, so it fails on large regions such as wall decorations and carpets in NYU images §3.1 fig. 3.

A pilot study on KITTI, built on the DPT framework, compared ViT-Base with ResNet-50 §3.1. ViT-Base is better on far objects (60–80 m delta1 0.727 vs 0.600) but worse on near objects (0–20 m delta1 0.955 vs 0.973), and worse overall (0.938 vs 0.952) tab. 1. The authors explain the near-range loss by the Transformer's weak spatial inductive bias and by its fixed-scale tokens §3.1.

Global attention over high-resolution feature maps is too memory-hungry. The paper calls this cost unbearable but gives no memory measurements §3.3. The working assumption is that a few key points are enough to describe the scene structure §3.3 §5.

Method

Overview. DepthFormer has an encoder with two branches on the same input image, a HAHI neck, and a decoder. The decoder is the baseline DenseDepth-style decoder with up-convolution layers §3.3 fig. 2.

1. Transformer branch. The image is split into non-overlapping patches. A linear layer embeds them, and then L Transformer layers run. Each layer applies multi-head self-attention and an MLP, with a LayerNorm before each and a residual connection around each §3.2 eq. 1. The paper uses Swin Transformer, which adds local windows, shifted windows and patch merging for hierarchical features §3.2.
2. Transformer features F. The paper says it follows DPT and samples and reassembles N = 4 maps from selected Transformer layers, written f^n with C_n channels §3.2 §4.3. See "Paper vs code" for what the code does.
3. Convolution branch G. The paper uses only the first ResNet block, with C_g output channels. It argues that deeper blocks wash out low-level features §3.2 §4.5 fig. 12.
4. Projection. Each Transformer map goes through a 1×1 convolution to a common width C_h. Then all levels are flattened into one token set X, with a learned level embedding added §3.3.
5. Deformable self-attention. Queries, keys and values are linear projections of X eq. 2. Each query attends to a few sampled points around a reference location, with learned attention weights and offsets eq. 3. The paper uses 8 reference points and 8 heads §4.3. The output is split back to the original level sizes, and the enhanced map is fused with the input map by concatenation and a 1×1 convolution §3.3 fig. 5. This gives F_o.
6. Cross-attention for affinity. G is projected by a 1×1 convolution to C_h and flattened to queries. Keys and values come from the enhanced Transformer tokens. Reference points are predicted from the query embedding with a linear layer and a sigmoid §3.3.
7. Fusion. The cross-attention output is reshaped to G's grid and fused with G by concatenation and a 1×1 convolution, which gives G_o §3.3.
8. Loss. Both F_o and G_o go to the decoder. The depth loss is a scale-invariant log error with weights λ = 0.85 and α = 10 §3.3 eq. 4.

# HAHI: enhance the Transformer maps F and model affinity with G  [§3.3]
# Input:  F, list of 4 maps (B, C_n, H_n, W_n), one per Transformer stage
#         G, convolution-branch map (B, C_g, H_g, W_g)
# Output: F_o, list of 4 maps; G_o, map with the shape of G

def hahi(F, G, C_h):
    # Project every level to the same width C_h with 1x1 convs  [§3.3]
    F_h = [conv1x1(f, out_channels=C_h) for f in F]
    # Flatten all levels into one token set X; add a level embedding  [§3.3]
    X = concat([flatten(f) + level_embed[n] for n, f in enumerate(F_h)])
    # Deformable self-attention: each query samples a few learned points,
    # with 8 reference points  [§4.3] [eq. 3]
    X_hat = deformable_attention(queries=X, values=X, num_points=8)
    # Split X_hat back into the original per-level shapes  [§3.3]
    F_enh = unflatten(X_hat, shapes_of(F_h))
    # Fuse enhanced and input maps per level: concat, then 1x1 conv  [§3.3]
    F_o = [conv1x1(concat([f, e])) for f, e in zip(F, F_enh)]
    # Affinity queries come from G, projected to C_h  [§3.3]
    Q = flatten(conv1x1(G, out_channels=C_h))
    # Keys and values are X_hat; reference points come from the query
    # embedding through a linear layer and a sigmoid  [§3.3]
    points = sigmoid(linear(query_embedding(Q)))
    G_att = deformable_attention(queries=Q, values=X_hat, points=points)
    # Fuse G_att with G: concat, then 1x1 conv  [§3.3]
    G_o = conv1x1(concat([unflatten(G_att, shape_of(G)), G]))
    return F_o, G_o

# Depth loss: scale-invariant log error  [§3.3] [eq. 4]
# Input:  pred (B, H, W), predicted depth; gt (B, H, W), 0 = no depth
# Output: scalar loss

def depth_loss(pred, gt, lam=0.85, alpha=10.0):
    # Keep only pixels with valid ground truth; T is their count  [§3.3]
    valid = gt > 0
    # Per-pixel log difference h_i  [eq. 4]
    h = log(pred[valid]) - log(gt[valid])
    # Mean of h squared minus lam times the squared mean of h  [eq. 4]
    spread = mean(h ** 2) - lam * mean(h) ** 2
    # Weight by alpha  [§3.3]
    return alpha * sqrt(spread)

Key equations, as written in the paper:

Transformer layer l, with the MLP line numbered eq. 1. The source numbers only that line:

\hat{z}^{l} = \mathrm{MSA}\left(\mathrm{LN}\left(z^{l-1}\right)\right) + z^{l-1}

z^{l} = \mathrm{MLP}\left(\mathrm{LN}\left(\hat{z}^{l}\right)\right) + \hat{z}^{l} eq. 1

Queries, keys and values from the flattened tokens X eq. 2:

\mathbf{Q} = X\mathbf{P}_Q, \quad \mathbf{K} = X\mathbf{P}_K, \quad \mathbf{V} = X\mathbf{P}_V eq. 2

Deformable attention for query q eq. 3:

\mathrm{DAttn}(x_q, x_v, p_q) = \sum_{k \in \Omega_k} A_{qk}\, x_v\left(p_q + \Delta p_{qk}\right) eq. 3

The paper normalises A_qk to sum to 1 and uses bilinear interpolation for fractional sampling positions §3.3.

Depth loss, with h_i = log d̃_i − log d_i and T the number of pixels with valid ground truth eq. 4 §3.3:

\mathcal{L}_{pixel} = \alpha \sqrt{\frac{1}{T}\sum_i h_i^{2} - \frac{\lambda}{T^{2}}\left(\sum_i h_i\right)^{2}} eq. 4

Hyperparameters

Values are as the paper states them unless the Notes column says otherwise. "Not stated" means the paper gives no value. Code-only values are marked and are not confirmed by the paper.

Results

Author-reported. Baseline numbers are copied from the original papers according to the captions tab. 2 tab. 3 tab. 5. §4.4 also says the authors reproduced AdaBins from its released code and weights §4.4. This conflicts with the captions. Setups:

• NYU Depth v2. 50K training pairs, Eigen center crop, depth capped at 10 m §4.1 §4.2.
• KITTI Eigen split. Garg crop, depth range 0–80 m §4.1 tab. 3.
• KITTI benchmark. Trained on the standard benchmark split, scored by the online server §4.4 tab. 4.
• SUN RGB-D. Trained on NYU only, no fine-tuning, 5050 test images §4.1 tab. 5.

tab. 2 NYU Depth v2, test set:

tab. 3 KITTI Eigen split, 0–80 m:

tab. 4 KITTI benchmark test set (online server, submitted Nov. 16, 2021 per §4.4):

tab. 5 SUN RGB-D, trained on NYU only, no fine-tuning (baseline numbers from the AdaBins paper per the caption):

tab. 6 Ablation on NYU. CB is the convolution branch. LP is 22K-ImageNet pre-training:

tab. 7 HAHI ablation on NYU, Swin-T with CB:

tab. 8 Detailed KITTI results for the pilot study and the Swin-T ablation. tab. 1 is the first six rows of this table:

Speed. The paper reports inference time for DepthFormer without HAHI on an RTX 3060 at 352×1216, and says it is faster and more accurate than the compared methods §4.5 fig. 14. No timing table is given.

Training cost. About 5 hours for 38.4k iterations on 8 V100 GPUs §4.3.

Paper vs code

Code: the official toolbox linked in the abstract and footnote §1 §5. The repository's LICENSE file is Apache License 2.0, and its header names the MMSegmentation authors. I cloned it at commit d8aff50e75c4b0c183361c7690959d0269b45a9d. I did not search further, because the paper links the repository.

Each difference below cites both sides. Items are ordered by how much they change the result.

• Loss weight α. The paper uses α = 10 §3.3. The code uses loss_weight = 1.0 and no override in the DepthFormer configs [code: configs/base/models/depthformer_swin.py#L38-L39] sigloss.py.
• Reference-point source. The paper predicts cross-attention reference points from the "affinity query embedding" §3.3. The code feeds a sinusoidal positional encoding built from an all-zero mask through a linear layer and a sigmoid hahi.py. The points therefore do not depend on image content hahi.py.
• Fusion convolutions. The paper fuses with 1×1 convolutions §3.3. The code uses 3×3 convolutions for both fusions hahi.py hahi.py hahi.py hahi.py.
• Common width C_h. The paper sets it to the median channel dimension of F §4.3. The code hard-codes 512 for Swin-L and 256 for Swin-T depthformer_swinl_22k_w7_nyu.py depthformer_swint_w7_nyu.py. For the Swin-L widths 192, 384, 768 and 1536, the median would be 576.
• Window size. The paper says 12 §4.3. The code uses 7 [code: configs/base/models/depthformer_swin.py#L13].
• Convolution branch. The paper uses the first ResNet block §3.2. The code builds the stem only (7×7 stride-2 convolution, BN, ReLU). The max-pool and residual stages run only when num_stages ≠ 0, and the base config sets num_stages = 0 depthformer_swin.py depthformer_swin.py [code: configs/base/models/depthformer_swin.py#L31].
• Transformer maps. The paper follows DPT and reassembles the selected layers §3.2. The code takes the four Swin stage outputs directly into HAHI. The DPT-style reassemble module is not on the DepthFormer path depthformer_swin.py.
• Learning rate. The paper gives 1e-4 for the whole network §4.3. The same section also gives 6e-5 for Transformer models and a "1-cycle" schedule. The code uses cosine annealing, 6e-5 for Swin-T and 1e-4 for Swin-L depthformer_swinl_22k_w7_nyu.py depthformer_swint_w7_nyu.py.
• Warm-up. The paper says 30% of iterations and also "500 iterations" §4.3. The code uses 12800 of 38400 iterations, which is 33% depthformer_swinl_22k_w7_nyu.py.
• Batch size. The paper says batch size 2 on 8 GPUs §4.3. The code's samples_per_gpu = 2 is per GPU, so the global batch is 16 on 8 GPUs unless the paper's 2 is the global value depthformer_swinl_22k_w7_nyu.py. I could not resolve this from the sources.
• Test-time augmentation. The mirror-averaging paragraph in the paper source is commented out §4.3. The active text does not describe test-time augmentation. The code averages the prediction with its horizontal flip at test time [code: configs/base/datasets/nyu.py#L29] [code: configs/base/datasets/kitti.py#L32] encoder_decoder.py.
• Pre-training. The paper says ImageNet pre-training §4.3. The Swin-L headline configs load an ImageNet-22K checkpoint depthformer_swinl_22k_w7_nyu.py.
• Prediction head. The paper does not say how the depth output is produced. The code uses ReLU plus min_depth = 1e-3 by default decode_head.py. Prediction is resized to the ground-truth size before the loss decode_head.py.
• Variance estimator. The paper's formula uses 1/T eq. 4. The code calls torch.var, which divides by T − 1 by default, and adds eps = 1e-3 inside the logs sigloss.py sigloss.py.
• Loss mask. The paper defines T as the number of valid pixels §3.3. The loss mask is target > 0 only, because max_depth is not passed to SigLoss sigloss.py. I did not check whether the loaders clip ground truth above the cap.
• Baseline numbers. The table captions say baselines come from the original papers tab. 2 tab. 3. §4.4 says AdaBins was reproduced §4.4. Which numbers the AdaBins row uses is therefore unclear.

Pitfalls

• Loss scale. The paper's α = 10 makes the loss 10× larger than the code's. The effect on training is untested. The optimizer config clips gradients at max-norm 35 depthformer_swinl_22k_w7_nyu.py, so the loss scale matters at least through that clip.
• Reference points. Because the code's reference points come from a fixed positional encoding, the cross-attention sampling pattern is the same for every image, given the learned weights. The paper's description suggests it depends on the query features. If you follow the released code, use its behaviour; the paper does not settle the question.
• Undefined terms. The paper does not define the "affinity query embedding", the reference points for self-attention, or the C_h "median" rule in the same terms as the code.
• Deformable kernels. The code uses MMCV's MultiScaleDeformableAttention CUDA op hahi.py. A pure PyTorch version is possible, but I did not check whether it matches the op numerically.
• Memory. The paper says global attention is too costly, but gives no memory numbers. The speed claim applies to DepthFormer without HAHI §4.5 fig. 14.
• Ablation arithmetic. The paper says multi-level self-attention gives a 4.9% RMS gain §4.5. From tab. 7 the single-level to multi-level step is 0.427 → 0.408, which is 4.4%. The paper's 2.2% DCA gain (0.408 → 0.399) does match.
• Benchmark claim. The paper says sqErrorRel improves by 16% §4.4. Against ViP-DeepLab's 2.19, DepthFormer's 1.82 is about 17% lower. The KITTI test numbers come from the online server and cannot be reproduced from the repository alone.
• Column labels. In tab. 1, tab. 6 and tab. 8, the third accuracy column is labelled δ2 twice. The values (0.998–1) show it is δ3.
• Interpretation of "first block". The paper says the best result comes from one ResNet block, and that more blocks hurt §4.5 fig. 12. The code's stem is not the same as a ResNet block with residual layers, so the ablation may not be reproducible from the code path alone.
• Budget. Training takes 38.4k iterations at batch size 2 on 8 V100 GPUs, about 5 hours §4.3. Check the batch-size question above before you budget.

Lineage

• Builds on: Swin Transformer for the encoder §3.2. AdaBins supplies the λ = 0.85 and α = 10 loss settings and the warm-up practice §3.3 §4.3. Deformable DETR (cited as zhu2020deformabledetr) supplies the deformable attention §3.3. DPT (cited as ranftl2021dpt) supplies the reassembly and the ViT-based depth framing §3.2. The DenseDepth-style decoder is cited as alhashim2018densedepth §3.3.
• Compared against: AdaBins, BTS, DPT, TransDepth, DORN, ViP-DeepLab, PWA, BANet and others in tab. 2 to tab. 5.
• Superseded by: not known from content on this wiki.