BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation

In plain words

Most depth models guess one distance for each pixel directly. BinsFormer first decides, for each photo, a small set of depth ranges ("bins") that suit that scene. Each pixel then gets a probability for every range, and its depth is the weighted average of the range centres. Choosing the ranges per image lets the model spend its resolution where the scene's depths actually are.

The bins come from a Transformer decoder that looks at the whole image and proposes them in one pass. The authors keep this global part apart from the detailed per-pixel features, so the two do not overwrite each other. Two more additions help. One extra query predicts what kind of room or street the photo shows. The bins are also refined at three image resolutions, from coarse to fine.

The model learns from photos with measured depth maps, and it is judged on standard indoor and outdoor depth benchmarks.

TL;DR

BinsFormer estimates depth from one RGB image as a probability-weighted average of bin centres that are generated per image. A Transformer decoder produces the bins from learned queries, and a separate per-pixel branch produces the probabilities §3.2.1 §3.2.2 §3.2.3. An auxiliary scene-classification query and multi-scale refinement are added §3.2.4 §3.2.5. Author-reported, the authors report that it outperforms the compared methods on KITTI, NYU and the SUN RGB-D cross-dataset test §4.3.1 §4.3.2 §4.3.3. The official code differs from the paper in several training details, listed under Paper vs code.

Problem

Per-pixel regression is the most direct formulation, but it converges slowly and gives weaker results §1. Classifying discretised depth gives sharp steps between levels §1. Classification-regression methods avoid both problems by predicting a probability over bins and taking a weighted sum of bin centres §1. Their bins were fixed, uniform or log-spaced, or learned once per dataset §1. AdaBins makes the bins adaptive, but the authors list three limits of it: the bins are predicted from the highest-resolution feature map, which limits global context; bins and per-pixel probabilities share one feature; and its chamfer loss adds an inductive bias §1. The method assumes a per-dataset minimum and maximum valid depth §3.2.3.

Method

The model has a pixel-level module, a Transformer module and a depth-estimation module, plus a scene-classification auxiliary task and multi-scale refinement §3.1 fig. 2. Bin generation is framed as a direct set-to-set prediction problem solved by a Transformer decoder §1 §3.2.2.

1. Per-pixel features. A backbone and a decoder give per-pixel features f_p of size H×W×C and S multi-scale features F §3.2.1. Any depth decoder can serve here. The code uses a light FPN §4.2.3 binsformer_head.py.
2. Bin and scene queries. N bin queries (64 by default §4.2.4 tab. 6) and one scene query enter the Transformer decoder, which attends to F §3.2.2 binsformer_head.py. Each query output feeds two heads: a bin-length logit and a bin embedding from a 3-layer ReLU MLP §3.2.2. The scene query feeds a 3-layer MLP that classifies the scene §3.2.4 transformer.py.
3. Bin lengths and centres. The bin-length logits pass through a softmax over the N bins §3.2.2. Each bin's centre is then computed from the cumulative lengths, with the dataset's minimum and maximum depth as bounds §3.2.3 eq. 1.
4. Probabilities. For every pixel, a softmax over the N bins of the dot product between f_p and each bin embedding gives the probability distribution P §3.2.3.
5. Depth. The depth at each pixel is the sum of centre times probability over the bins §3.2.3 eq. 2.
6. Scene classification. A cross-entropy loss on the scene query's output, with a small weight μ §3.2.4 eq. 4. The paper uses 25 indoor classes, taken from the folder names of NYU §4.2.1. KITTI has no scene loss §4.2.1.
7. Multi-scale refinement. The three feature scales are handled one at a time. Each scale has L = 3 decoder layers, nine layers in total §4.2.4. The paper says the queries start at zero and carry learnable positional embeddings §3.2.5 (the code differs; see Paper vs code). In each layer, a cross-attention block to that scale's features is followed by a self-attention block and a feed-forward network §3.2.5. The depth module runs after every layer for auxiliary supervision §3.2.5.
8. Losses. The depth loss is a scaled scale-invariant loss with λ = 0.85 and α = 10 §3.2.3 eq. 3. The total loss weights each layer's losses by a per-scale weight w_s §3.2.5 eq. 4.

Equations as numbered in the paper:

c(b_i) = d_{min} + \left(d_{max} - d_{min}\right)\left(\frac{b_i}{2} + \sum_{j=1}^{i-1} b_j\right) eq. 1

\hat d = \sum_{i=1}^{N} c(b_i)\, p_i eq. 2

\mathcal{L}_{reg} = \alpha \sqrt{\frac{1}{T}\sum_i g_i^{2} - \frac{\lambda}{T^2}\Big(\sum_i g_i\Big)^2}, \qquad g_i = \log \tilde d_i - \log d_i eq. 3

\mathcal{L}_{total} = \sum_{s=1}^{S} \Big( w_s \sum_{l=1}^{L} \big( \mathcal{L}_{reg}^{s,l} + \mu\, \mathcal{L}_{cls}^{s,l} \big) \Big) eq. 4

With μ = 1e-3 and w_s = {1/2^s, 1/2^(s-1), ..., 1} as written in the paper §3.2.5 eq. 4. The symbols b_i, c, d_min and d_max are defined in §3.2.3 eq. 1. T is the number of pixels with valid ground truth, and g_i is the log-depth error §3.2.3 eq. 3.

The paper has no algorithm box. The pseudocode below is written from the description above, in the paper's form. Where the official code differs, see Paper vs code.

# BinsFormer depth prediction for one image  [§3.2] [eq. 1] [eq. 2]
# Input:  features F (multi-scale); per-pixel features f_p (B, C, H, W)
# Output: per-layer depth (B, 1, H, W); per-layer scene logits (B, 25)

def predict_depth(features, per_pixel):
    # 1. Learned queries: index 0 is the scene query, 1..N the bin queries
    #    [§3.2.2] [§4.2.4]
    queries = embed_queries(N + 1)                       # (N+1, C_q)
    # 2. Run the decoder layers, one scale per layer  [§3.2.5]
    #    (the scale order is a Paper vs code item)
    layer_outputs = transformer_decoder(queries, features, per_pixel)
    depths = []
    scene_logits = []
    for out in layer_outputs:
        # 3. Scene logits from the scene query  [§3.2.4]
        scene_logits.append(mlp_scene(out[0]))          # (B, 25)
        # 4. Bin lengths: softmax over the N bin outputs  [§3.2.2]
        bin_lengths = softmax(linear(out[1:]), axis="bins")   # (B, N)
        # 5. Bin centres from cumulative bin lengths  [§3.2.3] [eq. 1]
        centres = bin_centres(bin_lengths, d_min, d_max)      # (B, N)
        # 6. Bin embeddings from a 3-layer ReLU MLP  [§3.2.2]
        bin_embed = mlp_bins(out[1:])                         # (B, N, C)
        # 7. Per-pixel probabilities: softmax over bins of a dot product
        #    [§3.2.3]
        logits = einsum("bnc,bchw->bnhw", bin_embed, per_pixel)
        probs = softmax(logits, axis="bins")                  # (B, N, H, W)
        # 8. Depth is the probability-weighted sum of centres  [§3.2.3] [eq. 2]
        depths.append(sum(probs * centres[:, :, None, None], axis="bins"))
    return depths, scene_logits

# Training loss, summed over every decoder layer  [§3.2.5] [eq. 4]
# Input: per-layer depths and scene logits; ground-truth depth and scene label
# Scale weights w_s and layer grouping: see Paper vs code

def total_loss(depths, scene_logits, depth_gt, scene_gt):
    total = 0.0
    for layer in range(num_layers):
        # 1. Scale-invariant loss on pixels with valid ground truth  [eq. 3]
        #    alpha = 10, lambda = 0.85  [§3.2.3]
        reg = sigloss(depths[layer], depth_gt)
        # 2. Scene cross-entropy, weight mu = 1e-3  [§3.2.4] [eq. 4]
        cls = cross_entropy(scene_logits[layer], scene_gt)
        # 3. Per-scale weight w_s applied to both terms  [§3.2.5] [eq. 4]
        total = total + scale_weight(layer) * (reg + mu * cls)
    return total

Hyperparameters

Values the paper gives are cited to the paper. "Not stated" means the paper does not give the value. Code values come from the Swin-Large NYU config unless another config is named.

Results

Author-reported. Setup: NYU Depth v2 with the listed backbones, evaluated with the pre-defined Eigen centre crop §4.1.1 tab. 3; KITTI Eigen split and the official online benchmark §4.1.1 tab. 1 tab. 2; SUN RGB-D, 5050 test images, evaluated with NYU-trained models without fine-tuning §4.1.1 tab. 4. In tab. 1 † marks ImageNet-22K pre-training and ‡ marks pre-training on an auxiliary depth dataset. The same markers appear in tab. 3, whose caption does not define them.

Headline numbers. On NYU with Swin-Large†, δ1 is 0.925, REL 0.094 and RMS 0.330 tab. 3, against 0.903, 0.103 and 0.364 for AdaBins tab. 3. On KITTI Eigen with Swin-Large†, δ1 is 0.974, REL 0.052 and RMS 2.098 tab. 1, against 0.964, 0.058 and 2.360 for AdaBins tab. 1. On the KITTI online benchmark, SILog is 10.14 against 10.80 for ViP-DeepLab, which the paper reports as a 6.1% improvement tab. 2 §4.3.1. Cross-dataset, δ1 on SUN RGB-D is 0.805 with Swin-Large† and 0.771 for AdaBins tab. 4.

tab. 1 KITTI, Eigen split, depth range 0–80 m. Higher δ is better; lower error is better.

tab. 2 KITTI online depth benchmark, test set. Lower is better. Numbers are from the benchmark website.

The backbone and configuration used for this row are not stated in the caption tab. 2.

tab. 3 NYU Depth v2. Higher δ is better; lower error is better.

tab. 4 Cross-dataset: trained on NYU Depth v2, tested on SUN RGB-D without fine-tuning.

tab. 5 NYU ablation of components. δ1 higher is better; REL and RMS lower is better. The backbone is not stated in the caption. Check marks are as in the table.

tab. 6 NYU: number of bins N. δ1 to δ3 higher is better; REL, RMS and log10 lower is better. The text says the gain saturates above N = 64, which the table shows as the default.

tab. 7 NYU: multi-scale refinement. Scales used and total decoder layers.

Figure-based claims in the paper, such as the uncertainty trend and the query visualisations, are described in words in §4.4.1 and fig. 5, and not reproduced here.

Paper vs code

Official code: github.com/zhyever/Monocular-Depth-Estimation-Toolbox, at commit d8aff50e75c4b0c183361c7690959d0269b45a9d. The paper's abstract and its footnote link this repository, so it is taken as the official code. Its LICENSE file is Apache-2.0 and carries an MMSegmentation copyright header, since the toolbox builds on that framework. No code is quoted in this card.

• Scale order. The paper runs the L = 3 layers of each scale one after another §3.2.5 §4.2.4. The code's default maps layer i to scale i mod 3, so the scale changes at every layer transformer.py. The NYU and KITTI Swin-Large configs do not set operation, so the code uses this default binsformer_swinl_22k_w7_nyu.py. The other option, //, groups the layers by scale as the paper describes transformer.py. Its use in the reported results is not stated.
• Query initialisation. The paper zero-initialises the queries and gives them learnable positional embeddings §3.2.5. The code uses learnable query features, which are Xavier-initialised, and learnable query positions binsformer_head.py binsformer_head.py binsformer_head.py.
• Scene loss weight. The paper gives μ = 1e-3 eq. 4. The Swin config sets the scene loss weight to 1e-2 binsformer_swinl_22k_w7_nyu.py. The head's own default is 1e-1 binsformer_head.py.
• Which layers get supervision and how they are weighted. The paper applies both loss terms to every layer, weighted by w_s eq. 4 §3.2.5. The code supervises the depth at the 3rd and 6th layers (0-based indices 2 and 5) with weights 1/4 and 1/2, and the final layer with weight 1 binsformer_swinl_22k_w7_nyu.py binsformer_head.py. The scene loss on the auxiliary layers is added without the aux weight binsformer_head.py. The main-layer scene loss is also unweighted binsformer_head.py.
• Scale weights. The paper writes w_s = {1/2^s, ..., 1} §3.2.5. The code's values 1/4, 1/2, 1 are consistent with the reading 1/2^(S−s) for S = 3. This reading is my inference, since the paper's formula is ambiguous.
• SI loss normalisation. The paper writes the 1/T and λ/T² form of eq. 3 eq. 3. The code computes the unbiased variance of g (torch.var) plus 0.15 times the squared mean sigloss.py. The constant 0.15 is 1 − λ with λ = 0.85. The code's ε = 1e-3 inside the logs is not in the paper sigloss.py sigloss.py.
• Valid-pixel mask. The code's mask is target > 0 only, with no upper depth bound, because the config does not pass max_depth to the loss sigloss.py binsformer_swinl_22k_w7_nyu.py. The paper says "valid" ground truth §3.2.3 without a definition.
• Depth bounds. The paper uses the dataset's minimum and maximum valid depth §3.2.3. The code uses a fixed 1e-3 and 10 m (NYU) or 80 m (KITTI) binsformer_swinl_22k_w7_nyu.py binsformer_swinl_22k_w7_kitti.py.
• Warm-up and schedule. The paper gives 30% linear warm-up and cosine decay §4.2.1. The code's warm-up is 12,800 iterations, which is 33%, under a OneCycle schedule binsformer_swinl_22k_w7_nyu.py.
• Dataset sizes. The paper says about 50K NYU pairs §4.1.1. The repository's NYU training list has 24,230 lines nyu_train.txt. The paper says about 26K KITTI Eigen training images §4.1.1. The repository list has 23,157 lines kitti_eigen_train.txt. The test lists have 653 NYU and 696 KITTI lines nyu_test.txt kitti_eigen_test.txt. The paper gives 697 KITTI test frames.
• Test-time flip. The paper does not mention test-time augmentation. The NYU test pipeline sets flip = True binsformer_swinl_22k_w7_nyu.py. The model's aug_test averages the predictions over its augmented inputs encoder_decoder.py. Whether the paper's numbers use this is not stated.
• Missing configs. The paper's table rows for ResNet-18/50, Swin-Base and Swin-Base† have no config in configs/binsformer/. Only Swin-Tiny and Swin-Large configs exist there, for NYU, KITTI and NYU-to-SUN. No BinsFormer config is present for the KITTI online benchmark, and none for the ablations .
• Consistent items. The 25 scene classes §4.2.1 are in the code too. Chamfer loss is off in the configs binsformer_swinl_22k_w7_nyu.py, which matches the paper's method description. No discrepancy found there.

Pitfalls

• Layer-to-scale mapping is ambiguous. The paper's description gives consecutive layers per scale, while the released default cycles through scales every layer. Choose one and state it. The code's configs use the cycling default.
• Under the cycling default, all three supervised layers see the finest scale. The supervised layers are the 3rd, 6th and 9th (0-based indices 2, 5 and 8). Under the modulo rule all three map to scale index 2 transformer.py. So the depth losses here act on the finest features only.
• Eq. 3 can go numerically negative. The paper's form subtracts two large terms. The code's variance form cannot yield a negative square-root argument, because the variance is non-negative sigloss.py. Implement the variance form, or clamp the argument.
• ε in the log. The loss takes log(depth + 1e-3), so predictions near zero get a fixed offset in log space sigloss.py.
• Bin normalisation is a no-op in exact arithmetic. The code normalises the softmax bin lengths again so they sum to one binsformer_head.py. Harmless, but it is an extra step not in the paper.
• Scene loss with batch size 1. The cross-entropy calls input.squeeze(), which removes the batch axis when the batch size is 1 celoss.py. Keep the batch size above 1, or fix the squeeze.
• Queries are learned, not zero. Implementers following the paper's zero-initialisation note will diverge from the code's learned queries §3.2.5 binsformer_head.py.
• Scene loss weight differs by a factor of ten. The paper's μ = 1e-3 and the config's 1e-2 give different training objectives. Results in the paper are not stated to use either §3.2.5 binsformer_swinl_22k_w7_nyu.py.
• Uncertainty is only described. The paper says uncertainty is computed by maximum likelihood "following" a cited work §4.4.1. No formula or code path for it is given here.
• Costs. Nine decoder layers over three scales, a six-layer deformable encoder and a Swin-Large backbone, trained at batch 16 for 38.4k iterations on eight V100s, about five hours by the paper's account §4.2.1 binsformer_swinl_22k_w7_nyu.py binsformer_swinl_22k_w7_nyu.py. The paper does not give the cost of the ResNet or Swin-Tiny variants.
• Unstated values to take from the code. Pixel-decoder width, encoder depth and width, positional encoding, decoder width and heads, crop and colour augmentation, gradient clipping and weight-decay exclusions are not in the paper. The code is the only source for them.
• Reported numbers. All tables are author-reported. The KITTI benchmark row's configuration is not stated tab. 2, and the ablation backbone is not stated tab. 5.

Lineage

• Builds on: AdaBins, whose adaptive bins and chamfer loss the paper revisits §1 §3.2.4. The Transformer decoder and set-to-set framing follow DETR §1 §3.2.2. The light FPN and the multi-scale deformable encoder follow MaskFormer, Mask2Former and deformable DETR §4.2.3 §4.2.4. The scale-invariant loss follows Eigen et al. §3.2.3.
• Compared against: per-pixel regression and classification methods, including DORN, BTS, DAV, TransDepth and DPT on NYU and KITTI tab. 1 tab. 3; Godard et al., Johnston et al., Gan et al., Yin et al. and PGA-Net on KITTI tab. 1; BANet, PWA and ViP-DeepLab on the KITTI benchmark tab. 2; and Chen et al. on SUN RGB-D tab. 4.