NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation

In plain words

Given one ordinary photo, the job is to guess how far away each pixel is. Many recent systems simply train a big network to output a depth map directly. This paper adds a classic idea from older computer vision: a conditional random field (CRF), which refines a rough guess by making neighbouring pixels agree with each other, and in the fully connected version every pixel can talk to every other pixel. That is too expensive for a whole image, so the image is cut into small blocks of patches and the pixels only talk within a block. A second pass uses blocks shifted by half a block, so that neighbours across a block border are also connected.

The "talking" is done with attention, the same mechanism used in transformers, so the strength of each link is learned rather than hand-designed. A network built this way takes a transformer image encoder, makes a rough depth guess at the coarsest scale, and then refines it step by step at four finer scales. The authors report that this refinement helps on indoor, outdoor and 360-degree panorama datasets.

TL;DR

• Decoder for monocular depth: fully-connected CRF energy computed inside windows of 7x7 patches, with the pairwise potential learned by multi-head attention §3.2 §3.3.
• Regular and shifted windows alternate so that information crosses window borders §3.2 §3.4.
• Four decoder levels, heads 32/16/8/4 from coarse to fine, pyramid pooling for global context at the top §3.4 tab. 5.
• Author-reported results that beat the other methods in the paper's tables on KITTI, NYUv2 and Matterport3D panoramas. The panorama result uses the perspective network without panorama-specific projections §4.2 §4.3 tab. 1 tab. 3 tab. 4.
• Code is public and MIT-licensed, but it differs from the paper in training schedule and inference details (see Paper vs code).

Problem

Monocular depth is ill-posed: one image is consistent with many 3-D scenes §1. Most deep methods regress the depth map directly and compete on network design §1. Classic CRF approaches exploit spatial and colour consistency, but deep methods use CRFs only between neighbouring nodes, because fully connected CRFs are too costly §1. The paper's goal is to make fully connected CRF energies feasible inside a deep network §1 §3.2.

The computation cost is stated per iteration: with h x w patch nodes, a fully connected energy needs hw(hw-1) pairwise terms, whereas a window of N x N patches needs hw(N^2-1) eq. 3. The paper assumes that the depth of a pixel is mostly determined by nearby pixels, so long-range links are not needed §3.2.

Method

Overview §3.4 fig. 3. An encoder (Swin transformer, variant not stated §3.4) produces features at four scales: 4x4, 8x8, 16x16 and 32x32 pixel patches §3.4. A pyramid pooling head (PPM, pool scales 1, 2, 3, 6 §3.4) gives the initial coarse prediction X from the top feature. Then each of the four levels runs the neural window FC-CRF module on the features F of that level and the previous prediction X, and outputs a refined X' fig. 3. Between levels, a pixel rearrangement (PixelShuffle) doubles the spatial size and divides channels by four §3.4 fig. 3.

Fully connected CRF energy §3.1 eq. 1 eq. 2. The energy sums a unary term per node and a pairwise term per node pair. The pairwise term is zero for i = j §3.1.

Window FC-CRF §3.2 fig. 2. The image is split into N x N windows of patches, and each patch is one graph node §3.2. All nodes in a window are connected; nodes in different windows are not §3.2. For shifted windows, the grid is moved by (N/2, N/2) patches, following Swin, and the energy is computed for both the original and the shifted windows, one after the other §3.2 §3.4.

Neural pairwise potential §3.3 eq. 7 eq. 8 eq. 9. Hand-designed Gaussian colour and position weights are replaced by learned ones. The pairwise term is written as a weight w times the difference of predictions eq. 7. For each node, the sum over other nodes is written as alpha times its own prediction plus beta times the others' predictions eq. 8. Queries and keys are computed per patch and the weights come from a softmax over query-key scores plus a relative position embedding P; the previous prediction X is multiplied by these weights to send messages eq. 9. Multi-head attention gives several such energies per level §3.3 §3.4.

Unary term §3.3 eq. 4 eq. 6. The unary potential is computed by a convolutional network from the image features §3.3.

Optimisation network §3.4. The energy output is passed through an optimisation network of two fully connected layers, which gives the refined map X' §3.4.

Decoder levels §3.4 app. A. Each level has two successive CRF steps (regular and shifted windows) §3.4. Window size N = 7 at all levels §3.4. The number of heads is 32, 16, 8, 4 from the top (coarsest) to the bottom (finest) level §3.4 app. A. Pixel rearrangement follows the top three levels §3.4.

Loss §3.4 eq. 10 eq. 11. A scale-invariant logarithmic (SILog) loss over the K valid pixels of an image, with variance factor lambda = 0.85 and scale alpha = 10 §3.4.

# Decoder over four levels  [§3.4] [fig. 3]
# Input:  encoder features at 1/4, 1/8, 1/16, 1/32 scale
# Output: depth map at 1/4 scale, resized to full resolution afterwards

def decode(features):
    # Global context at the coarsest level from pyramid pooling  [§3.4]
    prediction = pyramid_pooling(features[3])    # (B, 512, H/32, W/32)
    for level in [3, 2, 1, 0]:
        # Refine the coarse prediction with windowed FC-CRFs  [§3.4] [eq. 9]
        refined = neural_window_crf_level(features[level], prediction)
        if level > 0:
            # Move pixels to a 2x grid and divide channels by 4  [§3.4]
            prediction = pixel_shuffle(refined, 2)
    # Single-channel map in (0, 1), scaled by max depth
    # [code: newcrfs/networks/NewCRFDepth.py#L146]
    depth = sigmoid(conv3x3(refined)) * max_depth
    # Bilinear upsampling by 4 to full resolution  [§4.1]
    # [code: newcrfs/networks/NewCRFDepth.py#L163]
    return bilinear_upsample(depth, 4)

# One decoder level: window FC-CRF refinement  [§3.2] [§3.3] [eq. 9]
# Input:  features F (B, C, H, W); previous prediction X (B, C, H, W)
# Output: refined map X' (B, C, H, W)

def neural_window_crf_level(features, prediction):
    # Project features to the stream and the prediction to messages
    # [code: newcrfs/networks/newcrf_layers.py#L383-L392]
    # Channel numbers are not stated in the paper
    stream = conv3x3(features)                   # (B, C, H, W)
    messages = conv3x3(prediction)               # (B, C, H, W)
    # Two CRF steps: regular windows, then windows shifted by N // 2
    # [§3.2] [§3.4] [code: newcrfs/networks/newcrf_layers.py#L296]
    for shift in [0, 3]:
        # Attention input is layer-normalised; the residual uses the raw stream
        # [code: newcrfs/networks/newcrf_layers.py#L207-L208]
        normed = layer_norm_1(stream)
        # Cyclic shift, then split into non-overlapping N x N windows, N = 7
        # [§3.2] [code: newcrfs/networks/newcrf_layers.py#L219-L233]
        windows = partition(cyclic_shift(normed, shift), 7)
        window_messages = partition(cyclic_shift(messages, shift), 7)
        # Per-head queries and keys for every patch in a window
        # [§3.3] [eq. 9]
        queries, keys = project_queries_keys(windows)
        # Score every pair of patches in a window, scaled per head, plus
        # the relative position bias P
        # [code: newcrfs/networks/newcrf_layers.py#L121-L127]
        scores = queries @ transpose(keys) + relative_bias
        if shift > 0:
            # Block pairs that fall in different shifted regions
            # [code: newcrfs/networks/newcrf_layers.py#L349-L350]
            scores = scores + shift_mask
        # Weights over all patches in the window  [eq. 9]
        weights = softmax(scores)
        # Send messages from the previous prediction along the weights
        # [eq. 9]
        update = linear_projection(weights @ window_messages)
        # Undo the partition and the cyclic shift
        # [code: newcrfs/networks/newcrf_layers.py#L238-L246]
        update = reverse_partition(update, shift)
        # Residual connection on the raw stream
        # [code: newcrfs/networks/newcrf_layers.py#L254]
        stream = stream + update
        # Two-layer MLP with residual  [§3.4]
        # [code: newcrfs/networks/newcrf_layers.py#L255]
        stream = stream + mlp(layer_norm_2(stream))
    # Final layer normalisation  [code: newcrfs/networks/newcrf_layers.py#L430]
    return layer_norm_out(stream)

Key equations, as written in the paper:

E(\mathbf{x}) = \sum_i \psi_u(x_i) + \sum_{ij} \psi_{p}(x_i, x_j) eq. 1

\Omega(\text{FC-CRFs}) = hw\,\Omega(\psi_u) + hw(hw-1)\,\Omega(\psi_p) \Omega(\text{Window FC}) = hw\,\Omega(\psi_u) + hw(N^2-1)\,\Omega(\psi_p) eq. 3

\psi_{p_i} = \alpha(\mathcal{F}_i, \mathcal{F}_j, p_i, p_j)\, x_i + \sum_{j\neq i} \beta(\mathcal{F}_i, \mathcal{F}_j, p_i, p_j)\, x_j eq. 8

\psi_{p_i} = \text{SoftMax}(q K^T + P)\, X, \qquad \sum_i \psi_{p_i} = \text{SoftMax}(Q K^T + P)\, X eq. 9

\Delta d_i = \log \hat{d}_i - \log d^*_i eq. 10

\mathcal{L} = \alpha \sqrt{\frac{1}{K}\sum_i \Delta d_i^2 - \frac{\lambda}{K^2}\Big(\sum_i \Delta d_i\Big)^2} eq. 11

# Scale-invariant log loss over valid pixels  [§3.4] [eq. 10] [eq. 11]
# Input:  predicted depth (B, H, W); ground truth depth (B, H, W)
# Output: scalar loss

def silog_loss(predicted, ground_truth):
    # Keep pixels with a valid ground truth; the threshold is not in the paper
    # [code: newcrfs/train.py#L299-L301]
    valid = ground_truth > min_valid_depth
    # Log difference per valid pixel  [eq. 10]
    diff = log(predicted[valid]) - log(ground_truth[valid])
    # Variance-minimising form with variance_focus = 0.85  [§3.4] [eq. 11]
    # [code: newcrfs/utils.py#L107-L109]
    spread = mean(diff ** 2) - variance_focus * mean(diff) ** 2
    # Scale constant alpha = 10  [§3.4]
    return 10.0 * sqrt(spread)

Hyperparameters

Values are for the paper's experiments unless marked "code" (from the official training configs). "not stated" means the paper does not say.

Results

Author-reported. Setup: each table names its dataset and split. The baseline for the relative gains in the text is not always named (see Pitfalls). The paper reports the "Ours" rows only for its own method; "*" in the source marks extra pretraining data.

KITTI, Eigen split §4.3 tab. 1. Abs Rel is the main ranking metric. Sq Rel is computed differently here, per the caption.

Text: Abs Rel, Sq Rel, RMSE and RMSE_log drop by 10.3%, 18.4%, 9.8% and 10.2% §4.3. These percentages match the AdaBins row, although the text does not name it.

KITTI, official split §4.3 tab. 2. Validation metrics for the first block; online test server metrics (four only) for the second.

Caption: the method "ranks 1st among all submissions" on the online benchmark at submission time tab. 2. Ranking changes after submission and was not re-checked.

NYUv2 §4.3 tab. 3. Abs Rel and RMSE are the main ranking metrics. No extra pretraining data is used for "Ours".

Text: Abs Rel decreases by 7.8% and RMSE by 8.2%, relative to AdaBins §1 §4.3 (the numbers match the AdaBins row).

Matterport3D, panoramas (equirectangular) §4.3 tab. 4. The same perspective-image network is applied directly to 1024x512 panoramas §4.2.

Text: Abs Rel and Abs drop by 14.8% and 20.0% relative to UniFuse §4.3, which matches the UniFuse row.

Ablation, Eigen split §4.4 tab. 5. S = window shift, R = pixel rearrangement upscale, P = PPM head. The last three rows vary the head counts, with the full model at 32, 16, 8, 4.

Text: the baseline is a UNet with the same encoder, with PPM and pixel rearrangement removed §4.4. The window shift reduces Abs Rel from 0.055 to 0.054; rearrangement gives a small gain in accuracy and fewer parameters §4.4.

Efficiency, Eigen split, 352x1216 input app. B tab. 6. Models were retrained on 352x352 random crops for this table tab. 6. Time is for the decoder only, measured on RTX 2080 Ti; "Feature extract" is the encoder cost.

The paper's reference points for other CRFs: depth CRF post-processing 59.1 s [wang2015depth], segmentation FC-CRF 0.5 s, and 29 ms for the 7x7 window (time without encoder) app. B.

Paper vs code

Official code: aliyun/NeWCRFs at commit d327bf7ca8fb43959734bb02ddc7b56cf283c8d9, MIT license (Copyright 2022 Alibaba Cloud) LICENSE. Found via the code link on the paper's project page, which the abstract footnotes; the owner aliyun is Alibaba, and the author block lists Alibaba Group.

• Epochs. Paper: 20 §4.1. Code configs: 50 arguments_train_kittieigen.txt arguments_train_nyu.txt.
• Learning rate. Paper: 1e-4 decreasing to 1e-5 §4.1. Configs: 2e-5 arguments_train_kittieigen.txt. The code decays the rate with power 0.9 down to 0.1 times the start value unless an end value is given train.py train.py.
• Encoder. Paper: "swin-transformer", no size or pretraining stated §3.4. Configs: Swin-L with ImageNet-22k weights arguments_train_kittieigen.txt. The code also has tiny and base variants NewCRFDepth.py.
• Optimiser settings. Paper: Adam, no epsilon or weight decay given §4.1. Configs: weight decay 1e-2 arguments_train_kittieigen.txt and epsilon 1e-3 arguments_train_kittieigen.txt. The code default epsilon is 1e-6 train.py.
• Training input and crops. Not stated in paper §4. Configs: KITTI 352x1120 with bottom crop arguments_train_kittieigen.txt, NYUv2 480x640 arguments_train_nyu.txt, plus random rotation arguments_train_kittieigen.txt.
• Maximum depth. Not stated in paper. The code multiplies a sigmoid output by the maximum depth, 80 m for KITTI and 10 m for NYUv2 NewCRFDepth.py NewCRFDepth.py arguments_train_kittieigen.txt arguments_train_nyu.txt.
• Test-time flip. Not in paper. The test script predicts the flipped image too and averages the two predictions test.py. Online evaluation during training does the same train.py. Whether the paper's tables use it is not stated.
• Valid pixels. Paper: "pixels with valid depth" §3.4. Code: depth above 0.1 for NYUv2 and 1.0 otherwise train.py.
• Attention self-pairs. Paper's pairwise sum excludes j = i eq. 8 §3.1. The code softmax includes each patch's own score newcrf_layers.py.
• Attention scale. Eq. 9 has no scale factor. The code scales queries by head dimension to the power -1/2 newcrf_layers.py newcrf_layers.py.
• Window shift. Paper: N/2 patches §3.2. Code: integer 3 for N = 7 (window size // 2) newcrf_layers.py.
• Optimisation network. Paper: "two fully-connected layers" §3.4. Code: a feed-forward block with hidden width 4x the channels, GELU, residual and layer normalisation newcrf_layers.py newcrf_layers.py.
• Unary term. Paper: a convolutional network computes the unary potential §3.3. Code: there is no separate unary term; a 3x3 convolution projects the features into the stream that feeds queries, keys and the residual newcrf_layers.py. This reading is mine, from the code.
• Intermediate outputs. Paper: each level outputs an optimised depth map X' §3.4. Code: intermediate levels pass multi-channel feature maps (1024, 512, 256, then 128 after the last level); only the final head predicts depth NewCRFDepth.py.
• Padding. Not in paper. Feature maps are zero-padded to a multiple of 7 before windowing newcrf_layers.py. The unshifted blocks apply no mask, so padded positions take part in attention newcrf_layers.py.
• Head counts. Paper 32, 16, 8, 4 §3.4. Code matches NewCRFDepth.py. No discrepancy.
• Loss constants. Paper lambda = 0.85, alpha = 10 §3.4. Code matches train.py utils.py. No discrepancy.
• Pixel rearrangement. Paper: after top three levels §3.4. Code matches NewCRFDepth.py. No discrepancy.

Pitfalls

• Learning schedule. The paper's 20 epochs and 1e-4 to 1e-5 do not match the released configs (50 epochs, 2e-5). An implementer copying the paper gets a different model than the one the authors released. Pick one and report which.
• Unstated encoder. The paper never states which Swin variant or which pretraining weights it uses. The code uses Swin-L pretrained on ImageNet-22k arguments_train_kittieigen.txt. Results are not comparable across encoder sizes.
• Latent intermediate maps. The decoder's internal state is a multi-channel feature stream, not a depth map. Porting only the final head, or inserting a supervision loss on intermediate levels, changes the method.
• Test-time flip. Flip averaging is on at evaluation test.py. A reimplementation that evaluates single-pass will not match the reported numbers.
• Self-pairs and scale. The code's attention includes self-pairs and scales queries by head dimension. Both differ from eq. 8-9 as written.
• Padding. Zero padding to a multiple of 7 is not masked in unshifted blocks newcrf_layers.py. This matters for inputs where the feature grid is not a multiple of 7, such as 352x1120 at 1/4 scale (88x280).
• Output bound. Sigmoid times max depth bounds predictions to (0, max depth) NewCRFDepth.py NewCRFDepth.py. The paper does not state this; a different maximum changes the result.
• Log argument. The SILog argument stays non-negative because lambda = 0.85 is below 1 eq. 11 §3.4. Keep lambda below 1 or guard the square root.
• Efficiency text vs table. The text says the N = 48 window costs 250 ms and 5.3 G app. B, but the table lists 345 ms and 7.3 G, and OOM in training tab. 6. Use the table.
• Relative gains. The percentages in the text (KITTI, NYUv2, Matterport) match the AdaBins and UniFuse rows, but the text does not name the baseline §1 §4.3.
• Ranking claim. "Ranks 1st" on the KITTI online benchmark is at submission time tab. 2 §4.3. This card did not re-check the current leaderboard.
• Pretraining data. "Ours*" uses 50K extra images for Matterport3D pretraining §4.3; their source is not stated, so the number cannot be reproduced from this paper alone.
• Typos in §4.1. The paper prints beta_1 twice; the second is beta_2 = 0.999 by context.
• Figure and equation numbers. Equation and figure numbers here are counted from the LaTeX source order (main body, then appendix). The compiled PDF was not checked.

Lineage

• Builds on: fully connected CRFs [krahenbuhl2011efficient, §3.1]; Swin transformer for the shifted-window scheme and encoder [liu2021swin, §3.2] §3.4; pyramid scene parsing for global context [zhao2017pyramid, §3.4]; the SILog loss [eigen2014depth, §3.4]. None has a card on this site.
• Compared against: DORN, BTS, AdaBins, DPT, PWA, PackNet-SAN, BA-Full, Yin et al., Xu et al., Liu et al., DAV, OmniDepth, BiFuse, SliceNet, HoHoNet, UniFuse tab. 1 tab. 2 tab. 3 tab. 4.