Unifying Flow, Stereo and Depth Estimation

In plain words

Three common vision tasks ask the same question. Given two pictures, which pixel in one matches which pixel in the other? For video, the answer is motion (optical flow). For two side-by-side cameras, it is distance (stereo). For camera views taken from known positions, it is depth. Until now, each task has had its own network design, so a model trained for one task cannot simply be reused for another.

This paper treats all three as one matching problem. A shared network first builds a description of every pixel. Each picture can look at the other while doing this, which helps the description tell similar-looking spots apart. The final answer then comes from a plain comparison step that scores every pair of pixels and takes a weighted average. That step has no learned weights of its own, so the same learned network serves all three tasks, and a model trained on one task can start the next one. A last step fills in pixels that have no clear match, such as ones hidden behind an object.

TL;DR

• One model, with the same learned weights, predicts optical flow, rectified stereo disparity and depth from posed image pairs §1 §3.1.
• Features come from a CNN followed by six Transformer blocks with self-attention, cross-attention and a feed-forward layer §3.2.
• Matching is a softmax over correlations, with no task-specific parameters §3.1.
• Author-reported: a model with one extra 1/4-resolution refinement beats RAFT with 31 refinements on Sintel §4.1.3 tab. 3. The full system uses additional task-specific refinements and reports competitive results on 10 benchmarks §4.5.

Problem

Dense flow, disparity and depth were usually computed by task-specific architectures built around a cost volume and a convolutional regressor §1. Iterative methods such as RAFT need many sequential refinements, and cost-volume search ranges limit stereo models §2. The authors want one model with no task-specific learned parameters that can transfer between tasks §1.

The matching formulation assumes corresponding pixels are visible in both images. Occluded and out-of-boundary pixels violate this and need separate handling §3.3.

Method

Overview. Two images go in: video frames for flow, a rectified pair for stereo, or posed images for depth. A weight-shared CNN produces features at 1/8 of the input resolution. A Transformer enhances both feature maps. A parameter-free, task-specific matching layer produces the output, and a self-attention propagation layer fills unmatched regions fig. 2 §3.

Feature extraction §3.2

1. A weight-shared ResNet-like CNN gives 1/8-resolution features for each image.
2. Fixed 2D sine and cosine positional encodings, as in DETR, are added to both feature maps §3.2 eq. 14.
3. Six Transformer blocks run on both features. Each block has self-attention, cross-attention and a two-layer feed-forward network §3.2.
4. Attention runs inside fixed local windows, not over all pixels: each image is split into K×K windows, with K = 2 at 1/8 resolution. The window partition is shifted by half a window, (H/2K, W/2K), to link neighbouring windows §3.2 tab. 2.
5. Stereo uses 1D horizontal cross-attention, which is faster and more accurate than 2D §3.2 tab. 6.

Matching layers (all parameter-free) §3.1

• Flow: correlate every pixel of image 1 with every pixel of image 2 eq. 1, softmax over all H·W targets eq. 2, expected target coordinate eq. 3, flow = expected coordinate minus own coordinate eq. 4.
• Stereo: same idea along one scanline. Correlations are masked so that matches lie on one side only §3.1, then disparity = own x minus expected x eq. 5–8.
• Depth: a plane sweep over inverse-depth candidates. Image 2 features are warped to each candidate using the known cameras eq. 9, correlated with image 1 eq. 10–11, softmaxed over candidates eq. 12, and depth is the expected candidate depth eq. 13.

Propagation §3.3 fig. 3

Matches are unreliable for occluded and out-of-boundary pixels. A single self-attention layer over the features spreads the flow, disparity or depth from matched to unmatched pixels, weighted by feature similarity eq. 15. The paper does not separate matched from unmatched pixels; the layer is trained with ground-truth supervision alone §3.3.

Upsampling §3.3

The 1/8 output is upsampled to full resolution with RAFT's convex upsampling. A small two-layer convolution predicts 8×8×3×3 weights from the features and the low-resolution flow §3.3.

Refinement (optional, improves accuracy) §3.4

• Hierarchical matching refinement at 1/4 resolution §3.4.1: upsample the 1/8 flow, warp the second feature map, then match in a local range (9×9 window for flow, length-9 horizontal window for stereo). The 1/4 stage shares Transformer and propagation weights with the 1/8 stage. Propagation uses a 3×3 local window, and attention uses 8×8 windows. Depth does not use this step §3.4.1.
• Local regression refinement §3.4.2: RAFT-style iterations that predict residual updates from local correlation. Correlation is 2D for flow and stereo, and built from the current depth and relative pose for depth. Final models use 6 steps for flow, 3 for stereo and 1 at 1/8 for depth §3.4.2 §4.5.

Training loss §3.5 eq. 16–17

Every intermediate prediction is supervised. Earlier predictions get smaller weights, decaying with a factor γ = 0.9 as in RAFT §3.5. Flow uses L1. Stereo uses smooth L1 §3.5. Depth uses L1 on inverse depth plus a gradient term, each weighted 20 §3.5 §4.3. The paper's gradient equation is written on depth, while the released code uses inverse depth (see Paper vs code).

# Matching step for flow, stereo or depth  [§3.1] [eq. 1-13]
# Input:  features of both views (B, C, H, W); target coordinates
# Output: flow, disparity or depth (B, H, W) at 1/8 resolution

def match(features0, features1, task):
    # Scale the dot products by sqrt(C) before the softmax  [eq. 1]
    correlation = dot_products(features0, features1) / sqrt(C)
    # Flow: all pairs of pixels, shape (B, HW, HW)  [eq. 1]
    # Stereo: pairs on one scanline, shape (B, H, W, W), upper triangle masked
    # Depth: one score per depth candidate, shape (B, N, H, W)  [eq. 10-11]
    match_prob = softmax(correlation, over the last target axis)  # [eq. 2]
    # Expected target value per pixel, a weighted average  [eq. 3] [eq. 7]
    expected = match_prob @ target_values
    # Convert the expected target into the output quantity  [eq. 4] [eq. 8]
    return difference(expected, own_coordinate)  # flow, disparity or depth

# Propagation of matched values to unmatched pixels  [§3.3] [eq. 15]
# Input:  features (B, C, H, W); matched prediction (B, 2, H, W)
# Output: propagated prediction (B, 2, H, W)

def propagate(features, prediction):
    # Similarity between every pair of positions in one feature map  [eq. 15]
    # Note: the released code adds learned projections; see Paper vs code
    scores = features_flat @ features_flat^T / sqrt(C)  # (B, HW, HW)
    weights = softmax(scores, over the last axis)
    # Weighted average of the matched values over similar positions  [eq. 15]
    return weights @ prediction_flat  # (B, HW, 2) -> (B, 2, H, W)

# Depth candidates and warping  [§3.1.3] [§4.3]
# The depth range is discretised uniformly in inverse depth.
# Paper range: 0.5 to 10 m, 64 candidates [§4.3].

def depth_candidates(d_min, d_max, count):
    # Uniform in inverse depth, so near depths get finer steps  [§3.1.3]
    inverse_min = 1.0 / d_max
    inverse_max = 1.0 / d_min
    inverse_depths = linspace(inverse_min, inverse_max, count)
    return 1.0 / inverse_depths  # back to depth values

Hyperparameters

Values in the paper are cited to the paper. "Not stated" means the paper does not give the value; released code, where it defines one, is marked as code only.

Results

Author-reported. Setup: models are trained on the named datasets. Stereo ablations use Scene Flow, depth ablations use ScanNet, and benchmarks follow the fine-tuning in Hyperparameters. Table numbers count captioned tables in source order (the PDF prints them in Roman numerals). Only headline rows are shown. Full tables are in the cited tables.

Optical flow, RAFT versus GMFlow, trained on Chairs and Things tab. 3:

Time is measured at Sintel resolution 436×1024 tab. 3.

Sintel test, final model tab. 14: GMFlow+ scores 1.03 EPE on clean and 2.37 on final. On final, FlowFormer is better at 2.12. The paper calls GMFlow+ state of the art on clean.

KITTI 2015 optical flow test tab. 15: GMFlow+ scores 2.40 on non-occluded and 4.49 on all pixels, the best of the methods listed. The metric is not labelled in the table.

Stereo, Scene Flow versus RAFT-Stereo tab. 7:

Stereo benchmarks (selected rows):

• KITTI 2015 test, D1-all tab. 17: GMStereo 1.77 (all), 1.61 (non-occluded), 0.17 s. LEAStereo is best at 1.65 and 1.51.
• Middlebury test tab. 18: GMStereo has the best RMS of 6.45 and bad-2.0 of 7.14. The paper also reports a 0.73 s run at full resolution.
• ETH3D test tab. 19: GMStereo gets bad-1.0 1.83, bad-2.0 0.25 and bad-4.0 0.08. The paper reports 2nd, 2nd and 1st place respectively.
• Argoverse tab. 20: GMStereo leads with all:10 1.61, all:5 3.19 and all:3 6.86 at 190 ms.

Depth, ScanNet test tab. 21 (selected):

Depth, RGBD-SLAM, SUN3D and Scenes11 tab. 22 (Abs Rel): GMDepth 0.101 on RGBD-SLAM, 0.112 on SUN3D, 0.050 on Scenes11. IIB is best on RGBD-SLAM (0.095) and SUN3D (0.099). Scenes11 is the best of the methods listed.

Ablations (author-reported):

• Transformer components, flow, Things EPE tab. 2: full 6.67; without cross-attention 10.84; without position 8.38; without FFN 8.71; without self-attention 7.04.
• Propagation, flow tab. 2: Sintel clean unmatched EPE falls from 15.54 to 10.39.
• Stereo and depth ablations tab. 4: depth Abs Rel is 0.074 full and 0.095 without cross-attention. Stereo Things EPE is 1.22 full and 1.96 without cross-attention.
• Stereo 1D versus 2D cross-attention tab. 6: 1D gives Things EPE 1.22 at 50 ms, against 1.25 at 61 ms for 2D.
• Local window splits tab. 2: 2×2 gives Things EPE 6.67 at 53 ms; 1×1 gives 6.34 at 105 ms.
• Global versus local matching, flow tab. 2: global 6.67 EPE; 9×9 local 19.88.

Cross-task transfer, using a flow model as the starting point tab. 10:

• Stereo, Things EPE, no fine-tuning: 2.58 from flow initialisation, against 75.24 from random.
• Stereo, Things EPE, 100K steps: 1.00 from flow, against 1.11 from random.
• Depth, ScanNet Abs Rel, no fine-tuning: 0.198 from flow, against 0.536 from random.
• Depth, ScanNet Abs Rel, 100K steps: 0.063 from flow, against 0.069 from random.

The paper reports no gain from transferring stereo or depth models to flow §4.4.

Paper vs code

Code commit 943a99006950ab6c84ea72a72beb147fdd03c19a, repository autonomousvision/unimatch, MIT.

• Propagation has learned projections. The paper writes the propagation layer as softmax(F₁F₁ᵀ/√D) with no learned projections eq. 15 §3.3. The code's SelfAttnPropagation applies learned q_proj and k_proj before the dot product, and divides by √C attention.py attention.py. A code comment acknowledges the difference and says it was left in place so the released models need not be retrained.
• Stereo correlation scaling. The paper's stereo correlation has no normalisation term eq. 5. The code divides by √C matching.py.
• Stereo cross-attention variant. The paper uses 1D horizontal cross-attention §3.2. The released stereo scripts use self_swin2d_cross_swin1d, meaning 2D self-attention and windowed, shifted 1D cross-attention gmstereo_scale2_regrefine3_train.sh transformer.py. The full-width 1D option self_swin2d_cross_1d also exists transformer.py.
• Stereo ablation schedule. The paper uses 50K iterations, batch 64 and lr 1e-3 for the Scene Flow ablations §4.2. The released stereo script uses 100K steps, batch 16 and lr 4e-4 for Scene Flow gmstereo_scale2_regrefine3_train.sh.
• Flow loss exclusion. The code drops pixels with ground-truth flow of 400 px or more from the loss flow_loss.py. The paper does not say this.
• Learning rate schedule. The paper says cosine with warmup and gives no warmup length §4. The code uses a one-cycle cosine schedule with 5% warmup and no momentum cycling main_flow.py.
• Gradient clipping and weight decay. The code clips the gradient norm to 1.0 and uses weight decay 1e-4 main_flow.py main_flow.py. The paper says nothing about either.
• Depth candidates. The paper discretises inverse depth §3.1.3. The code confirms this by passing inverse depth limits to the model main_depth.py. No discrepancy found.
• Depth gradient loss space. The paper's gradient term eq. 17 applies the L1 loss to the gradients of the depth prediction §3.5. The training code calls the gradient loss without changing its default, so it takes gradients of inverse depth main_depth.py depth_loss.py.
• Depth loss weights and refinement counts. Weights of 20 and 20 and the per-task refinement counts match §3.5 §4.5 main_depth.py.

Pitfalls

• Quadratic global matching. Flow matching builds an (H·W)×(H·W) score matrix eq. 1. With the released 384×512 training crop, the 1/8 grid is 48×64 = 3,072 positions, so there are 3,072² ≈ 9.4 M scores per image pair gmflow_scale2_train.sh matching.py. The windowed Transformer does not reduce this matrix, and the paper does not quantify its memory.
• Learned propagation. Implementing the paper's propagation literally (no projections) will not reproduce the released weights, which use learned projections attention.py (see Paper vs code).
• Stereo variant choice. The paper's 1D cross-attention and the released windowed 1D variant are different models. Implement the one the released checkpoints use.
• Depth model size is inconsistent. GMDepth is listed as 7.3 M parameters and 40 ms in one comparison tab. 8, and 4.7 M parameters and 17 ms in another tab. 9. The paper does not say which configuration each uses.
• Refinement counts are task-specific. The final flow model adds 6 local-regression steps, stereo adds 3 and depth adds 1 §3.4.2 §4.5. The speed comparison with RAFT tab. 3 uses the base model with no local-regression steps, so it does not show the final system's timing.
• Generalisation to KITTI. On KITTI 2015 training data, GMFlow trails RAFT when trained only on Chairs and Things (EPE 7.77 vs 5.32 tab. 5). With Virtual KITTI 2 added, the gap shrinks (2.85 vs 2.45). The paper attributes this to RAFT's convolutional inductive bias needing less data §4.1.
• Sintel test protocol. Inputs are resized to 416×1024 before inference and the output is resized back §4.5.1. Scores are not comparable to runs at native resolution.
• Propagation helps flow and depth more than stereo's KITTI D1. In the stereo ablation, removing propagation gives the best KITTI D1 (10.09 against 10.53 for the full model), while its Things and KITTI EPE are worse tab. 4. The paper does not discuss this row.
• Not stated in the paper: AdamW betas and eps, warmup length, the crop sizes for the Argoverse stages, and step counts for several fine-tuning stages (see Hyperparameters).
• Unmatched regions are not labelled. The model is not told which pixels are unmatched. Propagation is learned from ground truth only §3.3.
• Real-time is out of reach. The authors state the full model is not real-time (20 FPS or more) §5.

Lineage

• Builds on GMFlow §1, the authors' earlier CVPR 2022 optical-flow model. This paper extends it to stereo and depth and studies cross-task transfer.
• Uses RAFT's convex upsampling and local-regression refinement §3.3 §3.4.2 and its loss decay γ = 0.9 §3.5.
• Uses Transformer self- and cross-attention §3.2, Swin's shifted windows §3.2, and DETR's sine positional encodings §3.2.
• Compared against: RAFT and RAFT-Stereo tab. 3 tab. 7, CREStereo, LEAStereo, FlowFormer tab. 14 tab. 17, DeFiNe tab. 8, DepthFormer tab. 9, DeepV2D and BA-Net tab. 21, and IIB tab. 22.
• Superseded by: not stated in this wiki.