Stereo matching

Overview

Stereo matching takes a left and a right image and returns a disparity map: for each left pixel, the horizontal shift to the matching pixel in the right image 2501.09898 §1. Disparity is the quantity every method predicts. The shared step is a matching score for each pixel at each candidate shift, stored in a cost volume, which the network then filters and reads out [arxiv:2501.09898 §3.2]. The methods differ in how that volume is filtered and refined, and in how much target-domain data they need.

The only card that lists this concept is FoundationStereo, a zero-shot model Card: FoundationStereo. The variants below are therefore the design options that paper tests, not a survey of the field.

Variants

Zero-shot model with a frozen monocular prior

Trained on synthetic pairs and then used on new data without fine-tuning 2501.09898 §3.5 tab. 2. Its features fuse a small CNN with a frozen monocular depth model at 1/4 scale 2501.09898 §3.1. Two volumes are built, a group-wise correlation with 8 groups and a concatenation volume 2501.09898 §3.2. Filtering is done by an axial-planar 3D hourglass and a disparity transformer, and the result is refined by three levels of ConvGRU updates 2501.09898 §3.2 2501.09898 §3.3. Card: FoundationStereo.

Design choices the paper ablates

These are the settings the paper compares, each on the Middlebury BP-2 metric (lower is better). The ablation models are trained on a random 100K-pair subset of the synthetic set and evaluated on the Middlebury training set 2501.09898 §4.5.

The paper says gains saturate around a disparity kernel of 17 [arxiv:2501.09898 §4.5].

When to use which

When the paper's code is used, the inference settings match the paper: 32 refinement steps and maximum disparity 416 2501.09898 §4.1. The released code also has no training code, so training values are only what the paper states Card: FoundationStereo.

Implementation sketch

Zero-shot inference in condensed form. It follows the paper's order of stages, not any released file.

# Zero-shot stereo inference, condensed  [arxiv:2501.09898 §3, §4.1]
# Input:  rectified left, right images (B, 3, H, W); iters = 32
# Output: disparity at full resolution (B, 1, H, W)

def stereo_zero_shot(left, right, iters):
    # Fused CNN and frozen monocular features at 1/4 scale  [§3.1]
    feat_left, feat_right = backbone(left, right)
    # Context features seed the recurrent state  [§3.1]
    hidden, context = context_branch(left)
    # Group-wise volume (8 groups) plus concatenation volume  [§3.2]
    volume = concat_channels(
        groupwise_volume(feat_left, feat_right, groups=8),
        shifted_concat_volume(feat_left, feat_right))
    # Hourglass filtering and disparity transformer, summed  [§3.2] [tab. 6]
    filtered = hourglass(volume) + disparity_transformer(volume)
    # Soft-argmin over disparity gives the initial estimate  [§3.2] [eq. 2]
    disp = soft_argmin(filtered)
    for _ in range(iters):
        # Look up the volume at the current disparity, then update  [§3.3]
        hidden, delta = gru_update(hidden, context, filtered, disp)
        # Residual disparity update  [eq. 10]
        disp = disp + delta
    # Convex up-sampling to full resolution  [§3.3]
    return convex_upsample(disp)

Contradictions

• In-domain Scene Flow EPE. The text gives 0.33 for the model and the best previous method at 0.41 2501.09898 §4.4. The table gives 0.34 for the same model 2501.09898 tab. 3.
• ETH3D BP-1 of the zero-shot model. The "Any (no targets)" row gives 0.5 2501.09898 tab. 2. The zero-shot row of the ETH3D table gives 1.52 for BP-1.0 2501.09898 tab. 4. The Scene Flow-only row gives 1.8 2501.09898 tab. 2. The paper does not say whether these rows describe the same checkpoint and training mix, so the numbers should not be compared as one result.
• Full-resolution Middlebury. At full resolution, BP-2 is 4.8, against 1.1 at half resolution 2501.09898 tab. 10. The paper does not explain why the larger input scores worse.

Papers

• FoundationStereo: Zero-Shot Stereo Matching: a zero-shot stereo network trained on a 1M-pair synthetic set with self-curation, using a frozen monocular prior, an axial-planar hourglass, a disparity transformer and ConvGRU refinement.