---
kind: concept
slug: stereo-matching
title: Stereo matching
summary: >-
  Estimating a disparity map from a rectified stereo pair by matching pixels along horizontal lines. Covers how a cost volume is built, filtered and refined, and when a zero-shot model that needs no target-domain training is worth its speed and licence limits.
tags: [depth-estimation, stereo, cost-volume, zero-shot]
papers: ["2501.09898"]
related: [cost-volume, cost-aggregation, domain-randomization, foundation-model, monocular-depth-estimation, multi-view-stereo, optical-flow]
status: draft
updated_at: 2026-11-03
---

## Overview

Stereo matching takes a left and a right image and returns a disparity map: for
each left pixel, the horizontal shift to the matching pixel in the right image
[arxiv:2501.09898 §1]. Disparity is the quantity every method predicts. The
shared step is a matching score for each pixel at each candidate shift, stored
in a cost volume, which the network then filters and reads out [arxiv:2501.09898
§3.2]. The methods differ in how that volume is filtered and refined, and in how
much target-domain data they need.

The only card that lists this concept is FoundationStereo, a zero-shot model
[Card: FoundationStereo](/papers/2501.09898). The variants below are therefore
the design options that paper tests, not a survey of the field.

## Variants

### Zero-shot model with a frozen monocular prior

Trained on synthetic pairs and then used on new data without fine-tuning
[arxiv:2501.09898 §3.5] [tab. 2]. Its features fuse a small CNN with a frozen
monocular depth model at 1/4 scale [arxiv:2501.09898 §3.1]. Two volumes are
built, a group-wise correlation with 8 groups and a concatenation volume
[arxiv:2501.09898 §3.2]. Filtering is done by an axial-planar 3D hourglass and a
disparity transformer, and the result is refined by three levels of ConvGRU
updates [arxiv:2501.09898 §3.2] [arxiv:2501.09898 §3.3]. Card:
[FoundationStereo](/papers/2501.09898).

### Design choices the paper ablates

These are the settings the paper compares, each on the Middlebury BP-2 metric
(lower is better). The ablation models are trained on a random 100K-pair subset
of the synthetic set and evaluated on the Middlebury training set
[arxiv:2501.09898 §4.5].

| Choice | Options (BP-2) | Chosen | Source |
|---|---|---|---|
| Monocular backbone | DINOv2-L 2.46; DepthAnythingV2-S 2.22, -B 2.11, -L 1.97 | DepthAnythingV2-L | [arxiv:2501.09898 tab. 5] |
| Feature fusion | variant (a) 6.48; (b) 2.22; (c) 1.97 | variant (c) | [arxiv:2501.09898 tab. 5] |
| Monocular model frozen | unfrozen 3.94; frozen 1.97 | frozen | [arxiv:2501.09898 tab. 5] |
| Transformer position encoding | RoPE 2.19; cosine 1.97 | cosine | [arxiv:2501.09898 tab. 6] |
| Transformer attention domain | full volume 2.25; disparity only 1.97 | disparity only | [arxiv:2501.09898 tab. 6] |
| Transformer placement | pre-hourglass 2.06; post-hourglass 2.20; parallel 1.97 | parallel | [arxiv:2501.09898 tab. 6] |
| Disparity kernel of the hourglass | 5: 2.10; 9: 2.06; 13: 2.01; 17: 1.97; 21: 1.98 | 17 | [arxiv:2501.09898 tab. 6] |
| Filtering modules | none 2.48; STA 2.21; STA+APC 2.16; STA+DT 2.05; all 1.97 | all three | [arxiv:2501.09898 tab. 7] |

The paper says gains saturate around a disparity kernel of 17 [arxiv:2501.09898
§4.5].

## When to use which

| Situation | Use | Why |
|---|---|---|
| No labelled data for the target scenes, and the scenes differ from common benchmarks | Zero-shot model trained on synthetic data, used without fine-tuning | The Scene Flow-only model beats the listed Scene Flow-only baselines on four zero-shot benchmarks [arxiv:2501.09898 tab. 2] |
| Labelled data for one target domain, and accuracy on that domain is the goal | Fine-tune on that domain | On ETH3D, BP-1 is 0.26 after fine-tuning and 1.52 zero-shot [arxiv:2501.09898 tab. 4]. Fine-tuning adds 50K steps [arxiv:2501.09898 §4.4] |
| Real-time or high frame-rate requirement | Not this model | 0.7 s per 375×1242 image on an A100 [arxiv:2501.09898 §5]. The paper says the full model is not real-time, and it is the slowest method in the Middlebury timing table at every resolution [arxiv:2501.09898 tab. 10] |
| Limited GPU memory | Lower the input resolution | Middlebury peak memory on a 3090 is 18.5 GB at full, 10.5 GB at half and 2.3 GB at quarter resolution [arxiv:2501.09898 tab. 10] |
| Commercial product | Check the licence before using the research weights or data | The paper's code and dataset are restricted to non-commercial research use. The card records a separate checkpoint licensed for commercial use, which should be checked on its own terms [Card: FoundationStereo](/papers/2501.09898) |
| Building a similar model from scratch | Budget for a large synthetic set and curation | The set has 1M pairs at 1280×720 and was rendered on 48 A40 GPUs over 10 days [arxiv:2501.09898 tab. 1] [arxiv:2501.09898 app. Tooling and Assets]. Training on the set lowers BP-2 from 2.34 to 1.15 [arxiv:2501.09898 tab. 7] |
| Synthetic data already available, and curation wanted | Keep the curation step | Removing ambiguous pairs (BP-2 above 60%) lowers BP-2 from 1.27 to 1.15 [arxiv:2501.09898 §3.5] [arxiv:2501.09898 tab. 8] |

When the paper's code is used, the inference settings match the paper: 32
refinement steps and maximum disparity 416 [arxiv:2501.09898 §4.1]. The released
code also has no training code, so training values are only what the paper
states [Card: FoundationStereo](/papers/2501.09898).

## Implementation sketch

Zero-shot inference in condensed form. It follows the paper's order of stages,
not any released file.

```pseudocode
# Zero-shot stereo inference, condensed  [arxiv:2501.09898 §3, §4.1]
# Input:  rectified left, right images (B, 3, H, W); iters = 32
# Output: disparity at full resolution (B, 1, H, W)

def stereo_zero_shot(left, right, iters):
    # Fused CNN and frozen monocular features at 1/4 scale  [§3.1]
    feat_left, feat_right = backbone(left, right)
    # Context features seed the recurrent state  [§3.1]
    hidden, context = context_branch(left)
    # Group-wise volume (8 groups) plus concatenation volume  [§3.2]
    volume = concat_channels(
        groupwise_volume(feat_left, feat_right, groups=8),
        shifted_concat_volume(feat_left, feat_right))
    # Hourglass filtering and disparity transformer, summed  [§3.2] [tab. 6]
    filtered = hourglass(volume) + disparity_transformer(volume)
    # Soft-argmin over disparity gives the initial estimate  [§3.2] [eq. 2]
    disp = soft_argmin(filtered)
    for _ in range(iters):
        # Look up the volume at the current disparity, then update  [§3.3]
        hidden, delta = gru_update(hidden, context, filtered, disp)
        # Residual disparity update  [eq. 10]
        disp = disp + delta
    # Convex up-sampling to full resolution  [§3.3]
    return convex_upsample(disp)
```

## Contradictions

- **In-domain Scene Flow EPE.** The text gives 0.33 for the model and the best
  previous method at 0.41 [arxiv:2501.09898 §4.4]. The table gives 0.34 for the
  same model [arxiv:2501.09898 tab. 3].
- **ETH3D BP-1 of the zero-shot model.** The "Any (no targets)" row gives 0.5
  [arxiv:2501.09898 tab. 2]. The zero-shot row of the ETH3D table gives 1.52 for
  BP-1.0 [arxiv:2501.09898 tab. 4]. The Scene Flow-only row gives 1.8
  [arxiv:2501.09898 tab. 2]. The paper does not say whether these rows describe
  the same checkpoint and training mix, so the numbers should not be compared as
  one result.
- **Full-resolution Middlebury.** At full resolution, BP-2 is 4.8, against 1.1
  at half resolution [arxiv:2501.09898 tab. 10]. The paper does not explain why
  the larger input scores worse.

## Papers

- [FoundationStereo: Zero-Shot Stereo Matching](/papers/2501.09898): a
  zero-shot stereo network trained on a 1M-pair synthetic set with self-curation,
  using a frozen monocular prior, an axial-planar hourglass, a disparity
  transformer and ConvGRU refinement.