Cost volume

Overview

A cost volume stores a matching score for each pixel and each candidate depth or disparity. The score says how well a pixel's appearance in one view agrees with the appearance of the same point in another view, if the point lies at that candidate. High scores at one candidate suggest a surface there 1903.04025 §1 2403.14627 §3.1.

Four of the five papers below read one value per pixel from the volume as the expectation over the candidates, a softmax-weighted average. Stereo matching uses the candidates as disparities, a horizontal pixel shift in a rectified pair 1903.04025 §1. A depth-indexed variant re-indexes the same kind of volume onto metric depth bins 1906.06310 §3. The plane-sweep case uses the candidates as depth planes, and the features are warped between posed views 2403.14627 §3.1. The construction methods differ enough to be separate variants below.

Optical flow has no small set of candidates to test. Its volume compares every source pixel with every target pixel, and the decoder reads it through learned tokens instead of an expectation 2203.16194 §3.1 §3.3. It is the fourth variant. The matching layer of the unifying paper also compares every pixel pair for flow, but reads it by expectation and never aggregates it in 3D. It is the fifth variant 2211.05783 §3.1.

Variants

Group-wise correlation volume

Stereo disparity from a rectified pair. The feature channels are split into groups, and each group is correlated with the other view at every disparity. Each group gives one channel of the volume, so the volume has N_g channels per disparity instead of one 1903.04025 §3.2, eq. 3. The correlation is divided by the number of channels in a group, so the scale depends on group size rather than on group count 1903.04025 §3.2, eq. 3.

The paper's setting uses 40 groups over 320 unary feature channels, which is 8 channels per group 1903.04025 §4.3 1903.04025 tab. 1. A second, concatenation-based volume is added to it. It uses features compressed to 12 channels per view, giving 24 more channels, so 64 in total [arxiv:1903.04025 §3.1] 1903.04025 tab. 1. The full model is called GwcNet-gc [arxiv:1903.04025 tab. 2].

The Scene Flow ablation in the paper shows the effect of each choice, with each volume fed to the same base network, which has no stacked hourglasses 1903.04025 §4.3 1903.04025 tab. 2:

The authors report timings at 480 × 640 on one Nvidia TITAN Xp [arxiv:1903.04025 tab. 2]. Going from 40 to 80 groups gives no gain in that sweep, and 160 groups gives a small one at 157.3 ms, against 122.2 ms for 40 1903.04025 §4.3 1903.04025 tab. 2.

Depth-indexed stereo volume

Stereo depth from a rectified pair, used when the output should be metric depth rather than disparity. The paper builds a disparity volume with a PSMNet-style network, then re-indexes it onto a depth grid with 1 m steps from 1 m to 80 m. Each depth bin is filled by bilinear interpolation from the disparity volume, at the disparity that the bin corresponds to, which is the focal length times the baseline divided by depth 1906.06310 §3. Three- dimensional convolutions then run on the depth grid. The output is a softmax- weighted sum over the bins, and the training loss is a smooth L1 on depth 1906.06310 §3.

The motivation is that a fixed disparity error costs more depth at range. In the paper's KITTI setting, a one-pixel error is about 0.1 m of depth at 5 m and 5.8 m at 50 m 1906.06310 §3 1906.06310 eq. 5. Training on disparity therefore spends its effort on near objects and gives too little weight to distant ones 1906.06310 §3.

The effect is reported on KITTI validation, car category, PointRCNN, moderate difficulty, IoU 0.7. With the depth-indexed network alone, AP_BEV rises from 56.0 for pseudo-LiDAR with PSMNet to 64.0 1906.06310 tab. 1. A depth loss on PSMNet alone gives 61.9 1906.06310 tab. 3. For PointRCNN the paper finds the depth loss helps more than the depth cost volume, and for vPIXOR and AVOD the cost volume helps as much or more 1906.06310 §5.2 1906.06310 app. D.1. The volume needs 64-beam LiDAR for its training depth, even when only the stereo network is used 1906.06310 §5.1.

Plane-sweep volume with scaled dot product

Feed-forward novel view synthesis from sparse posed views. Features of each other view are warped onto D depth planes sampled uniformly in inverse depth. Each warped feature is compared with the target view by a dot product, scaled by the square root of the channel count, and the results are averaged over the other views 2403.14627 §3.1 2403.14627 eq. 2-4. The volume has one plane per depth candidate at 1/4 resolution. The paper's experiments use D = 128 2403.14627 §4.1.

A softmax over the planes gives a per-pixel distribution. The coarse depth is its expectation, and the peak probability is kept as a matching confidence 2403.14627 §3.1 §3.2 2403.14627 eq. 6. The volume is then refined by a U-Net whose output is added back as a residual [arxiv:2403.14627 §3.1] 2403.14627 eq. 5.

The ablation on RealEstate10K shows the cost volume is the largest single contribution. Removing it drops PSNR from 26.39 to 22.83, while removing the refinement U-Net drops it to 25.45 2403.14627 tab. 3.

All-pairs volume with latent tokens

Optical flow between two frames. The volume is the dot product of every source pixel's feature with every target pixel's feature, so its shape is H × W × H × W at 1/8 resolution. The paper states no normalisation for these products 2203.16194 §3.1. It argues that a transformer applied to this volume directly would need thousands of tokens, which it calls unaffordable 2203.16194 §1. Each source pixel's H × W cost map is therefore patchified and summarised into K = 8 tokens of width D = 128. K learned codewords attend over the patch features to do this 2203.16194 §3.2 2203.16194 §4.1. Three alternating layers then mix the tokens within each pixel and across pixels. The source image's context features join the attention queries and keys in the across-pixel step 2203.16194 §3.2.

A recurrent decoder estimates flow at 1/8 resolution. At each step it crops a 9 × 9 window of the pixel's own cost map around its current match, combines the window with a positional embedding of the match, and cross-attends to the pixel's tokens. A ConvGRU predicts the flow update 2203.16194 §3.3 2203.16194 eq. 4-5. The number of recurrent steps is not stated in the paper 2203.16194 §3.3.

The paper's ablation is trained on a shorter RAFT-style schedule than its main results 2203.16194 §4.3. On KITTI-15 train F1-all, removing the intra- and inter-cost-map attention gives 16.16, and three layers of it give 15.15 2203.16194 tab. 2. With K = 8, raising the token width from 32 to 128 lowers the same metric from 18.71 to 17.64 2203.16194 tab. 2.

Matching layer shared across tasks

One parameter-free matcher serves optical flow, rectified stereo and posed depth, behind a shared Transformer encoder. Image features first pass through six Transformer blocks with self-attention and cross-attention 2211.05783 §3.2. The matcher then scores each pixel against its candidates, takes a softmax over all candidates, and reads out the expected target. Flow uses every pixel of the other image as a candidate. Stereo uses the pixels on one scanline, with correlations masked so that matches lie on one side. Depth uses inverse-depth planes, warped with the known cameras 2211.05783 §3.1. The paper's depth setting uses 64 planes from 0.5 m to 10 m 2211.05783 §4.3.

Unlike the group-wise and depth-indexed volumes, nothing aggregates the correlations in 3D. Learned capacity sits in the feature Transformer before the matcher. Matching assumes the point is visible in both views, so a self-attention layer spreads matched values to occluded and out-of-boundary pixels 2211.05783 §3.3.

The paper reports one set of weights across the three tasks. Depth initialised from a flow model reaches 0.198 Abs Rel on ScanNet with no fine-tuning, against 0.536 from random initialisation 2211.05783 tab. 10. The paper finds no gain when stereo or depth models are used to start flow 2211.05783 §4.4.

When to use which

Timings are not comparable across the papers. The stereo timings from the group-wise paper come from a TITAN Xp at 480 × 640 1903.04025 tab. 2. The plane-sweep timing includes encoding and rendering, and its hardware is not stated 2403.14627 tab. 1. The flow paper reports no timing or memory measurement in its main text 2203.16194 §4. The unifying paper reports V100 and A100 timings for its flow models 2211.05783 tab. 3, at the Sintel resolution, and its stereo timings use its own settings [arxiv:2211.05783 tab. 7].

Implementation sketch

The stereo and plane-sweep variants reduce to the same loop: build one slice per candidate, then turn the refined volume into a value by expectation. The group-wise version below follows the paper's eq. 3. The depth-indexed version is a resampling of a disparity volume and comes next. The plane-sweep version follows eq. 2 to 4. The all-pairs variant has no candidate loop, so it is sketched last. The matching layer is described in prose only, since its readout is the same expectation as the plane-sweep one.

# Group-wise correlation volume  [arxiv:1903.04025 §3.2, eq. 3]
# left, right: feature maps (C, H, W); right is shifted by d along x

def group_wise_volume(left, right, n_groups, d_max):
    # Split the C channels into equal groups  [§3.2]
    left_g = split(left, n_groups)  # (n_groups, C / n_groups, H, W)
    volume = []
    for d in range(d_max):
        # Pair pixel x in the left map with x - d in the right  [eq. 3]
        right_g = split(shift(right, d), n_groups)
        # Mean of the channel products within each group  [eq. 3]
        volume.append(mean(left_g * right_g, over=channels))
    return stack(volume, axis=1)  # (n_groups, d_max, H, W)

# Depth-indexed stereo volume  [arxiv:1906.06310 §3]
# disp_volume: disparity cost volume (D_disp, H, W) from a disparity network
# depths: 1 m to 80 m in 1 m steps  [§3]

def depth_indexed_volume(disp_volume, focal, baseline, depths):
    # Disparity that each depth bin corresponds to, f * b / z  [§3]
    disparities = focal * baseline / depths
    # Bilinear interpolation along the disparity axis  [§3]
    return interp_along_disparity(disp_volume, disparities)  # (D_z, H, W)

# Plane-sweep cost volume  [arxiv:2403.14627 §3.1, eq. 2-4]
# feats: features of all K views at 1/4 resolution, view i is the target

def plane_sweep_volume(feats, i, cams, depths):
    costs = []
    for depth in depths:  # D planes, uniform in inverse depth  [§3.1]
        # Warp each other view's features onto the plane at this depth
        # [eq. 2]
        warped = [warp(feats[j], cams, i, j, depth) for j in views_except(i)]
        # Scaled dot product per view, then average over the views  [eq. 3]
        corr = [sum(feats[i] * w, over=channels) / sqrt(C) for w in warped]
        costs.append(mean(corr))
    return stack(costs)  # (D, H/4, W/4)

# All-pairs volume with latent tokens  [arxiv:2203.16194 §3.1-3.2]
# feats_src, feats_tgt: features at 1/8 resolution, (H, W, C), C = 256

def all_pairs_tokens(feats_src, feats_tgt, codewords):
    # Dot product of every source pixel with every target pixel  [§3.1]
    # The paper states no normalisation for these products
    cost = einsum("hwc,ijc->hwij", feats_src, feats_tgt)  # (H, W, H, W)
    # Each source pixel's H x W cost map becomes K tokens of width D  [§3.2]
    tokens = [summarize(patchify(cost[y, x]), codewords) for y, x in grid]
    return stack(tokens)  # (H, W, K, D), with K = 8 and D = 128  [§4.1]

Readout for the stereo, depth-indexed, plane-sweep and matching-layer variants: a softmax over the candidate axis, then the expectation of the candidate values 1903.04025 §3.4, eq. 4 1906.06310 §3 [arxiv:2403.14627 §3.1, eq. 6] 2211.05783 §3.1, eq. 2-3. The all-pairs variant has no candidate axis, so its readout is the recurrent decoder described above.

Contradictions

Apart from the FlowFormer score below, the papers' results do not conflict. They solve different tasks: disparity on Scene Flow and KITTI, depth for 3D detection on KITTI, rendering quality on RealEstate10K, and flow, stereo and depth on Sintel, KITTI, Scene Flow, ScanNet and others. Their numbers are not comparable.

The unifying paper lists FlowFormer at 2.12 EPE on Sintel test final 2211.05783 tab. 14. The FlowFormer paper reports 2.09 for its own model on the same split 2203.16194 tab. 1. The two papers do not say whether these are the same checkpoint or how the 2.12 was measured, so the gap is unexplained. The flow row in the table above relies on the 2.09 figure.

Within the flow paper, the full model's Sintel train clean result is 1.01 in one table and 1.00 in another. Both are described as the same training schedule, and the paper does not explain the gap 2203.16194 tab. 1 [arxiv:2203.16194 tab. 3].

Two details of the released code differ from the text of the plane-sweep paper: the cost volume stays at 1/4 resolution, and the expectation averages inverse depths. The card's Paper vs code section records both 2403.14627 §3.1 2403.14627 eq. 6.

The depth-indexed paper writes its depth weights as a softmax of negative scores. The released code applies the softmax to the raw scores, without the minus sign. The card notes that the learned classifier can absorb the sign, but loading the pretrained weights needs the code's convention [arxiv:1906.06310 §3].

The unifying paper's stereo correlation has no normalisation term [arxiv:2211.05783 §3.1]. Its released code divides by the square root of the channel count. The card's Paper vs code section records this difference.

Papers

• Group-wise Correlation Stereo Network: builds the stereo volume from channel groups, with a concatenation volume added alongside.
• FlowFormer: A Transformer Architecture for Optical Flow: compresses each pixel's all-pairs cost map into 8 latent tokens, then decodes flow recurrently from them.
• Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving: re-indexes a stereo volume onto a 1 m depth grid and trains on depth, then corrects the depth with 4-beam LiDAR.
• Unifying Flow, Stereo and Depth Estimation: one parameter-free matcher after a six-block Transformer serves optical flow, rectified stereo and posed depth.
• MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images: builds a plane-sweep volume over 128 inverse-depth planes to predict depth for each Gaussian.