--- kind: concept slug: cost-volume title: Cost volume summary: >- Matching scores between views, one per candidate depth or disparity, or per pixel pair for flow. Variants: group-wise and depth-indexed stereo volumes, plane-sweep for posed views, all-pairs latent tokens for flow, and a parameter-free softmax matcher shared across tasks. Trade-off: explicit matching evidence against memory that grows with the candidates. Author-reported. tags: [stereo-matching, multi-view-stereo, depth-estimation, novel-view-synthesis, optical-flow] papers: ["1903.04025", "1906.06310", "2203.16194", "2211.05783", "2403.14627"] related: [cost-aggregation, multi-view-stereo, feed-forward-reconstruction, gaussian-splatting, optical-flow, dense-correspondence] status: draft updated_at: 2026-10-31 --- ## Overview A cost volume stores a matching score for each pixel and each candidate depth or disparity. The score says how well a pixel's appearance in one view agrees with the appearance of the same point in another view, if the point lies at that candidate. High scores at one candidate suggest a surface there [arxiv:1903.04025 §1] [arxiv:2403.14627 §3.1]. Four of the five papers below read one value per pixel from the volume as the expectation over the candidates, a softmax-weighted average. Stereo matching uses the candidates as disparities, a horizontal pixel shift in a rectified pair [arxiv:1903.04025 §1]. A depth-indexed variant re-indexes the same kind of volume onto metric depth bins [arxiv:1906.06310 §3]. The plane-sweep case uses the candidates as depth planes, and the features are warped between posed views [arxiv:2403.14627 §3.1]. The construction methods differ enough to be separate variants below. Optical flow has no small set of candidates to test. Its volume compares every source pixel with every target pixel, and the decoder reads it through learned tokens instead of an expectation [arxiv:2203.16194 §3.1] [§3.3]. It is the fourth variant. The matching layer of the unifying paper also compares every pixel pair for flow, but reads it by expectation and never aggregates it in 3D. It is the fifth variant [arxiv:2211.05783 §3.1]. ## Variants ### Group-wise correlation volume Stereo disparity from a rectified pair. The feature channels are split into groups, and each group is correlated with the other view at every disparity. Each group gives one channel of the volume, so the volume has N_g channels per disparity instead of one [arxiv:1903.04025 §3.2, eq. 3]. The correlation is divided by the number of channels in a group, so the scale depends on group size rather than on group count [arxiv:1903.04025 §3.2, eq. 3]. The paper's setting uses 40 groups over 320 unary feature channels, which is 8 channels per group [arxiv:1903.04025 §4.3] [arxiv:1903.04025 tab. 1]. A second, concatenation-based volume is added to it. It uses features compressed to 12 channels per view, giving 24 more channels, so 64 in total [arxiv:1903.04025 §3.1] [arxiv:1903.04025 tab. 1]. The full model is called GwcNet-gc [arxiv:1903.04025 tab. 2]. The Scene Flow ablation in the paper shows the effect of each choice, with each volume fed to the same base network, which has no stacked hourglasses [arxiv:1903.04025 §4.3] [arxiv:1903.04025 tab. 2]: | Volume | EPE (px) | Time (ms) | |---|---|---| | Concatenation only, 64 channels | 1.308 | 117.1 | | Full correlation, 1 group | 1.369 | 104.0 | | Group-wise, 10 groups | 1.230 | 112.8 | | Group-wise, 40 groups | 1.212 | 122.2 | | Group-wise, 160 groups | 1.188 | 157.3 | | Group-wise, 40 groups, with concatenation | 1.127 | 135.1 | The authors report timings at 480 × 640 on one Nvidia TITAN Xp [arxiv:1903.04025 tab. 2]. Going from 40 to 80 groups gives no gain in that sweep, and 160 groups gives a small one at 157.3 ms, against 122.2 ms for 40 [arxiv:1903.04025 §4.3] [arxiv:1903.04025 tab. 2]. ### Depth-indexed stereo volume Stereo depth from a rectified pair, used when the output should be metric depth rather than disparity. The paper builds a disparity volume with a PSMNet-style network, then re-indexes it onto a depth grid with 1 m steps from 1 m to 80 m. Each depth bin is filled by bilinear interpolation from the disparity volume, at the disparity that the bin corresponds to, which is the focal length times the baseline divided by depth [arxiv:1906.06310 §3]. Three- dimensional convolutions then run on the depth grid. The output is a softmax- weighted sum over the bins, and the training loss is a smooth L1 on depth [arxiv:1906.06310 §3]. The motivation is that a fixed disparity error costs more depth at range. In the paper's KITTI setting, a one-pixel error is about 0.1 m of depth at 5 m and 5.8 m at 50 m [arxiv:1906.06310 §3] [arxiv:1906.06310 eq. 5]. Training on disparity therefore spends its effort on near objects and gives too little weight to distant ones [arxiv:1906.06310 §3]. The effect is reported on KITTI validation, car category, PointRCNN, moderate difficulty, IoU 0.7. With the depth-indexed network alone, AP_BEV rises from 56.0 for pseudo-LiDAR with PSMNet to 64.0 [arxiv:1906.06310 tab. 1]. A depth loss on PSMNet alone gives 61.9 [arxiv:1906.06310 tab. 3]. For PointRCNN the paper finds the depth loss helps more than the depth cost volume, and for vPIXOR and AVOD the cost volume helps as much or more [arxiv:1906.06310 §5.2] [arxiv:1906.06310 app. D.1]. The volume needs 64-beam LiDAR for its training depth, even when only the stereo network is used [arxiv:1906.06310 §5.1]. ### Plane-sweep volume with scaled dot product Feed-forward novel view synthesis from sparse posed views. Features of each other view are warped onto D depth planes sampled uniformly in inverse depth. Each warped feature is compared with the target view by a dot product, scaled by the square root of the channel count, and the results are averaged over the other views [arxiv:2403.14627 §3.1] [arxiv:2403.14627 eq. 2-4]. The volume has one plane per depth candidate at 1/4 resolution. The paper's experiments use D = 128 [arxiv:2403.14627 §4.1]. A softmax over the planes gives a per-pixel distribution. The coarse depth is its expectation, and the peak probability is kept as a matching confidence [arxiv:2403.14627 §3.1] [§3.2] [arxiv:2403.14627 eq. 6]. The volume is then refined by a U-Net whose output is added back as a residual [arxiv:2403.14627 §3.1] [arxiv:2403.14627 eq. 5]. The ablation on RealEstate10K shows the cost volume is the largest single contribution. Removing it drops PSNR from 26.39 to 22.83, while removing the refinement U-Net drops it to 25.45 [arxiv:2403.14627 tab. 3]. ### All-pairs volume with latent tokens Optical flow between two frames. The volume is the dot product of every source pixel's feature with every target pixel's feature, so its shape is H × W × H × W at 1/8 resolution. The paper states no normalisation for these products [arxiv:2203.16194 §3.1]. It argues that a transformer applied to this volume directly would need thousands of tokens, which it calls unaffordable [arxiv:2203.16194 §1]. Each source pixel's H × W cost map is therefore patchified and summarised into K = 8 tokens of width D = 128. K learned codewords attend over the patch features to do this [arxiv:2203.16194 §3.2] [arxiv:2203.16194 §4.1]. Three alternating layers then mix the tokens within each pixel and across pixels. The source image's context features join the attention queries and keys in the across-pixel step [arxiv:2203.16194 §3.2]. A recurrent decoder estimates flow at 1/8 resolution. At each step it crops a 9 × 9 window of the pixel's own cost map around its current match, combines the window with a positional embedding of the match, and cross-attends to the pixel's tokens. A ConvGRU predicts the flow update [arxiv:2203.16194 §3.3] [arxiv:2203.16194 eq. 4-5]. The number of recurrent steps is not stated in the paper [arxiv:2203.16194 §3.3]. The paper's ablation is trained on a shorter RAFT-style schedule than its main results [arxiv:2203.16194 §4.3]. On KITTI-15 train F1-all, removing the intra- and inter-cost-map attention gives 16.16, and three layers of it give 15.15 [arxiv:2203.16194 tab. 2]. With K = 8, raising the token width from 32 to 128 lowers the same metric from 18.71 to 17.64 [arxiv:2203.16194 tab. 2]. ### Matching layer shared across tasks One parameter-free matcher serves optical flow, rectified stereo and posed depth, behind a shared Transformer encoder. Image features first pass through six Transformer blocks with self-attention and cross-attention [arxiv:2211.05783 §3.2]. The matcher then scores each pixel against its candidates, takes a softmax over all candidates, and reads out the expected target. Flow uses every pixel of the other image as a candidate. Stereo uses the pixels on one scanline, with correlations masked so that matches lie on one side. Depth uses inverse-depth planes, warped with the known cameras [arxiv:2211.05783 §3.1]. The paper's depth setting uses 64 planes from 0.5 m to 10 m [arxiv:2211.05783 §4.3]. Unlike the group-wise and depth-indexed volumes, nothing aggregates the correlations in 3D. Learned capacity sits in the feature Transformer before the matcher. Matching assumes the point is visible in both views, so a self-attention layer spreads matched values to occluded and out-of-boundary pixels [arxiv:2211.05783 §3.3]. The paper reports one set of weights across the three tasks. Depth initialised from a flow model reaches 0.198 Abs Rel on ScanNet with no fine-tuning, against 0.536 from random initialisation [arxiv:2211.05783 tab. 10]. The paper finds no gain when stereo or depth models are used to start flow [arxiv:2211.05783 §4.4]. ## When to use which | Situation | Use | Why | |---|---|---| | Rectified stereo pair, disparity output, fixed 3D aggregation budget | Group-wise correlation volume | At the same network width, 40 groups give 1.212 px EPE against 1.308 px for the concatenation volume, at 122.2 ms against 117.1 ms [arxiv:1903.04025 tab. 2] | | Stereo, lowest run-time over accuracy | Single full correlation | It runs at 104.0 ms, the fastest row, but has the highest EPE in the sweep, 1.369 px [arxiv:1903.04025 tab. 2] | | Sparse posed views, novel views wanted without depth labels | Plane-sweep volume with scaled dot product | Trained with a rendering loss only, and the volume is the largest contributor in the ablation [arxiv:2403.14627 §3.3] [arxiv:2403.14627 tab. 3] | | Optical flow, dense motion between two frames | All-pairs volume with latent tokens | The paper reports the best Sintel test AEPE in its table, 1.16 clean and 2.09 final, against 1.39 and 2.47 for GMA with warm start [arxiv:2203.16194 tab. 1]. The warm-start rows are not like-for-like, and the paper says so [arxiv:2203.16194 §4.1] | | Optical flow, parameter budget tight | All-pairs volume, small configuration | At 6.2M parameters it gives 1.20 on Sintel train clean, against 1.30 for GMA at 5.9M. The full model has 18.2M [arxiv:2203.16194 tab. 3] | | Poses, intrinsics or depth range unknown | Neither candidate variant | Both need calibration; the plane-sweep volume needs known extrinsics and near and far bounds [arxiv:2403.14627 §3] [arxiv:2403.14627 §4.1]. The all-pairs variant predicts 2D motion from an image pair and takes no such inputs [arxiv:2203.16194 §3.3] | | Stereo output must be metric depth, and far objects matter downstream | Depth-indexed stereo volume | The paper's depth-indexed network raises AP_BEV from 56.0 to 64.0 on KITTI validation, stereo only, against pseudo-LiDAR with PSMNet [arxiv:1906.06310 tab. 1]. The gain comes from a different architecture and loss than the volume alone, and the paper says the depth loss helps more than the volume for PointRCNN [arxiv:1906.06310 §5.2]. Training needs 64-beam LiDAR depth [arxiv:1906.06310 §5.1] | | One model for flow, stereo and posed depth, or transfer between them | Matching layer shared across tasks | The same weights serve all three tasks, and flow initialisation helps depth without fine-tuning, 0.198 against 0.536 Abs Rel [arxiv:2211.05783 tab. 10]. The paper reports no transfer gain in the other direction [arxiv:2211.05783 §4.4] | | Stereo with a Transformer encoder, Scene Flow accuracy and speed | Matching layer, stereo mode | GMStereo with 4 refinements gives 0.77 EPE at 86 ms with random init, against 0.86 EPE at 122 ms for RAFT-Stereo with 31 refinements [arxiv:2211.05783 tab. 7]. These rows compare whole models, not the volumes, so they do not show the volume's effect alone | | Memory or run-time is tight | Reduce the number of candidates or the input size, after measuring | Candidate volumes grow linearly with the number of candidates, and the all-pairs volumes grow with the square of the pixel count [arxiv:2203.16194 §3.1] [arxiv:2211.05783 §3.1]. The group-wise paper gives timings at fixed settings only, the plane-sweep paper does not report a sweep over D, and neither the flow paper nor the unifying paper gives memory figures [arxiv:1903.04025 tab. 2] [arxiv:2403.14627 §4.1] [arxiv:2203.16194 §4] [arxiv:2211.05783 §3.1] | Timings are not comparable across the papers. The stereo timings from the group-wise paper come from a TITAN Xp at 480 × 640 [arxiv:1903.04025 tab. 2]. The plane-sweep timing includes encoding and rendering, and its hardware is not stated [arxiv:2403.14627 tab. 1]. The flow paper reports no timing or memory measurement in its main text [arxiv:2203.16194 §4]. The unifying paper reports V100 and A100 timings for its flow models [arxiv:2211.05783 tab. 3], at the Sintel resolution, and its stereo timings use its own settings [arxiv:2211.05783 tab. 7]. ## Implementation sketch The stereo and plane-sweep variants reduce to the same loop: build one slice per candidate, then turn the refined volume into a value by expectation. The group-wise version below follows the paper's eq. 3. The depth-indexed version is a resampling of a disparity volume and comes next. The plane-sweep version follows eq. 2 to 4. The all-pairs variant has no candidate loop, so it is sketched last. The matching layer is described in prose only, since its readout is the same expectation as the plane-sweep one. ```pseudocode # Group-wise correlation volume [arxiv:1903.04025 §3.2, eq. 3] # left, right: feature maps (C, H, W); right is shifted by d along x def group_wise_volume(left, right, n_groups, d_max): # Split the C channels into equal groups [§3.2] left_g = split(left, n_groups) # (n_groups, C / n_groups, H, W) volume = [] for d in range(d_max): # Pair pixel x in the left map with x - d in the right [eq. 3] right_g = split(shift(right, d), n_groups) # Mean of the channel products within each group [eq. 3] volume.append(mean(left_g * right_g, over=channels)) return stack(volume, axis=1) # (n_groups, d_max, H, W) ``` ```pseudocode # Depth-indexed stereo volume [arxiv:1906.06310 §3] # disp_volume: disparity cost volume (D_disp, H, W) from a disparity network # depths: 1 m to 80 m in 1 m steps [§3] def depth_indexed_volume(disp_volume, focal, baseline, depths): # Disparity that each depth bin corresponds to, f * b / z [§3] disparities = focal * baseline / depths # Bilinear interpolation along the disparity axis [§3] return interp_along_disparity(disp_volume, disparities) # (D_z, H, W) ``` ```pseudocode # Plane-sweep cost volume [arxiv:2403.14627 §3.1, eq. 2-4] # feats: features of all K views at 1/4 resolution, view i is the target def plane_sweep_volume(feats, i, cams, depths): costs = [] for depth in depths: # D planes, uniform in inverse depth [§3.1] # Warp each other view's features onto the plane at this depth # [eq. 2] warped = [warp(feats[j], cams, i, j, depth) for j in views_except(i)] # Scaled dot product per view, then average over the views [eq. 3] corr = [sum(feats[i] * w, over=channels) / sqrt(C) for w in warped] costs.append(mean(corr)) return stack(costs) # (D, H/4, W/4) ``` ```pseudocode # All-pairs volume with latent tokens [arxiv:2203.16194 §3.1-3.2] # feats_src, feats_tgt: features at 1/8 resolution, (H, W, C), C = 256 def all_pairs_tokens(feats_src, feats_tgt, codewords): # Dot product of every source pixel with every target pixel [§3.1] # The paper states no normalisation for these products cost = einsum("hwc,ijc->hwij", feats_src, feats_tgt) # (H, W, H, W) # Each source pixel's H x W cost map becomes K tokens of width D [§3.2] tokens = [summarize(patchify(cost[y, x]), codewords) for y, x in grid] return stack(tokens) # (H, W, K, D), with K = 8 and D = 128 [§4.1] ``` Readout for the stereo, depth-indexed, plane-sweep and matching-layer variants: a softmax over the candidate axis, then the expectation of the candidate values [arxiv:1903.04025 §3.4, eq. 4] [arxiv:1906.06310 §3] [arxiv:2403.14627 §3.1, eq. 6] [arxiv:2211.05783 §3.1, eq. 2-3]. The all-pairs variant has no candidate axis, so its readout is the recurrent decoder described above. ## Contradictions Apart from the FlowFormer score below, the papers' results do not conflict. They solve different tasks: disparity on Scene Flow and KITTI, depth for 3D detection on KITTI, rendering quality on RealEstate10K, and flow, stereo and depth on Sintel, KITTI, Scene Flow, ScanNet and others. Their numbers are not comparable. The unifying paper lists FlowFormer at 2.12 EPE on Sintel test final [arxiv:2211.05783 tab. 14]. The FlowFormer paper reports 2.09 for its own model on the same split [arxiv:2203.16194 tab. 1]. The two papers do not say whether these are the same checkpoint or how the 2.12 was measured, so the gap is unexplained. The flow row in the table above relies on the 2.09 figure. Within the flow paper, the full model's Sintel train clean result is 1.01 in one table and 1.00 in another. Both are described as the same training schedule, and the paper does not explain the gap [arxiv:2203.16194 tab. 1] [arxiv:2203.16194 tab. 3]. Two details of the released code differ from the text of the plane-sweep paper: the cost volume stays at 1/4 resolution, and the expectation averages inverse depths. The card's Paper vs code section records both [arxiv:2403.14627 §3.1] [arxiv:2403.14627 eq. 6]. The depth-indexed paper writes its depth weights as a softmax of negative scores. The released code applies the softmax to the raw scores, without the minus sign. The card notes that the learned classifier can absorb the sign, but loading the pretrained weights needs the code's convention [arxiv:1906.06310 §3]. The unifying paper's stereo correlation has no normalisation term [arxiv:2211.05783 §3.1]. Its released code divides by the square root of the channel count. The card's Paper vs code section records this difference. ## Papers - [Group-wise Correlation Stereo Network](/papers/1903.04025): builds the stereo volume from channel groups, with a concatenation volume added alongside. - [FlowFormer: A Transformer Architecture for Optical Flow](/papers/2203.16194): compresses each pixel's all-pairs cost map into 8 latent tokens, then decodes flow recurrently from them. - [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](/papers/1906.06310): re-indexes a stereo volume onto a 1 m depth grid and trains on depth, then corrects the depth with 4-beam LiDAR. - [Unifying Flow, Stereo and Depth Estimation](/papers/2211.05783): one parameter-free matcher after a six-block Transformer serves optical flow, rectified stereo and posed depth. - [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](/papers/2403.14627): builds a plane-sweep volume over 128 inverse-depth planes to predict depth for each Gaussian.