Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching

In plain words

Stereo matching works out how far each point shifts between a left and a right photo. That shift, the disparity, gives depth. Modern networks build a table of match scores for every pixel and every candidate shift, then read off a smooth average shift. Today's networks train this table only indirectly, through the final shift, so many different tables can produce the right answer even when the scores themselves look unreasonable.

AcfNet makes the table itself the training target. For each pixel, the score should be lowest at the true shift and rise with distance from it. How sharp that dip is gets decided per pixel: sharp where the match is clear, flatter where the match is ambiguous, such as on plain surfaces or at occluded pixels. A special loss keeps training focused on the true shift rather than on the many wrong ones.

The authors report state-of-the-art results on Scene Flow and two KITTI benchmarks.

TL;DR

AcfNet is a PSMNet-based stereo network. It supervises each cost volume with a unimodal target centred on the ground-truth disparity, takes the target's width per pixel from a confidence head, and adds a stereo focal loss. Author-reported: Scene Flow EPE 0.867 px and 3PE 4.31% for the adaptive variant tab. 2. The paper says the method ranked 1st on KITTI 2012 and 4th on KITTI 2015 when recorded on 2019-08-20 § Introduction. The official code is MIT-licensed, and its configs differ from the paper in several places (see Paper vs code).

Problem

Stereo networks regress disparity as a soft argmin over a cost volume, and the loss is applied only to the disparity § AcfNet (Overview) eq. 1 eq. 2. The cost volume therefore has no direct constraint. The authors state that infinitely many cost distributions can give the same disparity, so the network may learn implausible ones, for example two local minima away from the true value § Introduction fig. 1. They argue that this under-constrained supervision invites overfitting § AcfNet (Overview). The setting assumes rectified image pairs and disparities in {0, ..., D-1} with D = 192 § AcfNet (Overview) § Experiments (Implementation details).

Method

AcfNet keeps PSMNet's stacked-hourglass backbone and 3D aggregation. It produces three cost volumes, one per output stage fig. 2 tab. 6. For each output, a confidence network gives a per-pixel confidence, which sets a per-pixel width. The ground-truth cost volume is then a unimodal distribution with that width. The stereo focal loss supervises the predicted cost volume against it. A smooth L1 loss supervises the disparity, and a confidence regulariser encourages high confidence on more pixels § AcfNet (Total loss function) fig. 2.

1. Soft argmin. At each pixel, costs c_d over d in {0, ..., D-1} give P̂(d) = exp(-c_d) / Σ exp(-c_d'), and the disparity is d̂ = Σ d · P̂(d). A lower cost means a better match § AcfNet (Overview) eq. 1.
2. Disparity loss. L_reg is the mean over pixels of smooth L1(d_gt - d̂), where smooth L1(x) = 0.5 x² for |x| < 1 and |x| - 0.5 otherwise § AcfNet (Overview) eq. 2 eq. 3.
3. Unimodal target. With ground truth d_gt and width σ > 0, the target is P(d) = softmax over d of (-|d - d_gt| / σ). A smaller σ gives a sharper peak § AcfNet (Unimodal distribution) eq. 4.
4. Per-pixel width. A 3×3 convolution, batch normalisation and ReLU are followed by a 1×1 convolution and a sigmoid, giving confidence f in [0, 1]. The width is σ_p = s (1 - f_p) + ε, so σ_p lies in [ε, s + ε], where s ≥ 0 sets the sensitivity and ε > 0 the lower bound § AcfNet (Confidence estimation network) eq. 5. Texture-less and occluded pixels are expected to get large σ § AcfNet (Confidence estimation network).
5. Stereo focal loss. L_SF is the mean over pixels of Σ_d (1 - P_p(d))^(-α) · (-P_p(d) · log P̂_p(d)). α ≥ 0 is the focusing parameter, and α = 0 gives cross-entropy. The loss is described as analogous to the focal loss for one-stage detection § AcfNet (Stereo focal loss) eq. 6.
6. Confidence regulariser. L_conf is the mean over pixels of -log f_p § AcfNet (Total loss function) eq. 8.
7. Total. L = L_SF + λ_reg · L_reg + λ_conf · L_conf § AcfNet (Total loss function) eq. 7.

# AcfNet training loss for one batch  [§ AcfNet]  [eq. 1]-[eq. 8]
# Inputs:  left, right (B, 3, H, W); gt (B, H, W) disparity
#          gt outside (0, D) is invalid and ignored
# Costs are distances here: lower cost = better match  [eq. 1]

def training_loss(left, right, gt):
    # Three cost volumes, one per aggregation output  [fig. 2] [tab. 6]
    costs = aggregated_costs(left, right)        # 3 x (B, D, H, W)
    # Candidate disparities 0 .. D-1  [§ AcfNet] (Overview)
    disps = arange(D).reshape(1, D, 1, 1)        # (1, D, 1, 1)
    loss = 0.0
    # Level weights are not in the paper; code uses (1.0, 0.7, 0.5)
    for level, cost in enumerate(costs):
        # Confidence f in [0, 1] per pixel  [§ AcfNet] (Confidence)
        conf = sigmoid(confidence_net(cost))     # (B, 1, H, W)
        # Width sigma_p in [eps, s + eps]  [eq. 5]
        sigma = s * (1 - conf) + eps             # (B, 1, H, W)
        # Unimodal target peaked at gt  [eq. 4]
        target = softmax(-abs(disps - gt) / sigma, dim=1)
        # Log of the predicted distribution  [eq. 1]
        log_pred = log_softmax(-cost, dim=1)
        # Focal factor (1 - P)^(-alpha)  [eq. 6]
        focal = (1 - target) ** (-alpha)
        # Stereo focal loss, averaged over valid pixels  [eq. 6]
        loss += level_weight[level] * mean_valid(
            -target * focal * log_pred
        )
        # Confidence regulariser -log f  [eq. 8]
        loss += level_weight[level] * lambda_conf * mean_valid(
            -log(conf)
        )
        # Soft-argmin disparity, then smooth L1  [eq. 1] [eq. 2] [eq. 3]
        pred = sum(disps * softmax(-cost, dim=1), dim=1)
        loss += level_weight[level] * lambda_reg * smooth_l1(gt - pred)
    return loss

Hyperparameters

Architecture

• Feature extraction: PSMNet-style stages conv0 to conv4 with dilations, four pooled branches (64, 32, 16 and 8 pixel grids) and a fusion to 32 channels at 1/4 resolution tab. 6.
• Cost volume: left features concatenated with right features shifted by each candidate disparity, giving 64 channels at D/4 × H/4 × W/4 tab. 6 cat_fms.py.
• Aggregation: a 3D convolution stack, three stacked hourglass modules, and three outputs. Each output is a 1-channel volume, and later outputs add the earlier ones tab. 6 AcfNet.py.
• Upsampling: a transposed 3D convolution with kernel 8, stride 4, to full D × H × W tab. 6 AcfNet.py.
• Confidence head: one per output, as described under Method tab. 6 cmn.py.

Results

Author-reported. Setup: Scene Flow has 35,454 training and 4,370 test pairs, with 16% occluded pixels § Experiments (Datasets). KITTI 2012 has 194 training and 195 test pairs, and KITTI 2015 has 200 training and 200 test pairs § Experiments (Datasets). The baseline is PSMNet re-implemented under the same protocol tab. 2.

Ablation, Scene Flow, EPE in px and 3PE in % tab. 2:

Loss comparison, uniform σ = 1.2 tab. 1: cross-entropy EPE 0.965 px; stereo focal loss EPE 0.920 px.

Cross-dataset check, trained on Scene Flow only and tested on KITTI training pairs tab. 3:

Benchmark comparison tab. 4. The caption defines the KITTI outlier threshold as max(3 px, 5% of the true disparity). The table does not say which split each KITTI row uses; the Introduction says the results were submitted to the evaluation servers § Introduction.

The table bolds GwcNet-gc for Scene Flow EPE (0.77) and AcfNet for the KITTI columns tab. 4. AcfNet's Scene Flow EPE appears as 0.87 here and 0.867 in tab. 2.

Variance behaviour: the histogram over Scene Flow test pixels is concentrated at small σ, with a long tail of larger σ fig. 4 § Experiments (Ablation studies). Sparsification: removing the 6.9% of pixels with the highest σ halves the average EPE fig. 5 § Experiments (Variance analysis).

Backbone transfer, Scene Flow tab. 5, from Appendix A:

§ Appendices (A. Effectiveness on different backbones)

Paper vs code

Official code: scene_flow_adaptive.py at commit 010aeb66e3ceaf3d866036b0ca751861df39432d. The paper's loss and σ equations match the code's formulas. The differences are in conventions, unstated details and schedules.

• Cost sign. The paper writes P̂ = softmax(-c) with c a distance eq. 1. The code applies softmax to the raw cost, so its cost is a similarity faster_soft_argmin.py stereo_focal_loss.py. The target is still softmax(-|d - d_gt| / σ) disp2prob.py disp2prob.py. The two agree only if the network output is negated when the paper's formulas are ported.
• Scale weights. The paper's eq. 7 has no weights on the three outputs. The code applies the regression, confidence and focal losses to all three outputs with weights (1.0, 0.7, 0.5) scene_flow_adaptive.py scene_flow_adaptive.py scene_flow_adaptive.py stereo_focal_loss.py.
• λ_reg. The text gives no value (figure 3c only). The code uses 0.1 scene_flow_adaptive.py.
• Averaging. The paper averages over all pixels in P eq. 6 eq. 8. The code averages over valid pixels, which are those with GT inside the range stereo_focal_loss.py conf_nll_loss.py.
• Confidence head width. Table 6 lists 48 channels in the CENet 3×3 convolution. The code uses in_planes // 3 with in_planes = 192, so 64 channels cmn.py scene_flow_adaptive.py.
• Scene Flow schedule. The paper trains 10 epochs at constant lr 0.001. The config runs 20 epochs with a linear warm-up and a step at epoch 20, and it evaluates the epoch-10 checkpoint scene_flow_adaptive.py scene_flow_adaptive.py scene_flow_adaptive.py.
• Batch size. The paper uses batch 3 on 3 GPUs. The adaptive Scene Flow config uses 1 image per GPU on 4 GPUs scene_flow_adaptive.py scene_flow_adaptive.py. The uniform config uses 2 per GPU on 4 GPUs scene_flow_uniform.py scene_flow_uniform.py.
• KITTI schedule. The paper gives 600 epochs with decays at 100 and 300. The config lists a third step at 600 and uses a constant 100-iteration warm-up and gradient clipping at 0.1 kitti_2015_adaptive.py kitti_2015_adaptive.py kitti_2015_adaptive.py.
• Crops. The paper does not state the training crop. The configs use 384×512 for Scene Flow adaptive, 256×512 for Scene Flow uniform, and 256×768 for KITTI adaptive scene_flow_adaptive.py scene_flow_uniform.py kitti_2015_adaptive.py.
• Error metric. The paper's KITTI table uses the benchmark outlier rule (max(3 px, 5%)) tab. 4. The shared error function counts errors above fixed 1, 2, 3 and 5 px, over GT strictly inside (0, D), and does not implement the relative rule pixel_error.py. Its visualisation uses tau = [3.0, 0.05] vis.py. KITTI numbers in this card come from the paper and are not reproduced by the shared function.
• License file. The repository's LICENSE is MIT, but its copyright line names "Facebook (2018)" LICENSE. The card records it as MIT as shipped; the code has not been checked for upstream provenance.

Pitfalls

• Pick one sign convention. Port either the paper's distance convention (softmax of -c) or the code's similarity convention (softmax of c). Mixing them trains the network to put its peak away from the true disparity, with no error message.
• The width is not detached. The paper does not say whether gradients from L_SF reach the confidence head. The code does not detach σ, so the focal loss also sends gradient through the target and the focal weight into the confidence head cmn.py stereo_focal_loss.py stereo_focal_loss.py.
• The focal exponent is negative. The factor (1 - P)^(-α) grows with P. It boosts entries near the true disparity, and entries with P near 0 keep a weight near 1. The text says easy negatives are suppressed; the formula does not do that by itself. Derived from eq. 4 with unit disparity spacing and σ = 1, the interior target peak is about 0.46, so the factor is at most about 22. This is our arithmetic, not a statement in the paper, and the code adds no clamp.
• Invalid pixels in the target. Pixels outside the valid range get target 0 plus a constant 1e-40 disp2prob.py. The focal loss masks them out, but the target is still built over all pixels first.
• Sparse KITTI ground truth. KITTI zeros mean "no ground truth". The code masks them in both losses smooth_l1_loss.py. The sparse setting also switches the downsampling of GT to max pooling stereo_focal_loss.py.
• Full-resolution volumes. The three volumes are upsampled to D × H × W with 192 channels for the confidence head cmn.py AcfNet.py. Memory cost grows with D.
• Uniform variant needs a fixed σ. The uniform variant uses σ = 1.2 and has no confidence net scene_flow_uniform.py. The paper found this value by grid search, but the grid is not described.
• Epoch count. Evaluate the checkpoint the config names, not the last epoch. The configs evaluate epoch 10 (Scene Flow) and epoch 600 (KITTI).

Lineage

• Builds on: PSMNet, the backbone and cost aggregation § AcfNet (Overview) tab. 6; GC-Net, for soft argmin § AcfNet (Overview); and the focal loss for one-stage detection, which the stereo focal loss adapts § AcfNet (Stereo focal loss).
• Compared against: PSMNet (re-implemented), GSM (concurrent LiDAR-guided cost filtering) § Experiments (Cost volume filtering comparisons), and the methods in tab. 4.
• Superseded by: not stated in the paper.