Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation

In plain words

To build a 3D model from several photos of a scene, a learned system has to guess how far away each visible point is, for every pixel. Earlier systems did this in one of two ways. One averages many possible distances, which can give precise answers but is hard to train and can overfit. The other picks the single most likely distance, which is easy to train but cannot give a precise number.

This paper combines the two. For each pixel, the system first decides which slice of the possible distances the surface sits in, and then only estimates where it lies inside that slice. Training uses a loss that gives more weight to the hard cases, so the many easy "this slice is not the right one" answers do not drown out the signal. The method is added to a stack that refines the answer in stages, from coarse to fine.

TL;DR

• Replaces the regression-or-classification choice for depth with a "unified" representation: one target per pixel, nonzero only in the depth slice that contains the ground truth §3.2.
• Trains with a new Unified Focal Loss that extends focal loss to continuous targets and rescales by relative error §3.3.
• Reports first place on DTU (Overall 0.315 mm) and on Tanks and Temples (Intermediate mean F-score 64.36, Advanced 38.96). Results are author-reported §4 tab. 1 tab. 2.
• Ablations attribute the gain to the representation and the loss, with an extra small gain from adaptive aggregation and finer DTU ground truth tab. 3.

Problem

Learning-based multi-view stereo (MVS) infers a depth map per view from a 3D cost volume built over a set of depth hypotheses §1 §3.1. Two families exist. Regression takes a soft-argmin (a probability-weighted sum of hypotheses), trained with an L1 loss on the depth. The paper says this indirect path tends to overfit and has many weight combinations that give the same depth §1. Classification takes the argmax hypothesis and is trained with cross-entropy on the probability volume. It constrains the cost volume directly, but it cannot give sub-hypothesis depth §1 §3.1.

The paper also notes that the target in this representation is sparse and imbalanced: one positive hypothesis among many hypotheses per pixel, and hard and easy samples differ §3.3.

Method

Setting and baseline pipeline §3.1. Image features are extracted with shared weights from N views. Depth hypotheses d_1 ... d_M are sampled over the depth range. Features are warped by homography eq. 1 and aggregated into one cost volume, either by variance eq. 2 or by adaptive weighting eq. 3. A softmax regularisation network turns the cost volume into a probability volume P. Regression reads depth as the P-weighted sum of hypotheses eq. 4. Classification reads depth as the argmax hypothesis eq. 5. Multi-stage (coarse-to-fine) methods repeat this at increasing resolution, and each stage samples its hypotheses from the previous stage's depth map §3.1.

Unified depth representation §3.2.

1. Target generation. For each pixel, find the hypothesis i whose interval [d_i, d_i + r) contains the ground truth D_gt, where r = d_{i+1} - d_i. The paper says the last hypothesis reuses the previous interval alg. 1. Set its target to 1 - (D_gt - d_i) / r, which the paper calls proximity. All other targets are 0 alg. 1 §3.2.
2. Prediction. Each of the M values of P is passed through a sigmoid, not a softmax, and interpreted as a unity value U-hat §3.2 §3.4.
3. Decoding. Choose the hypothesis o with the largest unity value. Set the offset to (1 - U-hat_o) * r and output depth = d_o + offset alg. 2 §3.2.

The paper frames this as classification first (choose the slice), then regression inside the slice (offset), trained jointly §3.2.

Unified Focal Loss §3.3. The paper starts from focal loss FL, defined for binary targets q in {0,1} eq. 6. It then uses binary cross-entropy in full form and a scaling factor that depends on the continuous target q, giving the generalised focal loss GFL eq. 7. The paper argues GFL ignores the size of the ground-truth target, so it cannot tell hard from easy positives of different strength. The "naive" UFL divides the error by the positive target q+ in (0, 1] eq. 8. Because that ratio can be very large, the final UFL eq. 9 rescales the factors with a sigmoid-like function S_b (base b) and uses separate weights for positives and negatives. The paper says FL is the special case q+ = 1 §3.3.

Training loss §3.4. The loss is a weighted sum over stages, each stage term being the average UFL at that stage eq. 10. The paper does not give the stage weights numerically.

Network §3.4 §4 fig. 4. The network follows CasMVSNet (a coarse-to-fine cost volume) with an FPN-like feature extractor. It uses three stages at 1/4, 1/2 and 1 of input resolution §4. Hypothesis counts and intervals shrink from coarse to fine §3.4 app. C. Adaptive aggregation is used §3.4 eq. 3. Multi-scale 3D CNNs regularise the cost volume. The probability volume at each stage is treated as the unity volume and decoded with alg. 2 §3.4. The paper says the pink parts of its framework figure come from existing methods and the green parts are new fig. 4.

Inference §4.1 §4.2. On DTU, depth maps are filtered with a probability threshold of 0.3 and at least 3 consistent views, then fused as in MVSNet §4.1. On Tanks and Temples, a dynamic geometric consistency check is used §4.2.

# Unity generation from ground-truth depth  [alg. 1]
# Input: depth_gt (H, W); hypotheses (M, H, W), increasing along M
# Output: unity_gt (M, H, W), at most one non-zero entry per pixel

def make_unity_target(depth_gt, hypotheses):
    # Start from all-zero targets  [alg. 1]
    unity_gt = zeros_like(hypotheses)
    for i in range(M):
        # Interval r = d_{i+1} - d_i; the last hypothesis keeps the
        # previous interval  [alg. 1]
        interval = interval_after(hypotheses, i)
        # The optimal hypothesis is the one whose interval [d_i, d_i + r)
        # contains the ground truth  [alg. 1]
        inside = (hypotheses[i] <= depth_gt) & (
            hypotheses[i] + interval > depth_gt
        )
        # Proximity: one minus the offset to ground truth, relative to r
        # [alg. 1]
        proximity = 1 - (depth_gt - hypotheses[i]) / interval
        unity_gt[i] = where(inside, proximity, 0)
    return unity_gt

# Unity regression at inference  [alg. 2]
# Input: unity_hat (M, H, W) after sigmoid; hypotheses (M, H, W)
# Output: depth (H, W)

def regress_depth(unity_hat, hypotheses):
    # Optimal hypothesis: largest unity per pixel  [alg. 2]
    optimal = argmax(unity_hat, axis=0)               # (H, W)
    # Interval after the optimal hypothesis; the last one uses the
    # previous interval  [alg. 2]
    interval = interval_after(hypotheses, optimal)    # (H, W)
    # Offset is the complement of the unity value times the interval
    # [alg. 2]
    offset = (1 - take(unity_hat, optimal)) * interval
    # Depth is the optimal hypothesis plus the offset  [alg. 2]
    depth = take(hypotheses, optimal) + offset
    return depth

# Unified focal loss for one stage  [eq. 9]
# Input: unity_hat (M, H, W) in (0, 1); unity_gt (M, H, W);
#        valid (M, H, W) mask; gamma; alpha_neg
# Output: scalar loss for this stage

def unified_focal_loss(unity_hat, unity_gt, valid, gamma, alpha_neg):
    # q+: positive target per pixel, 1 if none  [code: loss.py#L70]
    q_plus = max(unity_gt, axis=0)                    # (H, W)
    q_plus = where(q_plus > 0, q_plus, 1)             # (H, W)
    positive = unity_gt > 0                           # (M, H, W)
    # Relative error for positives, relative prediction for negatives
    # [eq. 9]
    pos_scale = scale_positive(abs(unity_gt - unity_hat) / q_plus)
    neg_scale = scale_negative(unity_hat / q_plus)
    # Focal factor; alpha+ = 1, so positives carry no alpha  [eq. 9]
    weight = where(
        positive,
        pos_scale ** gamma,
        alpha_neg * neg_scale ** gamma,
    )
    # Binary cross-entropy against the continuous target  [eq. 9]
    bce = -(unity_gt * log(unity_hat)
            + (1 - unity_gt) * log(1 - unity_hat))
    # Average over valid entries (see Pitfalls)  [code: loss.py#L77]
    return sum(weight * bce * valid) / sum(valid)


# Positive scale: maps x >= 0 into [1, 3)  [app. A eq. b]
def scale_positive(x):
    return 4 * (sigmoid_base5(x) - 0.5) + 1


# Negative scale: maps x >= 0 into [0, 1)  [app. A eq. c]
def scale_negative(x):
    return 2 * (sigmoid_base5(x) - 0.5)


# Sigmoid-like function with base b = 5  [app. A eq. a]
def sigmoid_base5(x):
    return 1 / (1 + 5 ** (-x))

Hyperparameters

Values marked "code" are not stated in the paper. They are the defaults or script arguments in the official repository at the pinned commit.

Datasets §4: DTU, 124 scenes, MVSNet split. Tanks and Temples intermediate (8 scenes) and advanced (6 scenes). BlendedMVS, 113 scenes, split into 106 training and 7 validation scenes.

Results

Author-reported. Setups: DTU evaluation set with 5 views §4 tab. 1. Tanks and Temples with 11 views, F-score §4.2 tab. 2. Ablations on DTU with 5 views, confidence 0.3 and 3 consistent views for all models tab. 3.

DTU evaluation set tab. 1 (lower is better; the paper bolds the best per column):

Tanks and Temples F-score tab. 2 (higher is better):

The paper's text reports 2.68 and 3.24 points above the previous best on the intermediate and advanced sets §4.2. Both values are consistent with the table: 64.36 − 61.68 and 38.96 − 35.72.

Ablation on DTU tab. 3 (5 input views unless noted; "Input" is the number of views; AA = adaptive aggregation; FGT = finer ground truth):

Other ablation tab. 4 (DTU evaluation set; the table gives no input-view count):

Supplementary tables, in the paper's order app. C:

• tab. A Base of the scaling function. Base e, range [1, 2): 0.354 / 0.282 / 0.318. Base 5, range [1, 2): 0.354 / 0.280 / 0.317. Base 5, range [1, 3): 0.352 / 0.278 / 0.315 (ACC / Comp / Overall, mm).
• tab. B Tunable parameters, no finer GT or AA. Negative α = 0.75 for all stages, λ = 2, 2, 2: 0.347 / 0.325 / 0.336. Negative α = 1 for all stages, λ = 2, 1, 0: 0.366 / 0.282 / 0.324. Negative α = 0.75, 0.5, 0.25 with λ = 2, 1, 0: 0.353 / 0.287 / 0.320. (See Pitfalls: the column is labelled λ but holds the γ values.)
• tab. C Offset vs proximity regression, unification only. Offset: 0.429 / 0.336 / 0.383. Proximity: 0.372 / 0.282 / 0.327.

Figures are described, not reproduced here. fig. 1 shows the three representations at one pixel. fig. 2 illustrates the unification. fig. 3 shows hardness of four sample types for GFL and UFL. fig. 4 is the framework. fig. 5 to fig. 7 are qualitative DTU and Tanks and Temples point clouds. fig. A is the distribution of scaling factors. fig. B shows the scaling value under different bases. fig. C shows the two unity-generation options.

Paper vs code

Code: github.com/prstrive/UniMVSNet at commit 7c1c9862cc6f2eb0b74c0569d58826d9416195b5 (MIT, Copyright 2022 Rui Peng). The paper links this repository §1. Files compared: loss.py, networks/mvsnet.py, networks/module.py, main.py, tools.py, datasets/dtu_yao.py and scripts/*.sh.

Matches:

• Unity regression alg. 2 and sigmoid-per-hypothesis are implemented as described mvsnet.py module.py.
• Stage-wise γ = 2, 1, 0 and α− = 0.75, 0.5, 0.25 from coarse to fine loss.py §4.
• Scale functions with base 5 and the [1, 3) and [0, 1) ranges loss.py app. A eq. b, c.
• DTU hypothesis counts 48, 32, 8 with interval ratios 4, 2, 1 dtu_train.sh app. C.
• Tanks and Temples: 64, 32, 8 hypotheses and 11 views tank_test.sh §4.

Differences and gaps:

1. Loss averaging. The paper says the stage loss is the average of UFL over valid pixels §3.4 eq. 10. The code expands the mask over all depth hypotheses and divides the sum by the number of valid pixel-hypothesis entries loss.py. The mean is therefore over pixels and hypotheses. The paper does not say whether its per-pixel UFL is summed over hypotheses. If it is, the code's value is smaller by the hypothesis count M at each stage, and that changes the effective stage weights.
2. Normaliser q+ for pixels without a positive target. The paper defines q+ in (0, 1] §3.3 but does not say what to use when no hypothesis contains the ground truth. The code sets q+ = 1 there loss.py.
3. Adaptive aggregation. Paper eq. 3 weights each term by W. The code weights each term by (W + 1) mvsnet.py, and W comes from two 1×1×1 convolutions with no activation module.py. The paper does not mention the +1 or the absence of a bound. The normalisation by N − 1 matches mvsnet.py.
4. Stage loss weights. The paper gives no numbers §3.4. The code uses 0.5, 1.0, 2.0 by default and in both training scripts main.py dtu_train.sh blendedmvs_finetune.sh.
5. Learning-rate warm-up. Not in the paper. The code ramps the rate linearly over 0.2 epochs from 1/3 of the target value tools.py. The schedule is applied per batch with fractional epochs model.py.
6. Interval scale. The paper says it follows CasMVSNet for the depth interval §4. The code multiplies the interval read from each camera file by 1.06 dtu_yao.py.
7. Confidence used for filtering. The paper filters on a probability threshold of 0.3 but does not say which quantity. For unification the code uses the maximum softmax over the logits, not the sigmoid unity values, as the confidence mvsnet.py.
8. Finer DTU ground truth. The paper uses a union of masks from CasMVSNet and MVSNet with an up-sampled mask from AA-RMVSNet, and processed results from that work app. B. A search of the repository for fgt, finer and related terms found no code for this. The DTU loader reads the standard masks from Depths_raw dtu_yao.py. Results with FGT cannot be reproduced from this repository alone.
9. Disparity threshold. The DTU fusion script sets a disparity threshold of 0.25, which is not in the paper dtu_test.sh.
10. Batch size and hardware. Not in the paper. The scripts use 2 GPUs with sync BN and a batch size of 2 per process for DTU, and batch size 1 for BlendedMVS dtu_train.sh blendedmvs_finetune.sh.

No discrepancies were found in the test-time hypothesis counts, the DTU evaluation resolution (1152 × 864) or the stated probability threshold of 0.3 with 3 consistent views dtu_test.sh §4.1.

Pitfalls

• Image size order. The paper writes 640 × 512 for DTU training §4. The script passes 512 640 dtu_train.sh. The dataset asserts both values are multiples of 32 dtu_yao.py and takes its default as [512, 640]. Confirm which order your loader expects before copying either number.
• Interval for the last hypothesis. alg. 1 and alg. 2 use the previous spacing for the last hypothesis. The code uses one constant interval per stage, ndepth × Δ / (ndepth − 1) module.py module.py. For uniform hypotheses these agree. For non-uniform sampling the paper gives no rule.
• Half-open interval. The target uses [d_i, d_i + r). A ground-truth depth exactly on a boundary falls into the next slice. Floating-point rounding can move borderline pixels.
• Ground truth outside all intervals. The target is all zeros for that pixel §4.3. Regression still returns some depth, so the offset value is not meaningful there. The paper says this supervision is correct, and its ablation shows unification is more robust in this case §4.3.
• Unbounded scaling factor. |q − u| / q+ has range [0, ∞) §3.3. Appendix A reports that a few large factors can dominate training and make convergence difficult, which is why S_b bounds the factor app. A. Most factors are below 4 app. C fig. A.
• Hyperparameter search is limited. The paper says it tried only a few base numbers and parameter sets, and that other settings may be better app. C.
• Table label. The column headed λ in tab. B holds the γ schedule (2, 1, 0), not stage loss weights. Do not read those values as λ_i.
• Sigmoid outputs are not normalised across hypotheses. Several hypotheses can have high unity. Decoding uses the single largest value. The confidence used for filtering is different (see Paper vs code, item 7).
• Cost claims. The conclusion says the two modules add no memory or computation cost §5. This is about the head and the loss. Adaptive aggregation adds a small weight network per source view module.py. The paper calls its parameter increase negligible §3.4.
• Ablation setting. All ablation models use confidence 0.3 and 3 consistent views tab. 3. Changing the filter changes the ranking tab. 4 app. C.
• Training cost. Hardware, training time and memory are not stated in the paper.

Lineage

• Builds on: MVSNet (the cost-volume pipeline and the DTU split) §3.1 §4; CasMVSNet (the coarse-to-fine backbone and depth configuration) §3.4 §4; focal loss §3.3; GFL, also named QFL and VFL in the paper, as the continuous-target form that UFL extends §3.3.
• Uses adaptive aggregation from the method cited as Pyramid MVS §3.4 §4.3.
• Compared against on DTU and Tanks and Temples: Furu, Gipuma, COLMAP, SurfaceNet, MVSNet, P-MVSNet, R-MVSNet, Point-MVSNet, AA-RMVSNet, CasMVSNet, CVP-MVSNet, UCS-Net, PatchmatchNet, ACMP, D²HC-RMVSNet, VisMVSNet and EPP-MVSNet tab. 1 tab. 2.
• Superseded by: not stated.