AA-RMVSNet: Adaptive Aggregation Recurrent Multi-view Stereo Network

In plain words

Multi-view stereo rebuilds a 3D scene from several photos taken from different places. A common approach guesses how far away each pixel is by testing many candidate distances and keeping the one where the photos agree best. AA-RMVSNet improves two parts of that pipeline.

First, it adapts the image features to the local texture. Flat, featureless areas get a wider view of their surroundings, while detailed areas keep a narrow one, so thin objects and smooth surfaces are handled better. Second, when it combines the evidence from several photos, it gives more weight to the photos that really see a given pixel, which helps where one object hides part of another.

To keep memory use low, it sweeps through the candidate distances one layer at a time with a recurrent network, instead of using a large 3D network over the whole volume.

TL;DR

• A learning-based depth-map MVS network, trained end to end on DTU. It predicts a probability over discrete depth hypotheses and is trained with cross-entropy §3.4.
• Two adaptive modules: intra-view aggregation (deformable convolutions at three scales) and inter-view aggregation (a pixel-wise weight for each source view) §3.1 §3.2.
• A hybrid CNN and ConvLSTM regulariser sweeps the depth axis recurrently, which the authors say allows more depth hypotheses at test time §3.3 §4.2.
• Author-reported: DTU overall error 0.357 mm, close to the best listed method (CVP-MVSNet, 0.351 mm) tab. 1. First place on the Tanks and Temples leaderboard as of Mar. 15, 2021 tab. 2.
• The official code is MIT-licensed and differs from the paper in several training and fusion settings. See "Paper vs code".

Problem

Learning-based MVS with fixed-grid 2-D features and fixed receptive fields struggles on thin structures and low-texture surfaces §1 §3.1. Averaging the per-view cost volumes gives every source view equal weight, which fails under occlusion and non-Lambertian lighting §3.2. Regularising the cost volume with full 3-D CNNs is memory-hungry, which limits resolution and the number of depth planes §1 §2.2.

Method

Overview §3. The input is one reference image and N−1 source images; N = 7 in the experiments §4.2. A weight-shared encoder with intra-view aggregation produces 32-channel features for every image §3.1 app. A. Each source feature is warped to the reference camera for each depth hypothesis with a differentiable homography eq. 1. Per-view cost volumes are squared feature differences eq. 2. Inter-view aggregation combines them into one cost volume eq. 4. A recurrent regulariser turns that volume into a per-pixel distribution over the D depth hypotheses §3.3 eq. 5–eq. 7. Training uses cross-entropy against the one-hot ground truth on valid pixels §3.4 eq. 8.

1. Intra-view aggregation §3.1 eq. 3 fig. 3 app. A. The encoder computes features at full, half and quarter resolution. Three deformable convolutions, one per scale, have unshared weights. Their outputs have 16, 8 and 8 channels after bilinear upsampling to full resolution, and are concatenated into a 32-channel map. The deformable convolution (eq. 3) samples the input at learned offsets Δp_k and weights each sample by a learned modulation Δm_k. The paper describes this as adaptive: the sampling points spread out in low-texture areas and stay on the surface at object boundaries app. B fig. 9.

2. Per-view cost volumes §3 eq. 1 eq. 2. For each depth hypothesis d, the homography H_i^(d) maps the reference pixel grid into source view i. The per-view cost is the element-wise squared difference between the warped source feature and the reference feature, c_i^(d) = (f_src_i^(d) − f_ref)², with 32 channels.

3. Inter-view aggregation §3.2 eq. 4 fig. 4 app. A fig. 10. A small network produces a pixel-wise weight ω_i in (0, 1) from each per-view cost (32 channels in, one channel out, with a sigmoid output). The aggregated cost is the average over the N−1 source views of (1 + ω_i) ⊙ c_i. Using 1 + ω rather than ω alone keeps pixels from being suppressed too strongly, which the authors say prevents over-smoothing §3.2. The weight network is a 3×3 conv (32 to 4 channels) with GroupNorm and ReLU, a residual block of two 1×1 convs on 4 channels, a 1×1 conv to one channel, and a sigmoid app. A.

4. Recurrent cost regularisation §3.3 eq. 5 eq. 6 eq. 7 fig. 2 app. A. The cost volume is sliced along D and the slices are processed in order. Each slice goes through a convolutional encoder-decoder, and five ConvLSTM cells pass their hidden state from one slice to the next. Their hidden channels are 16, 16, 16, 16 and 8 app. A. A final 3×3 conv gives one value per pixel and depth hypothesis, which is turned into a probability over d.

5. Loss §3.4 eq. 8. The ground-truth depth is one-hot encoded on the nearest hypothesis. The loss is the cross-entropy between that target and the predicted distribution, summed over valid pixels only. The paper defines valid pixels as those with reliable depth §3.4.

6. Filtering and fusion at test time §4.2. A pixel is kept when its photometric confidence (the predicted probability of the chosen depth) exceeds a threshold. A geometric consistency check follows the dynamic method of D2HC-RMVSNet §2.2. Depth maps are fused with a visibility-based method and mean averaging §4.2. The test-time depth search is a running maximum over d, so the full probability volume is not stored drmvsnet.py.

Pseudocode for the training forward pass (the paper's version; the code differs in a few points listed under "Paper vs code"):

# Training forward pass  [§3] [eq. 1-2] [eq. 4] [eq. 5-7]
# Input:  images (N, 3, H, W), N = 7, reference image first
# Input:  depth_hypotheses (D,), swept in this order
# Output: probability volume (D, H, W)

def aa_rmvsnet_forward(images, cameras, depth_hypotheses):
    # Extract adaptive features for every image  [§3.1] [eq. 3]
    features = [intra_view_features(image) for image in images]
    ref_feature = features[0]                    # (H, W, 32)
    src_features = features[1:]                  # N - 1 maps, (H, W, 32)

    # Start the recurrent regulariser with zero hidden states  [§3.3]
    hidden_state = zero_states()
    slices = []
    for depth in depth_hypotheses:
        # Squared feature differences after warping to this depth  [eq. 1-2]
        cost_views = [
            (warp(src, cameras, depth) - ref_feature) ** 2
            for src in src_features
        ]
        # Pixel-wise view weighting and averaging  [§3.2] [eq. 4]
        cost_slice = inter_view_aggregate(cost_views)     # (H, W, 32)
        # One recurrent regularisation step on this depth slice  [§3.3]
        output, hidden_state = regularise_slice(cost_slice, hidden_state)
        slices.append(output)                             # (H, W, 1)

    # Normalise over depth to get a probability per pixel  [§3.3] (paper silent)
    # Code applies a softmax over the depth axis [code: models/drmvsnet.py#L268-L269]
    return softmax(stack(slices), axis=depth_axis)

# Intra-view adaptive aggregation  [§3.1] [eq. 3] [fig. 3] [app. A]
# Input:  image (3, H, W); output: features (32, H, W)

def intra_view_features(image):
    # Five 3x3 conv + GroupNorm + ReLU layers, two with stride 2  [app. A]
    x0 = conv_gn_relu(image, out_channels=8)           # (8, H, W)
    x1 = conv_gn_relu(x0, out_channels=16)             # (16, H, W)
    x2 = conv_gn_relu(x1, out_channels=16)             # (16, H, W)
    x3 = conv_gn_relu(x2, out_channels=16, stride=2)   # (16, H/2, W/2)
    x4 = conv_gn_relu(x3, out_channels=16, stride=2)   # (16, H/4, W/4)
    # One modulated deformable conv per scale, unshared weights  [§3.1] [eq. 3]
    y2 = deform_conv(x2, out_channels=16)              # (16, H, W)
    y3 = upsample_bilinear(deform_conv(x3, out_channels=8), H, W)  # (8, H, W)
    y4 = upsample_bilinear(deform_conv(x4, out_channels=8), H, W)  # (8, H, W)
    # Concatenate into the 32-channel feature map  [§3.1] [fig. 3]
    return concat([y2, y3, y4])                        # (32, H, W)

# Inter-view adaptive aggregation at one depth  [§3.2] [eq. 4] [app. A]
# Input:  cost_views, list of N-1 arrays (H, W, 32)
# Output: aggregated cost (H, W, 32)

def inter_view_aggregate(cost_views):
    total = zeros_like(cost_views[0])                  # (H, W, 32)
    for cost in cost_views:
        # Pixel-wise weight in (0, 1) from a small CNN  [app. A]
        weight = attention_net(cost)                   # (H, W, 1)
        # Factor (1 + weight) lies in (1, 2) because weight is a sigmoid
        total = total + (1 + weight) * cost           # (H, W, 32)
    return total / len(cost_views)                     # (H, W, 32)

# One ConvLSTM step on a depth slice  [§3.3] [eq. 5-7]
# Input:  x (H, W, C); previous hidden h_prev and memory c_prev

def conv_lstm_step(x, h_prev, c_prev):
    # One 3x3 conv over the concatenated input and previous output  [eq. 5]
    gates = conv3x3(concat([x, h_prev]))               # (H, W, 4 * hidden)
    # Split into four signals, in the order used by the paper  [eq. 5]
    input_pre, forget_pre, output_pre, cand_pre = split(gates, 4)
    i = sigmoid(input_pre)
    f = sigmoid(forget_pre)
    o = sigmoid(output_pre)
    g = tanh(cand_pre)
    # Memory update  [eq. 6]
    c = f * c_prev + i * g
    # Cell output passed to the next layer and the next depth slice  [eq. 7]
    h = o * tanh(c)
    return h, c

# Cross-entropy depth loss over valid pixels  [§3.4] [eq. 8]
# Input:  prob_volume (D, H, W), depth_gt (H, W), valid (H, W) in {0, 1}

def depth_loss(prob_volume, depth_gt, valid):
    # One-hot target: the hypothesis nearest to the ground-truth depth  [§3.4]
    gt_index = argmin_over_d(abs(depth_hypotheses - depth_gt))   # (H, W)
    # Negative log-probability of the target at every pixel  [eq. 8]
    nll = -log(gather(prob_volume, gt_index))                    # (H, W)
    # Sum over valid pixels only  [eq. 8]; see Paper vs code for normalisation
    return sum(nll * valid)

Hyperparameters

Results

Author-reported. All numbers come from the paper. The code has not been rerun.

DTU evaluation set. Setup: 22 evaluation scans, accuracy and completeness in mm from the official MATLAB tool, lower is better, trained on the 79 training scans §4.1 §4.3. The row for AA-RMVSNet is the authors' result.

tab. 1. The table lists PointMVSNet twice with different numbers. The paper does not say which row is which. The text says the method has the best completeness and competitive overall error §4.3.

Tanks and Temples, intermediate set, F-score, higher is better. Rank is the average rank over the eight scenes and is the basis for the leaderboard position. Ranking as of Mar. 15, 2021 tab. 2 §4.3.

Ablation on DTU, with memory during inference tab. 3 §4.4. Setup: same settings as §4.2.

Effect of the number of views, depth hypotheses and resolution, on DTU tab. 5 app. E. Setup: as in the ablation, with the stated N, D and image size.

Other results. On BlendedMVS the authors report only qualitative results after fine-tuning §4.1 §4.3 app. F. Error maps on Tanks and Temples are shown in fig. 7 and training curves in fig. 8.

Paper vs code

Official repository: QT-Zhu/AA-RMVSNet, MIT licence, owner is one of the authors (Qingtian Zhu), commit 753206e22fc281a1bddeb3d34b5ed472dd960f2a. The README records a May 2022 update to the "network architecture & pretrained model" and says the new checkpoints are not compatible with the old codebase README.md README.md. The paper is the August 2021 arXiv version, so the code may describe a later architecture than the paper.

• Photometric threshold. The paper discards depths with probability below 0.3 §4.2. Code uses 0.35 for DTU fusion.py and 0.2 for Tanks and Temples fusion.py.
• Learning-rate schedule. The paper decays the learning rate by 0.9 each epoch §4.2. Code uses cosine annealing to 2e-6 over the number of epochs train.py, stepped at the start of each epoch train.py.
• Epochs. The paper does not state the number of epochs. The training script sets 10 train_dtu.sh.
• Batch size. The paper says batch size 4 on 4 GPUs §4.2. The training script sets batch size 2 on two GPUs. DataParallel splits that batch, so the total is 2 train_dtu.sh train_dtu.sh train.py.
• Number of training samples. The paper's count of 27,097 equals 79 scans × 49 references × 7 lightings §4.1. The loader adds a copy of each sample with the depth hypotheses in reverse order. both defaults to True, so the set is about twice as large, roughly 54,000 samples with all seven lightings dtu_yao.py dtu_yao.py train.py.
• Loss normalisation. Equation 8 is a sum over valid pixels eq. 8. Code divides each sample's sum by its count of valid pixels, then averages over the batch drmvsnet.py. The code comment says the original used a sum.
• Valid pixels. The paper's valid pixels are those with reliable depth §3.4. The loader's mask is ground-truth depth inside [depth_min, depth_end] dtu_yao.py. It does not test reliability.
• Regulariser input sign. The paper feeds the aggregated cost volume straight into the regulariser §3.2 §3.3. Code feeds the negative of the aggregated cost drmvsnet.py.
• Encoder channels. The paper gives encoder channels 8, 16, 16, 16, 16 app. A and in the caption of fig. 3. The code uses 8, 16, 32, 32, 32, so the deformable convolutions receive 32-channel inputs, not 16 drmvsnet.py drmvsnet.py. The output channels (16, 8, 8) agree.
• Depth range. The paper states 425 to 935 mm with D = 192 for training §4.2. The code sets the interval as the camera file's base interval times 1.06 dtu_yao.py and uses depth_min from the file dtu_yao.py. The camera files are not in the repository. The script comment says d × interval_scale = 203.52 train_dtu.sh. The metric range therefore cannot be checked from this repository.
• Ground-truth depth. The paper describes screened Poisson reconstruction, depth rendering and cross-filtering to build ground truth §4.2. The repository has no such step. It uses the preprocessed MVSNet data and points to the MVSNet repository for it README.md.
• Geometric filtering. The paper names the dynamic consistency check of D²HC-RMVSNet but gives no numbers §4.2. Code checks 1 px and 1% first, then uses reprojection distances below i/4 px and relative depth differences below i/1300 for i = 2..10 fusion.py.
• Image normalisation. The loader normalises each image to zero mean and unit variance per channel dtu_yao.py. The paper does not state this.
• Validation. The training loop evaluates on the DTU test list after every epoch train.py. It saves a checkpoint each epoch and does not select the best one train.py. The paper does not say how the final model was chosen.

Checked and matching: the per-view squared difference eq. 2 (models/drmvsnet.py#L255-L262), the inter-view weighting and averaging eq. 4 (models/drmvsnet.py#L257-L264), and the ConvLSTM gates and updates eq. 5–7 (models/module.py#L84-L90). Not compared line by line: the homography eq. 1, the deformable-convolution sampling eq. 3, and the Tanks and Temples and DTU data preparation.

Pitfalls

• Text versus table. The ablation text says intra-view aggregation improves completeness by 0.28 and inter-view aggregation adds 0.31 accuracy §4.4. The table gives 0.374 → 0.346 (a difference of 0.028) and 0.408 → 0.377 (0.031) tab. 3. The memory figures in the text (1.74 GB and 0.11 GB) match the table. Use the table values.
• Ranking claim. The Tanks and Temples first place is dated Mar. 15, 2021 tab. 2. This card did not check the current leaderboard.
• Deformable convolution. The code's intent is to scale the learning rate of the offset and modulation convolutions by 0.1, but the backward hook returns nothing, so it has no effect module.py module.py module.py. Offsets and modulation are zero-initialised module.py. The paper says nothing about either.
• Unstated implementation details. The code's ConvLSTM hidden states start at zero module.py. The warp uses bilinear sampling with zero padding module.py. A small offset (0.0001) is added to zero depths to avoid division by zero module.py. The paper states none of these.
• Softmax over depth. The paper does not say how the probability volume is normalised. Code applies softmax over the depth axis drmvsnet.py.
• Test-time memory. The test path keeps only a running maximum and a running sum of exponentials over depth, and reports the maximum divided by the sum as photometric confidence drmvsnet.py. The full volume is never stored, which is why the paper can use 512 hypotheses at test time.
• Dead branch. The regulariser branch that runs only when process_sq is False calls nn.Tanh(cost), which would raise a TypeError. The default path does not reach it drmvsnet.py.
• Tanks and Temples resolution. The paper gives 1024 × 544 or 960 × 544 without saying which scene uses which §4.2. The code uses 1024 for Lighthouse, M60 and Panther, and 960 for Family, Francis, Horse, Playground and Train eval_tnt.sh eval_tnt.sh tp_list_int_1024.txt tp_list_int_960.txt.
• Memory and time. Training needs 20.16 GB and takes about 3 days §4.2, on four Titan RTX GPUs. The paper does not say how many epochs that covers.
• Duplicate row. PointMVSNet appears twice in the DTU table with different numbers tab. 1. Which result is which is not stated.
• Table and figure numbers. The table, figure and equation numbers on this card follow the order in the LaTeX source. The compiled PDF was not checked.

Lineage

• Builds on: recurrent MVS networks R-MVSNet (a GRU that sweeps the depth axis) and D²HC-RMVSNet (ConvLSTM cells and a dynamic consistency check) §2.2 §4.2. Those papers are not on this wiki, so nothing from them is used here.
• Compared against: MVSNet, CasMVSNet, CVP-MVSNet, Vis-MVSNet, PVA-MVSNet and others in tab. 1 and tab. 2.
• Superseded by: not stated in the wiki.