NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

In plain words

NeuralRecon builds a 3D model of a room from an ordinary video taken with a camera that moves through it. The camera's path is assumed to be known already, from a separate tracking system. The video is cut into short groups of consecutive key frames, called fragments.

Older real-time methods work frame by frame. They guess a depth map for each frame and then merge those guesses. Because each guess is made alone, the scale can drift between frames and the same wall gets estimated several times. NeuralRecon instead predicts the 3D surface for a whole fragment at once, as a grid of small cubes that store the distance to the surface. A learned memory then carries what was seen in earlier fragments into the next one, so the pieces fit together.

The point of the design is speed without giving up coherence. The authors report that this runs at real-time rates and is more accurate than the depth-merging methods they compare with.

TL;DR

NeuralRecon is a real-time monocular reconstruction network that predicts a sparse signed-distance (TSDF) volume per fragment of 9 key frames, rather than per-frame depth maps. Image features are back-projected into a 3D grid, decoded coarse-to-fine with sparse 3D convolutions, and fused with earlier fragments by a 3D ConvGRU. Author-reported: 30 ms per key frame (33 key frames per second) on an RTX 2080 Ti, and F-score 0.562 on ScanNet (Atlas split, single-layer mesh) tab. 1. The official code is Apache-2.0, and its training configs differ from the paper in several details (see Paper vs code).

Problem

Real-time monocular reconstruction usually estimates a depth map on each key frame and fuses the maps into a TSDF volume afterwards §1. The authors identify two drawbacks. First, each key-frame depth is estimated from scratch, so scale can change between views and the surface can come out layered or scattered §1. Second, the same surface is estimated repeatedly in overlapping key frames, which wastes computation §1. The offline volumetric method Atlas is the main accuracy comparison, and the authors' method is about 10× faster than it §1 §4.2. The method assumes camera poses come from an external SLAM system §3.

Method

Overview. Key frames are chosen from the incoming stream. Each window of N key frames is one local fragment, and the network predicts the TSDF of that fragment directly in a 3D volume §3.1 §3.2. The fragment is then merged into a global volume and meshed with Marching Cubes at every time step §3.2.

Key-frame selection and fragments §3.1. A frame becomes a key frame when its relative translation and relative rotation both exceed thresholds. A window of N key frames forms a fragment. A cubic fragment bounding volume (FBV) encloses the view frusta of its key frames, with each frustum cut at depth d_max §3.1. Only voxels in the FBV are handled during reconstruction §3.1.

Feature volume §3.2. Each key frame goes through an image backbone that yields multi-level features. These are back-projected along rays into the fragment volume. Each voxel gets the average of the features from the views that see it, weighted by visibility §3.2. The unprojection is illustrated in fig. 3.

Coarse-to-fine decoding §3.2. The network works in 3 levels §3.2 §3.3. At each level it applies sparse 3D convolutions to the feature volume. Two MLP heads predict an SDF value and an occupancy score per voxel §3.2. Voxels whose occupancy is below θ = 0.5 are treated as empty and removed §3.2 §3.3. The survivors are upsampled by 2x with nearest-neighbour interpolation and concatenated with the next level's features §3.2 §3.3.

GRU fusion §3.2 GRU Fusion. The fragment's geometric features G_t are combined with a hidden state H_{t-1} that is read from the global hidden state inside the FBV. A ConvGRU, written with sparse convolutions, produces the updated hidden state H_t. The four GRU equations in this section are unnumbered in the paper. The updated hidden state is also written back to the global hidden state. The paper calls this a learned replacement for running-average TSDF fusion §3.2.

Integration §3.2. The last level's sparse TSDF is transformed to global coordinates and replaces the corresponding global voxels, because the fusion already happened inside the GRU §3.2. Marching Cubes is then run on the global volume at each time step §3.2.

Supervision §3.2 Supervision. Occupancy is trained with binary cross-entropy against ground truth. SDF is trained with an L1 loss on log-transformed values. Both losses are applied at every coarse-to-fine level §3.2.

# Fragment reconstruction, one window of N key frames  [§3] [§3.2]
# Input:  images (N, 3, H, W); poses (N, 4, 4); global hidden state per level
# Output: sparse TSDF of the fragment, written into the global volume

def reconstruct_fragment(images, poses, global_state):
    # Extract 2D feature maps for every key frame  [§3.3]
    feats = [backbone(image) for image in images]
    # Back-project features, average over the visible views  [§3.2]
    volume, visible = unproject_average(feats, poses)  # (V, C), (V,)
    # Start from the coarsest grid over the fragment bounding volume  [§3.2]
    coords = dense_grid(fragment_dims, interval=4)  # 16 cm (inferred)
    feat = gather(volume, coords)
    for level in [0, 1, 2]:
        # Sparse 3D convolution on the unprojected features  [§3.2]
        geo = sparse_conv_level(feat, coords, level)
        # Fuse with the hidden state from earlier fragments  [§3.2 GRU]
        hidden = gru_fuse(geo, global_state[level], coords)
        # Predict a truncated SDF and an occupancy logit per voxel  [§3.2]
        sdf = mlp_sdf(hidden)
        occ = mlp_occ(hidden)
        # Drop voxels with occupancy <= 0.5, i.e. logit <= 0  [§3.3]
        coords, feat = sparsify(coords, hidden, sdf, occ)
        # Upsample 2x with nearest neighbour, except after the last level
        coords, feat = upsample(coords, feat)
    # Replace the matching voxels of the global TSDF  [§3.2]
    write_global(global_state, coords, sdf)
    # Mesh the global volume with Marching Cubes  [§3.2]
    return marching_cubes(global_state)

# Key-frame test for one incoming frame  [§3.1] [§3.3]
# Input:  pose of the incoming frame; pose of the last key frame

def is_keyframe(pose, last_pose):
    # Translation larger than t_max = 0.1 m  [§3.3]
    moved = distance(pose.center, last_pose.center) > 0.1
    # Rotation angle larger than R_max = 15 degrees  [§3.3]
    turned = rotation_angle(pose, last_pose) > 15.0
    # Paper wording: both conditions  [§3.1]; code uses "or" (see Paper vs code)
    return moved and turned

# Supervision at one level  [§3.2 Supervision]
# Input:  predicted sdf, occupancy logit; ground-truth sdf, occupancy (0/1)

def level_loss(sdf, occ_logit, sdf_gt, occ_gt):
    # Balance free and occupied voxels  [code: models/neucon_network.py#L244]
    # POS_WEIGHT = 1.5 in train.yaml
    pos_weight = n_free / n_occupied * POS_WEIGHT
    # Occupancy: binary cross-entropy on logits  [§3.2]
    occ_loss = bce_with_logits(occ_logit, occ_gt, pos_weight)
    # SDF: L1 after signed log transform, on occupied voxels  [§3.2]
    occupied = occ_gt == 1
    sdf_loss = l1(signed_log(sdf[occupied]), signed_log(sdf_gt[occupied]))
    return occ_loss + sdf_loss

# Total training loss  [§3.2 Supervision] [code: config/train.yaml#L38]
# Level weights LW = [1.0, 0.8, 0.64]; the coarsest level has weight 1.0

def total_loss(levels):
    # Weighted sum of the per-level losses (not stated in the paper)
    return sum(LW[i] * level_loss(*levels[i]) for i in range(3))

Sparse convolutions are implemented with torchsparse §3.3. The equations for the hidden state are unnumbered; cite them as §3.2 GRU Fusion.

Hyperparameters

Values marked "paper" come from the text or tables. Values marked "code" are from the official repository at code_commit; the paper does not state them.

Results

Author-reported. Setup: ScanNet (V2) with two train/validation splits, one from Atlas (top block) and one from BA-Net (bottom block) §4.1 tab. 1 tab. 2. The 7-Scenes results use the model trained on ScanNet without further training §4.1 tab. 3. Depth-based baselines are fused with point-cloud or TSDF fusion, and the Atlas comparison uses double-layer meshes §4.1. Timing is per key frame on an RTX 2080 Ti §4.2.

3D geometry on ScanNet, Atlas split (top) and BA-Net split (bottom) tab. 1

2D depth on ScanNet, Atlas split tab. 2

2D depth on ScanNet, BA-Net split tab. 2

7-Scenes, 3D tab. 3, top

7-Scenes, 2D tab. 3, bottom

Timing per key frame, averaged over fragments tab. 4

Ablation on ScanNet, 3D metrics tab. 5

Efficiency and ablation text: §4.2 §4.3. The paper reports about 10× speed-up over Atlas and about 77× over Consistent Depth §4.2. The 2D and 3D metric definitions are in a supplementary document that is not in the source bundle we read §4.1.

Paper vs code

Code checked: official repository zju3dv/NeuralRecon at code_commit, Apache-2.0. The abstract links the project page at zju3dv.github.io/neuralrecon [abstract], and that page links the repository. The paper text itself does not name a code URL.

• Key-frame rule. Paper: a frame is a key frame when translation and rotation exceed the thresholds §3.1. Code: a frame is a key frame when the translation or the angle exceeds them generate_gt.py. The angle is computed from the optical axes generate_gt.py.
• Fragment windows. The paper does not say whether fragments overlap. Code builds consecutive non-overlapping windows of window_size key frames generate_gt.py.
• GRU operator. Paper: the four equations use only sparse convolutions §3.2 GRU Fusion. Code: each sparse convolution also adds a pointwise linear term of its input modules.py modules.py.
• Feature volume. Code concatenates one extra channel, the normalised mean depth of the voxel over its views, to the averaged image features back_project.py neucon_network.py. The paper does not mention it.
• Feature averaging. Paper: average weighted by visibility §3.2. Code: plain mean over the views where the voxel projects inside the image back_project.py. Voxels seen by at most one view are masked out of the loss and of the occupancy output unless FUSION.FULL is set neucon_network.py neucon_network.py.
• Loss. Paper: BCE for occupancy, L1 on log-transformed SDF, at all levels §3.2 Supervision. Code: BCE with a pos_weight of (free/occupied) × POS_WEIGHT neucon_network.py; L1 only on occupied voxels neucon_network.py; level weights 1.0, 0.8, 0.64 neuralrecon.py.
• Truncation. Paper: λ = 12 cm §3.3. Code: margin 3 voxels at each level, so 12 cm at the finest level and 24 and 48 cm at coarser levels generate_gt.py fusion.py.
• Augmentation. Code applies a random rotation about z and a random translation of the cropped volume during training main.py transforms.py. Not in the paper.
• Training schedule. The README describes two phases. Phase 1 (epochs 0–20) trains single fragments without GRU fusion. Phase 2 (epochs 21–50) enables GRU fusion. The switch is made by hand README.md. The shipped train.yaml sets 991 epochs, and FUSION_ON and FULL are True in it train.yaml train.yaml. The paper describes neither schedule.
• Fusion region. Paper: fusion inside the FBV, called FBV in the ablation tab. 5. Code: FULL: True keeps all valid FBV voxels as the updated set gru_fusion.py. Mapping FULL to the paper's FBV is our inference from the code.
• Config defaults. config/default.py uses N_VOX [128, 224, 192], N_VIEWS 5, POS_WEIGHT 1.0 and LR milestones 12, 24, 36 default.py default.py default.py default.py. The shipped train.yaml uses [96, 96, 96], 9 views, 1.5 and 12, 24, 48 train.yaml train.yaml train.yaml train.yaml.
• Dates. The README dates the code and pretrained models to 2021-05-02 README.md. The arXiv record is v1 of 2021-04-01 [raw/arxiv]. The code therefore post-dates this version, but no commit history was checked against the paper.

The values stated in the text (voxel size, λ, θ, d_max, t_max, R_max) match the shipped code, with the caveats above.

Pitfalls

• Equations are unnumbered. The GRU update is written in an unnumbered align* environment §3.2 GRU Fusion. Cite it as a section, not as an equation number.
• Coarse levels are underspecified. The paper gives only the finest voxel size (4 cm) and that there are 3 levels §3.2 §3.3. The coarse voxel sizes (16 and 8 cm) and the channel widths come from the code neucon_network.py modules.py.
• Sampling caps change the loss. During training, occupied voxels beyond 4096, 16384 and 65536 per batch item are randomly dropped at the three levels neucon_network.py. This changes which voxels get loss and is not in the paper.
• Hidden-state size is unused. FUSION.HIDDEN_DIM is defined but never read in the Python code. Hidden channel counts come from the per-level feature widths default.py neucon_network.py.
• Default config fails to build. The default BACKBONE2D.ARC is fpn-mnas. The model parses the part after the last hyphen as a float, which raises on mnas. Use fpn-mnas-1 from train.yaml default.py neuralrecon.py.
• Occupancy threshold on logits. Code thresholds the raw logit at 0, not the probability neucon_network.py. This is equivalent to θ = 0.5 on the sigmoid output, but a reimplementation that thresholds the probability must do the same.
• Dataset preparation is required. Training reads precomputed fragment pickles and TSDF files produced by generate_gt.py. Its defaults (window 9, margin 3 voxels at 4 cm, min angle 15°, min distance 0.1 m) match the values stated in the paper generate_gt.py.
• Timing scope. The 30 ms figure is the fragment reconstruction time divided by the number of key frames. Depth-based methods are timed without their TSDF fusion §4.2.
• Single- vs double-layer meshes. The Atlas comparison uses double-layer meshes §4.1. On the Atlas split, this method's double-layer F-score (0.516) is lower than its single-layer F-score (0.562) tab. 1.
• Images. Frames are resized to 640 × 480 and ScanNet images are padded vertically by 4 pixels main.py transforms.py. The paper states neither.
• Model weights. Pretrained weights are linked from the README and are not hosted in the repository README.md. This card does not use them.

Lineage

• Builds on: GRU as the recurrent cell §3.2 GRU Fusion; the volumetric feature back-projection of prior learned reconstruction work §3.2; sparse 3D convolution through torchsparse §3.3; key-frame and fragment selection following GPMVS §3.1.
• Replaces: the running-average linear TSDF fusion of KinectFusion, which the GRU takes the place of §3.2 §4.3.
• Compared against: real-time multi-view depth methods (MVDepthNet, GPMVS, CNMNet, N-RGBD), multi-view stereo methods (COLMAP, DPSNet, MVSNet, Atlas, Consistent Depth), and learning-based SLAM methods (DeMoN, BA-Net, DeepV2D) §4.1 §4.2 tab. 1 tab. 3.
• Superseded by: not stated in the sources of this card.