Dynamic neural radiance field

Overview

A dynamic neural radiance field is a NeRF that can render content that moves. Three designs appear on this wiki. The first adds time as an input to the field, so a 3D point and a moment give a colour and a density 2105.06468 §3.1 §3.3. The second splits the scene into a graph: a static background field plus one field per tracked object, each placed by tracking data 2011.10379 §3. The third also uses one field per tracked object inside a 3D box, and adds a semantic output to every field 2205.04334 §3.1. All three render new views with the NeRF compositing rule, where a pixel is a sum over ray samples of transmittance, opacity and colour 2105.06468 eq. 2 2011.10379 eq. 13 2205.04334 eq. 2. The NeRF concept page covers that rule in more detail.

The problems differ. For the time-input design, a monocular video shows each moment from one viewpoint only, so many fields fit the training frames, and most of them give artifacts from a new angle 2105.06468 §1 §3.1 fig. 4. For the graph design, a single network for a whole scene does not separate objects, so the objects cannot be edited on their own 2011.10379 §1. For the box design, the authors name three limits of earlier object-aware models: no panoptic output, learning from scratch for each scene, and one MLP shared across instances through per-object latent codes 2205.04334 §1 §2.

The three cards list this slug. Gao et al. describe the time-input design on monocular video 2105.06468. Ost et al. describe the graph design on driving scenes 2011.10379. Kundu et al. describe the box design, with semantics, on driving scenes 2205.04334. Their reported numbers come from different datasets and baselines, so they are not directly comparable; see Contradictions.

Variants

Time-conditioned static and dynamic fields (Gao et al.)

Two fields are trained together. The static field is a standard NeRF. Its training rays come only from pixels outside the moving-object mask 2105.06468 §3.2 eq. 1-4. The dynamic field is one MLP that takes a point and a time t, and outputs forward and backward 3D scene flow, density, colour and a blend weight 2105.06468 §3.3 eq. 5. The same MLP is queried at the flowed points and at times t+1 and t-1, so each 3D point gets up to three photometric constraints instead of one 2105.06468 §3.3 eq. 6-9.

The paper's regularisers do most of the work 2105.06468 §3.4:

• Motion matching: the projected scene flow must reproduce optical flow from RAFT between neighbouring frames 2105.06468 §3.4 fig. 5.
• Motion smoothness: small and temporally smooth flow, with forward and backward flow cancelling 2105.06468 eq. 10-11.
• Cycle consistency: following the flow forth and back returns to the start 2105.06468 eq. 12-13.
• Depth: a shift- and scale-invariant depth term against a monocular depth estimator, and a term tying dynamic depth to static depth 2105.06468 §3.4.
• Sparsity: low entropy of the rendering weights along each ray 2105.06468 §3.4.

The final image blends the two fields along each ray, so static points are meant to come from the static field and moving points from the dynamic one 2105.06468 §3.5 eq. 14. The paper's ablation on the Playground sequence reports that removing the depth-order term, the motion term or the rigidity term each lowers PSNR, from 23.65 for the full model to between 22.61 and 22.99 2105.06468 tab. 2. The depth-order term is the one the released code does not implement; see Contradictions.

Author-reported results use the Dynamic Scene dataset of Yoon et al. and score novel views from the first camera. On average over seven sequences the method reaches PSNR 24.74 and LPIPS 0.118. Li et al. reach 23.99 and 0.205 2105.06468 tab. 1. Li et al. are trained on 24 frames per sequence, this method on 12 2105.06468 supp. Comparisons with other methods. Card: Dynamic View Synthesis from Dynamic Monocular Video.

Scene graph of object fields (Ost et al.)

The scene is a directed acyclic graph. Its parts are a camera, a static background field, one dynamic field per object class, the latent codes and the edges. Edges carry either a rigid transform or an object scale, and an object's pose is read by walking from the world root to its node 2011.10379 §3 eq. 1-2.

The background is a NeRF-style field. The paper does not ray-march it through a volume. It samples a few planes parallel to the first camera's image plane, six in the paper 2011.10379 §4.1 §5.

Each object that moves rigidly during capture has its own field, defined in a local frame. That frame is the object's bounding box scaled to [-1, 1] 2011.10379 §3.2 eq. 8. Objects of one class share weights and differ only through a 256-dimensional latent code 2011.10379 §3.2 eq. 6. The density depends only on the object-frame point and the code, so it stays consistent across poses. The colour also takes the view direction and the object's global position 2011.10379 §3.2 eq. 7 eq. 9 eq. 10.

Training samples random rays from all frames. The loss is the squared colour error summed over rays, plus a zero-mean Gaussian prior on each latent code 2011.10379 eq. 14 §4.2. The paper balances ray sampling per batch, because a node's gradient depends on how many of its intersections the batch contains 2011.10379 §4.2.

Because the objects are separate nodes, changing the edges moves, rotates or removes them. The paper builds new layouts only from the road trajectories seen in training, which it calls the driveable space 2011.10379 §5.1. Object placement comes from tracking data, either hand labels or the output of a tracker 2011.10379 §5.1.

Author-reported on three KITTI tracking scenes, each trained on its own sequence 2011.10379 §5.1, the paper reports reconstruction PSNR 26.66 against 23.34 for NeRF 2011.10379 tab. 1. Card: Neural Scene Graphs for Dynamic Scenes.

Per-object fields with a semantic head (Kundu et al.)

The scene has a background MLP over a large scene box and one MLP per tracked object instance. Each object MLP is defined inside an oriented box, with a rotation, a translation at each timestamp and a time-invariant size. Those values are interpolated to any time 2205.04334 §3.1. Each MLP outputs density, colour and a one-hot class vector; the object's instance output is a one-hot vector for its own index 2205.04334 §3.1. The background semantic head ignores view direction, so labels do not change with the viewing angle 2205.04334 §3.1.

For each output channel, the field sums the contributions of every box the point hits, and uses the background when no box is hit 2205.04334 §3.1 eq. 1. The paper does not say what an object MLP returns outside its own box; the card records this as unstated 2205.04334 §3.1 eq. 1.

Boxes come from an RGB-only 3D detector and tracks from a 3D tracker 2205.04334 §3.1. The boxes are not trusted as given. All MLP weights and all track parameters are optimised together, because even ground-truth boxes may be noisy 2205.04334 §3.1 §3.3. After each step, every rotation is projected back onto SO(3) with an SVD 2205.04334 §3.1 app. A.

Training combines a colour term, the squared error summed over sampled rays, with a per-pixel softmax cross-entropy against predicted 2D segmentation maps 2205.04334 §3.3 eq. 3. The equation gives both terms weight 1. The appendix says the semantic term is scaled lower, without a value 2205.04334 app. A; see Contradictions.

Initialisation has two options 2205.04334 §3.4. Biased initialisation sets the density bias to -5.0 for the background and 0.1 for each object, with a softplus activation on density. For cars, and vans according to the appendix, a FedAvg meta-learner is first trained on 2D renders of ShapeNet car models, and its weights start each object MLP in a new scene 2205.04334 §3.4 app. A alg. 1. The paper says this is equivalent to Reptile 2205.04334 §3.4. On ShapeNet tests the learned prior converges faster after two epochs and copes better with sparse views 2205.04334 app. B fig. 14 fig. 15.

Author-reported on KITTI-360 static scenes, the method reaches mIoU 74.28 and PSNR 21.91, against 67.08 and 20.00 for the best listed baseline, FVS with PSPNet 2205.04334 tab. 2. Under the NSG protocol on dynamic KITTI, it reports PSNR 27.48 and SSIM 0.870, against 26.66 and 0.806 for NSG 2205.04334 tab. 3. Card: Panoptic Neural Fields.

When to use which

Implementation sketch

The paper's equations give the structure of the time-conditioned design. The released code differs in weights and blend convention, so take the numbers from one source and say which one you used.

# One training step, following the paper's equations  [arxiv:2105.06468 §3]
# Input:  frame at time t, its rays, RGB targets, masks, optical flow, depth
# Output: scalar loss

def training_step(t, rays, targets, masks, flows, depths):
    # Static field, trained only on rays from unmasked pixels  [eq. 1-4]
    colour_static = render_static(rays)
    # Dynamic field at time t: scene flow, density, colour, blend  [eq. 5]
    dyn = dynamic_field(rays, t)
    # Warp the dynamic field to the neighbouring times  [eq. 6-7]
    colour_next = render_dynamic(rays + dyn.flow_fw, t + 1)
    colour_prev = render_dynamic(rays + dyn.flow_bw, t - 1)
    # Blend static and dynamic into the full frame  [eq. 14]
    colour_full = composite(colour_static, dyn)
    # Photometric terms  [eq. 4] [eq. 9] [eq. 15]
    loss = photometric(colour_static, colour_full, dyn, colour_next,
                       colour_prev, targets, masks)
    # Scene flow reproduces optical flow between neighbours  [§3.4]
    loss += motion_matching(dyn, flows)
    # Slow, smooth and cycle-consistent flow  [eq. 10-13]
    loss += flow_regularisers(dyn)
    # Sparse rendering weights and depth  [§3.4]
    loss += sparsity(dyn) + depth_terms(dyn, depths, colour_static)
    return loss

The scene-graph design keeps the rendering rule but changes where samples come from. Samples are taken on the background planes and at the points on each box that a ray enters 2011.10379 §4.1 §5. Each object sample is queried in its object frame with its latent code 2011.10379 §3.2 eq. 7. The training loss is the colour error plus a latent prior eq. 14. Its object fields take a point, a latent code, a view direction and a pose eq. 7 eq. 9, not a time value, and the design has no scene flow or flow regularisers.

The box design follows the same per-box idea, but its training loss adds the semantic cross-entropy term, and it optimises the box parameters with the networks 2205.04334 §3.1 §3.3. It samples N = 1024 jittered points per ray and runs no hierarchical pass 2205.04334 §3.2 app. A.

Contradictions

• Loss weights, time-conditioned design. The supplement's weights and the released configuration differ for most terms. For example, the full photometric term is 1 in the supplement and 3.0 in the config, and the cycle term is 0.05 in one and 1.0 in the other [arxiv:2105.06468 supp. Training Losses]. The card records each pair on its loss-weights table; see the card.
• Blend convention, time-conditioned design. Equation 14 gives the static field the weight b and the dynamic field 1 - b. The text says static points have b near 1 2105.06468 §3.4 eq. 14. The released code gives b to the dynamic term card, Paper vs code.
• Depth term, time-conditioned design. The main text gives a robust depth loss. The supplement gives a pairwise depth-order loss instead, and the released code implements only the robust one 2105.06468 §3.4 [supp. Training Losses].
• Reported numbers, time-conditioned design. The paper reports PSNR 24.74 and LPIPS 0.118 on the seven sequences 2105.06468 tab. 1. The released PyTorch README reports PSNR 26.10 and LPIPS 0.082 for the same sequences, and says the paper's numbers come from the TensorFlow code see card, Paper vs code. Both are recorded; neither is resolved here.
• Loss normalisation and schedule, scene-graph design. The paper sums the squared colour error over the ray batch and penalises latents with a squared norm. The code averages the colour error and uses an unsquared norm for the latents. The paper says the learning rate decays linearly, and the code decays it exponentially card, Paper vs code.
• Semantic loss weight, box design. Equation 3 gives the colour and semantic terms weight 1. The appendix says the semantic term is scaled lower, and no value is given 2205.04334 eq. 3 app. A.
• Semantic gain, box design. The text reports a gain of +9.2 mIoU over SemanticNeRF 2205.04334 §4. Table 4 gives 56.5 against 45.3, a gain of 11.2 2205.04334 tab. 4. The row labelled "about SemanticNeRF" is the authors' own ablation, not SemanticNeRF itself.
• Leaderboard claim, box design. The text says the method outperforms every method on the leaderboard 2205.04334 §4. Table 2 compares only four baselines chosen by the authors 2205.04334 tab. 2.
• Numbers across the designs. The three papers use different datasets and baselines. The time-conditioned paper reports NeRF at average PSNR 21.62 on the Dynamic Scene dataset 2105.06468 tab. 1. The scene-graph paper reports NeRF at PSNR 23.34 on three KITTI scenes 2011.10379 tab. 1. The box paper uses KITTI-360 and the NSG protocol, reporting 27.48 for itself and 26.66 for NSG on the NSG protocol 2205.04334 tab. 3. The box paper's own numbers are author-reported, and this page does not rank the designs by them.
• Possible copied row, scene-graph design. In the novel-composition block, the SRN row repeats its reconstruction values exactly [arxiv:2011.10379 tab. 1]. The box paper's NSG-protocol table repeats the same SRN values 2205.04334 tab. 3. Neither paper says whether they were re-measured, so do not quote that row.

Papers

• Dynamic View Synthesis from Dynamic Monocular Video: static and dynamic NeRFs with scene flow, trained on one monocular video with moving-object masks.
• Neural Scene Graphs for Dynamic Scenes: one radiance field for the background and one per tracked object, shared per class through latent codes, for editable driving scenes.
• Panoptic Neural Fields: one MLP per tracked object in a learned 3D box, a background MLP with a semantic head, and a car prior learned by FedAvg, for driving scenes.