Dynamic View Synthesis from Dynamic Monocular Video

In plain words

A normal video shows a moving scene from one camera position, at one moment per frame. This paper asks whether a single casually filmed video can be turned into a 3D model that shows the scene from new viewpoints and at any point in time. Each moment is seen from only one angle, so many different 3D models fit the video equally well, and most of them look wrong from a new angle.

The method splits the scene in two. The still background is learned by a standard NeRF that ignores every pixel marked as moving. The moving parts are learned by a second network that takes a point in space and a time, and also predicts where that point moves in the frames just before and after. The two parts are blended. Several extra rules push the model towards a physically plausible answer: motion should be small and smooth, forward and backward motion should cancel out, depth should look right, and the background should not move. The method needs a mask of the moving objects in each frame. On the benchmark the authors use, it renders new views better than the earlier methods they compare with, in average image quality.

TL;DR

• Two fields: a static NeRF trained on unmasked pixels, and a dynamic NeRF that takes (position, time) and outputs density, colour, forward and backward 3D scene flow, and a blend weight §3.2 §3.3.
• The dynamic NeRF is also trained on the neighbouring times t-1 and t+1 by warping through the predicted scene flow, so each 3D point gets up to three photometric constraints §3.3 eq. 6-9.
• Regularisers: flow matched to RAFT optical flow, slow, temporally and spatially smooth, cycle-consistent scene flow, sparse rendering weights, and depth from a monocular depth estimator §3.4.
• Author-reported: average PSNR 24.74 and LPIPS 0.118 over seven sequences, against 23.99 and 0.205 for Li et al. tab. 1.
• The paper's loss weights, iteration count and blend convention differ from the released PyTorch code. See Paper vs code.

Problem

Free-viewpoint video of a dynamic scene normally needs a multi-camera rig or a depth-capture setup. Monocular video is cheap but leaves each moment with one image, so the scene's shape and motion are under-determined §1 §3.3.

A NeRF with time as an extra input (NeRF + time) fits the training frames but produces artifacts at new viewpoints §3.1 fig. 4. A static NeRF applied to a dynamic video gives blurred results fig. 3.

Assumptions and limits the paper names or shows:

• A binary mask of the moving foreground is supplied for every frame. It can come from a segmentation or motion-segmentation model, or from rotoscoping §3.1.
• The benchmark uses 12-frame sequences, one camera per frame, with COLMAP poses and near and far bounds §4.1.
• Failure cases remain when the scene is hard to model §4.4 fig. 11.
• Training takes about 22 hours per 12-frame sequence on one NVIDIA V100 [supp. Implementation Details].

Method

Inputs are a monocular video of N frames I_0 to I_{N-1} and a binary foreground mask M for each frame §3.1. The output is a model that renders any frame index and any camera §3.1.

Static NeRF §3.2 fig. 3. A standard NeRF MLP maps a 3D point and viewing direction to density and colour. Its training rays come only from pixels with M = 0 (static region). Volume rendering uses K quadrature samples drawn uniformly between the near and far bounds §3.2:

(\sigma^s, \mathbf{c}^s) = \text{MLP}_\theta(\mathbf{r}(u_k)) eq. 1

\mathbf{C}^s(\mathbf{r}) = \sum_{k=1}^{K} T^s(u_k)\,\alpha^s(\sigma^s(u_k)\,\delta_k)\,\mathbf{c}^s(u_k) eq. 2

T^s(u_k) = \exp\!\Big(-\sum_{k'=1}^{k-1} \sigma^s(u_k)\,\delta_k\Big) eq. 3

with \alpha(x) = 1 - \exp(-x) and \delta_k = u_{k+1} - u_k §3.2. The equation as typeset is ambiguous; see Pitfalls.

\mathcal{L}_\textit{static} = \sum_{ij} \big\lVert (\mathbf{C}^s(\mathbf{r}_{ij}) - \mathbf{C}^{gt}(\mathbf{r}_{ij}))\,(1 - \mathbf{M}(\mathbf{r}_{ij})) \big\rVert_2^2 eq. 4

Dynamic NeRF §3.3 fig. 4. One MLP takes (position, time t) and outputs forward scene flow s_fw, backward scene flow s_bw, density sigma^d_t, colour c^d_t and a blend weight b §3.3:

(\mathbf{s}_\textit{fw}, \mathbf{s}_\textit{bw}, \sigma^d_t, \mathbf{c}^d_t, b) = \text{MLP}_{\theta_d}(\mathbf{r}(u_k), t) eq. 5

The same MLP is queried at the flowed positions and neighbouring times, which warps the field to t+1 and t-1 §3.3:

(\sigma^d_{t+1}, \mathbf{c}^d_{t+1}) = \text{MLP}_{\theta_d}(\mathbf{r}(u_k) + \mathbf{s}_\textit{fw},\, t+1) eq. 6

(\sigma^d_{t-1}, \mathbf{c}^d_{t-1}) = \text{MLP}_{\theta_d}(\mathbf{r}(u_k) + \mathbf{s}_\textit{bw},\, t-1) eq. 7

Colour at any t' is rendered as in the static case eq. 8, and the dynamic photometric loss sums over the three times eq. 9:

\mathbf{C}^d_{t'}(\mathbf{r}) = \sum_{k=1}^{K} T^d_{t'}(u_k)\,\alpha^d(\sigma^d_{t'}(u_k)\,\delta_k)\,\mathbf{c}^d_{t'}(u_k) eq. 8

\mathcal{L}_\textit{dyn} = \sum_{t' \in \{t-1,\,t,\,t+1\}} \sum_{ij} \big\lVert \mathbf{C}^d_{t'}(\mathbf{r}_{ij}) - \mathbf{C}^{gt}(\mathbf{r}_{ij}) \big\rVert_2^2 eq. 9

Regularisers §3.4 fig. 5 fig. 6. These are the core contribution. The paper's regularisers, with the equations that are typeset in the main text:

• Motion matching: the scene flow, projected into the reference camera, must reproduce RAFT optical flow between neighbouring frames §3.4 fig. 5 [supp. Training Losses]. No equation in the main text.
• Motion regularisation: slow flow eq. 10, temporal smoothness, meaning the forward and backward flow cancel eq. 11, and spatial smoothness between neighbouring points §3.4:

\mathcal{L}_\textit{slow} = \sum_{ij} \lVert \mathbf{s}_\textit{fw}(\mathbf{r}_{ij}) \rVert_1 + \lVert \mathbf{s}_\textit{bw}(\mathbf{r}_{ij}) \rVert_1 eq. 10

\mathcal{L}_\textit{smooth} = \sum_{ij} \lVert \mathbf{s}_\textit{fw}(\mathbf{r}_{ij}) + \mathbf{s}_\textit{bw}(\mathbf{r}_{ij}) \rVert_2^2 eq. 11

• Cycle consistency: following the flow forward and back should return to the start eq. 12-13:

\mathcal{L}_\textit{cyc} = \sum \big\lVert \mathbf{s}_\textit{fw}(\mathbf{r}, t) + \mathbf{s}_\textit{bw}(\mathbf{r} + \mathbf{s}_\textit{fw}(\mathbf{r}, t), t+1) \big\rVert_2^2 + \big\lVert \mathbf{s}_\textit{bw}(\mathbf{r}, t) + \mathbf{s}_\textit{fw}(\mathbf{r} + \mathbf{s}_\textit{bw}(\mathbf{r}, t), t-1) \big\rVert_2^2

• Sparsity: the entropy of the rendering weights T^d alpha^d along each ray is minimised, so few samples dominate §3.4 [supp. Training Losses].
• Depth: a robust, shift-and-scale-invariant depth loss against a monocular depth estimator (Lasinger et al.), plus an L2 term that ties the dynamic depth to the static depth in static pixels §3.4. The supplement gives a pairwise depth-order loss instead [supp. Training Losses].
• 3D temporal consistency: colour and density of a point should match at its scene-flow neighbours, so that occluded background propagates across time §3.4 [supp. Training Losses].
• Rigidity: the blend weight b is meant to be near 1 for static points, which should force the scene flow to zero there §3.4.

Composition §3.5 fig. 7. Static and dynamic fields are blended into a full-frame colour. The main text's full-rendering equation eq. 14 uses the weights below, and the full loss is eq. 15:

\mathbf{C}^\textit{full}(\mathbf{r}) = \sum_{k=1}^{K} T^\textit{full}\!\Big( \alpha^d(\sigma^d\delta_k)(1-b)\,\mathbf{c}^d + \alpha^s(\sigma^s\delta_k)\,b\,\mathbf{c}^s \Big) eq. 14

\mathcal{L}_\textit{full} = \sum_{ij} \big\lVert \mathbf{C}^\textit{full}(\mathbf{r}_{ij}) - \mathbf{C}^{gt}(\mathbf{r}_{ij}) \big\rVert_2^2 eq. 15

The paper does not define T^full in the compiled text. The released code uses the form in the Paper vs code section below.

Training objective: the supplement lists all terms with weights [supp. Training Losses], reproduced in the Loss weights table below. The paper states that the weights are fixed across all results [supp. Training Losses].

# One training step, as the paper describes it  [§3.2] [§3.3] [§3.4] [§3.5]
# Input:  frame index i at time t, its rays, colour targets, masks, flows,
#         depths
# Output: scalar loss

def training_step(frame, t, rays, targets, masks, flows, depths):
    # Static field on rays from static pixels only  [§3.2] [eq. 1-4]
    colour_static = render_static(rays)
    # Dynamic field at time t: density, colour, flows and blend  [eq. 5]
    dyn = dynamic_mlp(rays, t)
    # Warp the dynamic field to the neighbouring times  [eq. 6-7]
    colour_next = render_dynamic(rays + dyn.flow_fw, t + 1)
    colour_prev = render_dynamic(rays + dyn.flow_bw, t - 1)
    # Composite static and dynamic with the blend  [eq. 14]
    colour_full = composite(colour_static, dyn)
    # Photometric terms  [eq. 4] [eq. 9] [eq. 15]
    loss = photometric(colour_static, colour_full, dyn, colour_next,
                       colour_prev, targets, masks)
    # Flow induced by the scene flow matches RAFT  [§3.4]
    loss += motion_matching(dyn, flows)
    # Slow, smooth and cycle-consistent scene flow  [eq. 10-13]
    loss += scene_flow_regularisers(dyn)
    # Sparse rendering weights and depth  [§3.4]
    loss += sparsity(dyn) + depth_terms(dyn, depths, colour_static)
    return loss

# Full-frame colour, with the code's convention for the blend  [eq. 14]
# Input:  per-sample densities and colours for both fields, sample spacings
# Output: colour of one ray

def composite(sigma_s, colour_s, sigma_d, colour_d, blend, deltas):
    # Per-sample opacities from densities and spacings
    #   [code: render_utils.py#L345-L346]
    alpha_s = 1.0 - exp(-relu(sigma_s) * deltas)
    alpha_d = 1.0 - exp(-relu(sigma_d) * deltas)
    # Per-sample factor, combining both fields  [code: render_utils.py#L378]
    keep = (1.0 - alpha_d * blend) * (1.0 - alpha_s * (1.0 - blend))
    # Transmittance before each sample  [code: render_utils.py#L378]
    transmittance = exclusive_cumprod(keep)
    # Here blend weights the dynamic share, the reverse of eq. 14
    #   [code: render_utils.py#L394-L395]
    weight_d = transmittance * alpha_d * blend
    weight_s = transmittance * alpha_s * (1.0 - blend)
    return sum(weight_d * colour_d + weight_s * colour_s)

Hyperparameters

Values are from the paper where marked § or [supp.], otherwise from the released code, which is labelled as such. "not stated" means neither source gives the value.

Loss weights. Supplement values against the released Playground config. The other six configs share these values for the slow term (checked by grep).

Results

Author-reported. Setup: the Dynamic Scene dataset of Yoon et al., novel views rendered from the first camera while time varies, scored with PSNR and LPIPS §4.2 tab. 1.

Source: tab. 1. The paper bolds the best result per column and underlines the second best; the table above drops that formatting.

Ablation, Playground sequence, PSNR / SSIM / LPIPS tab. 2:

Qualitative results: novel views never seen in training fig. 8; comparison with Li et al. on large motion, where the paper says its learned time-varying blend keeps the background clean fig. 9 §4.2; motion regularisation and depth loss in fig. 10; failure cases in fig. 11 §4.4.

Paper vs code

Official code: github.com/gaochen315/DynamicNeRF, MIT licence, commit c417fb2. The project page links it [project page: free-view-video.github.io]. The paper's supplement says the code will be released [supp. Training Losses].

Discrepancies. Each is checked against the file cited.

• Loss weights. The supplement's list does not match the released configs. The full photometric term is 1 in the supplement and 3.0 in the config [supp. Training Losses] config_Playground.txt. The cycle, sparsity and smooth terms are 0.05 in the supplement, and 1.0, 0.001 and 0.1 in the config config_Playground.txt. See the Loss weights table.
• Iterations. The paper gives 200,000 iterations [supp. Implementation Details]. The config uses N_iters = 300001 config_Playground.txt. The static pre-training loop runs N_iters steps before the joint stage run_nerf.py run_nerf.py.
• Static pre-training. The paper does not describe a separate static stage. The code trains the static NeRF alone first, then the joint model, when pretrain = True (set in every config) run_nerf.py.
• Blend convention. The main text puts the static weight on b and the dynamic weight on 1 - b eq. 14, and says static points have b near 1 §3.4. The code gives b to the dynamic term render_utils.py and its mask loss drives b to 0 on static pixels run_nerf.py.
• Rigidity. The paper multiplies the scene flow by the blend, (1 - b), to enforce rigidity §3.4. That line is commented out in the code render_utils.py. The code relies on the blend mask loss instead.
• Mask loss. The abstract says the blend is learned unsupervised [abstract]. The code adds an L1 term on the blend in static pixels and a mask term run_nerf.py. The paper does not list this term.
• Depth. The main text gives a robust depth loss and an L2 static term §3.4. The supplement gives a pairwise ranking loss [supp. Training Losses]. The code has the robust loss run_nerf.py and an L2 term against the static depth with that depth detached run_nerf.py. No ranking loss is implemented.
• 3D temporal consistency. Listed in the supplement with weight 0.05 [supp. Training Losses]. No corresponding term appears in the loss section of the code run_nerf.py.
• Extra losses in code. Two photometric terms at t-2 and t+2 are added from iteration 50,000 with 12 frames run_nerf.py run_nerf.py. A second spatial smoothness term runs alongside the temporal one run_nerf.py. The paper mentions neither.
• Entropy. The sparsity term in the paper is the entropy of T^d alpha^d §3.4. The code adds the entropy of the blend, and divides by the number of rays run_nerf.py run_nerf_helpers.py.
• Motion loss. The paper leaves the norm open [supp. Training Losses]. The code uses mean absolute error, weighted by a flow mask run_nerf.py run_nerf_helpers.py.
• Results reproduced in code. The README's PyTorch numbers are different from the paper's. Its average for the same seven sequences is PSNR 26.10 and LPIPS 0.082, against 24.74 and 0.118 in the paper [README.md#L107-L113]. The README says the paper's numbers come from the TensorFlow code.

Pitfalls

• Which weights to use. The paper says its weights are fixed across results, but the supplement and the released config disagree on most of them (see the Loss weights table). An implementer must pick one set and say which.
• Temp schedule. The released code multiplies the flow and depth terms by Temp = 10^-floor(step / (decay_iteration x 1000)), where decay_iteration = max(25, number of frames) run_nerf.py run_nerf.py. For 12 frames, Temp is 1 below 25,000 steps, 0.1 until 50,000, then 0.01. The mask loss stops at 25,000 run_nerf.py. The five-frame terms start at 50,000 run_nerf.py. None of this is in the paper.
• Summed versus averaged losses. The paper writes its photometric and regularisation losses as sums over pixels, rays and samples §3.2 §3.4. The code averages them, for example mean squared error over rays run_nerf_helpers.py and masked means elsewhere. The relative weights in the table are not the same as in the paper's sums. Use the code's normalisation if you need to match its numbers.
• Ray sampling. The paper uses every pixel §3.2. The code draws 1024 rays per step, half from moving pixels, which over-weights the dynamic region run_nerf.py.
• Equation 3 as typeset. The summand of the transmittance does not depend on the summation index k'. The released code computes transmittance as a product over earlier samples of per-sample opacities render_utils.py, which is the standard form. Implement that and cite eq. 3 only as the intent.
• T^full is undefined. The compiled main text uses T^full in eq. 14 but does not define it. The source has a commented-out definition as the exponential of the sum of both densities §3.5 [src: 3_method.tex]. The code uses the product form in composite above.
• Equation numbers. The supplement refers to main-text Equations 2, 8 and 13 for volume rendering [supp. Training Losses]. Counting the align rows in the source gives 14 for the full rendering, not 13. The numbers in this card come from the source count. Check them against the PDF before citing.
• Scene-flow bound. The code bounds scene flow with tanh [code: run_nerf_helpers.py#L145], so in NDC units the flow cannot exceed one unit per axis. The paper does not mention a bound. Large motions may be clipped.
• NDC coordinates. The code uses NDC coordinates for the sampled points config_Playground.txt and converts them to world units only for the smoothness losses run_nerf_helpers.py [code: run_nerf_helpers.py#L601-L617]. The smoothness terms use only the first 95% (spatial) or 90% (temporal) of samples along each ray.
• Density activation. The code's alpha uses ReLU on the density before the exponential render_utils.py. The paper does not state the activation.
• Mask quality. Masks are a required input. The released pipeline generates them automatically [README.md#L186] generate_motion_mask.py. The paper says the mask may come from segmentation or rotoscoping §3.1. The static NeRF and the blend both depend on it, so errors in the masks show up as errors in the output.
• Dataset sequences. The paper uses 9 sequences and excludes DynamicFace, which leaves 8. Table 1 reports 7 §4.1 tab. 1. The eighth sequence is not named and its result is not reported.
• Comparison fairness. Li et al. are trained on 24 frames per sequence, this method on 12 [supp. Comparisons with other methods]. Tretschk et al. need per-sequence tuning §4.2. The authors say their results may improve with tuning [supp. Comparisons with other methods].
• Yoon et al. results. The authors supplied these results [supp. Comparisons with other methods]. The released evaluation drops the first and last frame to match them [README.md#L118].
• Cost. About 22 hours per 12-frame sequence on one V100 [supp. Implementation Details]. Each step queries the dynamic MLP at up to five times and positions run_nerf.py.

Lineage

• Builds on NeRF (Mildenhall et al.) for the static field and the rendering equations §3.2. The paper's baseline "NeRF + time" adds time as an input §3.1.
• Uses RAFT optical flow for motion supervision §3.4, and a single-image depth estimator by Lasinger et al. for depth §3.4.
• Uses COLMAP for poses and bounds §4.1.
• Compared against: Yoon et al. tab. 1; Li et al. tab. 1 fig. 9; Tretschk et al. tab. 1.
• Superseded by: not known from content on this wiki.