---
kind: concept
slug: dynamic-neural-radiance-field
title: Dynamic neural radiance field
summary: >-
  A dynamic neural radiance field renders a moving scene from new views. One fits a monocular video with masks and predicts scene flow. Two others learn a field per tracked object in driving scenes, so objects can be moved or removed. All three need extra inputs. Training takes about a day for the first two; the third states no time.
tags: [novel-view-synthesis, neural-radiance-fields, dynamic-scenes, monocular-video, scene-flow, scene-graph, autonomous-driving, semantic-segmentation, object-tracking]
papers: ["2105.06468", "2011.10379", "2205.04334"]
related: [neural-radiance-field, novel-view-synthesis, explicit-feature-grid, multi-view-reconstruction, monocular-depth-estimation, 3d-scene-graph, implicit-neural-representation, meta-learning, semantic-segmentation, tracking-by-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

A dynamic neural radiance field is a NeRF that can render content that moves.
Three designs appear on this wiki. The first adds time as an input to the
field, so a 3D point and a moment give a colour and a density
[arxiv:2105.06468 §3.1] [§3.3]. The second splits the scene into a graph: a
static background field plus one field per tracked object, each placed by
tracking data [arxiv:2011.10379 §3]. The third also uses one field per tracked
object inside a 3D box, and adds a semantic output to every field
[arxiv:2205.04334 §3.1]. All three render new views with the NeRF compositing
rule, where a pixel is a sum over ray samples of transmittance, opacity and
colour [arxiv:2105.06468 eq. 2] [arxiv:2011.10379 eq. 13]
[arxiv:2205.04334 eq. 2]. The [NeRF concept page](/concepts/neural-radiance-field)
covers that rule in more detail.

The problems differ. For the time-input design, a monocular video shows each
moment from one viewpoint only, so many fields fit the training frames, and
most of them give artifacts from a new angle [arxiv:2105.06468 §1] [§3.1]
[fig. 4]. For the graph design, a single network for a whole scene does not
separate objects, so the objects cannot be edited on their own
[arxiv:2011.10379 §1]. For the box design, the authors name three limits of
earlier object-aware models: no panoptic output, learning from scratch for each
scene, and one MLP shared across instances through per-object latent codes
[arxiv:2205.04334 §1] [§2].

The three cards list this slug. Gao et al. describe the time-input design on
monocular video [arxiv:2105.06468]. Ost et al. describe the graph design on
driving scenes [arxiv:2011.10379]. Kundu et al. describe the box design, with
semantics, on driving scenes [arxiv:2205.04334]. Their reported numbers come
from different datasets and baselines, so they are not directly comparable; see
Contradictions.

## Variants

### Time-conditioned static and dynamic fields (Gao et al.)

Two fields are trained together. The static field is a standard NeRF. Its
training rays come only from pixels outside the moving-object mask
[arxiv:2105.06468 §3.2] [eq. 1-4]. The dynamic field is one MLP that takes a
point and a time t, and outputs forward and backward 3D scene flow, density,
colour and a blend weight [arxiv:2105.06468 §3.3] [eq. 5]. The same MLP is
queried at the flowed points and at times t+1 and t-1, so each 3D point gets up
to three photometric constraints instead of one [arxiv:2105.06468 §3.3]
[eq. 6-9].

The paper's regularisers do most of the work [arxiv:2105.06468 §3.4]:

- Motion matching: the projected scene flow must reproduce optical flow from
  RAFT between neighbouring frames [arxiv:2105.06468 §3.4] [fig. 5].
- Motion smoothness: small and temporally smooth flow, with forward and
  backward flow cancelling [arxiv:2105.06468 eq. 10-11].
- Cycle consistency: following the flow forth and back returns to the start
  [arxiv:2105.06468 eq. 12-13].
- Depth: a shift- and scale-invariant depth term against a monocular depth
  estimator, and a term tying dynamic depth to static depth [arxiv:2105.06468 §3.4].
- Sparsity: low entropy of the rendering weights along each ray
  [arxiv:2105.06468 §3.4].

The final image blends the two fields along each ray, so static points are
meant to come from the static field and moving points from the dynamic one
[arxiv:2105.06468 §3.5] [eq. 14]. The paper's ablation on the Playground
sequence reports that removing the depth-order term, the motion term or the
rigidity term each lowers PSNR, from 23.65 for the full model to between 22.61
and 22.99 [arxiv:2105.06468 tab. 2]. The depth-order term is the one the
released code does not implement; see Contradictions.

Author-reported results use the Dynamic Scene dataset of Yoon et al. and
score novel views from the first camera. On average over seven sequences the
method reaches PSNR 24.74 and LPIPS 0.118. Li et al. reach 23.99 and 0.205
[arxiv:2105.06468 tab. 1]. Li et al. are trained on 24 frames per sequence,
this method on 12 [arxiv:2105.06468 supp. Comparisons with other methods].
Card: [Dynamic View Synthesis from Dynamic Monocular Video](/papers/2105.06468).

### Scene graph of object fields (Ost et al.)

The scene is a directed acyclic graph. Its parts are a camera, a static
background field, one dynamic field per object class, the latent codes and the
edges. Edges carry either a rigid transform or an object scale, and an object's
pose is read by walking from the world root to its node [arxiv:2011.10379 §3]
[eq. 1-2].

The background is a NeRF-style field. The paper does not ray-march it through
a volume. It samples a few planes parallel to the first camera's image plane,
six in the paper [arxiv:2011.10379 §4.1] [§5].

Each object that moves rigidly during capture has its own field, defined in a
local frame. That frame is the object's bounding box scaled to [-1, 1]
[arxiv:2011.10379 §3.2] [eq. 8]. Objects of one class share weights and differ
only through a 256-dimensional latent code [arxiv:2011.10379 §3.2] [eq. 6]. The
density depends only on the object-frame point and the code, so it stays
consistent across poses. The colour also takes the view direction and the
object's global position [arxiv:2011.10379 §3.2] [eq. 7] [eq. 9] [eq. 10].

Training samples random rays from all frames. The loss is the squared colour
error summed over rays, plus a zero-mean Gaussian prior on each latent code
[arxiv:2011.10379 eq. 14] [§4.2]. The paper balances ray sampling per batch,
because a node's gradient depends on how many of its intersections the batch
contains [arxiv:2011.10379 §4.2].

Because the objects are separate nodes, changing the edges moves, rotates or
removes them. The paper builds new layouts only from the road trajectories seen
in training, which it calls the driveable space [arxiv:2011.10379 §5.1]. Object
placement comes from tracking data, either hand labels or the output of a
tracker [arxiv:2011.10379 §5.1].

Author-reported on three KITTI tracking scenes, each trained on its own
sequence [arxiv:2011.10379 §5.1], the paper reports reconstruction PSNR 26.66
against 23.34 for NeRF [arxiv:2011.10379 tab. 1]. Card:
[Neural Scene Graphs for Dynamic Scenes](/papers/2011.10379).

### Per-object fields with a semantic head (Kundu et al.)

The scene has a background MLP over a large scene box and one MLP per tracked
object instance. Each object MLP is defined inside an oriented box, with a
rotation, a translation at each timestamp and a time-invariant size. Those
values are interpolated to any time [arxiv:2205.04334 §3.1]. Each MLP outputs
density, colour and a one-hot class vector; the object's instance output is a
one-hot vector for its own index [arxiv:2205.04334 §3.1]. The background
semantic head ignores view direction, so labels do not change with the viewing
angle [arxiv:2205.04334 §3.1].

For each output channel, the field sums the contributions of every box the
point hits, and uses the background when no box is hit [arxiv:2205.04334 §3.1]
[eq. 1]. The paper does not say what an object MLP returns outside its own box;
the card records this as unstated [arxiv:2205.04334 §3.1] [eq. 1].

Boxes come from an RGB-only 3D detector and tracks from a 3D tracker
[arxiv:2205.04334 §3.1]. The boxes are not trusted as given. All MLP weights and
all track parameters are optimised together, because even ground-truth boxes
may be noisy [arxiv:2205.04334 §3.1] [§3.3]. After each step, every rotation
is projected back onto SO(3) with an SVD [arxiv:2205.04334 §3.1] [app. A].

Training combines a colour term, the squared error summed over sampled rays,
with a per-pixel softmax cross-entropy against predicted 2D segmentation maps
[arxiv:2205.04334 §3.3] [eq. 3]. The equation gives both terms weight 1. The
appendix says the semantic term is scaled lower, without a value
[arxiv:2205.04334 app. A]; see Contradictions.

Initialisation has two options [arxiv:2205.04334 §3.4]. Biased initialisation
sets the density bias to -5.0 for the background and 0.1 for each object, with
a softplus activation on density. For cars, and vans according to the appendix,
a FedAvg meta-learner is first trained on 2D renders of ShapeNet car models,
and its weights start each object MLP in a new scene [arxiv:2205.04334 §3.4]
[app. A] [alg. 1]. The paper says this is equivalent to Reptile
[arxiv:2205.04334 §3.4]. On ShapeNet tests the learned prior converges faster
after two epochs and copes better with sparse views [arxiv:2205.04334 app. B]
[fig. 14] [fig. 15].

Author-reported on KITTI-360 static scenes, the method reaches mIoU 74.28 and
PSNR 21.91, against 67.08 and 20.00 for the best listed baseline, FVS with
PSPNet [arxiv:2205.04334 tab. 2]. Under the NSG protocol on dynamic KITTI, it
reports PSNR 27.48 and SSIM 0.870, against 26.66 and 0.806 for NSG
[arxiv:2205.04334 tab. 3]. Card:
[Panoptic Neural Fields](/papers/2205.04334).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Driving-style scene with tracked 3D boxes per object, and objects must be moved, removed or recombined | Scene graph of object fields, or the box design | Both edit object placement. The scene graph reports novel-composition PSNR 25.11 against 18.25 for NeRF [arxiv:2011.10379 tab. 1] |
| Driving-style scene with tracked boxes, and per-pixel semantic labels or instance masks are needed with the render | Per-object fields with a semantic head | The only design here with a semantic output. Author-reported mIoU 74.28 against 53.01 for NeRF + PSPNet [arxiv:2205.04334 tab. 2] |
| Few views of a car, and fast fitting from a prior | Per-object fields with the car prior | Author-reported faster convergence and better sparse-view results on ShapeNet tests [arxiv:2205.04334 app. B]. Cars, and vans per the appendix. The prior needs its own meta-training [arxiv:2205.04334 §3.4] [app. A] |
| One casually filmed monocular video, and moving-object masks for each frame, with no object tracks | Time-conditioned fields | Author-reported average PSNR 24.74 against 23.99 for Li et al. [arxiv:2105.06468 tab. 1] |
| Neither tracks nor masks | Neither design | Tracks are costly to annotate, and masks need segmentation or rotoscoping [arxiv:2011.10379 §5.1] [arxiv:2105.06468 §3.1] |
| Changing object placement is not needed, only a new viewpoint of a fixed scene | Either, by data | The time-conditioned design renders any frame and camera [arxiv:2105.06468 §3.1]. The scene graph's view extrapolation is limited to about 2 m [arxiv:2011.10379 §5.1] [fig. 8] |
| Short training budget on one GPU | Neither is shown to be quick | Scene graph: about one day per scene on one TITAN Xp [arxiv:2011.10379 §5]. Time-conditioned: about 22 hours per 12-frame sequence on one V100 [arxiv:2105.06468 supp. Implementation Details]. The box design states no training time [arxiv:2205.04334 §5] |
| Need to reuse released code | Time-conditioned first | Its PyTorch code is MIT-licensed [card, Paper vs code](/papers/2105.06468). The graph code needs TensorFlow 1.15 and has no LICENSE file [card, Paper vs code](/papers/2011.10379). The box design has no official code [card, Paper vs code](/papers/2205.04334) |

## Implementation sketch

The paper's equations give the structure of the time-conditioned design. The
released code differs in weights and blend convention, so take the numbers
from one source and say which one you used.

```pseudocode
# One training step, following the paper's equations  [arxiv:2105.06468 §3]
# Input:  frame at time t, its rays, RGB targets, masks, optical flow, depth
# Output: scalar loss

def training_step(t, rays, targets, masks, flows, depths):
    # Static field, trained only on rays from unmasked pixels  [eq. 1-4]
    colour_static = render_static(rays)
    # Dynamic field at time t: scene flow, density, colour, blend  [eq. 5]
    dyn = dynamic_field(rays, t)
    # Warp the dynamic field to the neighbouring times  [eq. 6-7]
    colour_next = render_dynamic(rays + dyn.flow_fw, t + 1)
    colour_prev = render_dynamic(rays + dyn.flow_bw, t - 1)
    # Blend static and dynamic into the full frame  [eq. 14]
    colour_full = composite(colour_static, dyn)
    # Photometric terms  [eq. 4] [eq. 9] [eq. 15]
    loss = photometric(colour_static, colour_full, dyn, colour_next,
                       colour_prev, targets, masks)
    # Scene flow reproduces optical flow between neighbours  [§3.4]
    loss += motion_matching(dyn, flows)
    # Slow, smooth and cycle-consistent flow  [eq. 10-13]
    loss += flow_regularisers(dyn)
    # Sparse rendering weights and depth  [§3.4]
    loss += sparsity(dyn) + depth_terms(dyn, depths, colour_static)
    return loss
```

The scene-graph design keeps the rendering rule but changes where samples come
from. Samples are taken on the background planes and at the points on each box
that a ray enters [arxiv:2011.10379 §4.1] [§5]. Each object sample is queried
in its object frame with its latent code [arxiv:2011.10379 §3.2] [eq. 7]. The
training loss is the colour error plus a latent prior [eq. 14]. Its object
fields take a point, a latent code, a view direction and a pose [eq. 7] [eq. 9],
not a time value, and the design has no scene flow or flow regularisers.

The box design follows the same per-box idea, but its training loss adds the
semantic cross-entropy term, and it optimises the box parameters with the
networks [arxiv:2205.04334 §3.1] [§3.3]. It samples N = 1024 jittered points per
ray and runs no hierarchical pass [arxiv:2205.04334 §3.2] [app. A].

## Contradictions

- **Loss weights, time-conditioned design.** The supplement's weights and the
  released configuration differ for most terms. For example, the full
  photometric term is 1 in the supplement and 3.0 in the config, and the cycle
  term is 0.05 in one and 1.0 in the other [arxiv:2105.06468 supp. Training
  Losses]. The card records each pair on its loss-weights table; see the
  [card](/papers/2105.06468).
- **Blend convention, time-conditioned design.** Equation 14 gives the static
  field the weight b and the dynamic field 1 - b. The text says static points
  have b near 1 [arxiv:2105.06468 §3.4] [eq. 14]. The released code gives b to
  the dynamic term [card, Paper vs code](/papers/2105.06468).
- **Depth term, time-conditioned design.** The main text gives a robust depth
  loss. The supplement gives a pairwise depth-order loss instead, and the
  released code implements only the robust one [arxiv:2105.06468 §3.4] [supp.
  Training Losses].
- **Reported numbers, time-conditioned design.** The paper reports PSNR 24.74
  and LPIPS 0.118 on the seven sequences [arxiv:2105.06468 tab. 1]. The
  released PyTorch README reports PSNR 26.10 and LPIPS 0.082 for the same
  sequences, and says the paper's numbers come from the TensorFlow code [see
  card, Paper vs code](/papers/2105.06468). Both are recorded; neither is
  resolved here.
- **Loss normalisation and schedule, scene-graph design.** The paper sums the
  squared colour error over the ray batch and penalises latents with a squared
  norm. The code averages the colour error and uses an unsquared norm for the
  latents. The paper says the learning rate decays linearly, and the code
  decays it exponentially [card, Paper vs code](/papers/2011.10379).
- **Semantic loss weight, box design.** Equation 3 gives the colour and semantic
  terms weight 1. The appendix says the semantic term is scaled lower, and no
  value is given [arxiv:2205.04334 eq. 3] [app. A].
- **Semantic gain, box design.** The text reports a gain of +9.2 mIoU over
  SemanticNeRF [arxiv:2205.04334 §4]. Table 4 gives 56.5 against 45.3, a gain
  of 11.2 [arxiv:2205.04334 tab. 4]. The row labelled "about SemanticNeRF" is
  the authors' own ablation, not SemanticNeRF itself.
- **Leaderboard claim, box design.** The text says the method outperforms every
  method on the leaderboard [arxiv:2205.04334 §4]. Table 2 compares only four
  baselines chosen by the authors [arxiv:2205.04334 tab. 2].
- **Numbers across the designs.** The three papers use different datasets and
  baselines. The time-conditioned paper reports NeRF at average PSNR 21.62 on
  the Dynamic Scene dataset [arxiv:2105.06468 tab. 1]. The scene-graph paper
  reports NeRF at PSNR 23.34 on three KITTI scenes [arxiv:2011.10379 tab. 1].
  The box paper uses KITTI-360 and the NSG protocol, reporting 27.48 for itself
  and 26.66 for NSG on the NSG protocol [arxiv:2205.04334 tab. 3]. The box
  paper's own numbers are author-reported, and this page does not rank the
  designs by them.
- **Possible copied row, scene-graph design.** In the novel-composition block,
  the SRN row repeats its reconstruction values exactly [arxiv:2011.10379
  tab. 1]. The box paper's NSG-protocol table repeats the same SRN values
  [arxiv:2205.04334 tab. 3]. Neither paper says whether they were re-measured,
  so do not quote that row.

## Papers

- [Dynamic View Synthesis from Dynamic Monocular Video](/papers/2105.06468): static and dynamic NeRFs with scene flow, trained on one monocular video with moving-object masks.
- [Neural Scene Graphs for Dynamic Scenes](/papers/2011.10379): one radiance field for the background and one per tracked object, shared per class through latent codes, for editable driving scenes.
- [Panoptic Neural Fields](/papers/2205.04334): one MLP per tracked object in a learned 3D box, a background MLP with a semantic head, and a car prior learned by FedAvg, for driving scenes.