Neural Scene Graphs for Dynamic Scenes

In plain words

Driving videos show cars, pedestrians and the road moving past a camera. Earlier neural rendering methods store a whole scene in one network, which works for still scenes but does not separate the things that move. This paper splits the scene into a graph. One branch holds the static background. Every moving object gets its own small learned model of how it looks, and objects of the same kind share one network, told apart by a short learned code per object.

The graph also records where each object sits and how it is oriented. Because the objects are separate pieces, you can move them, remove them or place them in a different arrangement and render the result from new viewpoints. The model is trained from the video plus the tracked boxes of the objects, which can come from manual labels or from a tracker.

TL;DR

The paper represents a dynamic driving scene as a scene graph: a static background radiance field and one radiance field per tracked object, with objects of one class sharing weights through a learned latent code per object §3 §3.2. Object placement comes from tracking data through graph edges, so learned objects can be moved, removed or recombined and rendered from new viewpoints §5 fig. 7. Reported results are author-reported: on three KITTI scenes, reconstruction PSNR is 26.66 against 23.34 for NeRF tab. 1. The official repository has no LICENSE file, so use it as a reference and do not copy its code.

Problem

Implicit neural renderers encode a static scene in a single network. That representation does not allow hierarchical structure or dynamic scenes, and it does not decompose a scene into individual objects §1. The representation quality is also bounded by the variety and amount of training data §7. Manual annotation of object tracks is costly, so the paper also checks whether the output of existing trackers is good enough §5.1.

Method

The scene is a directed acyclic graph S = ⟨W, C, F, L, E⟩ eq. 1. The camera C, the static background field, the per-class dynamic fields, the latent codes L and the edges E are its parts. Edges carry either a rigid transform M = [R t; 0 1] or an object scale eq. 2 §3. To get an object's pose, walk from the world root to the object node §3.

1. Background node. A NeRF-style field F_bckg maps a 3D point to density σ and to an intermediate feature y. A second stage maps the view direction and y to colour eq. 3 eq. 4. The position and view-direction inputs are passed through Fourier encodings §3.1. The paper describes this field as living on sparse planes instead of a volume, without further detail §3.1.

2. Dynamic nodes. Each rigidly connected object that moves during capture is a node with its own radiance field, defined in its local frame §3.2. Objects of the same class share weights θ_c and differ through a 256-dimensional latent vector l_o §3.2 eq. 6.

3. Object coordinate frame. A world point x is mapped into the object frame by x_o = S_o T x, where S_o = diag(1/s_o) and s_o = [L, H, W] is the bounding-box size. Points inside the box then lie in [-1, 1]. The paper says the box is scaled to include the object's shadow §3.2 eq. 8 §5.

4. Object stage one. The density depends only on the object-frame point and the latent code §3.2 eq. 7.

5. Object stage two. The colour also depends on the view direction and on the object's global position p_o. The paper keeps pose out of the density so the density stays consistent across poses §3.2 eq. 9 eq. 10.

6. Background planes. Instead of ray marching the background, use N_s planes parallel to the initial camera image plane, equispaced between the near and far clip. The ray is intersected with each plane §4.1. The paper uses N_s = 6 §5.

7. Object samples. For each object hit by the ray, test the ray against the box (an AABB test, following Majercik et al.). Between entry and exit take N_d equidistant points. The paper uses N_d = 7 §4.1 eq. 11 §5.

8. Compositing. Sort all samples along the ray eq. 12. The pixel colour is the sum of T_i α_i c_i, where α_i = 1 − exp(−σ_i δ_i), δ_i is the spacing to the next sample and T_i = exp(−Σ_{k<i} σ_k δ_k) eq. 13 §4.1.

9. Training. Sample random rays from all frames. Each ray is linked to its scene graph and its ground-truth pixel. The loss is the squared colour error summed over rays, plus a zero-mean Gaussian prior on each latent code with weight 1/σ² eq. 14 §4.2. Gradients flow to every node the rays intersect. The paper says it balances ray sampling per batch, because the number of gradients a node receives depends on how many of its intersection points the batch contains §4.2.

10. Edits. Changing the edges moves, rotates or removes objects. New scene graphs are built only from the road trajectories observed in training, which the paper calls the driveable space §5.1.

Pseudocode. Ray rendering through the graph, per the steps above:

# Render one ray through the scene graph  [§4.1] [eq. 11-13]
# Input: origin and direction (3,); graph nodes and latent codes
# Output: RGB colour (3,)

def render_ray(origin, direction, graph, latents):
    # Background: sample the N_s = 6 planes between near and far  [§4.1]
    samples = plane_samples(origin, direction, graph.background_planes)
    # Each dynamic node hit: N_d = 7 points from entry to exit  [eq. 11]
    for node in graph.dynamic_nodes_hit_by(origin, direction):
        samples += box_samples(node, origin, direction, num_points=7)
    # Sort all samples by their distance along the ray  [eq. 12]
    samples = sort_by_distance(samples)
    # Density and colour from the node that owns each sample  [§3.1] [§3.2]
    densities, colours = query_nodes(samples, graph, latents)
    # Spacing to the next sample, the last one is unbounded  [eq. 13]
    spacing = distances_between_neighbours(samples)
    # Opacity of each sample  [eq. 13]
    alphas = 1 - exp(-densities * spacing)
    # Share of light not yet absorbed before each sample  [eq. 13]
    transmittance = exclusive_cumprod(1 - alphas)
    # Pixel colour is the weighted sum of the sample colours  [eq. 13]
    return sum(transmittance * alphas * colours)

# One optimisation step on a random ray batch  [§4.2] [eq. 14] [§5]
# Input: rays (N_rand, ...), target colours (N_rand, 3), latent codes
# Output: scalar loss

def training_step(rays, target_colours, graph, latents):
    # Render every ray of the batch through the graph  [§4.1]
    predicted = [render_ray(r, graph, latents) for r in rays]
    # Squared colour error summed over the rays  [eq. 14]
    image_loss = sum(squared_error(predicted, target_colours))
    # Prior weight 1/sigma^2 on each latent; sigma is not stated  [§4.2]
    prior_weight = 1 / sigma**2
    latent_loss = prior_weight * sum(squared_norm(z) for z in latents)
    # The paper trains with Adam  [§5]
    return image_loss + latent_loss

Hyperparameters

Values marked "code" come from the official repository, not from the paper. Check the Paper vs code section before using them.

Results

Author-reported. Setup: three KITTI tracking scenes §5.1. Each scene is trained on its own sequence. Baselines are SRN, NeRF and NeRF + time (a time input added to NeRF), trained with the forward-facing configuration §5.2. Novel composition: each method is trained with one frame left out per sequence §5.2.

The text reports no numbers for the qualitative results, the tracker comparison fig. 9, the view synthesis fig. 8 or the 3D detection §6 fig. 11. Those are described in words only.

Paper vs code

Code at 8d3d9ce9064ded8231a1374c3866f004a4a281f8 (main branch, shallow clone). The project page links this repository §5. Citations below point to the repository.

• Image loss scale. Paper: a sum of squared errors over the ray batch §4.2 eq. 14. Code: a mean over the batch and the RGB channels main.py neural_scene_graph_helper.py. The relative weight of the latent term therefore changes with the batch size.
• Latent penalty. Paper: (1/σ²)·‖z‖², a squared norm §4.2 eq. 14. Code: latent_balance·‖z‖₂, an unsquared norm main.py neural_scene_graph_helper.py. The example config uses 0.0001 config_kitti_0006_example_train.txt. The parser default is 0.01 main.py.
• Learning-rate schedule. Paper: linear decay §5. Code: exponential decay, factor 0.1 every 250 000 steps main.py config_kitti_0006_example_train.txt.
• Training length. Paper: about 350k iterations to converge §5. Code: the loop runs to 1 000 000 and saves weights every 10 000 steps main.py main.py. I read the loop to main.py#L1775 and found no stopping test there.
• Box scale. Paper: s_o is the box size, scaled to include shadows §3.2 §5, no factor given. Code: length and width ×1.9 in the example config, height unchanged config_kitti_0006_example_train.txt prepare_input_helper.py. Objects with label 4 get ×1.2 regardless of box_scale prepare_input_helper.py.
• Object origin. Paper: x_o = S_o T x with no vertical offset eq. 8. Code: the reference point is first moved down by half the box height, marked "vkitti2 specific" neural_scene_graph_helper.py.
• Density noise. Paper: not mentioned §3.2. Code: Gaussian noise added to the density during training, std 1.0 in the example config main.py config_kitti_0006_example_train.txt.
• 3D detection (§6). Paper: anchor positions and latent codes are optimised with an ℓ1 image loss §6. I did not find this optimiser in the repository. The only detection-related switch is --obj_detection, which makes the networks non-trainable main.py.
• README. The Quick Start uses .py config names README.md README.md. The files shipped are .txt, and the Training section uses .txt README.md.
• Framework. The code needs TensorFlow 1.15 requirements.txt and uses TF1-era calls such as tf.log neural_scene_graph_helper.py and tf.contrib.summary main.py. It will not run on TF2 without porting.
• Data. The example config loads only the image_02 folder config_kitti_0006_example_train.txt. The paper says each sequence comes from two camera views §5.1.
• Licence. The only LICENSE file is nerf_license/LICENSE (MIT, 2020, bmild), which sits in a subfolder LICENSE. The repository's own code has no LICENSE, so no code is quoted here.

Pitfalls

• Missing values. Learning rate, batch size, latent prior weight, near and far clip, Fourier frequencies and layer counts are not in the main text §4.2 §5. The paper defers them to a supplement that is not part of the source we have. The code values in the table are the only concrete numbers, and they may not match the paper's runs.
• Density and transmittance. The paper writes α = 1 − exp(−σδ) with σ used directly eq. 13. The code applies ReLU to the raw density before this main.py and a sigmoid to colour main.py. The code also sets the last spacing to 1e10 main.py and scales spacing by the ray direction's norm main.py. The paper defines δ_i = t_{i+1} − t_i and says nothing about the last sample or the scaling eq. 13.
• Sample count varies per ray. The number of object samples depends on how many boxes each ray hits §4.2. The paper says gradients per node depend on the number of intersections, and that ray sampling is balanced per batch §4.2. The code handles this with scatter operations and masks main.py.
• Cost. Training takes about one day per scene on one TITAN Xp §5. The README warns that the code needs large memory, and suggests reducing chunk, netchunk or the image size README.md. The paper says efficiency is a benefit §7 but gives no render timings in the text.
• View extrapolation. The novel-view result moves the ego camera about 2 m forward §5.1 fig. 8. The paper calls larger extrapolation future work §7.
• Camera-only tracking. Objects at long distance degrade with a camera-only tracker §5.1 fig. 9.
• Possible copy in the table. In the novel-composition block, the SRN row repeats the reconstruction values exactly (18.83, 0.590, 0.456) tab. 1. The paper does not say whether these were re-measured. Check against the supplement before quoting this row.
• Metrics. The temporal metrics tOF and tLP come from TecoGAN §5.2. The repository has a metrics.py metrics.py, which I did not check against those definitions.
• Equation numbers. Equations are numbered here by counting the LaTeX source in order, with each row of an align environment counted separately. Check against the compiled PDF before citing a number.

Lineage

• Builds on: NeRF (Mildenhall et al., 2020; arXiv:2003.08934 per the paper's bibliography) §3; Fourier features (Tancik et al., 2020) §3.1; DeepSDF (Park et al., 2019) for latent codes and the Gaussian prior §3.2 §4.2; multi-plane sampling (Zhou et al., 2018) §4.1; the AABB ray test (Majercik et al., 2018) §4.1.
• Compared against: SRN (Sitzmann et al., 2019), NeRF, and NeRF + time §5.2.
• Superseded by: not known from wiki content.