Editable Free-viewpoint Video Using a Layered Neural Representation

In plain words

A free-viewpoint video lets a viewer move the camera around a recorded scene after the fact. This paper records a stage with 16 cameras spread over a half circle, and the result can be watched from many angles and also edited.

The trick is to split the scene into layers: one layer for each performer and one for the background. Each layer is learned as its own neural model that represents its shape and colour, and also how it moves over time. Because the layers are kept separate, you can move one performer, copy them, play part of their motion earlier or later, or make them fade, and the other layers stay as they were. When the frames are drawn, the layers are blended together in depth order, so a performer that walks behind another is hidden correctly.

TL;DR

• Neural free-viewpoint video of large, dynamic scenes with several performers, from 16 synchronised RGB cameras covering up to 180 degrees §1 §3.
• Each dynamic entity and the background is a space-time neural radiance field (ST-NeRF); a deformation MLP maps points at time t into a shared canonical space §4.2.
• Editing (affine moves, duplication, retiming, transparency, insertion and removal) is done by changing the inputs to each layer, without retraining §5.
• Author-reported: PSNR 33.2161 on their captured dataset vs 28.2553 for NeRF-T, the best PSNR baseline tab. 1.
• The released code covers the rendering and training loop but not the scene parsing stage, and its training settings differ from the paper (see Paper vs code).

Problem

Novel-view synthesis for dynamic scenes either reconstructs a single moving object or handles only a static background. Methods that do model dynamic humans need dense capture or fail on large motions, and most neural methods are not editable §1. Earlier layered work handles retiming from a single monocular video only, with no 3D spatial editing §1 §6.2. The paper targets large scenes with several performers, captured sparsely with 16 cameras, with editing and free viewing at the same time §1 §3.

The hard parts it names are visibility (parts of one performer are hidden by another in some views) and the imbalance between a mostly static background and moving performers, which makes training slow if rays are sampled uniformly §4.4.

Method

Pipeline. The input is 16 synchronised RGB videos with calibrated cameras §3 §6. The method has four parts: a scene-parsing stage that produces per-frame label maps and 3D bounding boxes for every performer and the background §4.1 fig. 3; one ST-NeRF per layer §4.2 fig. 5; a renderer that segments each ray by layer box and composites all layers together §4.3 fig. 6 fig. 7; and a training scheme with a layered loss and motion-aware ray sampling §4.4. Editing reuses the trained layers by changing their inputs §5 fig. 9.

Scene parsing (§4.1). Depth is estimated per view with a patch-based multi-view stereo method, at half the input resolution §4.1 fig. 3. Each performer is tracked in every view with SiamMask, one tracker per object, and the 2D tracks are corrected with a trajectory prediction network §4.1. The correction is a weighted average over the cameras whose tracker confidence passes a threshold τ eq. 1:

{\bf g}_t^{b'} = q^b {\bf g}_t^b + \frac{1-q^b}{w} \sum_{c,\, q^c \geq \tau} q^c\, \Theta_{\mathrm{TP}}(\mathcal{G}_{t_0,t}^c, \mathcal{G}_{t_0,t_1}^b), \quad w = \sum_{c,\, q^c \geq \tau} q^c \qquad [\text{eq. 1}]

Here q^c is the tracker's confidence in view c, and Θ_TP is the trajectory prediction network §4.1. Silhouettes from the 2D boxes are carved into a coarse 3D shape, and a 3D box is fitted around it §4.1. Label maps are refined by discarding pixels whose depth is far from the performer's average depth in the previous frame §4.1. The background box is taken from the whole scene point cloud §4.1.

ST-NeRF per layer (§4.2). Each layer i is a continuous function of position p, view direction d and time t. A deformation MLP φ^d predicts an offset Δp from (p, t), and a radiance MLP φ^r predicts colour and density at the moved point §4.2 eq. 2 eq. 3. Frame index enters through positional encoding rather than per-frame latent codes §4.2:

\phi^{r}(\mathbf{p}+\Delta\mathbf{p}, \mathbf{d}, t, \theta^r) = (\mathbf{c}, \sigma), \qquad \Delta\mathbf{p} = \phi^{d}(\mathbf{p}, t, \theta^d) \qquad [\text{eq. 2}]

\phi(\mathbf{p}, \mathbf{d}, t, \Theta) = (\mathbf{c}, \sigma) \qquad [\text{eq. 3}]

Layered renderer (§4.3). For each camera ray, every layer's bounding box is intersected with the ray. Each intersection gives a depth segment S^i = {s_n^i, s_f^i} §4.3. A segment is valid only if the ray enters and leaves the box §4.3. Each valid segment is split into N evenly spaced bins and one point is drawn uniformly in each bin §4.3 eq. 4:

s_{j}^i \sim \mathcal{U}\!\left[ s_{n}^i + \tfrac{j-1}{N}(s_{f}^i - s_{n}^i),\; s_{n}^i + \tfrac{j}{N}(s_{f}^i - s_{n}^i) \right], \quad j = 1, \dots, N \qquad [\text{eq. 4}]

Points of each layer go to that layer's ST-NeRF §4.3. A second, fine pass draws points by inverse-transform sampling from the coarse weights §4.3. Samples of all valid layers are merged and sorted near to far §4.3 eq. 5. The pixel colour is the standard volume-rendering sum over this merged list §4.3 eq. 6:

\hat{\mathbf{C}}(\mathbf{r}) = \sum_{j=1}^{|\mathcal{P}|} T(\mathbf{p}_j)\left[ 1 - \exp\!\left(-\sigma(\mathbf{p}_j)\,\delta(\mathbf{p}_j)\right) \right] \mathbf{c}(\mathbf{p}_j), \quad T(\mathbf{p}_j) = \exp\!\left( -\sum_{k=1}^{j-1} \sigma(\mathbf{p}_k)\,\delta(\mathbf{p}_k) \right) \qquad [\text{eq. 6}]

with δ(p_j) = p_{j+1} − p_j §4.3. Coarse and fine renderings are both kept for training §4.4. The renderer handles occlusion between layers through this merged compositing §4.3 fig. 7.

Training (§4.4). Rays come from random views and times, 3000 per mini-batch §4.4. The RGB loss is a squared error on the coarse and fine renderings eq. 7:

\mathcal{L}_{rgb} = \sum_{\mathbf{r} \in \mathcal{R}} \left( \lVert \mathbf{C}(\mathbf{r}) - \hat{\mathbf{C}}_c(\mathbf{r}) \rVert_2^2 + \lVert \mathbf{C}(\mathbf{r}) - \hat{\mathbf{C}}_f(\mathbf{r}) \rVert_2^2 \right) \qquad [\text{eq. 7}]

The layered loss pushes each layer's accumulated opacity towards 1 on its own pixels, using only samples inside that layer's segment §4.4 eq. 8:

\mathcal{L}_{layer} = \frac{1}{2} \sum_{i=1}^{n_i} \left\lVert \Omega(\mathbf{r}, \mathbf{L}, i) - \alpha(\mathbf{r}, i) \right\rVert_2^2, \quad \alpha(\mathbf{r}, i) = \sum_{j=1}^{|\mathcal{P}^i|} \exp\!\left( -\sum_{k=1}^{j-1} \sigma_k \delta_k \right) \left( 1 - \exp(-\sigma_j \delta_j) \right) \qquad [\text{eq. 8}]

Ω is an indicator that is 1 when the pixel belongs to layer i in the label map L §4.4. The total loss mixes the two terms eq. 9:

\mathcal{L} = (1-\lambda)\,\mathcal{L}_{rgb} + \lambda\,\mathcal{L}_{layer} \qquad [\text{eq. 9}]

λ is 0.1, 0.05 and 0.01 over three warm-up epochs and 0 afterwards §4.4. The motion-aware sampling makes background and performer rays roughly balanced §4.4; the exact rule is not given.

# Layered ray rendering, one camera ray  [§4.3] [eq. 2] [eq. 4] [eq. 5]
# Input:  origin, direction (3,); frame t; one box per layer
# Output: pixel colour (3,)

def render_ray(origin, direction, t, boxes):
    # Intersect the ray with each layer's box; keep valid segments  [§4.3]
    segments = intersect_boxes(origin, direction, boxes)
    samples = []
    for layer, segment in enumerate(segments):
        if segment is None:
            continue
        # Stratified coarse depths, one draw per bin  [§4.3] [eq. 4]
        depths = stratified_depths(segment, num_coarse)
        points = origin + depths * direction    # (N, 3)
        # Deform into canonical space: p' = p + delta_p  [§4.2] [eq. 2]
        offsets = deform_nets[layer](points, t)
        canonical = points + offsets            # (N, 3)
        # Colour and density from the radiance network  [§4.2] [eq. 2]
        colour, density = radiance_nets[layer](canonical, direction, t)
        samples.append((depths, colour, density))
    # Merge the samples of all layers, sorted near to far  [§4.3] [eq. 5]
    depths, colours, densities = merge_sorted(samples)
    # Composite the merged list into one pixel colour  [§4.3] [eq. 6]
    colour, _ = volume_render(depths, colours, densities)
    return colour

# Fine depths: second draw from the coarse weights  [§4.3]
# Input:  coarse depths (N,); coarse weights (N,); number of fine draws M
# Output: fine depths (M,)

def fine_depths(depths, weights, num_fine):
    # Normalise the coarse weights into a distribution over depth  [§4.3]
    pdf = weights / sum(weights)
    # Cumulative distribution over the coarse depths  [§4.3]
    cdf = cumulative_sum(pdf)
    # Invert the CDF at uniform draws (inverse transform sampling)  [§4.3]
    return inverse_cdf(cdf, uniform(num_fine))

# Volume rendering over merged samples  [§4.3] [eq. 6]
# Input:  depths (M,), colours (M, 3), densities (M,); sorted near to far
# Output: pixel colour (3,), weights (M,)

def volume_render(depths, colours, densities):
    # Spacing to the next sample; the last spacing is not defined  [eq. 6]
    deltas = pad_last(depths[1:] - depths[:-1])
    # Density times spacing, per sample  [eq. 6]
    sigma_delta = densities * deltas
    # Opacity of each sample  [eq. 6]
    alphas = 1 - exp(-sigma_delta)
    # Transmittance: light left after all earlier samples  [eq. 6]
    transmittance = exp(-exclusive_cumulative_sum(sigma_delta))
    weights = transmittance * alphas
    colour = sum(weights * colours)
    return colour, weights

# Training loss for one batch of rays  [§4.4] [eq. 7] [eq. 8] [eq. 9]
# Input:  rays (B,); target colours (B, 3); lam from the warm-up schedule
# Output: scalar loss

def training_loss(rays, target, lam):
    # Squared error of the coarse and the fine rendering  [eq. 7]
    coarse_colour, fine_colour = render_batch(rays)
    rgb_loss = squared_norm(target - coarse_colour)
    rgb_loss += squared_norm(target - fine_colour)
    # Each layer's opacity should be 1 on its own pixels  [eq. 8]
    layer_loss = 0.5 * sum_over_layers(layer_opacity_error(rays))
    # Linear mix; lam is 0.1, 0.05, 0.01 in warm-up, then 0  [eq. 9] [§4.4]
    return (1 - lam) * rgb_loss + lam * layer_loss

Editing (§5). Each layer is driven by its input position and time, so edits are changes to those inputs §5:

• Affine transform A of a layer's box: the box is replaced by A∘B, and sample points and view directions are mapped back by A⁻¹ before the network is queried §5 eq. 10.
• Retiming: the timestamp is mapped to another frame, both for the box and for the network query §5 eq. 11.
• Duplication and insertion or removal of a layer change only the box set that is composited §5.
• Transparency: the density is multiplied by a scalar s, σ' = s·σ §5.

\phi(\mathcal{A}^{-1}\circ\mathbf{p},\; \mathcal{A}^{-1}\circ\mathbf{d},\; t, \Theta^i) = (\mathbf{c}, \sigma) \qquad [\text{eq. 10}]

\phi(\mathbf{p}, \mathbf{d},\; \mathcal{T}\circ t, \Theta^i) = (\mathbf{c}, \sigma) \qquad [\text{eq. 11}]

Hyperparameters

Values marked "not stated" are absent from the paper. Code values come from the official repository at code_commit. Those come from the two released configs (walking and taekwondo) unless noted.

Results

Author-reported throughout. The setups differ between tables, so the numbers are not directly comparable across tables.

Table 1 compares against baselines on the captured dataset, computed on the captured reference views §6.2 tab. 1. Higher is better for PSNR and SSIM, lower for MAE and LPIPS.

Setup: 16 cameras, eight captured dynamic scenes, metrics averaged over the 16 capture views for each timestamp (PSNR curve for the Walking sequence fig. 12). Visual comparisons are in §6.2 fig. 13 fig. 14.

Table 2 is the component ablation on the synthetic dataset, with a held-out view for PSNR_test §6.3 tab. 2:

Table 3 varies the number of input views on the synthetic dataset, with the central camera held out §6.3 tab. 3:

Editing results are shown as figures and a supplementary video, not as numbers §6.1 fig. 1 fig. 11. The paper reports that its layered approach recovers occluded performers where a layered monocular baseline (LNR) fails §6.2 fig. 14.

Paper vs code

Code: DarlingHang/st-nerf at commit e0c1c32b09d90101218410443193ddabc1f66d2f, shallow clone. Searched GitHub for the method name. The authors' project page (https://jiakai-zhang.github.io/st-nerf/) links this repository, and the README names this paper and its nine authors. The repository has no LICENSE file, so no code is quoted; code is cited for values and behaviour only. The project page says the release is partial ("test part of the code and part of our preprocessed dataset").

What the repository contains: the ST-NeRF model, renderer, sampler, volume renderer, deformation networks, training loop (engine/layered_trainer.py), demo scripts and two checkpoints under outputs/. What it does not contain: scene parsing (multi-view stereo, tracking, label refinement) and a training entry script. The dataset path is a placeholder config_walking.yml.

Differences:

• Learning rate. Paper: 1e-4 decaying to 1e-5 §4.4. Code: base 4e-4, with warm-up over 1000 steps and decay from step 3000, so at step 60000 the rate is about 1.7e-4 and it keeps decaying only slowly afterwards [code: configs/config_walking.yml#L4-L10] lr_scheduler.py layered_trainer.py. Derived from the code formula, not a logged run.
• Loss. Paper: L = (1−λ)·L_rgb + λ·L_layer with a three-epoch λ schedule §4.4 eq. 8 eq. 9. Code: the total is the sum of two mean-squared errors (coarse and fine) with no λ layered_trainer.py layered_trainer.py. The mean squared error is a mean, not the summed squared L2 of eq. 7 loss.py. The layered-loss term is not implemented. The code has a different mask penalty, applied only when REMOVE_OUTLIERS is set (taekwondo config) and only in the first three epochs layered_trainer.py config_taekwondo.yml.
• Coarse-only stage. Code has a branch for COARSE_STAGE with coarse-only loss. Epochs are numbered from 1 layered_trainer.py, so with COARSE_STAGE: 1 config_walking.yml the branch is never taken. The paper describes no coarse-only stage.
• Batch size. Paper: 3000 rays §4.4. Code: 2000 [code: configs/config_walking.yml#L6].
• Time into the radiance network. Paper: φ^r always receives t §4.2. Code: time is passed only when USE_SPACE_TIME is set layered_rfrender.py. The walking config leaves it off config_walking.yml; the taekwondo config turns it on config_taekwondo.yml.
• Background as a layer. Paper: the environment is an ST-NeRF like any performer §3 §4.2. Code: the background network has no time input and no time deformation in either config config_walking.yml config_taekwondo.yml layered_rfrender.py.
• Frame interpolation. Paper: "a simple nearest interpolation scheme" for new timestamps §6.4. Code: linear blend between the two neighbouring frames for the deformation input motion_net.py.
• Background ray sampling. Paper: the ratio is set so background and performer rays are roughly balanced §4.4. Code: a fixed random keep rate per layer, 0.0 (walking, background rays dropped) or 0.05 (taekwondo) [code: data/datasets/ray_dataset.py#L381-L384] ray_dataset.py.
• Image size. Paper: train on 960 × 540 crops, then refine at 1920 × 1080 §6. Configs set SIZE_TRAIN to 1920 × 1080 config_walking.yml; the 960 × 540 stage is not visible in the configs.
• Defaults vs configs. Code defaults (400 × 250 inputs, NEAR_FAR sampling, 64/80 samples) defaults.py defaults.py differ from the released configs, which set BBOX sampling and 90/30 samples. Use the configs.
• Matching parts. The bounding-box ray segmentation, the merged sorted-sample compositing and the affine and retiming input mappings match the paper layered_rfrender.py layered_rfrender.py layered_rfrender.py.

Pitfalls

• Equation, figure and table numbers here follow the order in the LaTeX source (no compiled PDF was checked). Verify against the PDF before quoting.
• Table 1 caption and text overstate the results. The caption says the method is best on all four metrics tab. 1; the text says it outperforms all baselines in PSNR, SSIM and MAE §6.2. The table shows NeRF-T with higher SSIM (0.9243 vs 0.9203), and this paper with the highest MAE of all methods (0.1178 vs 0.0219 for NeRF-T), which is worse, since lower is better.
• Table 1 is measured on input views ("all the captured reference views" §6.2). The held-out evaluation is only on the synthetic dataset §6.3.
• Synthetic camera count conflicts. The dataset section says 36 virtual cameras §6; the view ablation says 17 cameras in the synthetic dataset §6.3. Treat the synthetic setup as unclear.
• Unstated sample counts. N (coarse bins) and the number of fine samples are not given, and the near plane and the density cut-off are not stated either. The code uses 90/30 samples, near 0 and cut-off 1e-4; these are the values the released configs set, not values the paper gives.
• Undefined last spacing. Eq. 6 uses δ_j = p_{j+1} − p_j, which does not exist for the last sample. The code pads it with 1e10 [code: layers/render_layer.py#L19-L23] render_layer.py. With any positive density there, the last sample takes almost all remaining weight. Choose and document a padding rule before reimplementing.
• Activations not stated. The code applies ReLU to density [code: layers/render_layer.py#L11] and sigmoid to colour [code: layers/render_layer.py#L47]. Missing either changes the output.
• Density truncation and near clipping. Densities below 1e-4 are set to 0 layered_rfrender.py and samples before the near plane get zero density layered_rfrender.py. Neither step is in the paper.
• Layer-wise cost. Each layer runs its own networks on its own samples, so cost grows with the number of layers. Training takes 12 to 36 h on one RTX 3090 per scene, plus 2 to 3 days of refinement §6. A 1920 × 1080 frame with three layers takes about 2 min to render §6.
• Limitations the paper states: requires 16 cameras; only humans are tracked; tracking failures on severely occluded performers; a box-limited network cannot model effects outside the box, which causes ghosting when edited; no non-rigid edits and no slow motion; nearest-frame interpolation §6.4.
• Scene parsing is a black box in the code. Label maps and boxes are read from precomputed data, so reproducing the method end-to-end needs the authors' preprocessing, which is only partly released.
• Motion-aware sampling is vague. The paper only says background rays are balanced against performer rays §4.4; the released code uses fixed keep rates instead ray_dataset.py.
• No licence on code. The repository has no LICENSE file, so it cannot be reused or copied from here; only the described behaviour is cited.

Lineage

• Builds on: NeRF (Mildenhall et al. 2020) for volume rendering, the coarse-to-fine sampling and positional encoding §4.2 §4.3. The ray sampling is extended to one segment per layer §4.3.
• Deformation MLP: "similar to" Pumarola et al. 2020 §4.2.
• Scene parsing: patch-based multi-view stereo (Luo et al. 2019), SiamMask (Wang et al. 2019), trajectory prediction network (Wu et al. 2020), shape from space carving (Kutulakos and Seitz 2000) §4.1.
• Compared against: Neural Volumes, NeRF, NeRF-T (the paper's own time-input variant), AGI (Agisoft PhotoScan), HVR (segmentation-based stereo), and Layered Neural Rendering (Lu et al. 2020, monocular input, qualitative only) §6.2.
• Superseded by: none known from this wiki.