Monocular Dynamic View Synthesis: A Reality Check

In plain words

Many methods reconstruct a moving scene from a single camera video and then render it from new viewpoints. To score them, the usual test set is built from several cameras filmed at the same time. The training video is made by jumping between those cameras frame by frame, so the model quietly gets multi-camera information that a real phone could never record. The scores look better than they would for a real single-camera capture.

The paper fixes this in three ways. It measures how much multi-camera signal a video contains, based on how fast the camera moves compared with the scene. It scores images only on pixels that the training video actually saw. It also checks whether tracked points land in the right place, not just whether the picture looks right. Finally, it adds a new dataset recorded with one handheld phone, with more varied and faster motion. On that setup, the leading methods score noticeably lower, which the authors take as a sign that the field has more room to improve than the old numbers suggested.

TL;DR

• Existing monocular dynamic view-synthesis benchmarks train on frames that "teleport" between cameras, so they leak multi-view information §1, §4.1.
• The paper defines two effective multi-view factors (EMFs), a full one (Ω) and an angular one (ω), that quantify this leak §3.
• Under non-teleporting training, Nerfies and HyperNeRF lose about 1–2 dB of masked PSNR on the rectified Nerfies-HyperNeRF data §5.1; all methods score lower on the new iPhone dataset §5.2. All numbers are author-reported.

Problem

Dynamic view synthesis (DVS) from a monocular video asks for novel views at times where the test camera sees parts of the scene that the single training camera never saw §1. Because of that, the usual evaluation needs a second, simultaneous camera, and the training sequence is built by alternating frames from several cameras §1, §4.1. The authors call this camera teleportation. They argue that it turns a monocular problem into an effectively multi-view one. Standard full-image PSNR, SSIM and LPIPS then reward models for multi-view cues, and they do not measure whether the inferred deformation is correct §4.2.

Method

The paper contributes four pieces. Each is described below in the paper's own terms, with the code's version noted in Paper vs code.

1. Effective multi-view factors (§3). A monocular video behaves like a multi-view capture when the camera moves much faster than the scene.

• Full EMF Ω is the expected ratio of camera displacement to 3D scene flow between consecutive time steps, averaged over visible surface points eq. 1 §3. The scene flow is estimated from optical flow, depth and cameras; the appendix describes RAFT flow, DPT monocular depth aligned to COLMAP points, and a foreground mask from a video segmentation network app. A.
• Angular EMF ω assumes a single look-at point a (triangulated from the optical axes). It is the mean angle between consecutive view directions to a, scaled by the frame rate N eq. 2 §3. It needs only camera poses and frame rate, and the paper recommends reporting it always and Ω when possible §3. The appendix lists the assumptions behind ω: roughly constant object speed, the camera fixating the object, and roughly constant distance app. A.

2. Co-visibility masked image metrics (§4.2, app. B). For each test frame, a pixel is kept if it is visible in enough training frames. Visibility comes from a forward-backward optical-flow occlusion test with RAFT app. A, B. mPSNR, mSSIM (via partial convolution, following Liu et al.) and mLPIPS are computed only over the kept pixels §4.2, app. B.

3. PCK-T correspondence metric (§4.2, app. C). Keypoints are annotated on input frames so that each is visible during training. For each method, a 2D correspondence is read out between frames. For NeRF models with a canonical space, this uses root finding (Broyden) through the forward warp, and for NSFF it chains the predicted scene flow through time app. C. A keypoint counts as correct if it lands within a threshold of the ground truth §5.

4. iPhone dataset (§4.3, app. D). Fourteen sequences of non-repetitive motion (objects, humans, pets). Seven are multi-camera captures: one handheld iPhone for training and two static cameras with a wide baseline for validation. Seven are single-camera captures used only for correspondence evaluation. Training frames carry lidar depth from the phone, which is used for supervision. Keypoints are annotated by hand, using skeletons for humans and quadrupeds and 5–15 points for general objects app. D.

Reality-check protocol (§5). Models are retrained on the rectified Nerfies-HyperNeRF sequences with only the left camera (no teleportation), using the same held-out frames and co-visibility masks §5.1. On the iPhone dataset, a regularised variant adds random background compositing (+B), a depth loss (+D) and a sparsity term on the scene surface (+S) to HyperNeRF §5.2, ablation figure. Those variants are reported as "++" in the appendix figures app. I.

Pseudocode for the angular EMF, as written in the paper eq. 2:

# Angular EMF omega  [eq. 2] [§3]
# Input:  camera_positions (N, 3); lookat (3,) triangulated from optical axes
# Output: omega, scaled by the frame rate fps

def angular_emf(camera_positions, lookat, fps):
    # 1. Unit view direction from each camera to the look-at point  [§3]
    viewdirs = lookat - camera_positions          # (N, 3)
    viewdirs = viewdirs / norm(viewdirs)          # (N, 3), unit length
    # 2. Angle between consecutive view directions  [eq. 2]
    cosines = dot(viewdirs[:-1], viewdirs[1:])    # (N-1,)
    angles = arccos(clip(cosines, -1, 1))         # (N-1,), radians
    # 3. Mean angle over pairs, scaled by the frame rate  [eq. 2]
    return mean(angles) * fps

Pseudocode for the co-visibility mask, as written in the paper app. B:

# Co-visibility mask for one test frame  [app. B]
# Input:  occluded (N, H, W): 1 where a test pixel is occluded in train frame i
# Output: mask (H, W), 1 where the pixel counts as seen during training

def covisibility_mask(occluded, num_train):
    # 1. Count the training frames in which each test pixel is visible  [app. B]
    seen_count = sum(1 - occluded, axis=0)        # (H, W)
    # 2. Threshold beta = max(5, 0.1 N), with N training frames  [app. B]
    beta = max(5, 0.1 * num_train)
    # 3. Keep pixels seen in at least beta frames  [app. B]
    return (seen_count >= beta).astype(float32)   # (H, W)

The paper states the forward-backward occlusion test as an inequality on the chained forward and backward flows, with a relative tolerance of 0.01 and an absolute tolerance of 0.5 app. A. The code implements the same test and is cited in Paper vs code.

Hyperparameters

Results

Author-reported. Setup: Nerfies-HyperNeRF (7 multi-camera sequences, rectified to non-teleporting training) and the proposed iPhone dataset (7 multi-camera sequences). Baselines are the other models in the table, trained under the same recipe §5.

Rectified Nerfies-HyperNeRF, non-teleporting training, Ω = 1.30, ω = 51.53 §5.1, benchmark table:

Proposed iPhone dataset, Ω = 0.24, ω = 15.18 §5.2, benchmark table:

Headline findings, as the paper states them:

• Training with teleportation inflates every metric except PCK-T for NSFF §5.1, impact-of-multi-view figure. The paper reports a 1–2 dB masked PSNR drop and about 5% PCK-T drop for existing datasets without teleportation §1.
• Nerfies drops 4.4 dB in mPSNR, rises 69.6% in mLPIPS and falls 40.1% in PCK-T from the Nerfies-HyperNeRF numbers to the iPhone numbers §5.2.
• The paper reports that the +B/+D/+S regularisers consistently help, with random background compositing the most helpful §5.2, ablation figure. The ablation numbers are only in the figure, not in a table. The text does not state which variant the iPhone benchmark table reports.
• Per-sequence iPhone EMFs range from ω = 3.79 to 58.45 °/s and Ω = 0.03 to 0.75 app. A table. The multi-camera Nerfies and HyperNeRF sequences have ω between 110.51 and 251.37 °/s app. A table.

Paper vs code

Official repository: https://github.com/KAIR-BAIR/dycheck (Apache-2.0), commit ddf77a4e006fdbc5aed28e0859c216da0de5aff5, cloned from the default branch. The arXiv abstract links the project page hangg7.com/dycheck; the repository's README links back to the paper. The project page itself could not be read during compilation, so the link chain is partly inferred from the README.

Differences, each with both citations:

• Fine samples. Paper: 2S = 256 fine samples per ray for Nerfies and HyperNeRF app. E. Code: num_fine_samples defaults to 128 dynerf.py, and no gin config under configs/nerfies/dense or configs/iphone/dense overrides it.
• Co-visibility threshold. Paper: a pixel is kept when seen in at least β = max(5, 0.1 N) training frames app. B. Code: a pixel is kept when the visible count is strictly greater than max(5, int(0.1 · S)) process_covisible.py. The code uses a strict inequality and integer truncation of 0.1 S.
• Support set. Paper: N is the number of training frames app. B. Code: S is the length of the support set, which is the parser's full frame list process_covisible.py, not the training split.
• Image alpha. Code additionally requires the image alpha to be 255 when building the co-visibility mask nerfies.py. The paper does not mention alpha.
• mPSNR. Paper: mPSNR is the average of per-pixel PSNR over the kept region app. B. Code: PSNR is computed from the mean squared error over the kept region, which is a single PSNR value image.py. These differ, and the code's number is generally not the paper's definition.
• PCK-T threshold. Paper: α = 0.05 with no reference length §5. Code: threshold = 0.05 × max(image width, image height) keypoint.py.
• +S regulariser. Paper: "a sparsity regularization for scene surface" citing Mip-NeRF 360 §5.2. Code: the weight-0.01 term is the Mip-NeRF 360 distortion loss dist.py, enabled by train_step.use_dist_loss randbkgd_depth_dist.gin. The code also requires use_sample_at_infinity to be off when using the depth or distortion loss [code: dycheck/core/training.py#L68-L71, dycheck/core/training.py#L81-L84]. The random-background config turns it off randbkgd.gin.
• +D depth loss. Paper: "a depth loss on the ray matching distance" §5.2. Code: mean squared error between predicted and sensor depth, masked where depth is 0 depth.py.
• Full EMF estimator. Paper: one expectation over the ratios app. A. Code: the forward and backward ratio estimates are averaged emf.py.
• Angular EMF units. The code multiplies the mean arccos by 180/π and the frame rate, so the output is in degrees per second [code: dycheck/processors/emf.py#L63-L70]. The paper's equation is written without the conversion; the tables use °/s.
• Warp-alpha annealing. The code anneals a warp-alpha weight from 0 to 6 over 80,000 steps for the Nerfies-style models randbkgd_depth_dist.gin. The paper's implementation details do not mention it app. E.
• NSFF. The paper reports results from a third-party re-implementation because the official NSFF code failed to converge on the iPhone data §5. This repository has no NSFF training configs (configs/ contains ambient, dense and tnerf for each dataset). NSFF results therefore cannot be reproduced from this repository alone. The mapping of config names to the paper's model names is not stated in the paper; the names are used here as they appear.

Checked and consistent: batch size 6144, 250k iterations, learning rate 1e-3 to 1e-4, Adam, and the coarse sample count of 128 app. E match configs/nerfies/dense/mono.gin and dycheck/models/dynerf.py.

Pitfalls

• Teleporting must be removed from training, not only from evaluation. The rectified protocol trains on the left camera alone §5.1. Evaluating on the multi-camera splits with the old training data reproduces the inflated numbers the paper criticises §1.
• Co-visibility masks are noisy. The paper accepts this on purpose: a conservative threshold improves recall when flow is noisy, and the chicken example shows an occluded but visible region kept by the mask app. B. Results depend on the threshold and on RAFT, so an implementer should copy the threshold from the code, not from the prose.
• Masked LPIPS can be artificially low. The mask is applied by zeroing unseen pixels before AlexNet, and AlexNet's receptive field is 195 pixels. When the mask is small, the score is biased low app. B.
• EMF depends on estimated depth and flow. Ω needs RAFT, DPT and segmentation, so it is sensitive to their errors. The paper therefore adds ω, which needs only cameras and frame rate, but ω assumes a fixated look-at point app. A.
• Frame rates are hand-picked. For the existing datasets, the paper does not know the frame rate and chose it by reassembling the video until motion looked natural, then checked with the original authors app. A. ω changes linearly with fps, so per-sequence values carry this uncertainty.
• Keypoint annotation is manual. Keypoints are placed by hand on single-camera captures and are visible during training app. D. PCK-T results therefore depend on the annotators and on the reference length in Paper vs code.
• Background regularisation is fragile. Nerfies and HyperNeRF's background points need a manually cleaned point cloud, which the authors call a weakness of that regularisation app. D.
• Sensitivity to NDC. NSFF samples in normalised device coordinates, which the paper says causes layered artefacts for inward-facing scenes §5. The paper does not apply the iPhone enhancements to NSFF because of this §5.2.
• Cost. Training takes about 12 h for T-NeRF and 24 h for Nerfies or HyperNeRF on 4 A4000 GPUs or 2 A100s app. E. The NSFF number lacks a unit.
• Unstated details. Network width and depth, and the ablation numbers, are not stated in the paper's text (the ablation numbers appear only in a figure). Network details are available only in the code, which this card does not transcribe.

Lineage

• Builds on: the Nerfies and HyperNeRF deformation models, NSFF's scene flow and T-NeRF as the time-conditioned baseline §2, §5. These are named in the paper only; no cards exist for them in this wiki yet.
• Compared against: T-NeRF, NSFF (third-party re-implementation), Nerfies, HyperNeRF §5.
• Regularisers borrowed: random background compositing, depth supervision and the Mip-NeRF 360 distortion loss [§5.2, Paper vs code].
• Superseded by: not stated in this wiki.