HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

In plain words

6-DoF video lets a viewer move their head through a recorded scene. Volume methods can produce this, but they usually ask for hundreds of lookups into the scene for every ray of light, and that makes them slow or heavy in memory.

HyperReel splits the work. A small neural network looks at each camera ray and the current time, and decides a handful of places along that ray where the scene surface probably is. Only those few places are then looked up in the scene's stored volume, so each pixel needs few lookups. The points can also shift a little depending on the viewing direction, which helps with shiny or see-through objects. For video, the scene is stored as snapshots every few frames, and the points are moved between snapshots so any moment can be shown.

The result is a format that renders a few frames per second at megapixel size with ordinary deep-learning code, and stays small per frame. The card's numbers show that it is still far from the speed needed for VR.

TL;DR

• Replaces dense per-ray sampling with a ray- and time-conditioned network that predicts a few sample points per ray, placed on z-planes (forward-facing scenes) or spherical shells (other scenes) §3.1.
• Stores dynamic scenes as TensoRF-style keyframes every 4 frames, with predicted scene flow to move points between keyframes §3.2.
• Trains only on the input images with an L2 rendering loss plus L1 and total-variation regularisers §3.3. Author-reported: highest PSNR of the dynamic methods in the comparison table on all three datasets, at about 4 FPS for megapixel images, implemented in vanilla PyTorch. Neural 3D Video has higher SSIM and LPIPS on Technicolor and Neural 3D Video tab. 2.

Problem

Volume rendering needs many samples per ray for accurate images, and fast methods that use importance sampling or pruning still often need hundreds or thousands of queries per ray §3.1. Existing 6-DoF video formats trade off between quality, speed and memory: some are memory-efficient but take close to a minute per megapixel frame, some are fast but need gigabytes, and most do not capture strong view-dependent effects such as reflections and refractions §1. The method assumes that most scenes are made of solid surfaces, so only a few samples per ray carry weight §3.1.

Method

3.1 Sample prediction network

A feed-forward network maps a ray and time to the parameters of geometric primitives: axis-aligned z-planes for forward-facing scenes or concentric spheres at the origin for other scenes §3.1 §4 app. D.3 app. D.4. Each ray is intersected with each primitive, and the intersection points are the sample points eq. 4 eq. 5. The primitives are defined in world space, so two rays that see the same point can share one primitive §3.1. Rays are encoded in two-plane form for forward-facing scenes and in Plücker form otherwise eq. 3.

The network also outputs, for each sample point, a Tanh-activated 3-D offset and a scalar that is turned into a gate. The offset is scaled by the gate and added to the point eq. 6 eq. 7. Colour scale and shift terms are predicted per sample point to correct inconsistent white balance across views app. D.6.

# Render one ray at time tau  [§3.2.3] [eq. 13]
# Input:  origin (3,), direction (3,), time tau (scalar)
# Output: rgb (3,)

def render_ray(origin, direction, tau):
    # Sample network maps ray and time to primitive parameters  [§3.1]
    params = sample_net(encode_ray(origin, direction), tau)
    # Intersect the ray with each primitive  [§3.1] [eq. 4] [eq. 5]
    points = intersect_primitives(origin, direction, params)  # (N, 3)
    # Gated point offsets  [§3.1] [eq. 6] [eq. 7]
    gate = gate_fn(params.delta)                              # (N, 1)
    points = points + gate * params.offset                    # (N, 3)
    # Advect toward the nearest keyframe, one forward-Euler step  [eq. 12]
    tau_key = nearest_keyframe(tau)
    points = points + params.velocity * (tau_key - tau)       # (N, 3)
    # Keyframe volume gives colour and density  [§3.2.2] [eq. 10] [eq. 11]
    colour, sigma = query_keyframe_volume(points, tau_key, direction)
    # Quadrature over the N samples  [eq. 2] [eq. 13]
    weights = transmittance_weights(sigma, points)            # (N,)
    return sum(weights * colour)                              # (3,)

gate_fn is the sigmoid of the scalar in the paper §3.1. The code uses a different expression, which is listed under Paper vs code. The velocity sign convention follows the paper as written eq. 12. The code's sign is the opposite, and this is also listed under Paper vs code.

3.2 Keyframe-based dynamic volume

The volume is a TensoRF-style factorisation: three planes and three lines for appearance, and the same layout for density, with spherical-harmonic coefficients for colour §3.2.1 eq. 8 eq. 9. For dynamic scenes, the lines are indexed by time as well as one spatial coordinate, so every keyframe shares the planes §3.2.2 eq. 10 eq. 11. The paper notes that this costs about as much memory as a single static frame when the number of keyframes is small relative to the spatial resolution §3.2.2.

3.2.3 Rendering

Sample points are advected to the nearest keyframe with a single forward-Euler step, using predicted velocities eq. 12. Colour and density come from the keyframe volume at those advected points, and the usual quadrature gives the pixel eq. 13. The paper calls this a backward warp with a scene-flow field §3.2.3.

3.3 Optimisation

The total loss is an L2 term plus L1 on the density tensors and total variation on both tensors eq. 14 eq. 15. The L2 term is written as a sum over training rays and times eq. 15. Training uses a random subset of rays to fit in memory. In dynamic experiments, for frame numbers divisible by four the sampler alternates between all rays and rays from images downsampled by four. Other frames are downsampled by eight §3.3 alg. 1.

Training ray subsampling alg. 1

# Build the training ray set  [alg. 1]
# Input:  N videos, M frames each
# Output: rays_gt (K, 6), colours_gt (K, 3)

def build_training_rays(videos):
    # Keep rays and colours for every frame of every video  [alg. 1]
    rays_gt = []
    colours_gt = []
    for video in videos:
        for m in range(1, M + 1):
            # Frame m's pixels and their rays  [alg. 1]
            colours = get_frame(video, m)
            rays = get_rays(video, m)
            # Frames not divisible by 8 are downsampled by 4  [alg. 1]
            if m % 8 != 0:
                colours = nearest_downsample(colours, 4)
                rays = nearest_downsample(rays, 4)
                # Frames not divisible by 4 are also downsampled by 2  [alg. 1]
                if m % 4 != 0:
                    colours = nearest_downsample(colours, 2)
                    rays = nearest_downsample(rays, 2)
            colours_gt.append(colours)
            rays_gt.append(rays)
    return rays_gt, colours_gt

Hyperparameters

Appendix numbering follows the compiled paper: A is the appendix overview, B the website overview, C the evaluation details, D the sample network, E the keyframe volume, and F to H the extra comparisons and results.

Results

Author-reported. Setup: Technicolor, 5 sequences, 50 frames each, 2048 × 1088 pixels §4.2; Neural 3D Video, 6 sequences, downsampled by 2 §4.2; Google Immersive, 7 scenes §4.2. FPS is per megapixel tab. 2. Memory is MB per frame tab. 2. Neural 3D Video and StreamRF were evaluated only on one sequence (flame salmon) in their own papers tab. 2 footnote.

Dynamic comparisons tab. 2

Text claims: on Technicolor the paper reports about 1.5 GPU hours per sequence against more than 1000 GPU hours for Neural 3D Video §4.2. On Neural 3D Video it reports about 40 times faster rendering than NeRFPlayer §4.2. On Google Immersive it reports a 3 dB gain over NeRFPlayer §4.2.

Static comparisons, DoNeRF tab. 1

The text reports 6.5 FPS at 800 × 800 and 29 FPS for the Tiny model §4.1.

Ablations, Technicolor tab. 3

Point offsets tab. 4

Appendix comparisons

• LLFF 1008 × 756: NeRF 26.5 dB at 0.3 FPS; HyperReel 26.2 dB at 4.0 FPS and 58.8 MB per frame app. F. HyperReel is slightly below NeRF on this dataset, which the paper attributes to calibration error and sparse views app. F.
• DeepView on the Spaces dataset: DeepView 31.60 dB, SSIM 0.965, LPIPS 0.085, more than 100 FPS; HyperReel 35.47 dB, SSIM 0.968, LPIPS 0.080, 4.0 FPS app. F.

Paper vs code

Searched: the paper's project page links the repository https://github.com/facebookresearch/hyperreel, which is the code of the authors' lab, and the paper's code field records it. Checked at commit c38063a02bc2549549a80efbce9bb71ddf4a748b, licence MIT. The configs for the Technicolor dynamic setting were read, and the sample network, advection, point-offset, regulariser and optimiser code were checked. Other datasets and configs were not compared line by line.

1. Sample network learning rate. Paper: 0.0075 app. D.1. Code: the sample MLP is in the embedding_impl group technicolor_z_plane.yaml, which has learning rate 0.00075 technicolor_tensorf.yaml. The embedding group at 0.01 technicolor_tensorf.yaml may cover other parameters, which was not verified. An implementer who copies the paper's value gets 10 times the code's rate.

2. Final grid size. Paper: 640³ app. E. Code: N_voxel_final 512,000,000, which is 800³ technicolor_z_plane.yaml. The initial 128³ grid matches.

3. Point-offset gate. Paper: displacement is γ(δ)·e with γ a sigmoid, and δ is initialised negative so the gate starts near zero §3.1 eq. 6. Code: offset = tanh-activation × 0.25 × (1 − sigmoid(δ + 4)) point.py technicolor_z_plane.yaml technicolor_z_plane.yaml. Under this form, a negative δ opens the offset and a positive δ closes it. That is the reverse of the paper's negative-initialisation argument. The code also forces the gate to 1 (offset zero) at the start, with an ease-in over 3 epochs after a 1-epoch wait technicolor_z_plane.yaml. The paper does not describe that ease-in.

4. Advection sign. Paper: x ← x + v(τ_i − τ) eq. 12. Code: points + v × (τ − τ_i), where time_offset = t − base_t point.py point.py. As written, the sign is opposite. The paper calls the step a backward warp. A flow fitted under one convention will be wrong under the other. The code also scales the flow output by 0.25 twice, once in the network output activation technicolor_z_plane.yaml and once in the flow block technicolor_z_plane.yaml. The paper states neither factor.

5. Reconstruction loss. Paper: L2 written as a sum of norms, summed over rays and times §3.3 eq. 15. Code: mean squared error technicolor_tensorf.yaml losses.py. The paper's text gives no normalisation, so the scale of the loss relative to L1 and TV is not comparable.

6. L1 schedule. Paper: decays from 8e-5 to 4e-5 over 30,000 iterations app. D.1. Code: 8e-5 until iteration 4000, then a single step to 4e-5 that stays constant tensorf.py tensorf.py.

7. TV schedule. Paper: decays by 0.1 every 30,000 iterations app. D.1. Code: per-iteration factor of 0.1^(1/30000) tensorf.py, applied while the iteration is at most total_num_tv_iters tensorf.py tensorf.py. When unset, that cutoff is 120,000 iterations. This matches the paper's intent, but the cutoff is not stated in the paper.

8. Training length. Paper: about 1.5 GPU hours per 50-frame chunk §4. Code: training config says num_iters 4000 and num_epochs 40 technicolor_tensorf.yaml. The Technicolor run script sets num_epochs to 80 run_one_technicolor.sh. The regulariser config uses n_iters 30000. The iteration count is therefore not single-valued in the code, and the paper gives no iteration count.

9. Ray positional encoding. Paper: 1 frequency app. D.2. Code: 0 frequencies with identity kept technicolor_z_plane.yaml.

10. Components not in the paper. Code adds a skip connection at layer 3 technicolor_z_plane.yaml, a ReLU density with distance scale 16 technicolor_z_plane.yaml, and an output scale of 0.25 for offsets technicolor_z_plane.yaml. The paper states none of these values.

11. Component counts. Paper: 8 components for the (x,y) and (z,t) textures and 4 for the rest §4. Code: n_lamb_sigma [8,0,0] technicolor_z_plane.yaml. The mapping of these lists onto the paper's texture names was not verified.

12. Consistent items. Keyframe interval 4 §4, 50-frame chunks §4, 32 z-planes §4, 16,384 rays per batch app. D.1, volume learning rate 0.02 app. D.1, the upsampling schedule app. E, TV weight 0.05 app. D.1, L1 on density only app. D.1, and the colour scale and shift output app. D.6 all agree.

Pitfalls

• Sign and gate conventions. The advection sign and the gate form both differ between paper and code (items 3 and 4 above). An implementation written from the paper alone will move points the wrong way and gate them inversely, so check it against the code or against the rendered output.
• Learning rate for the sample network. The paper's 0.0075 is 10 times the code's 0.00075 for the same MLP group. Using the paper's value is untested in the code's settings.
• Schedules are not one number. L1, TV and learning-rate decay each have their own cut-offs, and some are step functions instead of ramps. The paper's description of them is approximate.
• Per-chunk training. Each 50-frame chunk is a separate model. Memory per frame (1.2 MB for dynamic scenes) only holds if the chunks are stored and rendered independently. The paper's static per-frame figures (58.8 MB) are a different order of magnitude.
• FPS normalisation. Tables give FPS per megapixel tab. 2 tab. 3. The text reports 6.5 FPS at 800 × 800 §4.1. Convert before comparing.
• SSIM is not comparable across papers. The paper uses scikit-image SSIM with data range 1. Some baselines use a data range of 2 or their own implementation, which inflates their scores app. C.3.
• LPIPS variant. AlexNet LPIPS is used throughout, as are all baselines in the paper app. C.2.
• Generalisation outside the convex hull. The sample network is supervised only by the rendering loss, and sample points need not be consistent across views. Quality drops for views outside the training cameras' convex hull, and for content seen from few views §5. Temporal jitter and noisy geometry appear in such cases §5.
• No streaming. The keyframe representation cannot be streamed the way NeRFPlayer or StreamRF can §5.
• Speed. The implementation is vanilla PyTorch, without custom CUDA. The paper says it falls short of the 72 FPS in stereo that VR needs §5.
• Refraction and reflection. The point offsets help with refractive scenes §4.3 tab. 4. The offset gate is the piece most likely to be misimplemented (see Paper vs code, item 3).

Lineage

• Builds on TensoRF, cited in the paper under the key ChenXGYS2022 §3.2.1. The paper gives no arXiv id for it, so none is recorded here.
• Compared against sampling-network methods: DoNeRF, AdaNeRF, TermiNeRF and NeuSample §3.1 §4.1. Also compared against static methods R2L, NeRF, Instant NGP, and against the 6-DoF video methods Neural 3D Video, NeRFPlayer and StreamRF, and the DeepView baseline §4.
• Related sampling ideas: Eikonal fields are described as a special case of the warp idea §3.1.
• Superseded by: not known from content already in this wiki.