HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

In plain words

Filming a person with a large camera rig gives many photos of the same moment from many angles, and a whole clip gives many moments. HumanRF turns such a recording into a 3D model of the person that can be shown from a viewpoint that was never filmed, at any moment of the clip. Its goal is high detail, high enough for film and games, and it is built for very large images.

The main idea is to cut a long clip into time chunks and give each chunk its own compact store of 3D detail. The store is split into compact parts, so only a few chunks need to sit in graphics memory at once. Small shared networks turn the stored values into colour and density, and the model is trained by rendering it and comparing the result with the real photos and with the person's silhouette.

The authors also release a dataset of 12-megapixel footage from 160 cameras, because most earlier public datasets stop at 4 megapixels or less.

TL;DR

HumanRF is a dynamic neural radiance field for full-body humans, trained per sequence from multi-view video. It splits the sequence into segments whose sizes come from a greedy occupancy-growth rule, and gives each segment a 4D feature grid built from four 3D hash grids and four 1D grids. The authors report the best PSNR, SSIM and VMAF against the baselines they tested on 4x-downscaled ActorsHQ, but per-frame Instant-NGP has better LPIPS and uses about 20 times more trainable parameters. Official code exists, but it is non-commercial.

Problem

Dynamic NeRF methods that use a deformation field, or a single representation for the whole sequence, struggle with long sequences with complex motion, especially for humans §1. Their results also degrade on fast-changing scenes §5.2. High-quality output needs high-resolution training data, which is hard to capture and to use §1. Earlier public multi-view human datasets provide images at 4MP or less §5.4 tab. 1. The authors state that single-representation methods do not scale to long sequences §3.2.

The method is supervised with multi-view images and foreground masks for each frame §3, and the main results use the capture rig of 160 synchronised cameras §4. Each sequence gets its own model; the paper does not train one model across sequences §5.6.

Method

HumanRF adds time to Instant-NGP-style hash encodings and factorises the space-time feature grid. The steps are:

1. Partition the sequence into segments with a greedy occupancy-growth rule, so that each segment covers a similar volume §3.2 eq. 2-4.
2. For each segment, store a 4D feature grid as a sum of four Hadamard products, each pairing a 3D multi-resolution hash grid with a 1D dense grid §3.1 eq. 1.
3. Feed the 32-dimensional feature to a density MLP that outputs a density and 15 geometry features §3.3 eq. 6.
4. Feed the geometry features and the first four spherical-harmonic bands of the view direction to a radiance MLP that outputs RGB §3.3 eq. 7. Both MLPs are shared across the whole sequence §3.3.
5. Render each ray with the volume-rendering integral, discretised by quadrature §3.3 eq. 5.
6. Train with a per-pixel Huber loss on colour and a binary cross-entropy loss on the accumulated opacity, weighted by 1e-3 §3.4 eq. 8-11.

Pseudocode for step 1, the segment partition. The rules below are those in the official code; the paper describes only the threshold test and the predefined size pool §3.2 §5.3.3.

# Greedy adaptive temporal partitioning  [§3.2] [eq. 2-4]
# Input:  occupancy grid per frame, threshold gamma = 1.25
# Output: list of segment sizes, each from {6, 12, 25, 50, 100}

def partition_sequence(grids, gamma):
    segment_sizes = []
    start = 0
    while start < len(grids):
        union = grids[start]
        split_found = False
        # Extend the candidate segment one frame at a time  [§3.2]
        for end in range(start, len(grids)):
            # Union of the occupancy grids so far  [eq. 2]
            union = logical_or(union, grids[end])
            count = end - start + 1
            # Do not test expansion before 6 frames
            # [code: humanrf/adaptive_temporal_partitioning.py#L87]
            if count < 6:
                continue
            # Expansion factor against the first frame  [eq. 3-4]
            expansion = (total_occupancy(union)
                         / total_occupancy(grids[start]))
            # Split when the volume grows past gamma, or at 100 frames
            # [§3.2] [code: humanrf/adaptive_temporal_partitioning.py#L90]
            if expansion > gamma or count >= 100:
                split_found = True
                break
        if split_found:
            # Round the count down to a predefined size
            # [code: humanrf/adaptive_temporal_partitioning.py#L29-L34]
            size = largest_predefined_not_above(count)
        else:
            # The sequence ended first: round the rest up
            # [code: humanrf/adaptive_temporal_partitioning.py#L37-L40]
            size = smallest_predefined_not_below(count)
        segment_sizes.append(size)
        # Frames after the cut start the next segment
        # [code: humanrf/adaptive_temporal_partitioning.py#L95]
        start += size
    return segment_sizes

Pseudocode for step 2, the 4D feature query. The hash grids are the multi-resolution hash encoding of Instant-NGP §3.1.

# 4D feature of one segment at a point and a time  [§3.1] [eq. 1]
# p: point in [0, 1]^3 (code adds 0.5 to a [-0.5, 0.5] box)
# tau: local frame index / segment length, in [0, 1)
# (not stated in the paper; see Pitfalls)

def feature_4d(p, tau, segment):
    # Four 3D multi-resolution hash grids, one per 3D sub-space  [§3.1]
    f_xyz = segment.hash_xyz(p.x, p.y, p.z)    # (L * F,)
    f_xyt = segment.hash_xyt(p.x, p.y, tau)    # (L * F,)
    f_yzt = segment.hash_yzt(p.y, p.z, tau)    # (L * F,)
    f_xzt = segment.hash_xzt(p.x, p.z, tau)    # (L * F,)
    # Four 1D dense grids, linearly interpolated  [§3.1]
    v_t = segment.vector_t(tau)                # (L * F,)
    v_z = segment.vector_z(p.z)                # (L * F,)
    v_y = segment.vector_y(p.y)                # (L * F,)
    v_x = segment.vector_x(p.x)                # (L * F,)
    # Hadamard products, summed  [eq. 1]
    return (f_xyz * v_t + f_xyt * v_z
            + f_xzt * v_y + f_yzt * v_x)

Pseudocode for step 3 and 4, the MLP heads for one sample.

# Density and colour at one sample  [§3.3] [eq. 6-7]
# density_mlp and radiance_mlp are shared across all segments

def sample_density_color(p, tau, segment, direction):
    feature = feature_4d(p, tau, segment)      # (L * F,) = (32,)
    # Density head: sigma and 15 geometry features  [eq. 6]
    h = density_mlp(feature)                   # (16,)
    sigma = density_activation(h[0])           # see Paper vs code
    geometry = h[1:]                           # (15,)
    # First four spherical-harmonic bands of the view direction  [§3.3]
    sh = spherical_harmonics(direction, 4)     # (16,)
    # Colour head: SH and geometry features  [eq. 7]
    color = radiance_mlp(concat(sh, geometry)) # (31,) -> (3,)
    return sigma, color

Pseudocode for step 5, the rendering sum along one ray.

# Composite samples along one ray  [eq. 5] [eq. 10] [§3.3]
# samples: list of (sigma, color, delta) ordered from the camera

def render_ray(samples):
    color = zeros(3)
    transmittance = 1.0
    for sigma, rgb, delta in samples:
        # Opacity of this step  [eq. 5]
        alpha = 1.0 - exp(-sigma * delta)
        # Weight of this sample, from the transmittance so far  [eq. 5]
        weight = transmittance * alpha
        color = color + weight * rgb
        transmittance = transmittance * (1.0 - alpha)
    # Colour and accumulated opacity  [eq. 10]
    return color, 1.0 - transmittance

Pseudocode for step 6, the training loss for one ray.

# Training loss per ray  [§3.4] [eq. 8-11]
# color, gt_color: (3,) in [0, 1]; opacity, gt_mask: scalars

def ray_loss(color, gt_color, opacity, gt_mask):
    # Huber loss, delta = 0.01, averaged over the three channels  [eq. 8]
    diff = abs(color - gt_color)
    photometric = mean(where(diff <= 0.01,
                             0.5 * diff ** 2,
                             0.01 * (diff - 0.5 * 0.01)))
    # Binary cross-entropy on the accumulated opacity  [eq. 9]
    # The second term is printed without the 1 - opacity; see Pitfalls
    mask = -(gt_mask * log(opacity)
             + (1.0 - gt_mask) * log(1.0 - opacity))
    # Weighted sum, beta = 1e-3  [eq. 11] [§3.4]
    return photometric + 1e-3 * mask

Key equations, stated in the paper: the decomposition eq. 1, the expansion factor eq. 4, the volume-rendering integral eq. 5, the Huber loss eq. 8 and the final loss eq. 11.

Hyperparameters

Values for the 4x-downscaled setting unless stated. "Paper" rows come from the paper and supplement; "code" rows come from the official repository only and are not stated in the paper.

Results

Author-reported. Setup: ActorsHQ, 8 actors, two sequences each, sequences of 20 to 1000 frames, 4x-downscaled, metrics on 14 test cameras §5.1 [supp §2.1]. Baselines: NDVG, HyperNeRF, TiNeuVox, Neural Body, TAVA, and per-frame Instant-NGP §5 tab. 2.

Quality against number of frames tab. 2. HumanRF has the highest PSNR, SSIM and VMAF at every length. Per-frame Instant-NGP has the lowest LPIPS from 50 frames on, and ties at 20.

Other author-reported numbers: HumanRF uses 5.2% of the parameters of per-frame Instant-NGP while outperforming it §5.3.3. Per-frame Instant-NGP uses 20 times more trainable parameters §5.2.

Feature grid comparison tab. 3, 100-frame sequences, default grid parameters. Hex4D and tNGP are the ablations defined in the supplement §5.3.1 [supp §3].

Dynamic Furry Animal dataset tab. 4. The method runs with L = 20 on this data §5.5. Starred rows are copied from the Artemis paper, as the paper states. The paper says all baselines except Neural Volumes use skeleton information.

Input resolution [supp §2.2]. Training at lower input resolution and rendering at full 12MP:

The main text reports that finer details appear with higher input resolution §5.4, and that some baselines get worse at higher resolution §5.4.

Paper vs code

Official code: synthesiaresearch/humanrf at commit c0c1fb7bc1c86cbb3b5905a5c68cebb9d66e4d31. The abstract footnote links the project page synthesiaresearch.github.io/humanrf, which redirects to actors-hq.com; a GitHub search for "humanrf" in the synthesiaresearch organisation returns this one repository. Its LICENSE is CC BY-NC 4.0, so the code is read only: nothing is quoted from it.

Confirmed to match: the decomposition sum and its pairing of grids eq. 1 tensor_composition.cu; the segment hash sizes 2^15 to 2^19 §5.3.3 humanrf.py; the expansion threshold 1.25 §3.2 model_args.py; the Huber delta and the three-channel mean §3.4 trainer.py; the mask weight 1e-3 §3.4 run_args.py; the learning rate decay to half over training [supp §1.3] run.py; 8 frames per batch [supp §1.3] run_args.py; the 4x training length of N * 1000 iterations [supp §1.3] example_humanrf.py; and the evaluation masks: PSNR on the foreground, LPIPS and SSIM on the cropped region [supp §2.1] evaluate.py.

Discrepancies and additions:

• Camera embeddings: the main text's radiance MLP input is 31 values eq. 7, with no camera term. The example config sets a 2-dimensional learned camera embedding, and its comment says the paper's numerical comparisons use 0 example_humanrf.py. The supplement says the 12MP videos use camera embeddings [supp §1.1]. At test and validation time the code feeds a zero vector instead humanrf.py.
• MLP depth: the paper says the density network has 3 layers and the radiance network 4 layers §3.3. Code defaults are 1 and 2 hidden layers, both 64 wide model_args.py. The tiny-cuda-nn option counts hidden layers only, so the code has 2 and 3 weight matrices. The paper does not say whether it counts input and output layers.
• Density scale: the paper defines density as the MLP output eq. 6. The code multiplies a truncated exponential of the output by 100 humanrf.py model_args.py.
• Sample pruning: the paper uses masks for empty-space pruning §3.4 and occupancy grids from masks §3.2 [supp §1.2]. The code also prunes samples at each step where the density is below a threshold volume_rendering.py.
• Random background: for RGBA training, the code draws a random colour per ray and composites both the ground truth and the render over it trainer.py trainer.py. The paper does not mention this.
• Optimiser: Adam with betas (0.9, 0.99) and eps 1e-15 run.py. The paper and supplement give only the learning rate.
• Full-resolution setting: the paper gives K_max = 8192 and L = 24 for 12MP §5.3.2. The repository ships only the 4x example config example_humanrf.py.
• Dataset loading: the README says the dataset needs an access file from the project site, so the data are not in the repository README.md.

No discrepancy found in the adaptive partitioning equations eq. 2-4; the code adds rules that the paper does not state (see Pitfalls).

Pitfalls

• Eq. 9 misprint. In the source the background term reads log(M̂) with no 1 -. The code uses log(1 - M̂) with clamping and a small epsilon loss.py. Implement the code's form, which is the only sensible binary cross-entropy.
• Segment sizes are not the frame that triggered the split. The count is rounded down to the largest predefined size, so the frames after that point start the next segment adaptive_temporal_partitioning.py. The expansion test starts only at 6 frames and forces a split at 100 adaptive_temporal_partitioning.py. None of this is in the paper. If the sequence ends first, the remainder is rounded up adaptive_temporal_partitioning.py, so the sizes can sum to more than the frame count; the model caps the last segment humanrf.py.
• Time coordinate. The time input is the local frame index divided by the segment length, so it lies in [0, 1) and differs by segment humanrf.py humanrf.py. The paper writes time as a global index.
• Unstated density scale. The paper gives no scale factor for density. The code's comment says a value around 100 speeds convergence model_args.py. The gradient clip inside the exponential is also code-only activation.py.
• Half precision. The four products are computed in FP32 from FP16 inputs, and the output is stored as FP16 tensor_composition.cu. Training uses a gradient scaler trainer.py.
• Memory and segment paging. Each training batch covers 8 frames, so at most 8 segments are needed on the GPU; the rest are moved to the CPU [supp §1.3] humanrf.py. The average segment has about 12 frames, roughly 64 million parameters or 256 MB on the GPU [supp §1.3].
• Hidden training cost. Training is per sequence. The authors report about one day on one RTX 3090 for a 1000-frame sequence [supp §1.3], and the parameter count grows linearly with the sequence length §5.3.3.
• Silhouette flicker. The masks come from independent per-frame mesh reconstructions, so silhouette edges flicker in time. The authors name this as a limitation §5.6.
• Motion outside training poses. The model does not give explicit control of the articulation outside the training poses §5.6.
• Validation metrics. The baseline comparison uses 4x-downscaled data, because some baselines fail at full resolution §5.1. The full-resolution results are for HumanRF alone §5.4.
• Light bloom. The filter that drops rays near light sources is off in the example config and is used only for the 12MP videos. Those videos are not part of the comparisons [supp §1.1] example_humanrf.py.

Lineage

• Builds on: NeRF (Mildenhall et al., 2020) for volume rendering §3; Neural Volumes for differentiable rendering §3; Instant-NGP (Müller et al., 2022) for the multi-resolution hash encoding and shallow MLPs §3.1 §3.3; TensoRF (Chen et al., 2022) for the vector-matrix decomposition, extended here to time §3; the mask loss follows Yariv et al. (2020) §3.4.
• Compared against: NDVG, HyperNeRF, TiNeuVox, Neural Body, TAVA and per-frame Instant-NGP §5 tab. 2; ablations tNGP and Hex4D §5.3.1; Artemis, on the Dynamic Furry Animal dataset §5.5 tab. 4.
• Concurrent work with 2D low-rank 4D decompositions: K-Planes, HexPlanes and Tensor4D §2.
• Superseded by: not known from content in this wiki.