Neural Articulated Radiance Field

In plain words

The paper asks how to build a neural 3D model of a jointed object, such as a person, that can be shown in new poses and from new camera angles. It learns only from photos taken with known joint positions. Earlier neural methods for this either ignore the joints or must learn from scratch how joint movement changes the surface, which generalises poorly to poses they have not seen.

The idea is to give each point in space only the joint frame of the part that most plausibly contains it. A small learned chooser decides, softly, which part a point belongs to. The rest of the model works in that part's own coordinates, so moving a joint moves the part without retraining. The same chooser also gives a part segmentation. The recommended version stays about as cheap as a single neural radiance field, and the paper adds an extension that learns shape and appearance codes across many bodies.

TL;DR

NARF is a neural radiance field for articulated objects such as human bodies, trained from images with known joint poses and no 3D shape supervision. Each point is softly assigned to one bone, and that bone's local frame is fed to a single NeRF, so a new pose can be rendered without a template mesh. The recommended variant, NARF_D, is close to the P-NeRF baseline in FLOPs and memory per ray tab. 1 and also yields part segmentation. All results are author-reported on synthetic renders, and the repository mixes MIT code with non-commercial NVIDIA code.

Problem

• Skinning-based models need a template mesh, expert supervision and 3D scan data, and their quality is limited by the mesh resolution §2.
• Implicit articulated shape models (NASA) need ground-truth occupancy and do not learn appearance §2.
• 2D pose-conditioned image generators cannot model view-dependent appearance §1 §2.
• NeRF handles static scenes. Dynamic NeRF variants do not use the kinematic structure, so they offer no explicit pose control §2.
• A naive pose-conditioned NeRF that concatenates the whole pose fails in two ways: it must learn forward kinematics implicitly, and every location depends on all bones, which hurts generalisation to unseen poses §1 §3.2.
• Assumptions: the background is pre-cleaned §3.2; training needs pose annotations for each image §1; parts are treated as rigid app. "Experiment on Real Human Images".

Method

Kinematic model §3.2. The object is a tree of joints connected by bones of fixed length. Each bone has a local rigid transform built from a rotation and a translation by the bone length. The global transform of each joint is the product of the local transforms along its path from the root, times the root transform. Each part therefore has a rigid pose (R, t) computed explicitly by forward kinematics.

Rigidly transformed NeRF (RT-NeRF) §3.3. For a part with pose (R, t), a global point x maps to the part's local frame as R^-1 (x - t), and the view direction maps to R^-1 d. Density is conditioned on the local point and the bone length ζ. Colour is conditioned on the local direction and a 6-value se(3) code ξ of the part's pose §3.3.

Three ways to combine parts §3.4.

• NARF_P (part-wise). One RT-NeRF per part. Densities and colours are combined with a softmax weighting, controlled by a temperature τ §3.4. Cost grows with the number of parts §3.4.
• NARF_H (holistic). All parts' inputs are concatenated into one NeRF. Cost is nearly constant in the number of parts, but every location depends on all parts, and no part segmentation can be derived §3.4.
• NARF_D (disentangled, recommended). A selector has one small occupancy network per part, working in that part's local frame. A softmax over parts gives each point a probability for each part. Each part's inputs are scaled by its probability, then concatenated and fed to one NeRF. The per-point argmax of the probabilities gives the part segmentation §3.4.

Rendering §3.1 §3.4. Volume rendering is the standard discrete NeRF rule along each ray. A foreground mask is the same sum of weights, without colour. A segmentation image takes, per pixel, the part with the highest probability.

Training §3.5. Each step samples a batch of camera rays. Points along each ray are chosen by NeRF's hierarchical sampling. The loss is squared colour error plus squared mask error over the rays §3.5. The paper does not state the number of samples per ray.

Autoencoder for appearance variation §5. A 2D CNN encodes an input image into one latent vector. The latent is split into a shape part and an appearance part. The shape part joins the density inputs and the appearance part joins the colour inputs of NARF_D. Encoder and decoder train jointly with the same loss §5.

Pseudocode for NARF_D at one sample point. Equation numbers are not used because the source numbering could not be verified (see Pitfalls).

# Disentangled NARF at one sample point  [§3.4]
# point (3,), direction (3,); per bone: frame (R, t) and se(3) code xi
# zeta: bone length; z_shape, z_app: latent codes from the autoencoder [§5]

def narf_d(point, direction, bones, xi_codes, zeta, z_shape, z_app):
    # Express the point and direction in each bone frame  [§3.3]
    local_points = []
    local_dirs = []
    for bone in bones:
        local_points.append(bone.R.T @ (point - bone.t))  # (3,)
        local_dirs.append(bone.R.T @ direction)          # (3,)

    # One occupancy logit per bone, from its local point  [§3.4]
    logits = []
    for local_point in local_points:
        logits.append(selector(gamma(local_point), gamma(zeta)))

    # Softmax over bones gives a soft part assignment  [§3.4]
    probs = softmax(logits)                              # (P,)

    # Scale each bone's encoded inputs by its probability  [§3.4]
    masked_points = concat(
        [gamma(p) * probs[i] for i, p in enumerate(local_points)])
    masked_dirs = concat(
        [gamma(d) * probs[i] for i, d in enumerate(local_dirs)])
    masked_xi = concat(
        [gamma(x) * probs[i] for i, x in enumerate(xi_codes)])

    # Density branch: masked points, bone length, shape latent  [§3.4] [§5]
    density, features = density_net(
        concat(masked_points, gamma(zeta), z_shape))

    # Colour branch: features, masked direction and pose, appearance latent
    colour = colour_net(
        concat(features, masked_dirs, masked_xi, z_app))
    return density, colour

# Discrete volume rendering along one ray  [§3.1]
# sigmas (N,), colours (N, 3); deltas (N,) are spacings between samples

def render_ray(sigmas, colours, deltas):
    # Opacity of each sample: 1 - exp(-sigma_j delta_j)  [§3.1]
    alphas = 1 - exp(-sigmas * deltas)                   # (N,)

    # Transmittance before sample j: exp(-sum over k < j of sigma_k delta_k)
    steps = sigmas * deltas
    transmittance = exp(-(cumsum(steps) - steps))        # (N,)
    weights = transmittance * alphas                     # (N,)

    # Colour, and the foreground mask from the same weights  [§3.1] [§3.4]
    colour = sum(weights[:, None] * colours, axis=0)     # (3,)
    mask = sum(weights)                                  # scalar
    return colour, mask

# Training loss over a batch of rays  [§3.5]
# Ground truth per ray: colour (3,) and foreground mask (scalar)

def loss(pred_colours, true_colours, pred_masks, true_masks):
    # Squared colour error plus squared mask error, summed over rays  [§3.5]
    colour_error = sum((pred_colours - true_colours) ** 2)
    mask_error = sum((pred_masks - true_masks) ** 2)
    return colour_error + mask_error

Hyperparameters

Values are from the paper unless the source column says otherwise. Code values are from the per-subject configuration of the official repository (see code_commit). Where the paper says nothing, the table says "not stated".

Results

Author-reported, on the paper's own synthetic renders of human bodies (THUman meshes, two subjects for the single-object study, 128 x 128 images). Metrics are mask L2 error (lower is better), PSNR and SSIM (higher is better) §4. The single-object table has four test settings. "Seen" and "novel" refer to pose and view relative to training §4.

Cost per ray, from the same table tab. 1:

Single object, tab. 1. Each cell: Mask / PSNR / SSIM.

Autoencoder over many bodies, supplement table app. "Ablation Studies for AutoEncoder". Each cell: Mask / PSNR / SSIM. See Pitfalls: the table's bold marks disagree with the text.

Rigid-part ablation for RT-NeRF, on a rendered bulldozer (480 training and 20 test images, 200 x 200) app. "Ablation Studies in RT-NeRF":

Selected rows from the NARF ablation table app. "Additional Ablation Studies in NARF", same single-object setting. Each cell: Mask / PSNR / SSIM. Omitted rows are the NARF_P size variants with 64, 128 and 256 widths, and the NARF_P temperature sweep with tau at 0.01, 0.1, 1 and 1000.

Qualitative results (no numbers reported) are on SURREAL, with the autoencoder trained on THUman app. "Cross Dataset Evaluation on SURREAL Dataset", and on the ZJU-MoCap real-human video, where the authors say quality is lower than on the synthetic data app. "Experiment on Real Human Images".

Paper vs code

Official repository nogu-atsu/NARF at code_commit. Its LICENSE file is named LISENCE LISENCE. Code paths below are the core model in NARF/models/, the training loop in train.py, and the per-subject configs.

• Loss reduction. Paper sums squared colour and mask errors over rays §3.5. Code uses mean squared error on the sampled pixels, with colour and mask weights of 1 loss.py loss.py.
• Batch and ray count. Paper states batch 16 and as many rays as fit in memory app. "Training Details". Configs use bs 4 and a 24x24 patch of random rays per image NARF_D.yml NARF_D.yml model_utils.py train.py.
• Selector scale factor. Paper's probabilities are a plain softmax over parts §3.4. Code rescales the probabilities by sqrt(P) divided by their L2 norm over bones, and this is on by default nerf_model.py default.yml. The per-subject NARF_D config does not override it, so it should apply there too (inferred from the config merge in utils.py).
• Feature normalisation. Code L2-normalises the density feature over channels and multiplies by sqrt(hidden width) before the colour branch nerf_model.py. The paper does not describe this.
• Where pose and direction enter the colour branch. Paper concatenates the density feature, masked direction, masked pose and appearance latent at the input of the colour function §3.4 §5. Code applies one colour layer to the density feature, then adds the pose and direction features, then applies an activation and a final layer nerf_model.py nerf_model.py.
• Output activations. Paper states none. Code applies ReLU to density through a custom function. For negative inputs its backward pass lets through a 0.1-scaled gradient, but only when the incoming gradient is negative activation.py nerf_model.py. Colour goes through tanh nerf_model.py.
• Colour range. Images are scaled to [-1, 1] in training and masks to [0, 1] dataset.py dataset.py. The paper does not state a range.
• Invalid positional-encoding inputs. Code zeroes the encoding of any input whose absolute value exceeds 1 nerf_model.py. The paper does not mention this.
• Mask-bone-length. The per-subject NARF_D config does not mask the bone length NARF_D.yml. This agrees with the paper's density equation, which masks only the point encoding §3.4.
• Temperature convention for NARF_P. Paper's weights are softmax of density divided by tau, with best tau = 100 app. "Ablation Studies in NARF". Code multiplies density by arf_temperature, default 0.1 default.yml nerf_model.py. That is tau = 10 in the paper's convention. The repository has no NARF_P config, so this default may not be the one used for the reported NARF_P numbers.
• Autoencoder latent split. Paper says the latent is decomposed into shape and appearance parts §5 without a ratio. Code splits it into equal halves nerf_model.py.
• Missing components. The repository has a P-NeRF class nerf_model.py. I found no D-NARF implementation in NARF/, so the D-NARF baseline numbers cannot be checked against code. The repository also has no config for NARF_P, NARF_H, or the P-NeRF and NARF_H autoencoders.
• Adam settings. Betas and epsilon are not stated in the paper. Code uses betas (0.9, 0.999) and eps 1e-12 train.py.
• Hidden width. Paper does not state it. Code has 256 in the THUman NARF_D config and 32 in the default NARF_D.yml default.yml.

Pitfalls

• Equation numbers in the supplement may not match the main text. The supplement refers to the P-NeRF baseline as Eq. 8 and the colour function of RT-NeRF as Eq. 11. Counting the numbered equations in the LaTeX source gives 9 for the P-NeRF baseline and 12 for the colour function. I did not compile the PDF, so this card cites sections only. Check equation numbers in the PDF before you cite them.
• The autoencoder table contradicts the text. The supplement says NARF_D outperforms the others on all metrics and settings app. "Ablation Studies for AutoEncoder". In its own table, the CNN baseline is best in 8 of 12 metric cells: mask in the first two settings, PSNR in all four, and SSIM in the first two. The bold mark on NARF_D's mask value of 163.5 is also wrong, since the CNN's 157.2 is lower.
• The NARF_P FLOPs differ between tables. Table 1 gives 2140M FLOPs per ray for NARF_P, and the ablation table gives 2149M for the same model.
• Mean versus sum. The code's loss is a mean, so its absolute value is smaller than the paper's sum by the number of sampled values. Colour and mask keep equal weights in both. Loss values from the code are not comparable with the paper's sum-based values.
• Colour range for metrics. Training uses colours in [-1, 1]. If you compute PSNR or SSIM on [0, 1] images, the numbers will not be comparable.
• Memory. The paper reports 382K elements per ray for NARF_D and 24 hours on four V100s. The configs train 576 rays per image with 96 samples per ray (48 coarse and 48 fine), so memory use is high. The code splits large batches into chunks and recomputes them with gradient checkpointing nerf_model.py.
• Softmax versus sigmoid selector. The NARF_D configs use a softmax with a constant temperature of 1 default.yml. Code also offers a sigmoid option nerf_model.py. The paper's table shows the sigmoid selector with mask errors close to softmax but lower PSNR and SSIM on novel pose and novel view, and concludes that softmax is better app. "Additional Ablation Studies in NARF".
• Pose must be known. The method needs per-image bone poses and bone lengths. The paper takes them from SMPL parameters in the THUman dataset §4. The authors list joint pose estimation as future work §6.
• Rigid parts. The authors attribute poor real-image quality to rigid parts not matching loose clothing, and they point to latent pose-independent deformation as future work app. "Experiment on Real Human Images".
• Licence of the code. The repository mixes MIT code with NVIDIA's Source Code License-NC, which allows research and evaluation use only LISENCE-NVIDIA. NARF's core model imports fused_leaky_relu from NARF/stylegan_op/ stylegan.py, so the code is not cleared for commercial use. This card quotes no code. Check the licence before you reuse any code.
• Unstated details. Number of samples per ray, network widths for the colour and density branches, the encoder architecture, and learning rates for the NARF_P and NARF_H autoencoders are not in the paper's main text. The supplement gives the learning rates only.

Lineage

• Builds on: NeRF for volume rendering and positional encoding §3.1. Its bibliography entry for NeRF has no arXiv id, so no link is given.
• Related to NASA: each selector network is an occupancy network in the local frame. The paper compares it with NASA's absolute occupancy §2 §3.4.
• Compared against: a 2D CNN baseline in the spirit of pose-stick-figure generation §4; P-NeRF §3.2; D-NARF, a simple extension of Nerfies to articulated objects §4.
• Superseded by: none in this wiki.