TAVA: Template-free Animatable Volumetric Actors

In plain words

To make a realistic, animatable model of a person or animal, the usual route is to fit it to a body template, a 3D scan or a hand-built rig. This paper skips the template. It learns the model from multi-view images of one subject plus the subject's tracked skeleton, and it builds a 3D volume that stores what the subject looks like in a neutral reference pose.

The model also learns how each point of the volume moves when the skeleton moves. When a new frame is drawn, each point along each camera ray is moved back to where it came from in the reference pose, its colour and density are read there, and the usual volume-rendering sum turns those values into a pixel. The same lookup links pixels in different views and poses to one shared reference point, so the model can also say which pixel in one picture shows the same spot as a pixel in another picture.

The authors test it on four human subjects from a public motion-capture dataset and on two synthetic animals. They also build validation sets whose poses are unlike the training poses, which is harder than the usual split.

TL;DR

• Template-free animatable neural volume: a canonical-space radiance field plus learned skinning weights, an extra non-linear offset term, and an inverse map found by root finding.
• Renders by taking, for each sample, up to K candidate canonical points and keeping the one with the highest density; a learned ambient-occlusion term depends on pose.
• Author-reported: better than the template-free baselines on ZJU-Mocap and on the animals, close to the template-based methods on ZJU-Mocap, and about 2x more accurate pixel correspondences on the animals.
• The released repository implements a Broyden-type solver instead of the Newton update in the paper, and its default config turns off the two bone losses. Read "Paper vs code" before reimplementing.

Problem

Template-based animatable models (for example SMPL-based methods) need a body template, and they do not apply to other creatures §4.2. Template-free neural methods need no template, but the representation is usually conditioned directly on pose, so they generalise poorly to poses that were not in training §1, §4.2. Previous ZJU-Mocap splits put almost identical poses in both the training and the validation sets, which hides this weakness §4.1. Finally, the compared baselines do not establish pixel correspondences across poses and views natively §4.2; TAVA gets them from the canonical space §3.5.

Method

TAVA has three parts: a canonical representation of the actor §3.2, a skinning-based deformation with an inverse map computed by root finding §3.3, and volumetric rendering with a pose-dependent ambient-occlusion term §3.4. Correspondences come for free from the canonical coordinates §3.5. Training uses an image loss plus two auxiliary losses on the bones §3.6.

Preliminaries. Mip-NeRF integrates the radiance field over a Gaussian per sample, and rendering accumulates colour along a ray with transmittance weights §3.1, eq. 1. The image loss is a squared error per ray §3.1, eq. 2.

Canonical representation §3.2, eq. 3. The colour and density network F_Θr maps a canonical point and its covariance to colour and density. It takes no view direction, because the actor is treated as Lambertian. A second network F_Θs maps a canonical point to B+1 skinning weights: one per bone plus one for background (empty space).

Forward deformation §3.3.1, eq. 4, eq. 5. A point moves by linear blend skinning, where each bone transform is weighted by the skinning weight, and the background weight multiplies the identity so that empty space stays put. A second network F_Θ∆ adds a learned, pose-dependent offset on top, intended for muscles and clothing.

Inverse deformation by root finding §3.3.2, eq. 6, eq. 7. Rendering needs, for each view-space sample x_v, the canonical point x_c that the forward map sends to x_v. The paper solves f(x_c) = 0 numerically eq. 6 with an iterative update eq. 7 (written as Newton's method). Each sample gets K initial guesses, taken from the inverse rigid transforms of its K nearest bones (K = 5 in the paper). Gradients with respect to the skinning and offset networks come from implicit differentiation, which is derived in app. A.

Rendering §3.4. For each sample, the canonical colour c*_i, density σ*_i and ambient occlusion a*_i are queried at each converged candidate. The ambient occlusion head F_Θa is the only part conditioned on pose P, so the canonical geometry and appearance stay pose-independent. The sample takes the values of the candidate with the highest density. The emitted colour is the candidate colour multiplied by the occlusion value, and the density is the candidate density. For samples where root finding fails (the paper reports 1% to 8% of samples), the paper interpolates from valid neighbours along the ray. It reports that this works slightly better than setting the density to zero.

Correspondences §3.5. Rendering also accumulates the canonical coordinates along each ray with the same weights. Two pixels from different views or poses are matched when their accumulated canonical coordinates are nearest neighbours.

Training loss §3.6. L = L_im + λ L_w + β L_∆, with λ = 1.0 and β = 0.1 in all experiments. L_w pushes the skinning weights of samples on the bones towards one-hot vectors of their bone. L_∆ pushes the offsets on those samples towards zero. Both are mean squared errors.

# Render one ray  [§3.4] [eq. 1] [§3.3.2] [§3.4]
# Input:  samples x_v (N, 3); covariances; pose P; trained model
# Output: pixel colour (3,)

def render_ray(samples, covariances, pose):
    # Find up to K canonical candidates for every sample  [§3.3.2]
    candidates, converged = root_find(samples, pose)  # (N, K, 3), (N, K)
    # Query colour, density and occlusion in canonical space  [§3.4]
    colours, densities, occlusion = canonical_query(candidates, pose)
    # Keep the candidate with the largest density  [§3.4]
    best = argmax(densities * converged, axis=1)      # (N,)
    colours = pick(colours, best) * pick(occlusion, best)
    densities = pick(densities, best)
    # Fill samples where no candidate converged from valid neighbours  [§3.4]
    colours, densities = interpolate_failed(colours, densities, converged)
    # Standard volume rendering along the ray  [eq. 1]
    return volume_render(colours, densities, samples)

# Root finding for one sample  [§3.3.2] [eq. 6] [eq. 7]
# Input:  view-space point x_v (3,); initial canonical guess x_c (3,); pose P
# Output: canonical point, or None if it does not converge

def solve_canonical(x_v, x_c, pose):
    for step in range(max_steps):
        # residual f(x_c) = forward(x_c) - x_v  [eq. 6]
        residual = forward(x_c, pose) - x_v
        if norm(residual) < tolerance:
            return x_c
        # Newton update with the 3x3 Jacobian  [eq. 7]
        # (the released code uses a Broyden update; see Paper vs code)
        jacobian = forward_jacobian(x_c, pose)
        x_c = x_c - inverse(jacobian) @ residual
    return None

Hyperparameters

Results

Author-reported. Setup: ZJU-Mocap subjects 313, 315, 377 and 386 §4.1, and two synthetic Blender animals (Hare and Wolf) §4.1. Metrics are PSNR and SSIM for image quality, and P2P (pixel error in px) for correspondence. The baselines are Animatable-NeRF and NeuralBody (SMPL-based, official code) and NARF, A-NeRF and Pose-NeRF (template-free, re-implemented in the same code base) §4.2.

ZJU-Mocap tab. 1:

Synthetic animals tab. 2:

Ablation on ZJU-Mocap tab. 3 (PSNR / SSIM):

Baseline re-implementation check on ZJU subject 313 app. D, PSNR / SSIM: the authors' re-implementation of A-NeRF scores higher than its official code on all three columns (32.14 / 0.976 against 28.36 / 0.945 for novel view), and the same holds for NARF on the in-distribution and novel-view columns. For NARF, the official code is slightly better on OOD PSNR (28.10 against 28.05).

On ZJU-Mocap, NeuralBody has the higher PSNR and SSIM for novel views and in-distribution poses. TAVA's advantage is on OOD PSNR (30.69 against 30.33), with SSIM tied at 0.969. The authors say the ZJU results are comparable to the template-based methods §4.2.

Paper vs code

Official repository facebookresearch/tava, commit a9576801e81aebcf242588be39315e27f915894e (BSD-3-Clause), linked from the abstract. The repository is the authors' own (Meta, the affiliation of several authors) and covers training, evaluation and the animal data. The shipped configs are a good guide to the paper's settings, but several values differ.

• Root solver. The paper writes Newton's method §3.3.2, eq. 7. The code uses Broyden's method with inverse-Jacobian updates, at most 50 steps, a convergence threshold of 1e-5 and a divergence threshold of 1, and it keeps the iterate with the smallest residual snarf.py.
• Candidates. The paper uses K = 5 bones §3.3.2. The code always adds an identity (background) candidate, so there are K + 1 = 6 by default snarf.py snarf.py.
• Bone losses are off by default. The paper uses λ = 1.0 and β = 0.1 §3.6. The default dynamic config sets both bone-loss multipliers to 0.0 mipnerf_dyn.yaml. The documented ZJU command sets loss_bone_w_mult=1.0 and loss_bone_offset_mult=0.1 experiment.md, and the documented animal command sets only loss_bone_w_mult=1.0 and turns the offset network off experiment.md.
• Coarse-level loss. The code adds the coarse-level squared error with weight 0.1 to the fine-level loss mipnerf_dyn.yaml trainer.py. The paper's loss has no coarse term.
• Bone sample ranges. The skinning-weight loss samples points at 10% to 90% of each bone, which excludes the joints snarf.py. The offset loss samples 0% to 100% snarf.py, and a comment there says the paper's own code sampled in world space, which the authors call unintended. The paper does not state either sampling rule.
• Pose input. The paper does not state the pose representation for the occlusion head. In the default (legacy) setting, the ZJU loader uses a 78-dim vector: 72 axis-angle body values, 3 global rotation values and 3 translation values zju_loader.py zju.yaml. A comment in the loader says the paper uses axis-angles; the paper text itself does not say so. A non-legacy option uses 6D rotations, marked "not verified".
• Occlusion activation. The occlusion head is passed through a sigmoid mipnerf_dyn.py mipnerf_dyn.py. The paper does not name an activation.
• Density and colour activations. Softplus on density with bias −1, and a sigmoid on colour with padding 0.001 on both sides mipnerf.py. Not stated in the paper.
• Skinning network. Four layers of width 128 on a degree-4 encoding snarf.py. The logits are scaled by 5 before the softmax snarf.py snarf.py. Neither detail is in the paper.
• Canonical and world range masks. Candidates farther than 0.3 from the canonical bones are dropped snarf.py zju.yaml. Samples within 0.3 of the bones in world space are kept for rendering mipnerf_dyn.py mipnerf_dyn.py. The paper mentions neither.
• Interpolation of failures. Implemented as described: failed samples are filled from valid neighbours along the ray mipnerf_dyn.py.
• Pose clustering. The paper uses V2V distance on the full ground-truth mesh §4.1. The code subsamples every 100th vertex clustering.py, uses a fixed seed per subject zju_parser.py, and clusters with PAM K-Medoids on a precomputed distance matrix clustering.py. The test chunk is 1/10 of the frames in the code clustering.py, which the paper does not state. The in-distribution split holds out 1/3 of each cluster, a 2:1 split clustering.py.
• Optimiser and schedule (not in the paper). Adam, learning rate 5e-4 decaying log-linearly to 5e-6 over 300 000 steps, with a warm-up delay of 2 500 steps starting at 1% of the rate trainer.py _basic.yaml. 1024 rays per batch on ZJU, with images resized by 0.5 zju.yaml.
• Evaluation size. The documentation says the paper evaluates 100 images per set benchmark.md. The paper does not state the number of evaluation images for PSNR and SSIM.
• Searches: the repository was found through the abstract link. No other official code was looked for.

Pitfalls

• Equation 1 has a sign typo. The printed transmittance term reads 1 - exp(σδ). The standard form, and the one in the code, is 1 - exp(-σδ) mipnerf.py.
• Root finding is the cost centre. Each sample needs up to K + 1 candidates, each run through an iterative solver with Jacobians of the forward map. The search runs without gradients, and gradients are then injected with the implicit-function formula snarf.py. Expect the per-ray cost to scale with K.
• Failures are silent. Roughly 1% to 8% of view-space points fail §3.4. Their density is interpolated, so a mis-set convergence or divergence threshold changes the image without an error.
• Do not leave the bone losses off. With the default config they are disabled and the skinning weights are not forced to be one-hot. The documented runs enable them.
• Occlusion can be switched off only for pictures, not for scores. Turning it off at test time gives brighter-than-ground-truth images, so it cannot be compared with the ground truth §4.3. The reported metrics keep it on. For the animals, occlusion and the offset term are switched off for training and test §4.2. Do not mix the two settings.
• Out-of-distribution is hard for every method. On ZJU-Mocap the paper reports a drop of 0.67 to 2.66 dB from in-distribution to OOD poses, and it attributes the drop to pose-dependent shading that all methods still learn §4.2. The animal drop is larger for the baselines (1.68 to 5.73 dB) than for TAVA.
• Dataset caveats. The authors report camera calibration errors and varying exposure in ZJU-Mocap that hurt every method, found after submission app. F.
• Requirements beyond the method. Training needs a tracked skeleton per frame and multi-view images §1. The code also reads foreground masks, and the ZJU parser marks a band around the mask border as ignored zju_parser.py.

Lineage

• Builds on: NeRF and Mip-NeRF for the radiance field and integrated encoding §3.1; SNARF (cited as chen2021snarf) for the root-finding inverse map and its implicit gradients §3.3.2; a background identity bone from HumanNeRF (cited as weng2022humannerf) §3.3.1.
• Compared against: NARF, A-NeRF and Pose-NeRF (template-free); Animatable-NeRF and NeuralBody (SMPL-based) §4.2.
• Superseded by: not stated in this wiki.