Linear blend skinning

Overview

Skinning moves the points of a model with the bones of a skeleton. In linear blend skinning, each point is moved by every bone in turn, and the results are averaged with per-point weights. The weights say how much each bone controls the point. This page covers one use of the idea: a canonical radiance field whose points are moved to the posed frame by learned weights. The only source is TAVA, which does this for animatable people and animals 2206.08929 §3.2 §3.3.1. Because this wiki holds one card on the topic, the variants below all come from that paper.

The invariant is the forward map. A point in canonical space is sent to view space by a weighted sum of bone transforms. The weights come from a network that reads the canonical point, and empty space gets an identity weight so that it stays where it is 2206.08929 §3.3.1, eq. 4-5. Rendering needs the other direction: for each sample in view space, the canonical point that maps to it. TAVA finds that point by solving an equation numerically, starting from several guesses 2206.08929 §3.3.2, eq. 6-7. The skinning step therefore sits in two places: the forward map, which defines the deformation, and the inverse map, which the renderer uses.

Two design choices sit on top of the basic map. The first is that the weights are learned, not fixed by a rig. The second is an extra offset that a separate network predicts from the pose. The paper gives the offset for muscles and clothing 2206.08929 §3.3.1. The paper's loss can also push the weights of points on a bone towards that bone, and push the offset of those points towards zero 2206.08929 §3.6.

Variants

Learned skinned canonical field (TAVA, full model)

A canonical colour and density network, plus a network that gives each canonical point one weight per bone and one for empty space 2206.08929 §3.2 eq. 3. The forward map adds the pose-dependent offset 2206.08929 §3.3.1. The renderer keeps, for each sample, the root-finding candidate with the highest density 2206.08929 §3.4. Card: TAVA: Template-free Animatable Volumetric Actors. On ZJU-Mocap, the author-reported out-of-distribution score is 30.69 PSNR, against 30.60 for NARF and 30.33 for NeuralBody, which uses an SMPL template 2206.08929 tab. 1. The paper's own numbers for the baselines come from re-implementations of NARF, A-NeRF and Pose-NeRF, and the paper's appendix compares them with the official code 2206.08929 app. D card, Paper vs code.

Without the non-linear offset (ablation)

The same model with the offset network removed. The author-reported novel-view PSNR drops from 33.11 to 31.86, and the out-of-distribution score drops from 30.69 to 30.56 2206.08929 tab. 3. The offset therefore matters most for new views in this experiment, and little for new poses. The paper's animal runs also turn the offset off 2206.08929 §4.2.

When to use which

Implementation sketch

The sketch follows the paper's equations. Names are ours. The released code differs in several details, listed on the card's Paper vs code section.

# Skinning weights for one canonical point  [arxiv:2206.08929 §3.2] [eq. 3]
# Input: canonical point x_c (3,); network F_s
# Output: B+1 weights, one per bone plus one for empty space

def skin_weights(x_c):
    # B+1 values; the normalisation used in the release is on the card
    return F_s(x_c)

# Forward map, canonical to view space  [arxiv:2206.08929 §3.3.1] [eq. 4-5]
# Input: canonical point x_c (3,); bone transforms T_1..T_B; offset net
# Output: view-space point x_v (3,)

def forward(x_c, T, pose, offset_net):
    w = skin_weights(x_c)
    # Empty-space weight multiplies the identity, so it stays put
    x_v = w[0] * x_c
    for b in range(1, B + 1):
        # Each bone moves the point, scaled by its weight
        x_v = x_v + w[b] * apply(T[b], x_c)
    # Pose-dependent offset for muscles and clothing  [§3.3.1]
    return x_v + offset_net(x_c, pose)

# Inverse map by root finding  [arxiv:2206.08929 §3.3.2] [eq. 6-7]
# Input: view-space sample x_v (3,); pose; initial guesses (K, 3)
# Output: canonical candidates that converged

def inverse_candidates(x_v, pose, guesses):
    found = []
    for x_c in guesses:
        # Iterate until the residual forward(x_c) - x_v is small  [eq. 6]
        # The paper writes Newton's update; the release uses Broyden's
        for step in range(max_steps):
            r = forward(x_c, T, pose, offset_net) - x_v
            if norm(r) < tol:
                found.append(x_c)
                break
            x_c = x_c - solve_jacobian(x_c, pose, r)  # [eq. 7]
    return found

Contradictions

None known. The wiki has one card on this technique, so no two sources disagree. Within TAVA, the released code differs from the paper on the root solver, the bone losses and the number of candidates. Read the card's Paper vs code section before comparing numbers with the paper.

Papers

• TAVA: Template-free Animatable Volumetric Actors: a learned skinning field in canonical space with a non-linear offset and root-finding inverse map. Built from multi-view images and a tracked skeleton, with no body template.