Motion Representations for Articulated Animation

In plain words

The goal is to make a still picture of a person, or of another jointed object, move the way someone else moves in a video. Earlier systems of this kind pick points on the outline of the object and follow them. That works reasonably for a face, but it breaks down for arms and legs, because a point on an outline is hard to track and says little about how a limb is shaped or angled.

This method instead learns soft blobs, each covering one part of the object, such as an arm or a leg. Nobody labels the parts. From each blob the model reads where the part sits, how it is stretched, and which way it points. Comparing these readings between the photo and the video gives the motion of each part. A separate estimate handles movement of the whole background, such as a camera pan, so that the object parts are not confused with it.

The model also keeps the shape of the object in the photo apart from the pose taken from the video. Without this split, the animated figure can take on the body shape of the person in the video. The model is trained on ordinary videos without labels, and it helps most on articulated objects. The cost is a more involved training recipe with several networks.

TL;DR

• Unsupervised image animation for articulated objects: each region's centre, shape and orientation are measured from the principal axes (SVD) of its heatmap, not regressed §3.2.
• A global affine background motion absorbs camera movement, so the regions stay on the object §3.3.
• Animation via disentanglement: two fully connected encoders and one decoder separate source shape from driving pose; trained after the other networks §3.6 app. C.
• Author-reported: lower L1, keypoint and identity errors than FOMM on TaiChiHD and TED-talks, and 96.6% user preference over FOMM on TED-talks tab. 2 tab. 3.

Problem

FOMM-style keypoints sit on object boundaries. Any boundary point is a valid match, so they are hard to track between frames, and they describe location and direction but not the shape of a part §1. The authors argue that regressed local affine parameters need higher-capacity networks and generalise poorly §3.2 §4.1. Camera motion leaks into the keypoints unless it is modelled separately §1 §3.3.

Standard animation copies the shape of the driving object into the output. Relative animation avoids this, but it needs the source and the first driving frame to show the object in roughly the same pose §3.6. Training pairs are drawn from the same video, so source and driving share an object shape §3.6.

Method

Overview. The model follows the first-order motion model (FOMM) §3.1 and changes three parts: how regions are measured §3.2, how background motion is represented §3.3, and how shape is separated from pose at animation time §3.6. The paper runs the region predictor, background predictor and pixel-wise flow predictor on quarter-resolution inputs app. C. A generator warps source features with the flow and inpaints occluded parts §3.4. The whole model is trained end to end with a reconstruction loss and an equivariance loss §3.5. The shape and pose networks are trained afterwards §3.6.

1. Region measurement (PCA-based). For each of K regions, a softmax heatmap over pixels gives a weight per pixel §3.2. The centre is the heatmap-weighted mean of pixel coordinates eq. 5. The covariance of the heatmap around that centre is decomposed with an SVD eq. 6. The linear part of the region transform is U S^(1/2), and the centre is the translation eq. 7. Shear is not captured, so the transform has five degrees of freedom rather than six §3.2.
2. Motion per region. For a source and a driving frame, the region transform is composed through the whitened reference frame §3.2 eq. 3. The composition is an affine map from driving to source coordinates.
3. Background motion. A separate encoder takes the source and driving frames and outputs six numbers, forming a 2x3 affine map §3.3. The map is applied to the identity grid as a background motion.
4. Dense flow. The flow predictor outputs K+1 assignment maps, one per region plus background, through a softmax over regions at each pixel. The flow at each pixel is the weighted sum of the K+1 sparse motions §3.4 eq. 4 eq. 8.
5. Rendering. The source is encoded, and its features are warped by the flow. Each skip connection is warped the same way and weighted by a confidence map, which also covers parts absent from the source §3.4 app. C.
6. Training. The loss is a multi-scale perceptual reconstruction term on VGG-19 features plus an equivariance term. Equivariance asks that a region measured on a randomly warped driving frame matches the warped region §3.5 eq. 9 eq. 10.
7. Animation via disentanglement. Two encoders map the driving region parameters (pose) and the source region parameters (shape) to 64-dimensional codes. A decoder combines them into modified driving region parameters §3.6 app. C. During training the driving parameters are randomly scaled per axis, which forces shape to come from the source branch. No scaling is applied at test time §3.6.

# Region measurement  [§3.2] [eq. 5] [eq. 6] [eq. 7]
# Input:  image (B, 3, h, w) at quarter resolution  [app. C]
# Output: means (B, K, 2), linear (B, K, 2, 2), covars (B, K, 2, 2)
def measure_regions(image):
    # Softmax over pixels of K logit maps, with a temperature  [§3.2]
    logits = region_network(image)                  # (B, K, h, w)
    heatmaps = softmax(logits / temperature)        # sums to 1 per region
    # Region centre: heatmap-weighted mean of coordinates  [eq. 5]
    means = sum(heatmaps * coords)                  # (B, K, 2)
    # Covariance of each heatmap around its centre  [eq. 6]
    offsets = coords - means                        # (B, K, h*w, 2)
    covars = sum(heatmaps * outer(offsets, offsets))  # (B, K, 2, 2)
    # SVD gives the principal axes and the singular values  [eq. 6]
    axes, singular = svd(covars)                    # (B, K, 2, 2), (B, K, 2)
    # Linear part is U S^(1/2); the centre is the translation  [eq. 7]
    linear = axes @ diag(sqrt(singular))            # (B, K, 2, 2)
    return means, linear, covars

# Dense flow from region motions and background  [§3.1] [§3.4] [eq. 3] [eq. 8]
# Input:  source and driving region sets, background affine (B, 2, 3)
# Output: flow (B, h, w, 2), mapping driving coordinates to source coordinates
def dense_flow(source, driving, background):
    for k in range(K):
        # Driving point -> source point, through the reference frame  [eq. 3]
        relative = source.linear[k] @ inverse(driving.linear[k])
        sparse[k] = relative @ (coords - driving.means[k]) + source.means[k]
    # Background motion: affine applied to the identity grid  [§3.3]
    sparse[0] = affine_apply(background, coords)
    # Softmax over the K+1 sparse motions, per pixel  [§3.4] [eq. 8]
    weights = softmax(assignment_logits, over=K + 1)   # (B, K+1, h, w)
    return sum(weights * sparse)                       # (B, h, w, 2)

# Training loss: reconstruction plus equivariance  [§3.5] [eq. 9] [eq. 10]
# Input:  generated frame, driving frame, region sets of the driving frame
#         and of a randomly warped copy of it
# Output: scalar loss
def training_loss(generated, driving, regions, warped_regions):
    # Multi-scale VGG-19 feature distance to the driving frame  [eq. 9]
    reconstruction = 0
    for scale in scales:
        reconstruction += vgg_distance(generated, driving, scale)
    # Warped frame's regions must follow the warp  [eq. 10]
    equivariance = abs(regions - warp(warped_regions))
    return reconstruction + equivariance

Key equations, as printed in the paper (numbering counted from the source, check against the PDF):

\mu^k = \sum_{z} M^k(z)\, z eq. 5

U^k S^k V^k = \sum_{z} M^k(z)\,(z-\mu^k)(z-\mu^k)^{\top} eq. 6

A^k_{X \leftarrow R} = \left[\, U^k (S^k)^{1/2},\ \mu^k \,\right] eq. 7

A^k_{S \leftarrow D} = A^k_{S \leftarrow R} \begin{bmatrix} A^k_{D \leftarrow R} \\ 0\ \ 0\ \ 1 \end{bmatrix}^{-1} eq. 3

O(z) = \sum_{k=0}^{K} W^k(z)\, A^k_{S \leftarrow D} \begin{bmatrix} z \\ 1 \end{bmatrix} eq. 8

\mathcal{L}_{\mathrm{rec}} = \sum_{l} \sum_{i} \left| V_i(F_l \odot \hat{D}) - V_i(F_l \odot D) \right| eq. 9

\mathcal{L}_{\mathrm{eq}} = \left| A^k_{X \leftarrow R} - \tilde{A}\, A^k_{\tilde{X} \leftarrow R} \right| eq. 10

In these equations, M is a heatmap, z a pixel coordinate, and W the K+1 assignment maps. X is S or D, R the reference frame, and tilde marks the randomly warped copy.

Hyperparameters

Results

Author-reported. Setup: video reconstruction on TaiChiHD (256 and 512), TED-talks (384), VoxCeleb (256) and MGif, with K = 10 regions for all methods. The authors trained for 50% more iterations than FOMM and retrained FOMM on the same schedule §4.3. Metrics are L1, average keypoint distance (AKD), missing keypoint rate (MKR) and average Euclidean identity distance (AED) §4.2.

Reconstruction (video reconstruction) tab. 2.

| Method | TaiChiHD 256: L1 | AKD | MKR | AED | TaiChiHD 512: L1 | AKD | MKR | AED | TED-talks: L1 | AKD | MKR | AED | VoxCeleb: L1 | AKD | AED | MGif: L1 | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | X2Face | 0.080 | 17.65 | 0.109 | 0.27 | - | - | - | - | - | - | - | - | 0.078 | 7.69 | 0.405 | - | | Monkey-Net | 0.077 | 10.80 | 0.059 | 0.228 | - | - | - | - | - | - | - | - | 0.049 | 1.89 | 0.199 | - | | FOMM | 0.056 | 6.53 | 0.033 | 0.172 | 0.075 | 17.12 | 0.066 | 0.203 | 0.033 | 7.07 | 0.014 | 0.163 | 0.041 | 1.27 | 0.134 | 0.0223 | | This method | 0.047 | 5.58 | 0.027 | 0.152 | 0.064 | 13.86 | 0.043 | 0.172 | 0.026 | 3.75 | 0.007 | 0.114 | 0.040 | 1.28 | 0.133 | 0.0206 |

The authors describe the VoxCeleb result as on par with FOMM (non-articulated faces) §4.3. Their text also reports a 52% preference in favour of their method on VoxCeleb; that number is not in any table §4.3.

Regions K and the PCA-based representation, TaiChiHD (256) tab. 1.

Ablation, TaiChiHD (256), K=10 tab. 4.

User preference tab. 3. Each study used 50 videos, each rated by 50 users, so 2500 preferences §4.3.

Toy rotated-rectangles experiment §4.1 fig. 5. With a supervised angle regression and training sets from 32 to 1024 images, the PCA-based head beats the regression head, by more than an order of magnitude at the large-data end, according to the authors. No table reports these numbers.

Co-part segmentation on TaiChiHD §4.5 fig. 6. Foreground IoU of 0.81 for this method, against 0.77 for MSCS and 0.55 for SCOPS, as the authors report.

Paper vs code

Repository linked from the paper's abstract (footnote): snap-research/articulated-animation, cloned at commit 03f1968bb1be1b0d39e4818db1e39cd969adc033. Its LICENSE.md is a Snap Inc. sample-code notice that grants no licence, so nothing is quoted from the code; every reference is by path and line.

Differences between the paper and the code:

• Softmax temperature. The paper gives a plain softmax §3.2. The code divides by 0.1 region_predictor.py taichi256.yaml.
• Sign of the relative transform. The compiled paper gives no sign rule for the SVD axes §3.2. The code multiplies by the sign of the (0,0) entry of the relative transform, so that entry becomes non-negative. In the flow predictor the whole relative transform is multiplied pixelwise_flow_predictor.py. In the shape-pose network the source linear part is multiplied, using the sign of the relative transform's (0,0) entry avd_network.py. The equivariance term applies the same rule model.py.
• Heatmap term in the flow input. The paper says one Gaussian approximating each region heatmap §3.4. The code uses the difference of two Gaussians, one at the driving centre and one at the source centre pixelwise_flow_predictor.py.
• Input channels. The paper gives H x W x (4K+3) §3.4. The code concatenates a zero-valued background heatmap channel, giving (K+1) x 4 = 4K+4 channels pixelwise_flow_predictor.py pixelwise_flow_predictor.py.
• Background resolution. The paper says the background predictor also runs at quarter resolution app. C. The code feeds full-resolution frames model.py bg_motion_predictor.py, and its config has no scale factor for it taichi256.yaml.
• Final blend with the source. The paper does not describe this step. The generator blends its final RGB output with the warped source image, weighted by the occlusion map generator.py.
• Equivariance term. The paper has one term, with no weights §3.5 eq. 10. The code has a shift term and an affine term, each with weight 10, and the affine term uses the inverse of the driving linear part and compares to the identity model.py taichi256.yaml. The random transform is an affine map (sigma 0.05) plus a thin-plate spline (sigma 0.005, 5x5 control points) model.py taichi256.yaml.
• Perceptual weights and layers. The paper gives no per-layer weights §3.5. The code uses weight 10 for each of five VGG-19 outputs, at four scales model.py taichi256.yaml taichi256.yaml. The five outputs are relu1_1 to relu5_1, as inferred from the layer slicing; the paper does not name them.
• Optimiser betas. The paper gives only the learning rate app. C. The code uses betas (0.5, 0.999) train.py.
• Batch sizes. The paper gives 48, 20 and 12 for 256, 384 and 512 app. C. The configs use 14 for 256, 6 for ted384 (paper: 20 at 384) and 20 for ted-youtube384 taichi256.yaml ted384.yaml ted-youtube384.yaml. The README reports one V100 and about two days for a 256 model README.md. The paper used four P100 GPUs app. C.
• Shape-pose schedule. The paper gives learning-rate drops at 660K and 880K pairs out of 1M app. C. The code drops at epochs 20 and 25 of 30, with num_repeats 150 taichi256.yaml. The two cannot be reconciled from the code alone.
• Shape-pose scaling range. The paper gives no range for the random scaling §3.6. The code draws each axis scale uniformly from [0.75, 1.25] and applies it to both the shift and the linear part of the driving regions train_avd.py.
• Relative animation. The paper defines relative animation only as in FOMM §3.1. The code scales the movement by the square root of the ratio of convex-hull areas of the region centres animate.py. The paper's results use standard animation and shape-pose animation only §4.3.
• Not in the repository. The metrics (L1, AKD, MKR, AED), the rotated-rectangles experiment and the co-part segmentation evaluation have no code here. The repo contains only the training and animation code, a metadata CSV for the TED-talks videos ted-metadata.csv, and sample assets. The TED-talks videos themselves are not in the repository.

Pitfalls

• Coordinates are normalised to [-1, 1] util.py. Region centres, covariances and flows use these units, not pixels. A pixel-unit implementation will get the Gaussians and the flow wrong by a constant factor.
• Axis signs. The SVD returns each axis only up to sign. The code fixes the sign of the relative transform (see Paper vs code). An implementation that skips this step may flip the orientation of a part between frames, but the paper does not test that. This is inferred from the code.
• Singular matrices. The code inverts the driving linear part pixelwise_flow_predictor.py and the heatmap covariance util.py with no epsilon term. A collapsed heatmap gives a singular matrix. Inferred from the code.
• The printed SVD. Equation 6 writes U S V without the transpose on V. The covariance is symmetric, so only U and S are used in the code region_predictor.py.
• Hidden draft text. The arXiv source contains commented-out paragraphs about a per-pixel angle correction for the SVD sign ambiguity. They are not part of the compiled paper, and the code does not implement them.
• Table inconsistency. The TaiChiHD (256) full-model L1 at K=10 is 0.047 in tab. 1 and in tab. 2, but 0.048 in tab. 4. Use the value from the table you cite.
• Cost grows with K. The flow predictor builds K+1 warped copies of the source at quarter resolution, so memory and compute grow linearly with K. Inferred from the code pixelwise_flow_predictor.py.
• Covariance for shape-pose outputs. After the shape-pose network, the covariance is recomputed as A A^T from the decoded linear part avd_network.py, not taken from a measured heatmap.
• Equation and figure numbers are counted from the LaTeX source. Check them against the PDF before citing.
• TED-talks dataset licence. The paper says the source videos were shared under CC BY-NC-ND 4.0, a non-commercial, no-derivatives licence, and that the authors cut and rescaled them app. E. This card does not provide the dataset, and its TED-talks numbers are the authors' reported results. Do not redistribute or train on that dataset without checking those terms.
• Articulated benefit. Gains are largest on TaiChiHD and TED-talks, where the objects are articulated. On VoxCeleb the authors report on-par results with FOMM §4.3 tab. 2.

Lineage

• Builds on: First Order Motion Model. It keeps the generator, flow and losses, and changes the motion representation, background handling and shape-pose disentanglement §3 app. C.
• Compared against: X2Face, Monkey-Net and FOMM tab. 2. Co-part baselines MSCS and SCOPS §4.5. None of these has a card in the wiki yet.