First Order Motion Model for Image Animation

In plain words

The goal is to make a still picture move the way someone else moves. You give the model a photo of a person (or a cat, or a robot arm) and a short video of a different person doing something, and it produces a video of the photo subject copying that motion.

The method does not need labels saying where the body parts are. It learns from ordinary videos of one kind of object. It picks a small set of points on the object that it can track on its own, and for each point it also learns how the nearby patch stretches and rotates. Those points and patches form a compact description of the motion. The points found on the photo are then moved by the motion seen in the driving video, and a generator paints the photo's appearance onto the new pose. Where a body part swings into view that the photo does not show, a learned mask tells the generator to fill it in from context rather than copy pixels that do not exist.

The approach matters because it works for any object in a category the model was trained on, without a 3D model or per-object annotation. The trade-off is that it assumes the source photo and the driving video show objects in roughly similar poses.

TL;DR

• Image animation: move the object in a source image by the motion of a driving video, with no annotation or 3D model §1 §3.1.
• Motion is a set of learned keypoints, each with a local affine (first-order Taylor) approximation, which extends the zeroth-order motion of Monkey-Net §3.1.
• An occlusion mask marks parts that must be inpainted. The paper trains with a reconstruction loss and two equivariance losses only §3.2 §3.3.
• Author-reported: lower reconstruction error and user-preferred results than Monkey-Net and X2Face on Tai-Chi-HD, VoxCeleb, Nemo and BAIR tab. 2 tab. 3.

Problem

Image animation needs to separate what an object looks like from how it moves. Earlier systems either assumed an object-specific model (a 3D face model with graphics rendering) or relied on pre-trained keypoint detectors built from costly annotations §1. Monkey-Net removed the need for annotations by learning keypoints, but it moves only the keypoint positions. This ignores local appearance change around each keypoint and produces poor results for large pose changes §1 §3.1.

The method assumes that source and driving object are of the same category and that, at test time, they are roughly aligned. Relative motion transfer is valid only if the two objects have similar poses; otherwise the transferred keypoints can fall at physically impossible locations §3.4.

Method

Overview. The model is trained to reconstruct training videos from a single frame plus a learned, keypoint-based representation of the motion in the video. Training uses frame pairs from the same video. At test time the source image and each frame of the driving video are processed independently §3 §3.4 fig. 2.

Step 1: keypoints with local affine transforms §3.1. The keypoint detector is a U-Net that predicts K heatmaps, one per keypoint. A softmax over each heatmap gives a confidence map, and the keypoint location is its expected value. The same network outputs four more channels per keypoint, which are averaged with the same weights to give a 2x2 Jacobian. The paper uses K = 10 §4. Each keypoint therefore carries 2 + 4 = 6 numbers, and the motion representation is K × 6 = 60 floats §4.

The motion model is a first-order Taylor expansion of the transform from a reference frame R to each image X at the keypoints eq. 1 eq. 2. Composing the transforms from driving frame D to source S gives the local approximation below, where the Jacobian J_k is taken at keypoint k eq. 3 eq. 4 eq. 5 app. A.

\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{D}}(z) \approx \mathcal{T}_{\mathbf{S} \leftarrow \mathbf{R}}(p_k) + J_k \,(z - \mathcal{T}_{\mathbf{D} \leftarrow \mathbf{R}}(p_k)) \qquad [\text{eq. 4}]

J_k = \left.\frac{d}{dp}\mathcal{T}_{\mathbf{S} \leftarrow \mathbf{R}}(p)\right|_{p=p_k} \left(\left.\frac{d}{dp}\mathcal{T}_{\mathbf{D} \leftarrow \mathbf{R}}(p)\right|_{p=p_k}\right)^{-1} \qquad [\text{eq. 5}]

With the Jacobian set to the identity this reduces to the zeroth-order motion of Monkey-Net §3.1.

Step 2: dense motion §3.1. For each keypoint, a heatmap H_k marks where the motion happens, built as the difference of two Gaussian-like bumps centred on the driving and source keypoints eq. 6. The source image is also warped by each local transform, giving K + 1 images (the extra one is the unwarped background). A U-Net takes the heatmaps and these warped images and predicts K + 1 masks. The dense backward flow from D to S is the mask-weighted sum of the K local motions, with the identity term M_0 z modelling static background eq. 7.

Step 3: occlusion-aware generation §3.2. The generator encodes the source image through two down-sampling blocks, warps the feature map with the dense flow, and multiplies it by an occlusion map in [0, 1]. The occlusion map is one extra output channel of the dense motion network, so it is estimated from the keypoint representation without its own loss. The masked features then pass through the remaining generator layers, which render the output frame eq. 8 §3.2.

Step 4: training losses §3.3. The main term is a perceptual loss from a pre-trained VGG-19, computed as an L1 distance between feature maps. The same term is repeated on an image pyramid built from the driving frame and the reconstruction. There are 20 loss terms in total eq. 9 §3.3. Two equivariance losses are added. The keypoint loss requires keypoints to move consistently when the image undergoes a known deformation eq. 10 eq. 11. The Jacobian version is written as a condition that a product of Jacobians equals the identity, which avoids the collapse to zero that a direct L1 term would cause eq. 12 eq. 13 §3.3. The three terms carry equal weights in all experiments §3.3.

Step 5: relative motion transfer at test time §3.4. Rather than copying the absolute keypoint positions of the driving video, the model moves each source keypoint by the displacement the driving keypoint made relative to its first frame, and applies the change in Jacobian in the same way eq. 14 eq. 15 §3.4. Absolute transfer is also possible; the paper says it makes the output inherit the driving object's proportions §3.4.

Equivariance transform used in training §3.3 app. B. The random deformation is an affine map plus a thin-plate-spline (TPS) on a 5×5 control grid, with the parameters drawn from normal distributions. The paper gives the spread of the affine part as 0.05 and of the TPS part as 0.005 app. B.

L_{rec}(\hat{\mathbf{D}}, \mathbf{D}) = \sum_{i=1}^{I} \left| N_i(\hat{\mathbf{D}}) - N_i(\mathbf{D}) \right| \qquad [\text{eq. 9}]

\mathbb{1} \equiv \left(\left.\frac{d}{dp}\mathcal{T}_{\mathbf{X} \leftarrow \mathbf{R}}(p)\right|_{p=p_k}\right)^{-1} \left.\frac{d}{dp}\mathcal{T}_{\mathbf{X} \leftarrow \mathbf{Y}}(p)\right|_{p=\mathcal{T}_{\mathbf{Y} \leftarrow \mathbf{R}}(p_k)} \left(\left.\frac{d}{dp}\mathcal{T}_{\mathbf{Y} \leftarrow \mathbf{R}}(p)\right|_{p=p_k}\right) \qquad [\text{eq. 13}]

Here N_i is the i-th channel of a VGG-19 layer and 𝟙 is the 2×2 identity eq. 9 eq. 13.

# Keypoint detector: K keypoints, each with a 2x2 Jacobian  [§3.1] [app. B]
# Input:  frame (B, 3, 256, 256), resized to 64x64 inside the network
# Output: value (B, K, 2), jacobian (B, K, 2, 2)

def detect_keypoints(frame):
    # One heatmap per keypoint, softmax over all pixels  [§3.1]
    logits = keypoint_unet(frame)                 # (B, K, 64, 64)
    heatmaps = softmax(logits / temperature)      # (B, K, 64, 64)
    # Keypoint = expected grid position under its heatmap  [§3.1]
    value = sum(heatmaps * grid)                  # (B, K, 2)
    # Four extra channels per keypoint, averaged with the same weights  [§3.1]
    jacobian_maps = jacobian_head(frame)          # (B, K, 4, 64, 64)
    jacobian = sum(heatmaps * jacobian_maps)      # (B, K, 2, 2)
    return value, jacobian

# Local motion of keypoint k, first-order Taylor  [eq. 4] [eq. 5]
# Input:  z (2,) pixel in the driving frame; keypoint values and Jacobians
# Output: (2,) matching location in the source frame

def local_motion(z, kp_source, kp_driving):
    # Jacobian of the driving-to-source map at the keypoint  [eq. 5]
    jacobian = kp_source.jacobian @ inverse(kp_driving.jacobian)
    # Affine approximation around the keypoint  [eq. 4]
    offset = z - kp_driving.value
    return kp_source.value + jacobian @ offset

# Dense backward flow D -> S and occlusion map  [eq. 6] [eq. 7] [§3.1] [§3.2]
# Output: flow (H, W, 2) with values in source coordinates; occlusion (H, W)

def dense_motion(source, kp_source, kp_driving):
    # Background term: identity, so M_0 z keeps static pixels in place  [eq. 7]
    candidates = [identity_grid]
    # Local motion for each keypoint, evaluated on the grid  [eq. 4]
    for k in range(K):
        candidates.append(local_motion(grid, kp_source[k], kp_driving[k]))
    # Heatmaps: difference of two Gaussian bumps, one per keypoint  [eq. 6]
    heatmaps = [gaussian_heatmap(kp_driving[k], kp_source[k])
                for k in range(K)]
    # Warp the source by each local motion, plus the unwarped one  [§3.1]
    warped_sources = [warp(source, c) for c in candidates]
    # Masks over K+1 parts from one U-Net  [§3.1]
    logits = dense_unet(concat(heatmaps, warped_sources))
    masks = softmax(logits, dim=parts)            # (K+1, H, W)
    # Mask-weighted sum of the K+1 local motions  [eq. 7]
    flow = sum(masks[j] * candidates[j] for j in range(K + 1))
    # Occlusion map in [0, 1], one extra output channel  [§3.2]
    occlusion = sigmoid(occlusion_head(logits))
    return flow, occlusion

# Occlusion-aware generation  [eq. 8] [§3.2]
# Input:  source image, flow and occlusion from dense_motion
# Output: generated frame

def generate(source, flow, occlusion):
    # Two down-sampling blocks  [app. B]
    features = encoder(source)
    # Back-warp the features with the dense flow  [eq. 8]
    warped = warp(features, flow)
    # Hadamard product with the occlusion map, to inpaint hidden parts  [eq. 8]
    masked = occlusion * warped
    # Remaining generator layers render the frame  [§3.2] [app. B]
    return decoder(masked)

# Training objective: reconstruction plus two equivariance terms  [§3.3]
# Input:  driving frame, its reconstruction, a random warp T
# Output: scalar loss, all three terms with equal weight  [§3.3]

def training_loss(driving, reconstructed, warp_transform):
    # Perceptual term over 4 pyramid scales and 5 VGG-19 layers  [eq. 9] [§3.3]
    perceptual = 0
    for scale in [1, 0.5, 0.25, 0.125]:
        perceptual += vgg_l1(pyramid(reconstructed, scale),
                             pyramid(driving, scale))
    # Keypoints must move with a known warp T  [eq. 10] [eq. 11]
    warped_frame = warp_transform(driving)
    kp_warped = detect_keypoints(warped_frame)
    equivariance_value = l1(kp_driving.value,
                            warp_transform(kp_warped.value))
    # Jacobians: product must equal identity, avoiding L1 collapse  [eq. 13]
    product = inverse(kp_driving.jacobian) @ T_jacobian @ kp_warped.jacobian
    equivariance_jacobian = l1(identity_2x2, product)
    return perceptual + equivariance_value + equivariance_jacobian

Hyperparameters

Results

Author-reported. Setup: four datasets, all at 256 × 256 §4: VoxCeleb (19,522 training and 525 test videos), Nemo (1,116 training and 124 evaluation videos), BAIR (42,880 training and 128 test videos, 30 frames each), and the authors' Tai-Chi-HD (3,049 training and 285 test chunks). Baselines are X2Face and Monkey-Net, the only model-free methods the authors list §4. Metrics: L1 distance, Average Keypoint Distance (AKD), Missing Keypoint Rate (MKR, Tai-Chi-HD only) and Average Euclidean Distance (AED) on video reconstruction §4.

Video reconstruction, lower is better tab. 3:

Paired user study, share of votes for this paper tab. 2. Each question was answered by 10 Amazon Mechanical Turk workers, over 50 input pairs §4:

Ablation on Tai-Chi-HD, video reconstruction tab. 1:

The paper reports the qualitative ablation in fig. 3. The pyramid loss improves every metric except AKD, the occlusion map improves all metrics over Pyr., and the Jacobian equivariance term is needed to keep the Jacobians stable §4. The MKR of the full model is 2.75 times smaller than the baseline's, as stated in the text §4.

Qualitative comparisons are in fig. 4 and in the supplement app. C. The paper states that its results are visually better than both baselines on Tai-Chi-HD, where Monkey-Net and X2Face warp the body as a blob §4. Comparisons are visual; no separate numbers are given for them.

Paper vs code

Code: official repository AliaksandrSiarohin/first-order-model at commit c0274845cb2dd8f0f2fe6da580d97b60fef90c91 (linked from the paper's footnote) §1. Checked against the training code, the Tai-Chi-HD and VoxCeleb configs, and the animation script.

• Heatmap H_k (eq. 6). The paper prints exp((T_D − z)² / σ) with no minus sign and no ½. The code uses a Gaussian with exponent −½ · d² / σ and σ = 0.01, then takes the driving bump minus the source bump dense_motion.py util.py dense_motion.py. Implement the code's form.
• Equivariance spread. The appendix says the TPS and affine parameters have variance 0.005 and 0.05 app. B. The code draws them with those values as standard deviations model.py model.py.
• TPS kernel. The code's thin-plate-spline term uses the L1 distance between points, not the Euclidean one, inside r² log r model.py. The paper gives no kernel formula.
• Reconstruction loss (eq. 9). Eq. 9 sums absolute differences over channels. The code takes the mean over all elements of each layer, scale by scale, with weight 10 per layer model.py.
• Batch size and learning-rate schedule. The appendix gives batch 20 and drops at T/2 and 3T/4 app. B. The shipped configs use batch 30 (Tai-Chi-HD) and 40 (VoxCeleb), and drop at epochs 60 and 90 of 100 taichi-256.yaml vox-256.yaml. The drops are applied by a MultiStep schedule with factor 0.1 train.py.
• Adam betas. The code uses betas (0.5, 0.999) for all three optimizers train.py. The paper gives only the learning rate.
• Relative vs absolute transfer. The caption of fig. 1 says absolute transfer is used for Tai-Chi-HD, but the shipped Tai-Chi-HD config sets relative movement and relative Jacobian to True taichi-256.yaml. Running the animation with that config therefore uses relative transfer, which is the mode §3.4 describes as the main method.
• Discriminator. The code trains a discriminator with weight 1 taichi-256.yaml. The paper does not describe one. In the shipped configs the generator's adversarial weight is 0, and the feature-matching term sits inside the check on that weight, so it is never added to the generator loss model.py taichi-256.yaml.
• Equivariance Jacobian (eq. 13). The code computes the mean absolute difference between the identity and the product of the inverse driving Jacobian, the warp Jacobian, and the transformed-frame Jacobian model.py. This matches eq. 13. The paper uses the same expression.
• Paper's claimed training budget. The paper gives about 2 days on 2 TitanX GPUs for Tai-Chi-HD and VoxCeleb app. B. Not verified here.

Pitfalls

• Eq. 6 sign and factor. As printed, eq. 6 grows with distance, so it is not a bump-shaped heatmap. Use the code's Gaussian form (see Paper vs code).
• Eq. 14 notation. The derivation in the appendix uses the first source frame S_1 in the reference term, but eq. 14 writes T_{S←R}(p_k). Read it as the source keypoints, which the appendix confirms app. A.
• Relative transfer needs aligned poses. Relative motion transfer assumes source and driving object have similar poses, and absolute transfer makes the output inherit the driving object's proportions §3.4.
• No temporal term. Each output frame is computed independently from the source and one driving frame §3.4. The paper does not add a term for temporal consistency.
• Matrix inversion in the dense flow. The dense motion network inverts the driving Jacobian with torch.inverse dense_motion.py. The paper assumes local bijectivity of the transform §3.1. The code has no explicit guard against near-singular Jacobians.
• Occlusion map resolution. The occlusion map is computed at 64 × 64 and resized bilinearly to the feature resolution before the multiplication generator.py. The paper does not state this.
• Flow resolution. The dense flow is computed at 64 × 64 and resized bilinearly to the feature grid generator.py app. B.
• Occlusion masks are hard to read. On BAIR the masks hide the robot arm, which the authors attribute to the moving object being the same in every frame. The masks also show low values on shadows, which warping cannot produce app. C.
• Keypoint count is fixed. K = 10 is used for all datasets §4. The baseline placed no keypoints on the arms, which the authors link to failure on large pose changes §4.
• Dataset preprocessing matters. VoxCeleb is built by tracking faces and cropping §4. Tai-Chi-HD is built from YouTube clips; the paper says the dataset will be released but does not give a URL §4. Availability was not checked here.
• Evaluation depends on third-party detectors. AKD and MKR use an external facial landmark detector on VoxCeleb and Nemo and a human-pose estimator on Tai-Chi-HD §4. Changing the detector changes the numbers.
• User studies are small. Each result is 50 input pairs with 10 raters per question §4.
• Costs. About 100k iterations at batch 20 to 40 on 256-pixel frames, roughly 2 days on 2 GPUs for Tai-Chi-HD and VoxCeleb app. B.
• Jacobian L1 collapse. A direct L1 on the Jacobian pulls its magnitude to zero, which is why eq. 13 is used instead §3.3.

Lineage

• Builds on: Monkey-Net (Siarohin et al., "Animating arbitrary objects via deep motion transfer", CVPR 2019), which uses zeroth-order keypoint motion. The paper sets J_k = 𝟙 to recover it §3.1. The bibliography gives no arXiv id.
• Builds on: equivariance constraints for unsupervised keypoints (Jakab et al., NIPS 2018; Zhang et al., CVPR 2018), extended here with Jacobian terms §3.3.
• Compared against: X2Face (Wiles et al., ECCV 2018), which uses absolute transfer and a larger motion embedding (128 floats) §4, and Monkey-Net §2 §4.
• Superseded by: not known from content already in this wiki.