MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

In plain words

Given one photo of a person and a sequence of body poses, the goal is a video of that same person moving through those poses, with the clothes, face and background of the photo kept intact. Older methods either warp the photo with optical flow, which breaks down when the pose changes a lot, or generate each frame separately, which makes clothes and backgrounds flicker from frame to frame.

MagicAnimate uses a diffusion image generator that is extended to handle a whole clip at once, so neighbouring frames can agree with each other. A second, trainable copy of the network reads the reference photo in detail and passes those details into the generator, so the person keeps their look. Poses come in as dense body-surface maps rather than a few joints. For long videos the clip is cut into overlapping pieces that are generated together and blended where they overlap, so the joins are smooth.

TL;DR

• Latent diffusion model (Stable Diffusion 1.5 UNet) with pose control via a ControlNet fed by DensePose §3.3 animation.yaml.
• Temporal attention layers make frames in a clip agree with each other §3.1.
• An appearance encoder, a second trainable UNet copy, feeds reference features into the generator's self-attention §3.2 eq. 2.
• Training code is not in the released repository; only inference code is. The training description is paper-only.
• Results are author-reported; the supplementary material that holds training details was not part of the source we compiled from.

Problem

Frame-warping animation methods lose temporal consistency and reference identity when the pose changes a lot §1. Diffusion methods that process each frame independently flicker, and CLIP image embeddings of the reference are too sparse to keep fine detail §1 §3. Long videos processed segment by segment show visible seams §3.3.

Method

Temporal consistency modeling §3.1. The 2D image UNet is inflated to a 3D temporal UNet by inserting temporal attention layers §3.1. A clip of K frames is reshaped so that attention runs across the time axis for each spatial position §3.1. Sinusoidal positional encodings tell the model where each frame sits in the clip §3.1. The attention itself is standard scaled dot-product attention over the frames §3.1.

Appearance encoder §3.2. A trainable copy of the base UNet, written F_a, processes the reference image at each denoising step and produces normalised attention hidden states for the middle and upsampling blocks §3.2 eq. 1. These states are concatenated to the keys and values of the generator's spatial self-attention, so queries from the video frames can read details from the reference §3.2 eq. 2. The paper contrasts this with ControlNet's residual injection §3.2.

Pose conditioning §3.3. DensePose maps are chosen over OpenPose keypoints because keypoints are sparse and less robust to rotation §3.3. A pose ControlNet F_p turns each pose frame into condition residuals added to the middle and upsampling blocks §3.3 eq. 3.

Denoising §3.3 eq. 4. The noise estimate for a K-frame segment is the temporal UNet's output, given the noisy latents, the timestep, the appearance features and the pose residuals §3.3 eq. 4.

Long video §3.3. The full N-frame video is split into overlapping K-frame segments. Their noise predictions are averaged on overlapping frames at every timestep §3.3. The paper says that all segments start from the same initial K-frame noise block, which it found improves quality §3.3. The last segment, when shorter than K, is padded with frames from the start §3.3.

Sampling loop, as the paper describes it and as the released code runs it. The overall pipeline is in fig. 2:

# Sample a video of N frames  [§3.3]
# Input:  reference (H, W, 3), poses (N, H, W, 3) DensePose
# Output: frames (N, H, W, 3)

def animate(reference, poses, steps=25, K=16, s=4):
    # Clean reference latent, encoded once  [§3.2]
    ref_latent = vae_encode(reference)  # (4, H/8, W/8)
    # One K-frame noise block, tiled over all N frames  [§3.3]
    block = random_normal(4, K, H / 8, W / 8)
    latents = tile(block, N // K)  # (4, N, H/8, W/8), N multiple of K
    for t in ddim_timesteps(steps):  # DDIM, eta = 0  [§3.3]
        # Pose residuals, computed per frame  [§3.3] [eq. 3]
        pose_res = pose_controlnet(latents, poses, t)
        # Windows of K frames, starting every K - s frames  [§3.3]
        noise_sum = zeros_like(latents)
        counts = zeros(N)
        for window in windows(N, K, step=K - s):  # wraps modulo N
            # Denoise the window with reference attention  [eq. 2] [eq. 4]
            noise = unet(latents[:, window], t, ref_latent,
                         pose_res[window])
            noise_sum[:, window] += noise
            counts[window] += 1
        # Average the overlapping predictions  [§3.3]
        noise = noise_sum / counts
        # Classifier-free guidance, scale 7.5  [§3.3]
        noise_uncond, noise_cond = split(noise)
        noise = noise_uncond + 7.5 * (noise_cond - noise_uncond)
        latents = ddim_step(noise, t, latents)
    return vae_decode(latents)

Training §3.4. Stage 1 trains the appearance encoder and the pose ControlNet without temporal layers, using a per-image denoising loss §3.4 eq. 5. Stage 2 trains only the temporal attention layers on K-frame clips §3.4 eq. 6. A joint image-video strategy samples single images from a large image dataset with probability threshold τ0 in stage 1, and mixes images and videos with thresholds τ1 and τ2 in stage 2 §3.4 eq. 7. The threshold values are not stated in the text.

Hyperparameters

Values from the released inference configuration are listed with their location. "not stated" means the paper text compiled here does not give a value.

Results

Author-reported. Setup: TikTok test set as in DisCo; TED-talks official split §4. Single-frame metrics and video metrics are as defined in §4.1. Rows marked (ref.) are GAN methods that use ground-truth frames as the driving signal and are shown only for reference tab. 1 caption. The paper marks best results in bold and second-best underlined; those markers are not carried over here.

TikTok tab. 1a:

TED-talks tab. 1b: L1 is reported as overall L1 and foreground L1 (L1_fg).

Headline claims in the text: on TikTok, FVD is 38.8% lower than DisCo's (179.07 vs 292.80) and FID-VID is 63.7% lower §4.1. On TED-talks, FVD is 28.1% lower than MRAA §4.1. On TED-talks the overall L1 is higher than MRAA's and DisCo's; the paper attributes this to DensePose carrying no background, and says the foreground L1 is comparable to MRAA's §4.1 tab. 1b.

Ablations on TikTok tab. 2: L1 is shown as ×10⁻⁴ in the table caption.

2a, temporal attention tab. 2a:

2b, appearance encoder tab. 2b:

2c, image-video joint training tab. 2c: Spat and Temp are the table's column labels for joint training on each stage; the text applies it to the appearance-encoding and temporal-modeling stages §4.2.

2d, inference-stage video fusion tab. 2d:

2e, shared initial noise across segments tab. 2e:

Applications, shown only as figures fig. 5: unseen-domain animation of paintings and film frames, combination with DALL·E 3 generated references, and two-person animation. These are qualitative and have no metrics §4.3.

Paper vs code

Code: magic-research/magic-animate at commit d2bc3bc3c9cc3dadeedef8a02668dd5582d10a01, BSD-3-Clause LICENSE. Found through the project page the paper links (showlab.github.io/magicanimate), which links the repository. The arXiv abstract says "Code and model will be made available" and links nothing. Model weights are not in the repository; the README points to Hugging Face README.md.

Discrepancies between paper and code:

• Appearance encoder input. Eq. 1 writes the encoder's input as the noisy latent z_t eq. 1. The pipeline encodes the clean reference image with the VAE and passes that un-noised latent, together with the timestep, at each denoising step pipeline_animation.py pipeline_animation.py.
• Initial noise. The paper says all segments share one K-frame noise block §3.3. The code samples one 16-frame block and tiles it across the whole video pipeline_animation.py. Frame f and frame f+16 therefore start from identical noise. Windows start every 12 frames (16 − 4) context.py, so a window starting at frame 12 sees noise block entries 12 to 15 and then 0 to 11, not the block's first 16 entries in order.
• Padding of the last segment. The paper pads a short last segment with the first frames §3.3. The demo driver pads the pose sequence to a multiple of 16 by repeating its last frame animation.py, and the windowing wraps around modulo N context.py.
• Pose residual injection. The paper adds pose residuals to the middle and upsampling blocks §3.3. The UNet adds them to every down-block skip residual as well as to the middle block unet_controlnet.py unet_controlnet.py.
• Pose ControlNet windows. Pose residuals are computed once per frame over non-overlapping windows (overlap 0) and cached; the UNet then runs on overlapping windows (overlap 4) pipeline_animation.py pipeline_animation.py pipeline_animation.py. The paper does not describe this.
• Joint training cases. The text describes two cases, image sampling for r ≤ τ1 and video otherwise §3.4. Eq. 7 has three cases, with a single video frame for τ1 ≤ r ≤ τ2 eq. 7. The released code has no training loop, so neither reading can be checked.
• Step count key. The driver reads S (value 1) into a variable named step, which the call does not use animation.py. The pipeline call uses steps (25) animation.py animation.yaml.
• Training. The paper gives two training objectives eq. 5 eq. 6 and a joint image-video case selection eq. 7 §3.4. The repository has inference scripts only (scripts/animate.sh, scripts/animate_dist.sh, demo/) and no training code animate.sh.
• Code-only details (not in the paper):
  • In the reference attention, the unconditional branch of classifier-free guidance skips the reference features and attends only to its own tokens mutual_self_attention.py.
  • The mask that splits the conditional and unconditional halves hard-codes 16 frames per window mutual_self_attention.py; a different window length falls back to a shape-based mask mutual_self_attention.py.
  • Temporal layers come from a motion module placed at resolutions 1, 2, 4 and 8, with the UNet's own temporal-attention flag off inference.yaml inference.yaml. The paper does not give these positions.
• Consistent details. The sinusoidal positional encoding §3.1 matches the code motion_module.py. The reference features are normalised hidden states, as §3.2 describes mutual_self_attention.py.

Pitfalls

• Missing training details. The text points to the supplementary material for the dataset preprocessing and implementation details §4. The supplementary file in the source tree contains only a title, and main.tex has it commented out, so learning rate, optimiser, batch size, number of steps, τ0 to τ2 and frame sampling are not stated anywhere we have. An implementer cannot reproduce training from this card.
• Joint training claim. The text says joint training "consistently increases" quality §4.2. In tab. 2c, FVD is lowest without the spatial joint training (156.32 with Temp on, 158.16 with both off), and the full model has 179.07. Joint training improves FID and FID-VID in that table, but not FVD.
• Initial noise. The paper and code differ on how the shared noise is indexed. Check this when you reimplement the windows. The ablation in tab. 2e shows the same-noise setting is better on FID and FID-FVD but worse on L1.
• Reference latent. Eq. 1 conditions on z_t, but the code feeds the clean reference latent, as the Paper vs code section notes. Match the code if you want the released behaviour; match eq. 1 if you want the text's version. The two are not checked against each other.
• Compute. Each denoising step runs the appearance-encoder UNet once, and the pose ControlNet and temporal UNet on every window, with classifier-free guidance doubling the batch pipeline_animation.py. Long videos multiply the window passes by the number of windows. The driver casts all modules to fp16 animation.py and moves the appearance encoder to CUDA animate.py.
• DensePose input. The repository ships pre-extracted DensePose videos but no DensePose estimator . The paper cites DensePose §3.3 without a version. Pose extraction is a separate step you must match yourself.
• Weights and their licence. The released checkpoints (pose ControlNet, appearance encoder, temporal module) and the Stable Diffusion 1.5 and VAE base models are downloaded separately README.md. Their licences were not checked for this card. Do not treat the BSD-3-Clause code licence as covering them.
• Repository age. The commit used here may postdate the paper, and the paper says code "will be made available" [abstract]. Behaviour in other commits is not checked.
• Author-reported only. All numbers are the authors' own. The DisCo, TPS* and MRAA* numbers are cited from the DisCo paper, and the other baselines were run or trained by the authors; TPS and MRAA were trained with DensePose as the driving signal tab. 1a caption §4.1. The TED-talks L1 is worse than MRAA's, and the paper attributes this to missing background in DensePose §4.1.

Lineage

• Builds on: Stable Diffusion, which the code uses as the backbone animation.yaml; ControlNet for pose conditioning §3.3; temporal attention modules as in AnimateDiff §3.1; reference-attention ideas from MasaCtrl and Reference-only ControlNet §3.2.
• Compared against: TPS and MRAA (GAN-based warping) §4.1; DisCo (diffusion, per-frame) §4.1; IP-Adapter combined with pose ControlNet, with and without temporal layers §4.1.
• Superseded by: not stated in the source text used here.