EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

In plain words

Most systems that make a still photo talk first work out a skeleton of the face or a fixed head motion from the sound, and then paint the picture onto that skeleton. The skeleton keeps things steady, but it also keeps the face stiff, and every person moves a little differently.

EMO skips the skeleton. It takes one photo and the sound, and a model that learned to draw images from noise draws each video frame directly, guided by the sound. A second copy of the network looks at the photo so the person still looks like themselves. A few loose hints, such as roughly where the face is and roughly how fast the head should turn, keep the picture from wandering without forcing a fixed motion. The result is a video that can include head movement, expression and singing, not only moving lips.

TL;DR

• One reference image and one audio clip go in; a video comes out. No 3D face model, landmark sequence or pose template is used §1 §3.
• The backbone is a Stable Diffusion 1.5 style UNet with temporal layers. A ReferenceNet keeps identity, audio attention drives motion, and a face-box mask and a head-speed embedding act as weak conditions §3.2.
• Long videos are made from 12-frame clips that share 4 motion frames with the previous clip §3.2.4 §4.1.
• Author-reported on HDTF: FID 8.76 (best of the table) and FVD 67.66. SyncNet is lower than Wav2Lip and SadTalker, which the authors state tab. 1.
• The official repository has no code, so every implementation detail comes from the text. Read "Paper vs code" and "Pitfalls" before building.

Problem

Talking-head methods usually separate head motion from facial expression and constrain both with an intermediate signal: a 3D morphable model, blendshapes, 2D landmarks or a predefined pose sequence. The authors argue that these signals limit how expressive the result can be, because the signal cannot describe every facial movement and does not carry each person's style §1 §3.2.5.

The paper asks whether a diffusion model can learn the audio-to-motion mapping directly from data, without those signals. A second problem is stability. A model with only audio conditioning can drift or move too fast, so the authors add conditions that are weak on purpose §1 §3.2.5.

Assumptions: the training data is talking or singing video of a single person per frame, with a stable camera. The authors say they chose data this way [supp §B.1]. The paper shows anime and 3D portraits as well, although its training data is realistic video only §4.3.

Method

Overview. The method is a diffusion model that denoises a block of frames in parallel. Four inputs shape it: the reference image through ReferenceNet, audio features through audio layers, a face-region mask through the Face Locator, and a speed value through speed layers §3.2 fig. 2. Clips are generated one after another, and the last frames of one clip become the motion frames of the next §3.2.4.

Base model. The method starts from Stable Diffusion. An autoencoder maps images to latents, and the denoiser is trained with the standard noise prediction loss, written with text features c §3.1 eq. 1. The paper does not use a text prompt in its own model; the cross-attention layers of the SD 1.5 UNet are replaced by reference-attention layers §3.1 §3.2.1.

Step 1: backbone. The backbone is a UNet with the SD 1.5 layout. Each block has reference-attention, then audio-attention, and temporal layers are added at every resolution level §3.2.1 §3.2.2 §3.2.4.

Step 2: audio layers. Features from several blocks of a pretrained wav2vec model are concatenated per frame. For frame f, the features of the frames from f−m to f+m are joined, with m = 2 §3.2.2 §4.1. Each audio-attention layer is a cross-attention from the latent to these features, placed after the reference-attention layer §3.2.2.

Step 3: ReferenceNet. A second UNet with the same structure as the backbone reads the reference image and takes its self-attention feature maps. During denoising, the backbone's reference-attention layers attend to these maps. Both networks start from the SD weights. The ReferenceNet runs once per clip, and its features are reused across the denoising steps §3.2.3 §3.2.4.

Step 4: temporal modules. The temporal layers reshape a feature map of shape (b, c, f, h, w) to (b·h·w, f, c) and apply self-attention along the frame axis f §3.2.4. They are initialized from AnimateDiff §3.3.

Step 5: motion frames. The n motion frames of the previous clip are sent through ReferenceNet, and their feature maps are merged with the temporal-layer inputs along the frame axis. The first clip uses zero-valued motion frames §3.2.4. The paper describes their features entering the temporal layers; it does not say whether the motion frames are noised or denoised §3.2.4.

Step 6: Face Locator. The face region is the union of the face boxes of all frames in the clip, giving one mask M. A light stack of convolutional layers encodes M, and the encoded mask is added to the noisy latent before the backbone §3.2.5. The mask only allows a region; it is not a hard constraint §3.2.5 §4.5.2.

Step 7: speed layers. The head rotation speed of each frame is labelled from the 6-DoF head pose §4.1. The speed range is split into d buckets, each with a centre c_i and a radius r_i. For frame f, the value for bucket i is tanh(3·(w_f − c_i)/r_i). The values for the 2m+1 frames around f are stacked and passed through an MLP to a speed feature map §3.2.5. Inside the temporal layers, this speed feature is attended to in the same way as the audio. At inference, one constant speed value is set for all frames of a clip §4.1.

Step 8: training stages. Training has three stages §3.3:

1. Image pretraining: backbone, ReferenceNet and Face Locator are trained on single frames. The reference is another random frame of the same clip.
2. Video training: temporal modules and audio layers are added. Clips of n + f = 16 consecutive frames are used, the first n being motion frames.
3. Speed training: speed layers are added, and only the temporal modules and the speed layers are trained. The audio layers are left out, because the authors found that training them together weakened the audio's control of motion §3.3.

Inference. One clip of f = 12 frames is denoised with DDIM for 40 steps §4.1. The denoised latents are decoded to frames; the last 4 frames seed the next clip §3.2.4 §4.1.

Pseudocode (training step). Written by us from the steps above; the noise term and the loss follow eq. 1. Not stated in the paper: the noise schedule, the number of timesteps and the exact conditioning of the loss.

# EMO training step (one clip)  [§3.1] [§3.3] [eq. 1]
# Input:  clip (n + f frames), reference frame, audio features, face mask
# Output: scalar loss

def training_step(clip, reference, audio, face_mask, speed=None):
    # Encode every frame of the clip into latents  [§3.1]
    latents = vae_encode(clip)                  # (n + f, C, h, w)

    # Sample Gaussian noise and a timestep, then noise the latents  [§3.1]
    noise = sample_gaussian(latents.shape)
    timestep = sample_timestep()
    noisy = add_noise(latents, noise, timestep)

    # The Face Locator mask is added to the noisy latent  [§3.2.5]
    noisy = noisy + face_locator(face_mask)

    # ReferenceNet features from the reference frame  [§3.2.3]
    reference_features = reference_net(reference)

    # Speed layers are used only from the third training stage  [§3.3]
    predicted = backbone(
        noisy,
        timestep,
        reference_features,
        audio,
        speed,
    )

    # Noise-prediction loss  [eq. 1]
    return mean_squared_error(predicted, noise)

Pseudocode (inference over a sequence). Written by us from the steps above.

# EMO inference over an audio track  [§3.2.4] [§4.1]
# Input:  reference image, audio features, face boxes, speed value
# Output: frames for the whole track

def generate_video(reference, audio, face_mask, speed, f=12, n=4):
    # The first clip starts from zero-valued motion frames  [§3.2.4]
    motion_frames = zero_frames(n)
    video = []
    for clip_audio in split_into_clips(audio, step=f):
        # ReferenceNet runs once per clip; its features are reused  [§3.2.4]
        reference_features = reference_net(reference, motion_frames)

        # Start the f new frames from Gaussian noise  [§3.1]
        latents = sample_gaussian((f, C, h, w))

        # Denoise with DDIM, 40 steps  [§4.1]
        for timestep in ddim_timesteps(num_steps=40):
            latents = ddim_step(
                backbone,
                latents,
                timestep,
                reference_features,
                clip_audio,
                face_mask,
                speed,
            )

        # Decode the clip and keep its last n frames as motion frames  [§3.2.4]
        frames = vae_decode(latents)
        video.extend(frames)
        motion_frames = frames[-n:]
    return video

Hyperparameters

Values from the main text unless marked otherwise. "Not stated" means the paper does not give the value.

Results

Author-reported. Setup: the HDTF test split and 1k internet clips §4.2. Diffused Heads is compared only qualitatively §4.2 fig. 6.

Quantitative comparison tab. 1. Each cell is written "HDTF / internet"; the table does not label the columns, so this order is inferred from the text and the w/o row, which has values only in the HDTF position.

Arrows: FID, FVD and E-FID lower is better; SyncNet higher is better. F-SIM is read as closeness to GT §4.2. The w/o row is trained on VFHQ, HDTF and CELEB-V instead of the full data §4.4.

User study (20 participants, 1 to 5 scale; lip sync and vividness) [supp §A.1]

Ablation: speed layers (HDTF, mean head rotation velocity of generated clips) tab. 2

Ablation: Face Locator is qualitative only. A small face region keeps the head nearly still, a wider region allows more head swing, a taller region allows nodding, and a uniform white mask lets the face appear anywhere §4.5.2 fig. 7.

Other reported results.

• Lip sync: Wav2Lip has the highest SyncNet score on HDTF (5.79); the authors attribute this to training with SyncNet as a discriminator §4.4.
• Qualitative: Wav2Lip gives blurry mouths and a nearly static head; DreamTalk with the authors' style clips distorts faces and limits motion §4.3 fig. 3.
• Portrait styles: 14 characters of realistic, anime and 3D styles from text-to-image models, driven by the same audio. The model was trained on realistic video only §4.3 fig. 4.
• Long duration: clips of about 1 minute of singing are shown; identity is said to be kept across them §4.3 fig. 5.

Paper vs code

No official code. The author block links a project page, and that page links a GitHub repository, HumanAIGC/EMO, at commit ffead74c374d10ecb3d075839f866c863d35e941. That repository contains only a README, a banner image and a demo video. It has no source code, no weights and no LICENSE file, so it cannot be cited for any hyperparameter or algorithm step. Two other repositories under the same organisation that carry the project name (HumanAIGC/emote-portrait-alive and HumanAIGC/emote-portrait-alive-2) have a top-level listing of a README, an HTML page and asset folders, and no LICENSE file; their asset folders were not inspected. Searches: the project page (WebFetch), and the GitHub API for the repository names above. No third-party reimplementation is used in this card.

Pitfalls

• No code, no weights. Every implementation detail comes from the text. Items the text leaves open include the ReferenceNet attention wiring, the shape of the face-mask encoder, the speed MLP, the wav2vec checkpoint and the loss used when conditions change §3.2 §3.2.5 §3.2.2.
• Speed-layer parameters are missing. The number of buckets d, the centres c_i, the radii r_i and the speed feature size l are not stated §3.2.5. The units of the speed value (0.1 to 1.3 in §4.5.1) and of the velocity in tab. 2 are not stated either. Two implementers would pick different values and get different motion.
• Speed scale is steep near the centre. With the factor 3 in tanh, the bucket value is 0.76 at a distance of r_i/3 from the centre and about 0.96 at 2r_i/3. This follows from the formula §3.2.5; the paper does not discuss it.
• Speed values are noisy labels. The authors say the labelled speed is noisy and the generated motion only approximates the assigned level §3.2.5.
• Speed table needs careful reading. The "No speed" row has the highest mean velocity (1.365), above every speed-conditioned row (0.878 to 1.246) tab. 2. The text credits the speed layers with lower variance §4.5.1, but does not explain the higher mean without them. The reduction in variance is the main evidence.
• Stage 3 excludes the audio layers on purpose. Training them together is reported to weaken audio control of motion, but no numbers are given §3.3. Follow the freezing rule in §3.3 exactly.
• "Target image" is undefined. §3.2.4 says the target image and the motion frames are fed into ReferenceNet once. This card reads "target image" as the reference image, which is an assumption.
• Motion frames can leave seams. The authors say the method may show discontinuities at clip boundaries, because nothing strong controls the motion. They do not test a frame-replacing alternative in the text [supp §A.2].
• Jump cuts in training data. Speech videos contain jump cuts that the model can copy. The authors segment clips with scene detection and use the speed input to avoid jump cuts at inference. The claim that a coherent speed input avoids them is not backed by an experiment in the text [supp §B.2.1].
• Denoising step count is a sensitive choice. 40 DDIM steps are used §4.1. The supplement's guidance (fewer than 20 steps gives artifacts; more than 35 is stable) is qualitative [supp §A.3]. The DDIM variant (eta, timestep spacing) is not stated.
• Timing figures disagree. The main text gives about 15 s per 12-frame batch §4.1; the supplement gives about 18 s per 12 frames on an A100 at 40 steps [supp §C]. By arithmetic from these figures, one minute of 30 FPS video (150 clips) takes roughly 45 minutes to generate at the 18 s figure. Real-time use is not claimed.
• Lip sync is not the strength. On HDTF, SyncNet (3.89) is below Wav2Lip (5.79) and SadTalker (4.82) tab. 1. The authors say so and prioritise E-FID and FVD. If lip accuracy is the goal, these baselines are stronger.
• F-SIM bolding follows closeness to GT. The text says F-SIM is closer to GT is better §4.2. The table bolds MakeItTalk on HDTF (76.91, GT 77.44) and EMO on internet data (77.16, GT 72.64) under this rule. The caption does not explain the bolding.
• E-FID details are missing. It uses expression parameters from a face reconstruction method §4.2. The parameter count and the number of frames or clips behind each metric are not stated.
• No variance or seeds. FID, FVD, SyncNet and E-FID are reported without confidence intervals, seeds or repeated runs. The number of generated samples per method is not stated.
• Ablation for "w/o 250 h data" is partial. Only the HDTF column is given. The row is trained on VFHQ, HDTF and CELEB-V §4.4 and shows lower FVD and E-FID on HDTF than the full model (102.78 vs 67.66 FVD; 0.215 vs 0.116 E-FID) tab. 1.
• User study is small. 20 participants, ratings from 1 to 5 [supp §A.1]. The clips shown, the rating protocol and any spread across raters are not stated. The lip-sync gap to DreamTalk (4.17 vs 4.05) is small.
• Diffused Heads is excluded from the numbers. The authors call its outputs suboptimal: its model was trained on CREMA, which has green backgrounds, and errors build up across frames §4.2. No quantitative comparison exists.
• Failure modes. Body parts such as hands can appear where none should, and subtitle-like patterns can appear when training data had subtitles. About 3% of training frames contain hands [supp §C]. Audio is the only control signal; the authors say emotion cannot be set directly [supp §C].
• Loss conditioning is not restated. eq. 1 is the Stable Diffusion loss with text features c. The paper does not write the loss with EMO's own conditions (reference features, audio, mask, speed) §3.1.
• Dataset availability. The 250-hour collection is not stated to be released. The authors say the source videos are publicly accessible [supp §B.1]. Re-collecting this data is not reproducible from the paper.

Lineage

• Builds on: Stable Diffusion (latent diffusion) for the base model §3.1. Concepts: diffusion-models, latent-diffusion. No paper card in this wiki.
• Builds on: AnimateDiff for the temporal-module initialisation §3.3; no card in this wiki.
• Builds on: a ReferenceNet design from prior work on consistent identity §3.2.3, which the paper cites as two references (one of them Animate Anyone). No cards in this wiki.
• Builds on: wav2vec for audio features §3.2.2; MediaPipe for face boxes §4.1; scene detection after SVD for clip segmentation [supp §B.2.1]. No cards in this wiki.
• Compared against: Wav2Lip, SadTalker, DreamTalk, MakeItTalk tab. 1; Diffused Heads, qualitatively only §4.2.
• Related: 3D-model-based single-image methods, such as VividTalk and SadTalker, which use 3DMM or blendshape intermediates §2. No cards in this wiki.
• Superseded by: not stated in this wiki.