VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

In plain words

Making a still photo talk along with a voice recording is hard to make look real. Earlier systems mostly got the lips right and left the rest of the face and the head still, or moved them in ways that looked stiff. VASA-1 aims at the whole face: blinks, glances, expression changes and small head movements that go with the speech.

The method has two parts. First, it learns a compact way to describe a face as a few separate pieces: who the person is, how the head is turned, the 3D look of the face, and how the face is moving. It learns this from many talking-face videos, and it is trained so that the pieces do not leak into each other. Second, a generator learns to produce the movement pieces from the sound. Because it works on these compact movement codes rather than on pixels, the final picture can be rendered quickly. The generator works in short overlapping chunks, so the result can be streamed while it is being made.

TL;DR

VASA-1 is an audio-driven talking-face model that works on a learned, disentangled face latent space. A 3D-aided encoder-decoder gives identity, head pose, facial dynamics and a canonical 3D appearance volume; a diffusion transformer generates pose and dynamics latents from wav2vec 2.0 audio features, with optional gaze, head-distance and emotion conditions. The authors report the best lip sync, pose-audio alignment, motion intensity and FVD against MakeItTalk, Audio2Head and SadTalker on two benchmarks they built, plus 512x512 video at up to 40 FPS online on one RTX 4090. The code is not public and the paper does not describe training losses or several architecture details, so reproduction needs guesses §3.1 §4.

Problem

• Lip-sync accuracy had been the focus of prior work, while expressive facial dynamics and natural head motion were largely left out, so results looked rigid §1.
• Many prior systems model different facial factors with separate models, some regressive and some generative, and train on few identities §1 §3.2.
• Video diffusion gives high quality but is too slow for interactive, low-latency use §1.
• The 3D-aided face representation that the authors build on is expressive, but the authors report that its original losses left facial dynamics and head pose poorly disentangled, and identity and motion imperfectly separated §3.1.

Method

Overview. The input is one face image, an audio clip, and optional control signals. The model does not render frames directly. It generates a sequence of head-pose and facial-dynamics latents conditioned on the audio, then a decoder turns those latents into frames together with the face's appearance volume and identity code §3 fig. 2.

1. Face latent space §3.1. Four separate encoders each pull one factor out of a frame: a 3D appearance volume that is warped into a canonical pose (V_app), identity (z_id), head pose (z_pose), and facial dynamics (z_dyn). One decoder rebuilds the frame from these. The architecture follows MegaPortraits, and the paper refers readers there for details §3.1.

2. Training losses for the latent space §3.1. Training swaps latents between two frames of one video. The base reconstruction losses come from MegaPortraits. Two losses are added:

   • a pairwise pose and dynamics transfer consistency loss l_consist, inspired by DPE: the head pose of frame i is moved onto frame j and the dynamics of j onto i, and the two results are pushed to agree §3.1;
   • a cross-identity loss l_cross_id: the motion of a frame of subject d is transferred onto a frame of subject s, and the cosine similarity between ArcFace features of the source face and the result is maximised §3.1.

3. Holistic motion generator §3.2. A motion sequence over a window of W frames is X = {[z_pose_i, z_dyn_i]}, paired with Wav2Vec2 audio features A §3.2. A transformer H is trained as a diffusion model on X. It predicts the clean motion X^0 directly rather than the noise §3.2 eq. 1:

   \mathbb{E}_{t\sim \mathcal{U}[1, T],~\mathbf{X}^0,\mathbf{C}}\left(\|\mathbf{X}^0-\mathcal{H}(\mathbf{X}^t, t, \mathbf{C})\|^2\right) eq. 1

   The forward process adds Gaussian noise in the usual way §3.2.

4. Conditions §3.2. The condition set is C = [X^pre, A^pre; A, g, d, e]: the last K frames of motion and audio from the previous window, the current audio, the main gaze g, the head distance d, and the emotion offset e. All conditions are concatenated with the noisy input along time.

   • g is a spherical-coordinate gaze direction, extracted per frame with L2CS and then clustered with a histogram-based method §3.2.
   • d is a normalised scalar for the face scale, from a 3D face fitting method §3.2.
   • e is a global offset; the authors say it is not meant to change emotion away from what the audio implies. They use averaged emotion coefficients from HSEmotion §3.2.

5. Classifier-free guidance §3.2. During training each condition is dropped at random. At inference the prediction is combined as follows eq. 2:

   \hat{\mathbf{X}}^0 = \Big(1 + \sum_{\mathbf{c}\in\mathbf{C}}\lambda_\mathbf{c}\Big)\cdot \mathcal{H}(\mathbf{X}^t, t, \mathbf{C}) - \sum_{\mathbf{c}\in\mathbf{C}}\lambda_\mathbf{c}\cdot \mathcal{H}(\mathbf{X}^t, t, \mathbf{C}|_{\mathbf{c}=\emptyset}) eq. 2

   Here λ_c is the guidance scale of condition c, and C|_{c=∅} replaces c by an empty condition §3.2.

6. Generation at test time §3.3. The portrait is encoded once into V_app and z_id. The audio is split into windows of length W. Motion windows are generated one after another with the diffusion transformer, each conditioned on the previous window, and the decoder turns the motion into frames §3.3.

# Swap-based training of the face latent space  [§3.1]
# Input: two frames i and j from one video
# Output: consistency loss (other loss terms and weights not stated)

def latent_consistency_loss(frame_i, frame_j):
    # Encode each frame into appearance, identity, pose and dynamics  [§3.1]
    app_i, id_i, pose_i, dyn_i = encode(frame_i)
    app_j, id_j, pose_j, dyn_j = encode(frame_j)

    # Put pose of i onto j, and dynamics of j onto i  [§3.1]
    pose_on_j = decode(app_j, id_j, pose_i, dyn_j)
    dyn_on_i = decode(app_i, id_i, pose_i, dyn_j)

    # The two transfers should agree; the distance is not stated  [§3.1]
    return discrepancy(pose_on_j, dyn_on_i)

# One training step of the motion diffusion transformer  [§3.2] [eq. 1]
# motion: clean latents [z_pose, z_dyn] per frame, shape (B, W, D)
# audio: Wav2Vec2 features, shape (B, W, F)

def training_step(motion, audio, conditions):
    # Draw a diffusion time step uniformly from 1..T  [§3.2] [eq. 1]
    t = sample_time_step()
    # Noise the clean motion with the forward process  [§3.2]
    noisy = add_gaussian_noise(motion, t)
    # Drop each condition at random; previous-window ones more often  [§3.2]
    conditions = random_drop(conditions)
    # The transformer predicts the clean motion itself  [§3.2]
    predicted = transformer(noisy, t, conditions)
    # Squared norm between prediction and clean motion  [§3.2] [eq. 1]
    return squared_norm(motion - predicted)

# Classifier-free guidance for one motion window  [§3.2] [eq. 2]
# scales: per-condition guidance weights, e.g. {"audio": 0.5, "gaze": 1.0}
# The sampler is not named in the paper; 50 steps is the default  [§4]

def sample_window(conditions, scales):
    # Start the window from Gaussian noise  [§3.3]
    motion = random_normal(window_shape)
    for t in time_steps():
        # Prediction with all conditions present  [eq. 2]
        cond_pred = transformer(motion, t, conditions)
        # Scale up the conditional prediction  [eq. 2]
        guided = (1 + sum(scales.values())) * cond_pred
        # Subtract one prediction per condition, that condition emptied  [eq. 2]
        for name in scales:
            empty = empty_condition(conditions, name)
            guided -= scales[name] * transformer(motion, t, empty)
        # Sampler update from guided to the next step  [§3.2]
        motion = update(motion, guided, t)
    return motion

# Sliding-window generation of a whole clip  [§3.3]
# Window length W and overlap K are not stated

def generate_clip(portrait, audio_windows, scales):
    # Encode the portrait once: appearance volume and identity code  [§3.3]
    appearance, identity = encode_face(portrait)
    previous_motion = empty_motion()
    previous_audio = empty_audio()
    frames = []
    for audio in audio_windows:
        # Condition on the last K frames of previous motion and audio  [§3.2]
        conditions = build_conditions(previous_motion, previous_audio, audio)
        motion = sample_window(conditions, scales)
        # Decode each generated motion frame into an image  [§3.3]
        frames.extend(decode_motion(appearance, identity, motion))
        previous_motion = motion[-K:]
        previous_audio = audio[-K:]
    return frames

Hyperparameters

Results

Author-reported. Setup: a 460-clip subset of VoxCeleb2 (46 subjects, 10 clips each, 5 to 15 s, 80% under 10 s) and OneMin-32 (32 clips of one minute from 17 people, mostly coaching and lectures) §4. Deterministic baselines produce one video per audio; SadTalker and the method produce three samples and average the metrics. Pose metrics are recomputed on generated frames §4.1. The paper says the real-video row is included for reference §4.1.

VC2 = VoxCeleb2 subset, OM32 = OneMin-32. The paper does not evaluate FVD on VoxCeleb2 §4.1. S_C is SyncNet confidence, S_D SyncNet distance, CAPP the authors' audio-pose score, ΔP mean adjacent-frame pose change §4.1. The authors note that the method's lip-sync scores exceed those of real videos, and attribute this to audio CFG §4.1 §4.3.

Ablations §4.3:

• Audio and gaze CFG and sampling steps tab. 4. Columns: S_C ↑, S_D ↓, CAPP ↑, ΔP, FVD25 ↓, gaze angular error E_θg ↓, head-distance error E_s ↓.

  λ_A	λ_g	steps	S_C	S_D	CAPP	ΔP	FVD25	E_θg	E_s
  0.0	0.0	50	7.087	7.391	0.414	0.291	117.425	5.730	0.004
  0.0	1.0	50	7.134	7.345	0.421	0.290	116.547	5.329	0.004
  0.0	2.0	50	7.108	7.386	0.414	0.298	117.784	5.064	0.005
  0.5	1.0	50	7.957	6.635	0.465	0.316	105.884	5.253	0.005
  1.0	1.0	50	8.218	6.437	0.474	0.342	104.886	5.333	0.005
  2.0	1.0	50	8.295	6.397	0.455	0.395	104.293	5.531	0.005
  0.5	1.0	10	8.293	6.363	0.523	0.243	117.060	5.469	0.006

  Bold marks the default configuration. The real-video reference is S_C 7.192, S_D 7.254, CAPP 0.559, ΔP 0.405, FVD25 29.244 tab. 4.

• CAPP sensitivity to frame shift tab. 2: 0 frames 0.608; ±1 0.462; ±2 0.206; ±3 0.069; ±4 0.082. The shift test uses about 2.1K 3-second pairs from VoxCeleb2 §4.3.

• CAPP sensitivity to pose-change scale tab. 3: ×0.2 0.368; ×0.5 0.584; ×1.0 0.608; ×1.5 0.587; ×3.0 0.505.

• Identity loss l_cross_id: mean face-identity cosine similarity over generated frames of 5,400 cross-identity clips rises from 0.72 to 0.80 §4.3.

• Consistency loss l_consist: without it, subtle dynamics such as side glances and lip asymmetry are hard to reproduce separately from head pose §4.3 fig. 4.

• Training data scale: 10% of data (50K clips) keeps lip-sync and pose-sync near the full model, with worse FVD and ΔP §4.3 tab. 1.

Qualitative: controllability over gaze, head distance and emotion offset §4.2 fig. 3; disentanglement of identity and motion app. B fig. A.1; disentanglement of head pose and dynamics app. B fig. A.2; out-of-distribution inputs (artistic photos, singing, non-English speech) §4.2 fig. A.3; comparisons on four spoken phrases app. B fig. A.4 to fig. A.7.

Inference: 512x512 frames at 45 FPS in offline batch mode, up to 40 FPS in online streaming with 170 ms preceding latency, on one RTX 4090 §4. The paper also reports a detector that separates real from generated video with 97.8% accuracy, as a safeguard note app. A.

Paper vs code

No official code. The project page linked in the abstract contains no code link and no statement about code or weights release. A GitHub repository search for VASA-1 returned 571 results; the first 19 were read, and none is owned by Microsoft or by an author. A search for vasa restricted to the microsoft organisation returned no results. Several third-party repositories with the name exist; none is used here.

Pitfalls

• No code, no weights. Everything must be reconstructed from the text. The encoders, decoder and training losses are largely deferred to MegaPortraits and to the cited 3D-aided reenactment work §3.1, and those papers are not in this wiki.
• Training losses and weights are not given. The base losses, their weights, and the distance used in the consistency and identity terms are not stated §3.1.
• Eq. 1 and the diffusion details. The model predicts X^0, not the noise §3.2. The number of time steps, the noise schedule, and the sampler are not stated. A standard DDPM or DDIM implementation is a guess.
• Eq. 2 is ambiguous. The sum over conditions uses C|_{c=∅} with c inside the sum, so the paper does not say whether each unconditional term empties one condition or all of them. The per-condition reading is what the notation says. Per-condition scales are given only for A and g; the scales for d and e are not stated.
• Training dropout doubles as the first-window case. The 0.5 drop rate for X^pre and A^pre is there so the model handles the first window, which has no preceding motion or audio §3.2. The paper does not say how that empty condition is encoded.
• Windows. The window length W and the overlap K are not stated. The model handles audio shorter than the window by dropping the last frames of A during training §3.2.
• Dimensions. The sizes of z_pose, z_dyn and the motion vector are not in the published text, so the shapes in the pseudocode are symbolic.
• Metric caveats. CAPP is the authors' own score, trained with a CLIP-style objective on 2K hours of their data §4.1. Its response to frame shifts and pose scaling is reported in tab. 2 and tab. 3. With audio CFG, the SyncNet lip-audio scores exceed those of real videos. The authors attribute this to CFG §4.1 §4.3; a reader should weigh that accordingly.
• Baseline runs are not described. The paper does not say which checkpoints or settings were used for MakeItTalk, Audio2Head and SadTalker §4.1.
• Data is not public. About 3.5K subjects come from an internal set §4. Reproducing the diffusion model at the stated scale needs equivalent data.
• Pose estimator for evaluation is not named in the published text §4.1.
• Scope limits stated by the authors. The method covers the human region only up to the torso; hair and clothing are not modelled; artifacts such as texture sticking can appear §5.
• Version. The compiled source is arXiv v2 (updated 2024-10-31). The license recorded by wikillm fetch is CC BY 4.0.

Lineage

• Builds on: the 3D-aided face reenactment framework of MegaPortraits and the one-shot free-view talking-head work §3.1; the diffusion transformer approach §3.2; classifier-free guidance §3.2 (arXiv:2207.12598 in the bibliography); Wav2Vec2 audio features §3.2. The disentanglement consistency term is inspired by DPE §3.1.
• Compared against: MakeItTalk, Audio2Head and SadTalker §4.1.
• Concurrent work named by the authors: EMO arXiv:2402.17485 and AniPortrait, which also adapt video diffusion to talking faces §2.
• Superseded by: not stated in the wiki content.