wikillm

Paper · 2022-04-18 · cs.CV

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion

Predicts 3D facial motion of a listener from a speaker's audio and face motion. A VQ-VAE turns listener motion into a 200-entry codebook, a transformer predicts the next code with cross-modal attention over speaker audio and motion, and sampling yields several plausible listeners. Author-reported: lower FD and P-FD than nearest-neighbour and Glow-based baselines on 72 h of interview video.

Vector quantizationCross-attentionNucleus (top-p) samplingMotion predictionmotion-synthesisfacial-animationvq-vaelistener-modelingdyadic-interaction

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

In a video call, the person listening rarely sits still. They nod, smile, raise an eyebrow or look away, and those small reactions tell the speaker that someone is paying attention. This paper asks a computer to produce that listener behaviour: you give it the speaker's face movements and voice, and it produces the matching face movements a listener might make.

The hard part is that listeners do not react the same way every time. So the method does not output one answer. It learns a small vocabulary of short facial motion pieces, each a little chunk of how a face moves. A model then predicts which piece should come next, given what the speaker just did and said. Picking pieces by sampling gives several different, believable listeners from the same speaker.

The authors also collected about 72 hours of interview videos to train on. Their reported comparisons say the generated listeners look more natural and better in sync with the speaker than simple baselines such as copying the speaker or replaying a nearest matching clip.

TL;DR

  • Task: given a speaker's facial motion and audio, autoregressively predict the listener's 3D facial motion §3.1.
  • Method: a sequence VQ-VAE with transformers builds a codebook of 200 listener motion codes §3.2; a cross-modal transformer fuses speaker audio and motion §3.3; an autoregressive predictor outputs a distribution over next codes §3.4.
  • Result (author-reported): best FD, P-FD and L2 among listed baselines on their in-the-wild data tab. 1.
  • Code is public, but has no license file, and the dataset is not released under a stated license. Treat the implementation as reference material, not as copy-paste source.

Problem

Earlier listener models used hand-written rules, simple hand-defined motions such as smiles or nods, or database lookups, and most were built on lab motion-capture data §1 §2. The authors want realistic listener motion that stays in sync with the speaker and can produce several plausible answers without drifting away from realistic faces over long sequences §1 §3.2.

Method

The model has three parts, each with its own citation.

  1. Motion representation §3.1. Each frame is a 3D Morphable Model expression vector (β, with jaw) concatenated with a normalised head rotation R in Euler angles eq. 3. Shape coefficients are discarded. The head rotation is aligned to the mean frontal pose of each video. The paper's dimension count is 53 for β and 3 for rotation, giving 56 per frame.

  2. Listener VQ-VAE codebook §3.2 fig. 3. A transformer encoder E maps a window of T listener frames to τ = T/w code vectors of dimension d_z. Each vector is snapped to its nearest entry of a codebook Z with K entries eq. 4. A transformer decoder D maps the quantised sequence back to motion eq. 5. Training minimises a reconstruction term, a term that pulls the encoder output toward its code, and a commitment term that pulls the code toward the encoder output eq. 6. Stop-gradient operators (sg) separate the two codebook terms.

  3. Speaker encoder with cross-modal attention §3.3. Speaker audio is a mel-spectrogram sampled at four times the motion frame rate. Speaker motion is not quantised. Audio supplies the queries of a cross-attention block, and speaker motion supplies keys and values eq. 7. After this, temporal convolutions downsample the speaker sequence to τ+1 positions (see Paper vs code for the count).

  4. Listener predictor §3.4 fig. 2. The listener's past is first turned into codebook indices by the frozen encoder and nearest-entry lookup eq. 8. A transformer reads the speaker embedding and the listener indices together and outputs a distribution over the K codes for the next step eq. 9. Training uses cross-entropy against the codes of the ground-truth future eq. 10. Teacher forcing is used, and past positions are randomly masked so the model learns to continue from its own predictions.

Generation is autoregressive. At each step a code is sampled from the predicted distribution, looked up in the codebook, and decoded to motion §3.4. The paper evaluates on 3D meshes; a person-specific renderer turns them into video but is not part of the scored output §5 app. B.

Training is done per listener identity: one codebook and one predictor per person §5 app. D.

Pseudocode
# Predictor training step  [§3.4] [eq. 8] [eq. 9] [eq. 10]
# Input:  speaker_motion (B, 40, 56), speaker_audio (B, 160, 128),
#         listener_motion (B, 64, 56): frames 1-32 past, 33-64 future
# Output: scalar cross-entropy loss

def training_step(speaker_motion, speaker_audio, listener_motion):
    # Quantise the past listener window to codebook indices  [§3.2] [eq. 8]
    past_codes = nearest_code_index(listener_motion[:, :32])     # (B, 4)
    # Target: codes of the 32 future frames
    # [code: src/utils/load_utils.py#L72-L76]
    future_codes = nearest_code_index(listener_motion[:, 32:])   # (B, 4)
    # Fuse speaker audio and motion with cross-modal attention  [§3.3] [eq. 7]
    speaker_emb = cross_modal_encoder(speaker_audio, speaker_motion)
    # Distribution over K codes at each output position  [§3.4] [eq. 9]
    logits = predictor(past_codes, speaker_emb)                  # (B, 9, K)
    # Cross-entropy on the first 4 positions only  [eq. 10]
    # [code: src/train_vq_decoder.py#L83-L86]
    return cross_entropy(logits[:, :4], future_codes)
Pseudocode
# Test-time code sampling  [§5]
# [code: src/vqgan/vqmodules/gan_models.py#L134-L178]
# Input:  logits (B, K) for the next code
# Output: one code index per batch item

def sample_next_code(logits, temperature=0.7, top_p=0.9):
    # Scale logits by temperature  (code default, not stated in paper)
    scaled = logits / temperature
    # Keep the smallest set of codes whose probabilities sum above top_p
    kept = nucleus(scaled, top_p)
    # Draw one code from the renormalised distribution
    return multinomial(softmax(kept))

Hyperparameters

Values in the paper are listed first. Where the released configuration differs, the code value is listed too.

NameValueWhereNotes
Frames per listener window T64§5VQ training splits each 64-frame window into two 32-frame halves train_vq_transformer.py
History length t32§5predictor uses 32 past frames train_vq_decoder.py
Window w (future context and code span)8§5patch size 8 train_vq_decoder.py
Codebook size K200§5l2_32_smoothSS.json
Code dimension d_z256§5l2_32_smoothSS.json
Train / val / test split70% / 20% / 10%§5
Motion features per frame56§3.153 expression+jaw and 3 rotation; see Paper vs code for slice order
Mel audio16 kHz, mel features, 4 audio feature steps per motion frameapp. Bmax-pool by 4 in the code fact_model.py
VQ encoder/decoder width512app. Bcode config: 256 l2_32_smoothSS.json
VQ encoder/decoder heads / layers8 / 12app. Bcode config: 8 / 12, intermediate size 384 l2_32_smoothSS.json
VQ convolutions3 layers, kernel 5app. B
VQ optimizerAdam, betas (0.9, 0.98), batch 32app. Bcode uses AdamW with the same betas gan_models.py
VQ learning rate and warm-up2.0, 4,000 stepsapp. Bcode config: 2.0, warm-up 100,000 l2_32_smoothSS.json
VQ training length1,000 epochs (about 1 day on 8 GPUs)app. Bcode caps at 5,000,000 epochs and keeps the best validation checkpoint l2_32_smoothSS.json
Commitment weightnot statedeq. 6code uses 0.25 gan_models.py
Predictor width / heads / layers200 / 10 / 5app. Bmatches code config delta_v6.json
Speaker cross-modal width / heads / layers1024 / 8 / 12app. Bcode: 400 (2×200), 10 heads, 2 layers fact_model.py
Predictor learning rate and warm-up0.01, 4,000 stepsapp. Bcode config: 0.01, warm-up 100,000 delta_v6.json
Predictor training length1,000 epochs (about 12 h on 8 GPUs)app. Bcode config: 500,000 epochs as cap delta_v6.json
Batch size (predictor)32app. Bdelta_v6.json
Masking of past listener inputrandom cut-off, 50% of samples maskedapp. Bcode draws the cut-off with 4× extra weight for no masking fact_model.py
Test-time samplingnucleus sampling, parameters not stated§5code uses temperature 0.7 and top-p 0.9 gan_models.py
Quantisation error on test setL2 11.32 (expression), 1.02 (rotation)app. Bauthor-reported
Metric clusters for SIk = 15 (expression), k = 9 (rotation)§5k-means on training data
TLCC search range0 to 60 frames (up to 2 s)§5
Delayed-mirror lag17 frames (about 0.5 s)§5chosen from TLCC
Training-data smoothingnot stated in paper—bilateral filter, window 5, colour and space sigma 20 load_utils.py

Results

Author-reported, on the authors' in-the-wild interview data, with the average over four listener identities §5 tab. 1. The LFI baseline was retrained on the authors' data §5 app. D.

SettingMetricBaselineThis paperSource
ExpressionL2 (lower is better)43.18 (Median), 45.76 (NN motion), 50.07 (LFI)33.16tab. 1
ExpressionFD ×10³ (realism)20.66 (NN motion), 43.63 (LFI)3.55tab. 1
ExpressionP-FD ×10³ (pair realism)21.94 (NN motion), 54.34 (LFI)5.15tab. 1
ExpressionPCC (synchrony; ground truth 0.09)0.89 (LFI), 1.00 (Mirror)0.07tab. 1
RotationL26.35 (Median), 6.44 (NN motion), 9.00 (LFI)4.75tab. 1
RotationFD ×10²2.78 (NN motion), 6.51 (Random)0.81tab. 1
RotationP-FD ×10²4.06 (NN motion), 12.36 (LFI)0.87tab. 1
Ablation, expressionL2 / FD ×10³ / P-FD ×10³36.06 / 16.60 / 18.49 (no VQ, audio+motion)33.16 / 3.55 / 5.15tab. 2
Ablation, expressionL2 / FD ×10³ / P-FD ×10³38.05 / 4.01 / 5.67 (concatenation instead of cross-attention)33.16 / 3.55 / 5.15tab. 2
Human A/B studypreference for this method75.3% over NN motion; 71.1% over the audio+motion ablation; 50.1% against ground-truth avatars—§5

The paper's text describes the Mirror baseline as producing excessive synchrony and the Random Walk over codes as realistic but out of sync with the speaker §5. A listener-agnostic variant, trained across listeners, reports FD 30.01 and P-FD 31.36 against 57.29 and 57.91 for its best baseline app. D.

Paper vs code

Code: github.com/evonneng/learning2listen at a0825d58b91bae63a5550bac115b5f7ba03ff36c, owned by the first author. It is linked from the project page and the paper's abstract. Paths below are relative to the repository root.

Searched for code: the abstract points to the project page (evonneng.github.io/learning2listen), which links the repository. No other repository was used.

Discrepancies, each with both sides cited:

  • VQ width: paper 512 app. B; config 256 l2_32_smoothSS.json.
  • Speaker cross-modal transformer: paper 1024 wide, 8 heads, 12 layers app. B; code uses width 2×200 = 400, 10 heads and 2 layers fact_model.py.
  • Learning-rate warm-up: paper 4,000 steps for both models app. B; configs use 100,000 steps for both l2_32_smoothSS.json delta_v6.json.
  • Learning-rate schedule: the code uses an inverse-square-root warm-up schedule scaled by the model width. With the predictor's config (lr 0.01, width 200, warm-up 100,000), the learning rate peaks at about 2.2×10⁻⁶ at step 100,000 optim.py. This is derived from the code, not stated in the paper, and the paper's 4,000-step warm-up would give a much higher peak.
  • Optimizer: paper says Adam for the VQ-VAE app. B; code uses AdamW gan_models.py. No weight decay argument is passed, so the library default applies.
  • Commitment term: the paper's loss has no weight on the commitment term eq. 6; the code multiplies it by 0.25 gan_models.py and leaves the codebook term at weight 1 quantizer.py.
  • Reconstruction loss: the paper writes a norm, and says squared in the text eq. 6; the code uses L1 on expression, rotation and jaw separately, with jaw weight 1 gan_models.py.
  • Feature slice order: the paper puts the 3 jaw dimensions inside β (dimensions 0–52) and head rotation at 53–55 §3.1. The code's loss treats 50–52 as rotation and 53–55 as jaw gan_models.py. Check this before slicing data.
  • Predictor target length: the main text says the predictor outputs w = 8 future frames §3.4. The code's target is the 32-frame future window, encoded to 4 codes load_utils.py.
  • Predictor loss: the paper says the first 4 outputs are used and the other 3 act as a temporal regulariser app. B. The code computes cross-entropy only on the first cut_point outputs, which equals the number of future codes train_vq_decoder.py. No regularising term appears in the loss.
  • Speaker downsampling: the appendix describes three convolutions, each followed by max-pooling, going from 32 to 4 positions app. B. The code has a stride-2 convolution without pooling, then two convolutions with pooling fact_model.py. The speaker input is 40 frames (t + w) load_utils.py. By the arithmetic of these layers the output has 5 positions, which matches the paper's τ + 1 notation §3.3. This was not run.
  • Masking: the appendix says past listener input is masked for 50% of samples app. B. The code samples a cut-off from 0 to 20 with more weight on "no mask" fact_model.py. The mapping from cut-off to attention mask is not verified.
  • Training-data smoothing: both training scripts apply a bilateral filter to the motion data load_utils.py train_vq_decoder.py. The paper does not describe this.
  • Feature count in the README: the README describes 184-dimensional features (50 expression, 6 pose, 128 detail) README.md. The model code uses 56 fact_model.py.
  • Sampling defaults: the paper says nucleus sampling without parameters §5. The code's defaults are temperature 0.7 and top-p 0.9 gan_models.py, and inference uses only the first predicted position gan_models.py.
  • Evaluator count: the main text says 150 evaluators in total §5. The appendix says 50 pairs × 3 evaluators × 3 comparisons = 450 app. D.
  • Table numbering: the appendix refers to the ablations as "Table 1" in one place and "Table 2" in another app. B app. D. This card uses the order in which the tables appear in the main text.

Public code limits:

  • The repository has no LICENSE file. Do not copy code snippets from it; implement from this card and the citations.
  • The model code calls .cuda() directly; a GPU is required.
  • Visualisation to photorealistic video is not released [README.md#L22] [README.md#L186-L188].
  • The README says Python 3.6 with CUDA 9.0 was used; newer environments may need edits.

Pitfalls

  • Data is not clearly licensed. The paper is CC BY 4.0, but the dataset and the raw videos are not covered by any stated license. The repository says raw videos are available only on request, and the data download links are on a university site with no license [README.md#L54-L61] [README.md#L89-L91]. Do not redistribute the data without a separate check.
  • Warm-up and learning rate in the paper and the configs disagree by a factor of 25. Use the configs if you want to reproduce the released models, and state which one you used.
  • Per-listener models. Each codebook and predictor is trained on one listener. Generalisation across listeners is only reported in one appendix experiment app. D.
  • Hidden costs. Training used 8 GPUs for about a day (VQ-VAE) and about 12 hours (predictor) app. B. Each listener needs its own codebook and predictor §5.
  • Sampling settings are unstated in the paper. The diversity metrics depend on how codes are sampled, and the paper gives no temperature or top-p. The code's defaults are temperature 0.7 and top-p 0.9 gan_models.py.
  • Quantisation adds error. On the test set the codebook adds error (L2 11.32 for expression, 1.02 for rotation) app. B. The paper reports an L2 of 17.13 for quantising ground-truth motion in its multi-sample study app. D.
  • Synchrony metric. The PCC for smile synchrony uses a lip-curvature signal §5 fig. 4; the rotation PCC uses head nods §5.
  • Human study setup. Evaluators were asked which listener seems to pay more attention, using a fixed set of 50 moments; the study covers one identity only §5.
  • Speaker and listener roles. The data was split into speaker and listener segments by a voting method using active-speaker detection and source separation app. C. Segments with both people speaking were removed. The split quality is a source of noise that the paper does not measure.
  • Face-model dependence. Expression and pose come from DECA's FLAME-based estimates, not from ground-truth motion capture §4. Errors in those estimates become the training targets.

Lineage

  • Builds on: VQ-VAE for discrete latent codes §2 §3.2; cross-modal attention for audio and visual fusion, applied to motion here §3.3; a transformer predictor adapted from FACT, as the code comments say fact_model.py.
  • Compared against: nearest-neighbour motion and audio retrieval, random clips, median face, mirroring and delayed mirroring, a Glow-based listener model (Let's Face It, Jonell et al.), and a random walk over the codebook §5 tab. 1.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion” by Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, Shiry Ginosar (arXiv:2204.08451), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/evonneng/learning2listen at commit a0825d58b91b (none stated (no LICENSE file in the repository)). Any quoted code is shown under that license.