Can Language Models Learn to Listen?

In plain words

When two people talk, the listener nods, smiles or frowns in response, and those small reactions make the conversation feel connected. This work asks how much of that reaction can be predicted from the words the speaker says alone, without hearing their voice or seeing their face.

The trick is to treat small facial movements as if they were words. A model first learns a fixed list of 256 "gesture tokens" that cover typical listener motion. A pretrained text language model, GPT-2, is then fine-tuned to read the speaker's words, with their timing, and to output the listener's gesture tokens one at a time, interleaved with the words as they are spoken.

The model does not use audio or video of the speaker. The authors report that it still picks up on jokes, punctuation and sentence structure, and that it produces listener motion that people often prefer over simple retrieval baselines. It is not a complete model of listening, because it cannot see or hear the speaker.

TL;DR

• Task: given a speaker's transcript with word timestamps, generate the listener's 3D facial expression and head pose, one 8-frame chunk at a time §3, §3.1.
• Method: a VQ-VAE turns listener motion into discrete tokens §3.2; GPT-2 Medium, initialized from text-only pretraining, predicts those tokens interleaved with the speaker's words §3.3.
• Result (author-reported, Trevor Noah listener): the full model is best on FD, P-FD and L2 Affect; an unconditional model has lower L2 but is worse elsewhere tab. 1.
• Human A/B test: listeners preferred the method over nearest-neighbor retrieval 70.1% of the time and over the unconditional model 92.8% §5.2.
• Relevant if you need a text-only baseline for listener or conversational-gesture generation, or want a recipe for adding a new discrete token vocabulary to a pretrained LM.

Problem

Listener responses are multimodal. Earlier work on listener motion used speech audio and the speaker's motion §1, §5.2. The authors ask how far lexical content alone can go, since transcripts are far more plentiful than paired conversational motion data §1.

Each sample covers one of the speaker's turns §3.1. Only motion starting at least 3 seconds after the turn begins is used §4.1. Visual and audio inputs are not used, so cases where the speaker's face or prosody carries the cue, such as sarcastic jokes, fail §6.4 fig. 3.

Method

Overview §3. The listener face is represented per frame by expression coefficients and head rotation from a 3D morphable face model §3.1 eq. 1. Two components are trained in sequence:

1. A VQ-VAE encodes windows of listener motion into discrete tokens and decodes tokens back to motion §3.2 app. B.
2. An autoregressive language model predicts the next motion token given the speaker's words, their timestamps, and previously generated motion tokens §3.3.

Details, in order:

1. Face representation. Each frame is the expression vector beta_t of size d_m, plus a normalized head rotation given as Euler angles. The shape coefficients are discarded §3.1 eq. 1. The paper does not state d_m.
2. VQ-VAE. The encoder maps a normalized motion sequence to latents, one per 8 frames (three downsampling layers) app. B. Each latent is replaced by its nearest codebook entry, giving the listener token q_t §3.2. The codebook has 256 entries app. B. The decoder maps codebook embeddings back to continuous motion §3.2.
3. VQ-VAE losses. Reconstruction (smooth L1), a velocity term (smooth L1 on frame-to-frame differences), and an embedding term that pulls latents toward their codes §3.2 app. B. The paper also uses exponential moving average updates and codebook reset during training §3.2.
4. Sequence layout. The input sequence starts with history words spoken before the listener segment. Then, for each motion token, the speaker's words that fall in the preceding 8-frame window are placed before that motion token. A space token precedes each motion token §3.3. Each motion token is a new vocabulary entry, initialized randomly; the output layer is replaced by a new affine projection onto the 256 motion tokens §3.3.
5. Language model. GPT-2 Medium, initialized from pretrained text weights. Training uses teacher forcing with cross-entropy over next-token prediction at motion positions. Each input motion token is corrupted with probability 0.5 app. B. The paper does not describe the corruption's effect on text tokens; in code, only motion inputs are corrupted train_t2m_trans.py.
6. Inference. Greedy decoding: each predicted motion token is the argmax, not a sample §3.3. The predicted tokens are decoded by the VQ-VAE decoder to 3DMM coefficients §3.3.

Pseudocode, following the paper's equations and the authors' code. Names are ours.

# VQ-VAE training step  [§3.2] [app. B]
# Input:  face windows (B, 32, d_m + 3); mean and std from training data
# Output: scalar loss; updates encoder, decoder and codebook

def vq_training_step(face_windows, mean, std):
    # Normalize each coefficient with training-set statistics  [§3.2]
    motion = (face_windows - mean) / std           # (B, 32, d_m + 3)
    # One latent per 8 frames, three downsampling layers  [app. B]
    latents = encoder(motion)                      # (B, 4, d_c)
    # Replace each latent by its nearest codebook entry  [§3.2]
    codes, quantized = nearest_code(latents, codebook)
    # Update the codebook by exponential moving average; reset unused
    # entries  [§3.2] [code: models/quantize_cnn.py#L50-L73]
    update_codebook_ema(codebook, latents, codes)
    recon = decoder(quantized)                     # (B, 32, d_m + 3)
    # Reconstruction and velocity: smooth L1  [§3.2] [app. B]
    loss_recon = smooth_l1(recon, motion)
    loss_velocity = smooth_l1(diff_in_time(recon), diff_in_time(motion))
    # Commitment: pull latents toward their (frozen) codes  [§3.2]
    loss_commit = mse(latents, stop_gradient(quantized))
    return loss_recon + 0.5 * loss_velocity + 0.02 * loss_commit

# Interleave speaker words with listener motion tokens  [§3.3] [app. B]
# Input:  history words; timed words (word, end frame); motion codes
# Output: one token list for the language model

def build_sequence(history_words, words, word_ends, motion_codes):
    # History words come first, before the listener segment  [§3.3]
    sequence = list(history_words)
    next_word = 0
    for t, code in enumerate(motion_codes):       # t = 0 .. n - 1
        # Words that end before this motion token's 8-frame window go
        # right before it  [§3.3] [code: dataset/dataset_TM_train.py#L176-L196]
        while next_word < len(words) and word_ends[next_word] < 8 * t:
            sequence.append(words[next_word])
            next_word += 1
        # A space token precedes each motion token  [§3.3]
        sequence.append(motion_token(code))
    return sequence

# One teacher-forced training step of the listener model  [§3.3]
# Input:  token ids (B, L); is_motion marks the motion positions
# Output: scalar cross-entropy loss

def lm_training_step(tokens, is_motion):
    # Corrupt motion tokens in the input with probability 0.5; text
    # is never corrupted  [app. B] [code: train_t2m_trans.py#L228-L244]
    keep = bernoulli(0.5, shape=tokens.shape)
    random_motion = random_motion_ids(tokens.shape)
    inputs = where(is_motion & ~keep, random_motion, tokens)
    logits = language_model(inputs)                # (B, L, 256)
    # Next-token cross-entropy, only where the target is a motion
    # token  [§3.3] [code: train_t2m_trans.py#L283-L290]
    targets = tokens[:, 1:] - text_vocab_size      # codes 0 .. 255
    scored = is_motion[:, 1:]
    return cross_entropy(logits[:, :-1][scored], targets[scored])

# Greedy generation of listener motion  [§3.3]
# Input:  prefix of history and speaker tokens, with empty motion slots
# Output: one motion code per slot, decoded afterwards by the VQ-VAE

def generate_listener(sequence, slots):
    for slot in slots:                             # in time order
        # Predict the motion code after everything generated so far
        logits = language_model(sequence[:slot])   # (slot, 256)
        # Greedy: take the most likely code, no sampling  [§3.3]
        # [code: models/t2m_trans.py#L232-L233]
        sequence[slot] = motion_token(argmax(logits[-1]))
    return sequence

Model and training data flow, with all configuration values, is in the Hyperparameters section.

Hyperparameters

Values marked "paper" come from the paper; values marked "code" are the repository's defaults or the configuration in its README, which the paper does not state. Where the paper and the code disagree, see Paper vs code.

VQ-VAE

Listener language model

Data and processing

Results

Author-reported. Setup: L2L dataset, Trevor Noah listener, 543 test segments §4.1. Metrics are described in §4.2. Baselines: nearest-neighbor text retrieval with all-mpnet-base-v2 embeddings, a random training segment (Random Train), an unconditional model, and the same model without GPT-2 pretraining §4.2. Random VQ also appears in the tables but the text does not define it tab. 1. Values are mean ± standard error tab. 1.

Table 1: quantitative comparison tab. 1

The paper reports the Shannon index of facial gestures as 2.52 for the method against 2.68 for ground truth §5.1. It says the unconditional model's lower L2 is outweighed by its worse results on every other metric, which confirms the text input is used §5.1.

Table 2: ablations, Trevor Noah tab. 2

"Pretrained init" means the language model starts from GPT-2 weights; "Interleaved" means text and motion tokens are merged by timestamp; "Word order kept" means the text is in its real order; "Text input" is the text the model receives.

The paper says Full's small gain over Unaligned is not significant, so the model does not depend on token proximity §6.1.

Human evaluation (Amazon Mechanical Turk, A/B) §5.2

• Videos: grayscale 3DMM meshes, at least 8 s each §5.2.
• Preference for the method: over NN 70.1% (150 evaluators); over Uncond 92.8%; over ground-truth avatars 49.7%; over L2L 55.7% §5.2.
• Sequences: 47 hand-picked moments of strong reactions from the held-out test data §5.2. Each comparison used 3 evaluators per test §5.2 and app. C.
• Attention checks: three duplicated questions with an obvious mismatch; submission blocked if answers change app. C.

Ablation and extended results (appendix)

• Conan and Stephen, with a single "global" VQ trained on all three listeners app. C: Full has FD 35.75 on Conan (best on FD and P-FD there) and FD 34.61 on Stephen. On Stephen, Unaligned is best on FD (33.47) and P-FD (35.90) app. C. The paper says pretraining matters less on the shorter Stephen and Conan segments app. C.
• Effect of history app. C, with the Trevor setup:

• Punctuation and nods app. D: how often each mark appears in the five text tokens before a small nod, a big nod, or a plain-face sequence. Before small nods, 51% of cases include a period, exclamation or question mark; before plain faces, 15%.
• Overlap check app. E: after removing overlapping clips from the Trevor test set, the text-conditioned model stays best of the four methods rerun. Full: L2 0.53 ± 0.02, FD 22.13 ± 0.88, P-FD 23.83 ± 0.90, L2 Affect 7.77 ± 0.62 app. E.

Paper vs code

Repository: sanjayss34/lm-listener at code_commit. It is linked from the authors' project page and owned by one of the authors. The repository has no LICENSE file README.md. Searches run: the paper's project page (evonneng.github.io/projects/text2listen/) linked the repository; no other link was needed.

• Codebook learning. Both use a stop-gradient on the codebook side in the embedding term and update the codebook by exponential moving average §3.2 quantize_cnn.py quantize_cnn.py.
• Reduction. The paper writes its losses as sums over time §3.2. The code uses smooth L1 with mean reduction and mean squared error for commitment losses.py quantize_cnn.py. The weights 1, 0.5 and 0.02 therefore act on a different scale than the paper's sums.
• Velocity loss. For face data the code pads frame 0's velocity with zeros, so the mean runs over all T frames, including a zero term, rather than over the T-1 differences in the paper losses.py train_vq.py.
• Codebook reset. The paper says "codebook reset" without a rule §3.2. The code resets entries whose moving count falls below 1 to random batch vectors quantize_cnn.py.
• Corruption. The paper's appendix describes corruption of motion inputs app. B. The code applies it only at motion positions train_t2m_trans.py.
• Decoding. Both use greedy decoding §3.3 t2m_trans.py.
• Training length. The paper says at most 50,000 gradient steps app. B. The README command uses 100,000 iterations with gradient accumulation 2, which is 50,000 optimizer steps README.md train_t2m_trans.py. The README's LR milestone (150,000) is never reached, so there is no decay.
• Warm-up. The paper gives warm-up only for the VQ-VAE app. B. The README's transformer setup uses a MultiStep schedule with no warm-up train_t2m_trans.py.
• Checkpoint. The paper says the last checkpoint is used app. B. The code saves both a best-by-validation and a last checkpoint eval_trans.py eval_trans.py. The paper does not say which one its tables used.
• History length. The text says the sweet spot is 8 s of history §6.3. The table labels the 3 s configuration "Ours" app. C, and the README uses 3 s README.md.
• Affect window. The paper's affect metric averages over a 1-second window §4.2. Validation during training uses a sliding window of 90 frames (3 s at 30 fps) by default option_transformer.py eval_trans.py. The standalone evaluation script uses a default of 30 frames evaluate_listener.py, non-overlapping evaluate_listener.py.
• Diversity. The paper samples 30 pairs §4.2. The code samples 30 pairs if the sequence has more than 30 frames, otherwise 10 evaluate_listener.py.
• Start offset. The paper says "at least 3 seconds" §4.1. The README uses 96 frames, which is 3.2 s at 30 fps README.md.
• Test data. The README says the test sets were updated in June 2026 to remove overlapping clips README.md. Tables 1 and 2 used the original test sets; only four methods were rerun on the updated set app. E.
• Extra configuration. A docstring in the dataset loader shows a configuration with speaker and audio inputs dataset_TM_train.py. That is not in the paper.

Pitfalls

• History length is ambiguous. The text says 8 s is the sweet spot §6.3, but the configuration reported as "Ours" uses 3 s app. C and the README uses 3 s README.md. Pick one and cite the source for it.
• Affect window differs between paper and code. One second in the paper §4.2; 3 s in training-time validation and non-overlapping windows in the evaluation script eval_trans.py evaluate_listener.py. Numbers from the two paths are not comparable.
• Repository has no license. Do not copy code from it. Cite by path and line only README.md.
• Global VQ needs manual setup. For the Conan and Stephen experiments, the README says to copy the speaker's dataset folder and overwrite mean.npy and std.npy with the shared statistics README.md. The README presents this as required, since the normalization statistics must match the shared VQ.
• EMOCA must be patched. The README says the EMOCA emotion model must be changed to accept precomputed coefficients instead of images, by editing gdl/models/EmoDeca.py README.md. The affect metrics depend on that change.
• Full variation differs between tables. The Full row reads 0.116 ± 0.005 in Table 1 and 0.12 ± 0.005 in Table 2 tab. 1 tab. 2. Use the value from the table you cite.
• Human-evaluation counts. The paper gives 47 curated moments §5.2, while the appendix describes 50 A/B tests per comparison app. C. The paper says "significantly outperformed" §5.2 without a statistical test in the text.
• Speaker dimension not stated. The paper does not state d_m (expression dimension). The code's diversity computation uses the first 56 columns of the motion array evaluate_listener.py. Confirm the layout before reimplementing.
• Padding and masking. Sequences shorter than 8 s are padded, and history words are truncated when the 480-token limit is reached app. B. The paper does not say how padding enters the loss; the code scores only positions inside each sequence train_t2m_trans.py.
• Embedding initialization. The paper says the new motion token embeddings are randomly initialized §3.3. The code grows the GPT-2 vocabulary with the library's resize call t2m_trans.py; I did not find an explicit initializer for the new rows, so the library default applies.
• Text-only limits. Gestures driven by prosody or the speaker's face are not modeled §6.4. The model does not always produce laughter for humorous text §6.4.
• Cost. GPT-2 Medium is 345M parameters, with a 480-token context app. B. Inference is one forward pass per motion token §3.3 t2m_trans.py.

Lineage

• Builds on: the VQ-VAE for the motion tokenizer; the discretization approach of T2M-GPT, which the README credits for part of the code; GPT-2 as the language model; and the Learning to Listen (L2L) dataset and baseline, which are prior work §1 §4.1.
• Compared against: nearest-neighbor retrieval with Sentence-BERT-style embeddings, random training segments, an unconditional model, and L2L §4.2 §5.2.
• Superseded by: not known from content on this wiki.