GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling

In plain words

When people talk, their hands, arms, torso and legs move together. Most computer systems that generate such movement from speech handle each body part on its own, so the arms and hands can end up doing unrelated things.

GestureLSM splits the body into a few regions and turns each region's motion into a short sequence of codes. A transformer then works on all regions at once: at each moment it lets the regions see each other, and then it lets each region look at what it did in earlier frames. Speech conditions the whole process through attention.

The generator is trained with a flow-matching method: it learns to move from random noise to real motion along a path. A shortcut training rule teaches it to take larger steps along that path, so it needs few steps to produce a motion. The authors report that this gives better gesture quality than the baselines they compare with and runs faster than them.

TL;DR

• Speech-driven full-body gesture generator on BEAT2. Body regions are tokenised with residual vector quantisation, and a transformer with spatial and temporal attention generates the tokens. §3.1 §3.2
• Generation uses flow matching with a shortcut (self-consistency) objective and Beta-distributed training times. §3.3 eq. 4 eq. 6 eq. 7
• Author-reported: FGD 4.088 (best of the nine compared methods in Table 1), BC 0.714, and AIST 0.039 at 8 sampling steps. tab. 1 §4.4
• Code (MIT) is available. It differs from the paper in several places, including the speaker scope of the reported numbers. See Paper vs code.

Problem

Gesture generators that model each body part separately, or that model the whole body as one vector, produce unnatural and disconnected motion. Diffusion and autoregressive generators that run dozens of steps are slow. §1

The paper asks for coherent full-body gestures that are both high quality and fast to generate. §1 §3

Method

The method has three parts. Stage 1 tokenises the motion. Stage 2 is the gesture generator with spatial and temporal attention. Stage 3 trains it with a latent shortcut flow-matching objective. §3 fig. 3 fig. 4

Body-region tokenisation §3.1

The motion of each body region (upper body, hands, and legs in the text) is encoded by a convolutional encoder with two residual blocks. Each frame's vector is replaced by its nearest entry in a codebook, and a decoder maps the codes back to joint motion. §3.1 eq. 1 The paper uses residual vector quantisation (RVQ) but does not state the number of quantiser stages. §3.1 Per-region codebooks have 1,024 entries of width 128. app. B

The codebook assignment, for frame vector \mathbf{v}_t and codebook \{\mathbf{c}_i\}, is \tilde{\mathbf{v}}_t = \mathbf{c}_{i^*} with i^* = \arg\min_i \|\mathbf{c}_i - \mathbf{v}_t\|_2 §3.1. The decoder reconstructs joints from the quantised vectors, and this reconstruction is the round trip \mathcal{D}(\mathcal{Q}(\mathcal{E}(\cdot))) eq. 1.

Speech conditioning §3.1

Speech is represented by onset and amplitude features from the audio, and by word-level semantics from the transcript. Each has its own encoder. The two streams are combined by element-wise concatenation. The fused speech features enter the gesture features through several cross-attention layers in which the gesture features are queries and the speech features are keys and values. §3.1 The paper uses three cross-attention layers. app. B

Spatial and temporal attention §3.2

The generator is a transformer. Each block applies spatial attention first and temporal attention second, then a feed-forward network. §3.2 app. B

• Spatial attention runs across body regions inside one frame. Time steps are treated as independent batch items, so the regions attend to each other at each moment. §3.2 eq. 2
• Temporal attention runs along time for each region separately. Body regions are batch items. §3.2 eq. 3
• Positional information is added before each attention. §3.2

\mathcal{A}_s = \mathrm{SoftMax}\!\left(\frac{Q_s K_s}{\sqrt{d}} + \mathbf{P}\right) V_s \qquad \text{[eq. 2]}

\mathcal{A}_t = \mathrm{SoftMax}\!\left(\frac{Q_t K_t}{\sqrt{d}} + \mathbf{P}\right) V_t \qquad \text{[eq. 3]}

Eqs. 2 and 3 are printed without the key transpose and with a single positional term \mathbf{P}. eq. 2 eq. 3 The sizes are n body regions, T time steps and feature dimension d. §3.2

The paper uses 8 spatial-temporal blocks. app. B

Flow matching and the latent shortcut §3.3

The flow path is linear: x_t = (1-t)\,x_0 + t\,x_1, where x_0 is Gaussian noise, x_1 is data and v_t = x_1 - x_0. §3.3

Flow matching regresses a network onto the velocity v_t. §3.3 eq. 4

\mathcal{L}^{F}(\theta) = \mathbb{E}_{x_0, x_1}\left[\left\| \bar{v}_\theta(x_t, t) - (x_1 - x_0) \right\|^2\right] \qquad \text{[eq. 4]}

A shortcut model also receives a step size d and predicts the displacement to x_t + d steps ahead along the flow. §3.3 eq. 5

x'_{t+d} = x_t + f_\theta(a, x_t, t, d) \qquad \text{[eq. 5]}

Self-consistency asks that one step of size 2d equal the average of two steps of size d. §3.3 eq. 6

f_\theta(a, x_t, t, 2d) = \tfrac{1}{2}\, f_\theta(a, x_t, t, d) + \tfrac{1}{2}\, f_\theta(a, x'_{t+d}, t, d) \qquad \text{[eq. 6]}

Here a is the speech condition. The paper's Eq. 6 writes the second argument as t; the code evaluates the second half-step at t + d. LSM.py eq. 6

Training times are drawn from a Beta distribution with density f(t;\alpha,\beta) on [0,1]. §3.3 eq. 7 The paper uses \alpha = 2 and \beta = 1.2. §3.3 tab. 2(c) The paper reports that the loss is highest near t = 1, which is why the skew emphasises that end of the path. §3.3 fig. 5

f(t;\alpha,\beta) = \frac{t^{\alpha-1}(1-t)^{\beta-1}}{\mathrm{B}(\alpha,\beta)}, \qquad t \in [0,1] \qquad \text{[eq. 7]}

Sampling uses Euler steps on the learned velocity. §3.3 The paper chooses 8 steps for inference. §4.4 tab. 2(f)

Training procedure as the code implements it

The paper does not spell out how the flow loss and the shortcut loss are combined in a batch. The official code does the following. [code: models/LSM.py#L352-L452]

# Shortcut flow training step  [§3.3] [eq. 4] [eq. 6]
# Input:  latents (B, T, D) of RVQ codes  [§3.1]
#         speech features for the batch  [§3.1]
# Output: scalar loss  [code: models/LSM.py#L352-L452]

def training_step(latents, speech):
    # Gaussian noise, the start of the flow  [§3.3] [code: models/LSM.py#L367]
    noise = sample_gaussian(latents.shape)
    # Time per sample from Beta(2, 1.2), skewed towards t = 1  [§3.3] [eq. 7]
    time = sample_beta(alpha=2.0, beta=1.2, size=batch)
    # The first 3/4 of the batch is flow matching with step size 0;
    # the rest draws d from {1/2, 1/4, ..., 1/128}
    # [code: models/LSM.py#L352-L365]
    flow_count = int(batch * 3 / 4)
    step = sample_step_size(size=batch)
    step[:flow_count] = 0.0
    # Straight path from noise (t = 0) to data (t = 1)  [§3.3] [eq. 4]
    x_t = time * latents + (1 - time) * noise
    # Flow part: velocity regression, with the MSE divided by t  [eq. 4]
    # [code: models/LSM.py#L386-L392]
    speech_flow = drop_speech(speech[:flow_count], prob=0.1)
    velocity_pred = denoiser(x_t[:flow_count], time[:flow_count],
                             step[:flow_count], speech_flow)
    velocity_target = latents[:flow_count] - noise[:flow_count]
    flow_loss = mean(mse(velocity_pred, velocity_target)
                     / time[:flow_count])
    # Consistency part: 80% of its samples get the null speech embedding
    # [code: models/LSM.py#L398-L403]
    speech_cons = force_null(speech[flow_count:], prob=0.8)
    # Target: two half steps of size d, averaged; no gradient  [eq. 6]
    # [code: models/LSM.py#L414-L435]
    with no_gradient():
        speed_a = denoiser(x_t[flow_count:], time[flow_count:],
                           step[flow_count:], speech_cons)
        x_mid = x_t[flow_count:] + step[flow_count:] * speed_a
        speed_b = denoiser(x_mid, time[flow_count:] + step[flow_count:],
                           step[flow_count:], speech_cons)
        speed_target = (speed_a + speed_b) / 2
    # Prediction for one double step 2d from x_t  [eq. 6]
    # [code: models/LSM.py#L437-L442]
    speed_pred = denoiser(x_t[flow_count:], time[flow_count:],
                          2 * step[flow_count:], speech_cons)
    consistency_loss = mse(speed_pred, speed_target)
    # The two losses are summed without weights  [code: models/LSM.py#L452]
    return flow_loss + consistency_loss

Sampling with guidance §3.3 app. B

The code starts from Gaussian noise and runs Euler steps. At each step the model is evaluated with and without speech, and the outputs are combined with a guidance scale of 2. app. B LSM.py [code: models/LSM.py#L207]

# Euler sampling with classifier-free guidance  [§3.3] [app. B]
# Input:  speech features; steps (8 in the reported results)  [tab. 2(f)]
# Output: latents (B, T, D), decoded by the RVQ decoders  [§3.1]

def sample(speech, steps=8, guidance_scale=2.0):
    # Start from Gaussian noise  [§3.3] [code: models/LSM.py#L292]
    x = sample_gaussian(latent_shape)
    # Uniform step; the same size is the shortcut d  [code: models/LSM.py#L297]
    dt = 1.0 / steps
    for index in range(steps):
        # Time at the start of this step  [code: models/LSM.py#L298]
        time = index * dt
        # Conditional and null-speech predictions
        # [code: models/LSM.py#L151-L209]
        cond = denoiser(x, time, dt, speech)
        uncond = denoiser(x, time, dt, null_speech)
        # Guided velocity  [app. B] [code: models/LSM.py#L207]
        velocity = uncond + guidance_scale * (cond - uncond)
        # Euler update  [§3.3] [code: models/LSM.py#L318]
        x = x + dt * velocity
    return x

Decoding to joints §3.1

Latent codes are decoded by the frozen RVQ decoders for each body region, and the decoded joints are the output. §3.1 shortcut_rvqvae_trainer.py

Hyperparameters

Values marked "paper" come from the text or appendix. Values marked "code" come from the official configuration or scripts and are not stated in the paper.

Results

Author-reported on BEAT2, with the EMAGE train/validation/test split. §4.1 The paper states the data as 60 hours from 25 speakers. §4.1 See Pitfalls for the speaker scope of the reported numbers.

Metrics §4.2 app. C:

• FGD, Fréchet Gesture Distance, computed on latent features from a pretrained network app. C eq. 8:

\mathrm{FGD}(\mathbf{g}, \hat{\mathbf{g}}) = \|\mu_r - \mu_g\|^2 + \mathrm{Tr}\!\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right) \qquad \text{[eq. 8]}

• Diversity: the average L1 distance between clips, with global translation removed. app. C eq. 9
• Beat Constancy (BC): beat alignment between speech onsets and upper-body velocity minima, with a Gaussian of width \sigma. app. C eq. 10 The paper does not state \sigma. app. C The paper argues that BC should be compared with ground truth (0.703), not maximised. app. E
• Efficiency: AIST (average inference time per sentence), recorded as AITS in the metric section. §4.2 tab. 1

\mathrm{L1\ div.} = \frac{1}{2N(N-1)} \sum_{t=1}^{N} \sum_{j=1}^{N} \left\| p_t^{i} - \hat{p}_t^{j} \right\|_1 \qquad \text{[eq. 9]}

\mathrm{BC} = \frac{1}{g} \sum_{b_g \in g} \exp\!\left(-\frac{\min_{b_a \in a} \|b_g - b_a\|^2}{2\sigma^2}\right) \qquad \text{[eq. 10]}

Eq. 9 is printed with indices i and j and a sum over t that does not match its normalisation. eq. 9 Use the official metric code where the two differ.

Main comparison tab. 1

Columns: FGD (\downarrow), BC (target: ground truth 0.703, \rightarrow), Diversity (\uparrow), MSE (\downarrow), AIST (\downarrow). "--" means the paper gives no value. Bold marks the best value in the paper. The caption says FGD and BC are multiplied by 10^{-1} and MSE by 10^{-7}, and does not say how to read the scaling. tab. 1

The text says GestureLSM achieves state-of-the-art results on all metrics and is closest to ground truth on BC §4.2. In the table, its BC (0.714) is the closest to 0.703, and its FGD is the lowest. Its diversity (13.24) is not the highest among the body-only rows: ProbTalk has 13.27 tab. 1. The face variant adds facial MSE (1.021). tab. 1

The paper reports a user study with 20 participants and 320 clips (80 per method, for GestureLSM, EMAGE, ProbTalk and DiffSHEG) on realness, synchrony and smoothness, scored 1 to 5. The mean opinion scores are plotted, not tabulated. §4.2 fig. 7

Ablations tab. 2

Each subtable reports FGD, BC and Diversity unless noted. Bold marks the best value in the paper, which is the proposed row in most cases.

(a) Module contribution. Removing spatial attention, temporal attention or positional encoding each hurts FGD. tab. 2(a)

(b) Architecture. tab. 2(b)

(c) Time sampling. tab. 2(c)

(d) Generator model type (FGD, Div., AIST). tab. 2(d)

(e) Speed-up methods (FGD, Div., Tr-T, AIST). Tr-T is the training time in units of the base model's training. tab. 2(e)

(f) Sampling steps (FGD, BC, Div., AIST). tab. 2(f)

Additional ablations tab. 3

These are in the appendix. app. D tab. 3

(a) Speech features (FGD, BC, Div.). Cross-attention is best. tab. 3(a)

(b) Guidance scale (FGD, BC, Div.). tab. 3(b)

(c) Motion representation (FGD, BC, Div.). tab. 3(c)

(d) Attention order (FGD, BC, Div.). tab. 3(d)

(e) Beta skew (FGD, BC, Div.). tab. 3(e)

Paper vs code

Official repository: andypinxinliu/GestureLSM, linked from the paper's project page. It is owned by the first author's account and has an MIT LICENSE (copyright Pinxin Liu). The clone is at commit f52ac2f. The code also has a MeanFlow variant and a diffusion variant that this card does not cover.

Discrepancies, each with both citations:

• Speaker scope. The paper describes BEAT2 as 25 speakers §4.1. The shipped config trains on speaker id 2 only (training_speakers: [2], configs_new/shortcut_rvqvae_128.yaml#L55). The README says the reported statistics are the one-speaker results, and that the pretrained models use one speaker. README.md
• Flow loss weighting. Eq. 4 is an unweighted mean squared error eq. 4. The code divides the flow MSE by each sample's t before averaging LSM.py. The MSE is one batch scalar here, so the weight is the mean of 1/t over the flow samples.
• Spatial-first order. The paper and the block comment say spatial then temporal §3.2 transformer.py. The attention module defaults to spatial_first=False, which runs temporal attention first transformer.py [code: models/layers/transformer.py#L437-L443], and the block does not pass the flag transformer.py. Table 3(d) reports spatial-first as the better order tab. 3(d).
• Positional encoding. The paper adds spatial and temporal positional encodings §3.2. The code applies rotary embeddings inside the attention transformer.py [code: models/denoiser.py#L118-L121].
• Text encoder. The paper uses BERT for transcript semantics §3.1. The code uses a pretrained word-embedding table from weights/vocab.pkl modality_encoder.py, and the config names t_pre_encoder: fasttext shortcut_rvqvae_128.yaml.
• RVQ training budget. The appendix states 30,000 iterations app. B. The script uses 300,000 with a learning-rate schedule at 200,000 train_rvq.sh.
• Sampling steps. The reported results use 8 steps tab. 2(f). The shipped config uses n_steps: 2 [code: configs_new/shortcut_rvqvae_128.yaml#L78]. Nothing in the repository states which setting produced Table 1.
• Guidance flag. The config sets do_classifier_free_guidance: False shortcut_rvqvae_128.yaml, but the model applies guidance whenever the scale exceeds 1 LSM.py LSM.py. The flag has no effect.
• Consistency-stage schedule. The trainer passes train_consistency=True only after epoch 100 shortcut_rvqvae_trainer.py. The argument is never read in train_forward LSM.py, so the shortcut loss runs from epoch 0.
• Null-speech rate. The code comment says 50/50 [code: models/LSM.py#L396-L397]. The call uses p = 0.8 for the null branch [code: models/LSM.py#L399].
• Shortcut target. The second half-step is evaluated at t + d [code: models/LSM.py#L423-L425]. Eq. 6 writes t eq. 6.
• Seed latents. The code also conditions the denoiser on 8 seed vectors taken from the first 4 latent frames of the clip [code: configs_new/shortcut_rvqvae_128.yaml#L91] [code: shortcut_rvqvae_trainer.py#L258]. The paper does not mention this input.

Not compared: the BC and diversity metric code. metric.py

Pitfalls

• Speaker scope. Read the main text as "25 speakers" but the numbers as "speaker 2" until you have checked the training data and the split. Training with the shipped config uses speaker 2. Reproducing Table 1 needs the same scope. §4.1 shortcut_rvqvae_128.yaml [code: README.md#L135-L137]
• Attention order and form. Eqs. 2 and 3 do not show the key transpose and give one positional term. Use standard scaled dot-product attention with rotary embeddings, as the code does, and test the order yourself, because the code's default order is temporal first. eq. 2 eq. 3 [code: models/layers/transformer.py#L437-L443]
• Eq. 4 weighting. The paper's flow loss has no 1/t weight. The code has one. Pick one and state it. eq. 4 LSM.py
• Target is not differentiated. The code computes the shortcut target under no-gradient LSM.py. The paper does not say this.
• Two losses, no weights. The code sums the flow and consistency losses with weight 1 LSM.py. The paper gives no relative weight. The batch is split 3/4 flow and 1/4 consistency LSM.py. The paper gives neither split.
• Few steps are cheap, but not free. The paper's step-count table shows FGD 6.235 at 1 step and 4.040 at 20 steps tab. 2(f). The 8-step setting in Table 1 is a trade-off, not a free choice.
• Epoch-gate dead code. The epoch > 100 gate in the trainer has no effect shortcut_rvqvae_trainer.py LSM.py.
• Units and naming. The paper calls the efficiency metric AIST in the table and AITS in the text, and reports it per sentence in one place and per frame in another tab. 1 §4.2. The FGD and BC scaling in the caption is not explained tab. 1.
• Beat Constancy. The paper says higher BC is better, and then argues that it should match ground truth (0.703) §4.2 app. E. The Table 1 BC column has its target arrow pointing to ground truth.
• Missing details. Not stated in the paper: \sigma in BC, batch size, step-size set, RVQ stage count, encoder shape, context-frame handling and the optimiser's betas and warm-up. The code supplies values for all of them (see Hyperparameters), but these are not paper values.
• Long clips. The code splits long sequences into windows with overlapping pre-frames at test time shortcut_rvqvae_trainer.py. The paper does not describe this.
• Other results. The README's results table reports different numbers on another scale (for example a shortcut at 20 steps with FGD 0.4040). Do not compare it with Table 1. README.md
• Speed claim. The 0.039 s figure is per sentence in the table and per frame in the text. Measure your own setup before you rely on it. tab. 1 §4.2

Lineage

• Builds on: flow matching and rectified flow §3.3 (cited there as lipman2022flow and liu2022flowstraightfastlearning), shortcut models §3.3 (cited as shortcutmodels), and residual vector quantisation §3.1 (cited as rvq).
• Data split from EMAGE §4.1. Reset strategy for the codebook follows Contextual Gesture app. B.
• Compared against: HA2G, DisCo, CaMN, DiffSHEG, TalkShow, ProbTalk, EMAGE, MambaTalk and SynTalker tab. 1.
• Superseded by: not known from this wiki.

Figures

The paper's figures are described above and cited in place. No figure is reproduced here. fig. 1 shows a teaser of the method. fig. 2 plots FGD against inference time. fig. 3 shows the overall pipeline. fig. 4 shows the shows the shortcut self-consistency idea eq. 6. fig. 5 compares time-sampling schedules. fig. 6 shows qualitative comparisons. fig. 7 is the user study. fig. 8 shows an avatar application. fig. 9 tracks training dynamics. §3 §4 app. E