Cross-Modal Retrieval for Motion and Text via DropTriple Loss

In plain words

The task is to search a library of short 3D human motions with a sentence, or to describe a motion clip with the closest sentence from a database. Both directions use one shared space in which a sentence and the motion it describes should sit close together.

The model has two separate encoders, one for motion and one for text, and each produces a point in that shared space. Training pulls each matching pair together and pushes mismatched pairs apart. The authors noticed that many motions in a batch share an action, such as "run forward", so a mismatched pair can describe nearly the same thing. Pushing those apart teaches the model the wrong lesson. Their fix is to first detect such look-alike pairs and remove them from the set of mismatches, then push away the hardest of the mismatches that remain.

Training starts with a simpler rule for the first few epochs, because the harder rule converges slowly from scratch. On the two standard benchmarks the authors report that the method beats the simpler and the hardest-negative baselines in summed recall, and that fine-tuning the text encoder helps further.

TL;DR

Use this card if you need a text-to-motion or motion-to-text retrieval model trained with a triplet loss. The method is a small change to the loss: remove look-alike negatives before hard-negative mining. It is evaluated only on HumanML3D and KIT-ML, and the paper reports no seeds or error bars. The official code's defaults differ from the paper in several places, listed under Paper vs code.

Problem

Matching sentences to 3D human motion sequences is harder than image-text retrieval, because a motion is a sequence and many motions share sub-actions §1. Common triplet losses treat every non-matching pair as a negative. The hardest-negative variant (MH loss) makes this worse: when several candidates describe the same action, it pushes apart pairs that are actually related §1 §2.3.2. The paper also reports slow convergence for hard-negative training from scratch §2.3.3.

Method

Task. Motion is a sequence of frames, each a pose vector, and text is a sequence of words. Motion retrieval means: given a text query, rank motions. Text retrieval means: given a motion query, rank texts. Recall at K (R@K) is the metric §2.1.

Motion encoder. Each frame is embedded linearly into a 256-d space. A learnable token is prepended to the sequence, a sinusoidal positional encoding is added, and a Transformer encoder (called TMR Enc) runs over the sequence. The output at the token position is the motion feature. A projection W_h maps it to the joint space (1024-d). Both embeddings are L2-normalised §2.2 §3.3. The pose vector is the redundant representation of the HumanML3D data release: root angular and linear velocities, root height, joint positions, velocities and 6D rotations, and four binary foot-contact flags §3.2.

Text encoder. DistilBERT is the backbone. The [CLS] token (768-d) is the text feature, and a projection W_g maps it to the joint space. DistilBERT is frozen unless fine-tuning is enabled §2.2 §3.3.

Similarity. The score is the inner product of the two normalised embeddings, which equals cosine similarity eq. 1 §2.2.

Losses. For a batch of matched pairs, the positive score is S(m_i, t_i), and every other pair in the batch is a negative. The sum-of-hinges loss (SH) sums the margin violations of every negative eq. 2. The max-of-hinges loss (MH) keeps only the hardest negative in each direction eq. 3 §2.3.1.

False negatives. A negative is a false negative if it is too similar to the positive. Motion-side similarity is S(m_i, m_j), and it is compared with threshold δ_hetero. Text-side similarity S(t_i, t_s) is compared with δ_homo, and a text that passes the test also brings its paired motion into the false-negative set eq. 4 eq. 5 §2.3.2. The paper uses separate thresholds for each modality §2.3.2.

DropTriple Loss. The false negatives are removed from the candidate set. The hinge then uses the hardest remaining negative in each direction eq. 6 §2.3.3. The paper notes that a pruned candidate set can be empty, in which case it reports no optimization target and the loss stays at zero §3.4.1.

Training schedule. Negatives are searched within each mini-batch, not the whole training set §2.3.3. For the first ρ = 5 epochs the model is trained with SH loss, then the pruned or hardest-negative loss takes over §2.3.3 §3.3.

# DropTriple training loss for one mini-batch  [eq. 2-6]  [§2.3]
# Input:  motion_emb (B, D), text_emb (B, D), both L2-normalised
# Output: scalar loss

def training_loss(motion_emb, text_emb, epoch,
                  margin=0.2, warm_up=5,
                  delta_hetero=0.7, delta_homo=0.9):
    # Cross-modal scores S(m_i, t_j) for every pair in the batch  [eq. 1]
    scores = motion_emb @ text_emb.T                  # (B, B)
    # Positive pairs (i, i) lie on the diagonal  [§2.1]
    positives = diagonal(scores)                      # (B,)
    # Off-diagonal entries are the negatives  [§2.3.3]
    eye = identity(batch_size)                        # (B, B)
    # Warm-up: keep every negative and sum the hinges  [eq. 2] [§3.3]
    if epoch < warm_up:
        hinge_motion = relu(margin - positives[:, None] + scores)
        hinge_text = relu(margin - positives[None, :] + scores)
        return sum((hinge_motion + hinge_text) * ~eye)
    # Intra-modal similarities S(m_i, m_j) and S(t_i, t_j)  [§2.3.2]
    motion_sim = motion_emb @ motion_emb.T            # (B, B)
    text_sim = text_emb @ text_emb.T                  # (B, B)
    # Pair (i, j) is a false negative if either similarity is high  [eq. 5]
    # One shared mask for both directions, as in the code; see Pitfalls
    false_neg = (motion_sim > delta_hetero) | (text_sim > delta_homo)
    # Keep only off-diagonal pairs that are not false negatives  [eq. 6]
    candidates = ~false_neg & ~eye                    # (B, B)
    # Mask pruned pairs to -inf so the max skips them  [eq. 6]
    pruned = where(candidates, scores, -inf)          # (B, B)
    # Hardest surviving negative: t'' for motion i, m'' for text i  [eq. 6]
    hardest_text = row_max(pruned)                    # (B,)
    hardest_motion = column_max(pruned)               # (B,)
    # Hinge against the positive; -inf gives zero, so empty sets add nothing
    loss_text = sum(relu(margin - positives + hardest_text))
    loss_motion = sum(relu(margin - positives + hardest_motion))
    return loss_text + loss_motion

Key equations, as written in the paper:

S(m, t) = h(\mathcal{T}_m; W_h) \cdot g(\mathcal{T}_t; W_g) eq. 1

\mathcal{L}_{SH} = \sum_{i}\sum_{\hat t \in \mathcal{Q}_T} [\alpha - S(m_i,t_i) + S(m_i,\hat t)]_+ + \sum_{i}\sum_{\hat m \in \mathcal{Q}_M} [\alpha - S(m_i,t_i) + S(\hat m,t_i)]_+ eq. 2

\mathcal{Y}_M = \{ m_j \mid S(m_i,m_j) > \delta_{hetero} \} \cup \{ \mathcal{F}(t_s) \mid S(t_i,t_s) > \delta_{homo} \} eq. 5

\mathcal{L}_{Drop} = \sum_{i}[\alpha - S(m_i,t_i) + S(m_i,\mathbf{t}'')]_+ + \sum_{i}[\alpha - S(m_i,t_i) + S(\mathbf{m}'',t_i)]_+ eq. 6

The paper defines Y_M (motion negatives) in eq. 5, but Y_T (text negatives) is referred to without an equation §2.3.3. α is the margin, t'' and m'' are the hardest surviving negatives eq. 6.

Hyperparameters

Results

Author-reported; no seeds or variance are given. Setup: HumanML3D (test split of 1,000 motions) and KIT-ML (500 motions) §3.1; DistilBERT features; R-sum is the sum of R@1, R@5 and R@10 in both directions eq. 7 §3.1. "Motion retrieval" in the paper means text query to motion; "text retrieval" means motion query to text.

Table 1, HumanML3D tab. 1

Table 1, KIT-ML tab. 1

"(f)" means DistilBERT was fine-tuned tab. 1. Headline numbers: on HumanML3D, the fine-tuned model reaches R@10 62.9 for motion retrieval and 71.5 for text retrieval, and R-sum 276.4 §3.4 tab. 1. The paper states that DropTriple beats SH and MH on both datasets §3.4; see Pitfalls for the exceptions.

Qualitative tables show the top-10 retrieved motions for three text queries tab. 2, and the top-5 retrieved texts for three motion queries tab. 3, for SH and DropTriple. The ground truth is marked in each table. Figure 1 illustrates the task with a "run forward" query and a hard negative fig. 1; Figure 2 gives the architecture and marks the false negatives removed by DropTriple fig. 2. Figure 3 shows intra-modal similarity matrices over epochs, and the authors use it to explain why training with DropTriple and no warm-up stalls fig. 3 §3.4.1. Figure 4 compares R-sum and loss with and without warm-up on HumanML3D fig. 4. Figure 5 plots R-sum against the two thresholds on both datasets; small thresholds make the method no better than MH loss fig. 5 §3.4.1.

Paper vs code

Code: official repository linked in the abstract, eanson023/rehamot, at code_commit above, Apache-2.0 LICENSE. The repository is a PyTorch release with the Hydra configs. Each difference is listed with both citations.

• Similarity. The paper defines the score as cosine on the [CLS] or first token eq. 1 §2.2. The default config is sim: 'cpsmapping', which scores token sequences and not single vectors rehamot.yaml rehamot.py. The cosine mode exists and matches the paper's formula rehamot.py. The paper does not say which mode produced Table 1.
• Thresholds. The paper gives per-dataset values (KIT-ML 0.6 / 0.9; HumanML3D 0.7 / 0.9) §3.3. The code default is 0.75 for both, and the README's run commands do not override it rehamot.yaml README.md README.md.
• Batch size. The paper gives 64 (HumanML3D) and 32 (KIT-ML) §3.3. The README commands use 50 and 25 README.md. The machine default is 50 server.yaml.
• Motion layers on KIT-ML. The paper uses 1 layer for KIT-ML §3.3. The code default is 3, and the KIT-ML data config does not override it rehamot.yaml kit-ml.yaml.
• Optimizer. The paper says AdamW §3.3. The code builds torch.optim.Adam with no weight decay rehamot.py.
• LR decay epoch. The paper says "at the 30th epoch" §3.3. The loop uses the 0-based index, so the decay happens at the 31st epoch train.py train.py.
• Fine-tuning LR. The paper gives 2e-4 for DistilBERT during fine-tuning §3.3. When finetune is on, the code multiplies the LR of every parameter whose name starts with text_model by 10 rehamot.py distilbert.py. Read from the code, that gives 2e-3 when the base LR is 2e-4. This is inferred from the code and was not run.
• Length cutoff. The paper excludes sequences over 1000 frames §3.3. The HumanML3D config sets 500 frames human-ml-3d.yaml humanml3d.py. KIT-ML uses 1000 kit-ml.yaml kitml.py.
• Pruning. The paper removes false negatives from the candidate set §2.3.3 eq. 6. The code sets their similarity to zero, and the hinge max can still take the zeroed cell losses.py losses.py.
• Checkpoint selection. The paper does not say which checkpoint it reports §3.3. The loader named val_loader is built from the test split utils.py. The best checkpoint by R-sum on that split is saved as model_best train.py.
• Extended schedules. The paper trains the MH baseline for 30 extra epochs with a later decay at epoch 45, and fine-tuning for 30 extra epochs §3.3. The released config has one schedule (60 epochs, decay at 30) server.yaml. The extra runs are not in the repository.
• Matches. Warm-up length (5), margin (0.2), joint dimension (1024), motion latent (256), frozen DistilBERT by default, and the summed loss match the paper rehamot.yaml rehamot.yaml losses.py.

Pitfalls

• Which similarity? Implement Eq. 1 (cosine) to match the paper's description. The default config uses the token-level cpsmapping score losses.py, which builds a (B, B, frames, tokens) tensor and is much heavier. Which mode the table numbers come from is not stated.
• Pruning is removal, not zeroing. The paper's argmax runs over the pruned set eq. 6. In the code, a pruned cell is set to zero losses.py. Its hinge is then margin − positive, which is positive when the positive score is below the margin. So the code can still count pruned pairs. The sketch above removes them, as the paper describes.
• Y_T is not written. The paper defines Y_M only and says the text side is "similar" §2.3.2 §2.3.3. The code applies one mask, built from motion and text similarities, to both directions losses.py. The sketch follows the code. A separate, symmetric Y_T is an alternative.
• Dataset-wide vs in-batch. Section 2.3.2 says F(t_s) retrieves motions from the training set. Section 2.3.3 and the code restrict negatives to the mini-batch losses.py. The loss only sees the batch, so F must be a batch lookup in practice.
• Batch size matters twice. The number of in-batch negatives depends on batch size, and the paper and code disagree on it (see Paper vs code). Adam is largely invariant to a constant loss scale, but the loss is a sum over the batch losses.py losses.py, so it grows with batch size.
• Fixed thresholds. The introduction says a threshold comes from sorting intra-modal similarities within the negative set §1. The method and code use fixed thresholds eq. 5. The example value 0.7 in §2.3.2 is not a setting; per-dataset values are in §3.3.
• Empty candidate sets. When every negative is pruned, the paper reports no optimization target and a loss that stays at zero §3.4.1. The code's zeroed cells can still produce a hinge, as noted above.
• Warm-up is needed. The paper reports that DropTriple without warm-up stays at zero loss, because almost every negative is pruned early in training §3.4.1 fig. 3. Use the 5-epoch SH warm-up.
• Table claims. The text says DropTriple improves R@1, R@5 and R@10 over SH by 5.3%, 9.9% and 8.8% §3.4. The table gives 5.3, 8.9 and 8.8 (the R@5 gap is 84.8 − 75.9). It also says Med R drops by 2.0; the table gives a total drop of 4 across both directions. On KIT-ML, DropTriple is below SH on text R@5 (30.7 vs 33.1) and text R@10 (47.8 vs 48.0) tab. 1.
• Summed R-sum. Some printed R-sums differ from the sum of their printed components by up to 0.3 (for example SH on KIT-ML: components sum to 183.3, printed 183.0). Rounding can explain this. Compare the printed R-sum column tab. 1.
• Metric definition. m2t counts a motion query as correct when any of its captions ranks in the top K; the rank is the best of the positives evaluation.py. The paper says a retrieval succeeds if "the correct sample" is in the top K §3.1. The paper does not say which rule produced its table; the released code implements the first.
• Labels. The code's log lines say "Motion to text" for the function called as motion retrieval train.py. The card follows the paper's terms, in which motion retrieval means text query to motion.
• Pose dimension. The paper's per-term sizes, with 22 joints, give 4 + 66 + 66 + 132 + 4 = 272, not the stated 263 §3.2. The 263-dim layout must come from the HumanML3D data release, which is not in this repository.
• Notation. θ is said to contain W_f, but the projection is W_h §2.2.
• No variance. The paper gives no seeds, repeat runs or error bars §3.3 §3.4 tab. 1. Differences of a point or two may be noise.
• Unequal budgets. Per §3.3, the MH row and the fine-tuned row get extra epochs, so the rows are not trained for the same time.
• Fine-tuning cost. Fine-tuning DistilBERT adds 30 epochs on top of 60 §3.3. Keep the encoder frozen when comparing against the non-fine-tuned rows.

Lineage

• Builds on: the hardest-negative triplet loss (MH loss) [faghri2017vse++] as named in the paper, the SH triplet loss [kiros2014unifying], and motion-text encoders following TEMOS §1 §2.2. None of these papers has a card here, so nothing from them is used.
• Compared against: SH loss and MH loss in Table 1 tab. 1.
• Superseded by: not known from content on this wiki.
• Related task: motion-text retrieval on HumanML3D and KIT-ML §3.1.