wikillm

Paper · 2023-08-22 · cs.CV, cs.IR, cs.LG

Multi-event Video-Text Retrieval

Me-Retriever picks 16 key frames per video by K-medoids over CLIP frame features and trains a CLIP two-stream retriever with a MeVTR loss: video-to-text and text-to-video softmax terms, weighted by a dynamic alpha. Author-reported on ActivityNet Captions, video-to-text R@1 is 8.52 (avg variant) versus 6.60 for CLIP4Clip (mean).

Cross-modal retrievalContrastive learningKeyframe selectionvideo-text-retrievalmulti-event-videoclipk-medoidscontrastive-loss

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Video-text retrieval means finding the right video for a sentence, or the right sentence for a video. Most models assume that one video matches one caption. Real videos often contain several things happening, and a short search query or a web page description covers only one of them. Standard training then pulls all of a video's captions toward one video embedding, so those captions end up looking alike.

This paper defines the setting where one video has several matching captions, calls it multi-event video-text retrieval, and proposes a simple model for it. The model does not look at every frame. It groups the frames into clusters and keeps one representative frame per cluster, so that the representatives stand for different moments of the video. The text and the frames are then compared in a shared space. Training changes the objective so that the captions of one video are not treated as rivals of each other.

The authors report that the simple model beats the compared models on most video-to-text measures on ActivityNet Captions, and that it stays competitive on text-to-video retrieval. They also show that earlier models lose a lot of accuracy when used in this multi-event setting without retraining.

TL;DR

Me-Retriever represents a video by 16 medoid frames chosen by K-medoids over CLIP frame embeddings §4.1 §5.1. It scores a video against a caption by averaging (avg) or maximising (max) the cosine similarity between the caption and each key event eq. 5 eq. 6. Training uses the MeVTR loss, a weighted sum of a video-to-text and a text-to-video softmax term, with the weight set dynamically §4.2 eq. 2. Author-reported, the avg variant has the best video-to-text recall from R@1 to R@50 on ActivityNet Captions tab. 2; on Charades-Event the max variant has the best median rank and R@50 tab. 3. On text-to-video it is comparable to CenterCLIP and CLIP4Clip rather than clearly better tab. 4 tab. 5.

Problem

A two-stream video-text model learns one embedding per video and one per caption and assumes a one-to-one match. In multi-event data a video has several captions, each covering one segment. Training then tries to make all captions of a video close to one video embedding, which can make the caption embeddings collapse toward each other §1 §3.3. The paper measures this: the average cosine similarity between captions of the same video is 0.864 with variance 0.0028 for CLIP4Clip (mean), against 0.789 with variance 0.0059 for Me-Retriever (avg) on the whole test set §5.3. Earlier models trained on standard video-text retrieval also lose much accuracy when evaluated on the multi-event task without retraining tab. 1.

Problem formulation: a video v_i is described by a set of texts T_i, and the task is to rank videos for a text query and to rank all relevant texts for a video query §3.1.

Evaluation metrics §3.2:

  • Recall@k-Average: the fraction of a video's correct texts found in the top k, averaged over the test set. Reported for text-to-video and video-to-text.
  • Recall@k-One-Hit: whether any correct text is in the top k.
  • Recall@k-All-Hit: whether all correct texts are in the top k.

Method

Overview: the CLIP visual encoder produces frame embeddings, with the [CLASS] token of the last hidden layer used as the frame embedding fig. 2. A clustering module selects key events from these frames. The CLIP text encoder embeds each caption, using the [EOS] token fig. 2. A similarity module scores each video against each caption, and the MeVTR loss trains the whole model §4 fig. 2.

Step 1: frame features. Each frame of video v_i gives one CLIP embedding; the set is written f^1_i ... f^|v_i|_i §4.1.

Step 2: key event selection. Cluster the frame embeddings with K-Medoids into K = 16 clusters §5.1. A medoid is an actual frame, which the paper uses as the key event's representation §4.1. The paper uses cosine similarity as the metric for medoid selection §4.1; the code's default differs (see Paper vs code). Clustering stops after at most 60 iterations or when the shift falls below 1e-5 §5.1. The key events of a video are K_i = {k^1_i, ..., k^K_i} §4.1.

Pseudocode
# Key event selection  [§4.1] [§5.1]
# Input:  frames (T, d) frame embeddings from the CLIP visual encoder
# Output: key events (K, d), the medoid frames

def select_key_events(frames, K):
    # Pairwise distances between all frames of the video  [§4.1]
    distances = pairwise_distance(frames)            # (T, T)
    # Initial medoids: the code uses a deterministic KKZ rule
    # [code: modules/cluster/cluster_utils.py#L106-L118]
    medoids = initial_medoids(frames, distances, K)  # (K,)
    for step in range(60):                           # iteration limit  [§5.1]
        # Assign every frame to its nearest medoid  [§4.1]
        assignment = nearest_medoid(distances, medoids)   # (T,)
        # Per cluster, pick the frame with the smallest total distance
        # to its members  [§4.1]
        new_medoids = cluster_medoids(distances, assignment, K)
        # Stop when the medoids barely move  (threshold 1e-5)  [§5.1]
        if medoid_shift(frames, new_medoids, medoids) < 1e-5:
            break
        medoids = new_medoids
    return frames[medoids]                           # (K, d)  [§4.1]

Step 3: text embeddings. Each caption is encoded by the CLIP text encoder and represented by its [EOS] token fig. 2. Each video has its set of captions as positives; ActivityNet uses all captions of a video as its events, and Charades-Event treats each annotated text as an event §5.1.

Step 4: similarity. Two variants, named by how they combine the key-event similarities:

s_{avg}(v_i, t_j) = \mathrm{avg}\big(\cos(k^i_1, w_j), \dots, \cos(k^i_K, w_j)\big) eq. 5

s_{max}(v_i, t_j) = \mathrm{max}\big(\cos(k^i_1, w_j), \dots, \cos(k^i_K, w_j)\big) eq. 6

Me-Retriever(avg) uses eq. 5 and Me-Retriever(max) uses eq. 6 §4.2. For bijective pairs (one caption per video) the usual cosine similarity is used §4.2.

Pseudocode
# Video-text similarity  [§4.2] [eq. 5] [eq. 6]
# Input:  key_events (K, d), text (d,) text embedding w_j
# Output: scalar score for the pair

def sim_avg(key_events, text):
    # Cosine of the text with each key event  [eq. 5]
    cosines = [cos(event, text) for event in key_events]   # (K,)
    return mean(cosines)                                    # avg variant

def sim_max(key_events, text):
    # Largest cosine over the key events  [eq. 6]
    cosines = [cos(event, text) for event in key_events]   # (K,)
    return max(cosines)                                     # max variant

Step 5: MeVTR loss. The loss is a weighted sum of a video-to-text term and a text-to-video term eq. 2. Both use a softmax over the batch with temperature τ eq. 3 eq. 4:

\mathcal{L}_{MeVTR} = \mathcal{L}_{v2t} + \alpha \, \mathcal{L}_{t2v} eq. 2

\mathcal{L}_{v2t} = -\frac{1}{|\mathcal{B}|} \sum_{i}^{|\mathcal{B}|} \sum_{k}^{|\mathcal{T}_i|} \frac{1}{|\mathcal{T}_i|} \log \frac{\exp(sim(\mathbf{K}_i, \mathbf{w}_k)/\tau)}{\sum_{j \in \mathcal{B}^{-} \cup \{k\}} \exp(sim(\mathbf{K}_i, \mathbf{w}_j)/\tau)} eq. 3

\mathcal{L}_{t2v} = -\frac{1}{|\mathcal{B}|} \sum_{i}^{|\mathcal{B}|} \log \frac{\exp(sim(\mathbf{K}_i, \mathbf{w}_i)/\tau)}{\sum_{j=1}^{|\mathcal{B}|} \exp(sim(\mathbf{K}_j, \mathbf{w}_i)/\tau)} eq. 4

Here B is the training batch, B^- the negatives in the batch, T_i the positive captions of video i, and τ the temperature §4.2. The paper states that for a video item, the other positive captions in T_i are excluded from the softmax, except the current caption t_k; with this, the text-to-video term reduces to an ordinary softmax §4.2.

Pseudocode
# MeVTR loss  [§4.2] [eq. 2] [eq. 3] [eq. 4]
# Input:  sim (B_text, B_video) scores; mask[t, v] = 1 if caption t
#         belongs to video v; alpha the weight of the text-to-video term
# Output: scalar loss

def mevtr_loss(sim, mask, alpha):
    # Video-to-text term; each video's captions are positives, and
    # other captions of the same video are left out of the denominator  [eq. 3]
    loss_v2t = v2t_term(sim, mask)
    # Text-to-video term: ordinary softmax over the batch's videos  [eq. 4]
    loss_t2v = t2v_term(sim, mask)
    # Weighted sum  [eq. 2]
    return loss_v2t + alpha * loss_t2v

Step 6: dynamic weight. Instead of a fixed alpha, the paper sets alpha to the ratio L_v2t / L_t2v, computed during training §4.2. The paper says the two losses differ in scale because of the positive/negative ratio, so a fixed alpha needs tuning §4.2. Fixed weights from {0.5, 1.0, 1.5, 2.0, 3.0} are compared, and the authors report that the dynamic weight is more stable across metrics fig. 5 fig. 6. They find the cutoff between 1.0 and 2.0 empirically: smaller alpha favours video-to-text and larger alpha favours text-to-video §5.3.

Training data and procedure. Each video is trained on all of its captions as positives. Each experiment runs for 5 epochs §5.1. The paper reports no other training details (see Hyperparameters).

Hyperparameters

NameValueWhereNotes
Key events K16 (fixed)§5.1 README.mdAuthors did not use fewer or more; shorter sequences cannot cover videos with up to 27 events §5.1
K-Medoids max iterations60§5.1Code CLI default is 100 main_task_retrieval_multi.py; see Paper vs code
Clustering stop threshold1e-5§5.1Same value as the code default main_task_retrieval_multi.py
Epochs5§5.1Code default is 20 main_task_retrieval_multi.py; the repository has no training command
Loss weight alphadynamic: L_v2t / L_t2v§4.2Fixed grid {0.5, 1.0, 1.5, 2.0, 3.0} compared fig. 5 fig. 6; code flag --dynamic_alpha main_task_retrieval_multi.py
Temperature taunot stated§4.2 eq. 3 eq. 4Code uses the learned CLIP logit scale, clamped to at most log(100) main_task_retrieval_multi.py
CLIP backbonenot stated§4Evaluation command uses ViT-B/32 README.md
Frames per videonot stated§4.1Evaluation command uses 64 README.md; dataloader default is 100 dataloader_activity_multievent.py
Max words per captionnot stated—Evaluation command uses 77 README.md
Max captions per video27§5.1Dataloader cap matches dataloader_activity_multievent.py
Learning ratenot stated—Code default 1e-4 main_task_retrieval_multi.py; evaluation command sets --coef_lr 1e-3 README.md
Batch size (training)not stated—Code default 256 main_task_retrieval_multi.py
Warm-up proportionnot stated—Code default 0.1 main_task_retrieval_multi.py
Gradient clippingnot stated—Code clips the norm to 1.0 main_task_retrieval_multi.py
ActivityNet Captions10,009 train videos; 4,917 val1 videos / 17,505 texts§5.1Video-text pairs from YouTube; all captions of a video are its events
Charades-Event5,338 train videos / 12,408 texts; 1,334 test videos / 3,720 texts§5.1Built from Charades with TALL text annotations; each text is one event

Results

Author-reported. Setup: ActivityNet Captions and Charades-Event, CLIP-based models, each baseline trained with its original code and evaluated on the multi-event protocol tab. 1 §5.1. Baselines: CLIP4Clip (mean and tight), CenterCLIP, FROZEN-in-time, ClipBERT and Singularity tab. 2 tab. 4. Cells show Average / One-Hit / All-Hit where the paper gives them; "-" means not reported.

Video-to-text, ActivityNet Captions tab. 2:

ModelMedian rankR@1R@5R@10R@50
CLIP4Clip (mean)39.06.60 / 20.74 / -20.32 / 47.02 / 3.4629.37 / 61.26 / 6.7158.41 / 89.77 / 26.24
CLIP4Clip (tight)49.55.23 / 16.45 / -16.74 / 41.94 / 1.7725.50 / 57.37 / 4.2354.22 / 88.00 / 21.52
CenterCLIP38.06.93 / 21.62 / -20.58 / 47.98 / 3.6629.90 / 61.72 / 6.9658.88 / 90.34 / 26.28
FROZEN-in-time330.00.30 / 0.67 / -0.83 / 2.10 / 0.221.60 / 3.37 / 0.675.89 / 12.97 / 2.55
CLIPBERT174.51.80 / 6.10 / -6.48 / 18.97 / 0.1810.93 / 29.00 / 0.8731.34 / 61.42 / 8.72
Singularity64.54.83 / 14.68 / -15.1 / 37.79 / 1.9522.23 / 49.50 / 4.4148.96 / 81.94 / 19.38
Ours (avg)35.08.52 / 26.91 / -23.56 / 54.18 / 3.8632.95 / 67.13 / 7.7961.26 / 92.50 / 27.80
Ours (max)49.06.98 / 22.72 / -18.72 / 46.37 / 2.3027.04 / 60.62 / 5.7654.19 / 88.37 / 20.90

Video-to-text, Charades-Event tab. 3:

ModelMedian rankR@1R@5R@10R@50
CLIP4Clip (mean)441.51.32 / 2.92 / -4.07 / 8.32 / 1.805.65 / 11.84 / 2.5516.84 / 32.01 / 7.65
CLIP4Clip (tight)391.00.80 / 1.80 / -2.45 / 6.30 / 0.754.19 / 10.27 / 1.3515.26 / 30.43 / 6.37
CenterCLIP408.01.32 / 3.00 / -3.71 / 7.87 / 1.576.33 / 13.42 / 2.6217.37 / 33.43 / 7.72
FROZEN-in-time810.00.30 / 0.67 / -0.83 / 2.10 / 0.221.60 / 3.37 / 0.675.89 / 12.97 / 2.55
ClipBERT1339.50.15 / 0.45 / -0.60 / 1.50 / 0.151.19 / 3.30 / 0.225.31 / 11.92 / 1.87
Singularity621.50.95 / 2.17 / -2.84 / 5.92 / 1.424.35 / 9.52 / 1.9512.22 / 25.94 / 4.87
Ours (avg)406.01.39 / 3.22 / -5.04 / 9.90 / 2.477.02 / 13.72 / 3.3717.62 / 32.08 / 8.70
Ours (max)365.61.01 / 2.62 / -4.03 / 9.00 / 1.726.39 / 13.72 / 2.7019.08 / 36.66 / 8.10

Text-to-video, ActivityNet Captions tab. 4. Recall only:

ModelMedian rankR@1R@5R@10R@50
CLIP4Clip (mean)11.015.2536.4349.1176.64
CLIP4Clip (tight)15.010.1429.0741.9372.50
CenterCLIP11.015.5337.2149.7877.19
FROZEN-in-time111.01.826.8011.7633.86
ClipBERT48.04.2014.5723.3551.02
Singularity (FT)22.09.2924.9636.2665.55
Ours (avg)11.015.9937.1349.6076.66
Ours (max)13.014.9334.4446.3674.65

Text-to-video, Charades-Event tab. 5. Recall only:

ModelMedian rankR@1R@5R@10R@50
CLIP4Clip (mean)136.01.536.4810.5429.38
CLIP4Clip (tight)127.01.455.549.3530.11
CenterCLIP131.01.936.6110.6730.13
FROZEN-in-time277.00.351.993.8214.52
ClipBERT442.00.351.773.3111.10
Singularity179.51.535.228.5223.87
Ours (avg)126.01.886.6910.5929.97
Ours (max)112.02.027.6112.7434.49

Ablations, ActivityNet Captions, text-to-video tab. 6:

SettingMedian rankR@1R@5R@10R@50
Ours (best)11.015.1735.9448.3975.85
w/o key event selection12.015.0335.6747.9475.06
w/o MeVTR loss (cross-entropy instead)12.014.4534.9047.2575.24

Ablations, ActivityNet Captions, video-to-text tab. 7 (Average recall only):

SettingMedian rankR@1R@5R@10R@50
Ours (best)35.08.5223.5632.7961.26
w/o key event selection41.57.4021.0030.2757.71
w/o MeVTR loss (cross-entropy instead)41.06.7020.0528.9657.45

Other reported results:

  • Performance by subset: the paper splits the ActivityNet test set by duration (test-S, -M, -L, -XL) and by number of events (test-E1 under 4, E2 with 5 to 12, E3 above 12). It reports the gain of Me-Retriever (avg) over CLIP4Clip (mean) on video-to-text per subset: the gain is usually larger on E1 and E2, decreases slightly for the videos with the most events, and varies only slightly with duration §5.2 fig. 4.
  • Weighting: with dynamic alpha, Recall@5 on video-to-text stays more stable across metrics than with fixed weights fig. 5 fig. 6 §5.3.
  • The authors describe the video-to-text result as the main strength and the text-to-video result as comparable to baselines, and attribute the max variant's text-to-video gap on ActivityNet to possible overfitting, without testing it §5.2.

Paper vs code

Repository: https://github.com/gengyuanmax/MeVTR, cloned at commit f9a23428e54009767ac2651cd3b9adda8c67d036. The paper links it in its abstract. The repository has no LICENSE file. The README covers evaluation only and gives no training command README.md.

  • Medoid distance. Paper: cosine similarity as the metric for medoid selection §4.1. Code: --cluster_distance defaults to euclidean main_task_retrieval_multi.py; the evaluation command does not override it README.md. The --pre_norm option that would L2-normalise before clustering is defined but never read main_task_retrieval_multi.py. So the evaluation clusters raw, unnormalised CLIP features with euclidean distance.
  • Iteration limit. Paper: 60 §5.1. Code: the CLI default is 100 main_task_retrieval_multi.py; the clustering function's own default is 60 fast_kmeans.py. The evaluation command does not set it, so the CLI value applies.
  • Initial medoids. Paper: "K frames are chosen as initial medoids", with no rule given §4.1. Code: a deterministic KKZ rule starts from the frame with the largest L2 norm and then adds the frame farthest from the chosen medoids cluster_utils.py. The argument is named kmediods++ main_task_retrieval_multi.py, which suggests k-means++ seeding, but the seeding used is KKZ.
  • Loss selection. Paper: other positives of a video are excluded from the video-to-text softmax §4.2 eq. 3. Code: the default --loss balanced selects CrossEnMulti, which normalises over all captions and does not remove other positives main_task_retrieval_multi.py modeling_multievent.py until_module.py. The exclusion is in CrossEnMulti_unbalanced, selected by --loss unbalanced until_module.py. That class also shifts each column by the mean of its maximum and minimum before the softmax, which the paper does not mention until_module.py. Which setting produced the reported numbers is not stated.
  • Average similarity. Paper: average of the cosine similarities eq. 5. Code: the key-event embeddings are each normalised, averaged, normalised again, and then dotted with the normalised caption embedding, scaled by the logit scale modeling_multievent.py. This equals the average cosine divided by the length of the mean key-event vector. For a fixed video, the ranking of captions is unchanged. For a fixed caption, the scores across videos are rescaled per video, which changes text-to-video ranking and the training softmax. The paper does not describe this step.
  • Max similarity. Paper: maximum over key events eq. 6. Code: the evaluation path takes the maximum over the scaled cosines modeling_multievent.py, selected by --sim_header maxP modeling_multievent.py. The training path for maxP is a different function, which I did not trace in full.
  • Temperature. Paper: τ is named but its value is not stated §4.2. Code: uses the learnable CLIP logit scale, and after each optimiser step clamps it to at most log(100) main_task_retrieval_multi.py.
  • Dynamic alpha. Paper: alpha equals L_v2t / L_t2v §4.2. Code: after the forward pass, alpha is set from the current batch's two losses (detached with .item()) and used from the next step on. Enabling it needs --dynamic_alpha, which the evaluation command does not use main_task_retrieval_multi.py main_task_retrieval_multi.py.
  • Epochs. Paper: 5 §5.1. Code: the default is 20 main_task_retrieval_multi.py. No training command is in the repository, so the paper's training configuration cannot be reconstructed from the code.
  • Backbone and frames. Paper: neither the CLIP variant nor the frame count is stated. Code: evaluation uses ViT-B/32 and 64 frames README.md README.md.
  • Key-event count at evaluation. Paper: 16 §5.1. Code: the evaluation command passes --post_cluster_centroids 16 README.md. The CLI default is 1, and --post_process defaults to none, so running without these flags uses every frame, not key events main_task_retrieval_multi.py.
  • Consistent items. The 27-caption cap §5.1 matches the dataloader cap dataloader_activity_multievent.py. The metric definitions in §3.2 match the code: average is the mean hit rate metrics.py, one-hit is the best rank metrics.py, all-hit is the worst rank metrics.py.

Pitfalls

  • Key events are not a pure clustering. Clustering runs on the full frame sequence without the frame mask modeling_multievent.py; the mask is applied only after selection modeling_multievent.py. Inferred from the code, not run: a padded frame can become a medoid and then drop out, so short videos may have fewer than 16 effective key events. The paper does not discuss this.
  • Metric choice may change the key events. Euclidean on unnormalised features, as the evaluation command uses, is not the cosine metric in the paper, and medoids can differ between the two. A reimplementation that follows the paper should decide which one to use and say so.
  • Loss flag matters. The default --loss balanced does not implement the exclusion described in the paper. Use --loss unbalanced to match the paper's description, and check which setting produced the reported numbers before comparing.
  • Scale of the average score. Eq. 5 and the code's normalised-mean score differ by a per-video factor. Implement the code's version if you want to reproduce the released model, and eq. 5 if you follow the text.
  • Table inconsistencies. The text-to-video ablation gives Ours (best) R@1 15.17 tab. 6, but the main text-to-video table gives Ours (avg) R@1 15.99 tab. 4. The video-to-text ablation gives R@10 32.79 tab. 7, against 32.95 in the main table tab. 2. The paper does not explain either difference.
  • Gains are in points. The text reports improvements of "1.12%" and "1.82%" from removing modules. These equal absolute differences in R@1 on video-to-text tab. 7 (8.52 minus 7.40 and minus 6.70), not relative gains §5.3.
  • Ablation scope. Ablations are only on ActivityNet Captions tab. 6 tab. 7. The paper does not say how frames are aggregated when key-event selection is removed.
  • Dynamic alpha timing. The code updates alpha from the previous batch, not the current one. Expect small differences if you update it within the same step.
  • Charades results are small. Absolute recall on Charades-Event is low for every model (text-to-video R@1 at most 2.02 tab. 5), so small differences may not matter. The tables give single values without error bars.
  • Table numbering. Equation, table and figure numbers in this card are counted from the LaTeX source in order; I did not check them against the compiled PDF.
  • Not checked. I read the ActivityNet data loader and the main model file, but not the Charades data loader or the full maxP training path.

Lineage

  • Builds on: CLIP (Radford et al.) and its video adaptation CLIP4Clip (Luo et al.); the data preparation follows the CLIP4Clip codebase README.md.
  • Compared against: CLIP4Clip (mean and tight), CenterCLIP (Zhao et al. 2022), FROZEN-in-time (Bain et al. 2021), ClipBERT (Lei et al. 2021), and Singularity. The table caption cites lei2022revealing among the compared models; the paper's bibliography gives arXiv:2206.03428 for it tab. 4.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Multi-event Video-Text Retrieval” by Gengyuan Zhang, Jisen Ren, Jindong Gu, Volker Tresp (arXiv:2308.11551), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/gengyuanmax/MeVTR at commit f9a23428e540 (not stated (no LICENSE file in the repository; no code quoted)). Any quoted code is shown under that license.