Align and Prompt: Video-and-Language Pre-training with Entity Prompts

In plain words

Video-text models learn by matching clips to their captions. Many earlier models pooled the whole clip into one summary and matched only at that coarse level, so they missed which object or scene in the picture a word refers to. Others relied on object detectors, which are slow and only know a small set of object names.

ALPRO adds two training signals. The first pulls each clip toward its own caption and away from the others in the batch, so the two halves of the model agree before they are combined. The second uses a helper model that has learned to label a clip with words, such as "dog" or "sky", by comparing it with phrases like "a video of a dog". The helper labels random patches of a video, and the main model learns to guess those labels from the patch it sees. The labels come from video-text pairs alone, so no extra annotation is needed. The result is a model that is strong on finding a clip from a sentence and on answering questions about a video.

TL;DR

• ALPRO pre-trains a video-language model on 5.5M web video-text pairs using four objectives §3.2 eq. 6.
• Its new parts are a video-text contrastive loss (VTC) and prompting entity modeling (PEM), which predicts entity pseudo-labels for random video crops §3.2.1 §3.2.2.
• Pseudo-labels come from a prompter that is pre-trained with VTC and then frozen, so no detector or human labels are needed §3.2.2.
• Author-reported: 33.9 R@1 on MSRVTT 1k after fine-tuning tab. 2 and 42.1 MSRVTT-QA accuracy tab. 4.
• Code is official (BSD-3-Clause) and mostly matches the paper, but the pseudo-label discard rule is implemented differently (see Paper vs code).

Problem

Earlier sparse video-language models encode video and text separately and then fuse them with a transformer, or match them with a dot product. The authors argue that the two unimodal embedding spaces do not line up before fusion, which makes cross-modal interaction less effective §1 §3.2.1.

Methods that add fine-grained, region-level supervision mostly use object detectors. The paper says these are costly, give imprecise results on video frames, and cover only a restricted set of categories; it cites detectors trained on MSCOCO that recognise fewer than a hundred categories §1 §3.2.2. Masked language modeling has no direct visual counterpart in existing video-language pre-training §3.2.2.

The paper assumes the sparse-sampling paradigm (a few frames per video) so that the video backbone can be trained end to end §1.

Method

The system pairs the network being pre-trained with a second, auxiliary network. That auxiliary network, the prompter, writes soft entity labels that supervise the main network during pre-training §3.1 fig. 2.

Architecture §3.1.

• Visual encoder: a 12-layer TimeSformer at 224 pixels. Sparsely sampled frames are split into patches, processed with temporal and spatial self-attention, and mean-pooled over time. The output includes a video [CLS] token.
• Text encoder: a 6-layer transformer over text tokens, with its own [CLS].
• Multimodal encoder: a 6-layer transformer that takes the concatenated video and text tokens. Positional embeddings are already added in the unimodal encoders.
• The prompter has its own video and text encoders §3.1.

Video-text contrastive loss (VTC) §3.2.1.

1. Project the video and text [CLS] tokens to a common normalised 256-d space with two linear layers and take the dot product as the similarity eq. 1.
2. Treat matched pairs in a batch as positives and all other pairs as negatives, with a learnable temperature τ eq. 2 eq. 3.
3. Average the video-to-text and text-to-video terms §3.2.1.

s(V,T)=g_v(\mathbf{v}_\mathrm{cls})\cdot g_t(\mathbf{t}_\mathrm{cls}) eq. 1

\mathcal{L}_\mathrm{v2t}=-\log\frac{\exp(s(V_i,T_i)/\tau)}{\sum_{j=1}^{B}\exp(s(V_i,T_j)/\tau)} eq. 2

\mathcal{L}_\mathrm{t2v}=-\log\frac{\exp(s(T_i,V_i)/\tau)}{\sum_{j=1}^{B}\exp(s(T_i,V_j)/\tau)} eq. 3

Prompter (entity pseudo-labels) §3.2.2.

1. Pre-train the prompter's two unimodal encoders with VTC on video-text pairs, then freeze them §3.2.2.
2. Keep M text prompts made from templates such as "A video of {ENTITY}.", where each entity is a frequent noun §3.2.2 app. A.1.
3. Encode each prompt once. For each entity, average the [CLS] embeddings of its prompts over the 12 video templates in the ensemble §4.5 app. A.1.
4. For a random video crop, compute the softmax over the similarities between the crop's [CLS] embedding and the M prompt embeddings. This is the pseudo-label q eq. 4.

q_{\hat{V},m}=\frac{\exp(s(\hat{V},T_m)/\tau)}{\sum_{m'=1}^{M}\exp(s(\hat{V},T_{m'})/\tau)} eq. 4

Prompting entity modeling (PEM) §3.2.2.

1. Draw a random crop that covers 30%-50% of the frame area §3.4.
2. Compute the pseudo-label q for the crop with the frozen prompter.
3. Mean-pool the multimodal-encoder outputs at the crop's positions §3.2.2.
4. Predict the M entities from the pooled vector with an MLP classifier §3.2.2.
5. Use the cross-entropy between the prediction p and the pseudo-label q as the loss eq. 5.

\mathcal{L}_\mathrm{pem}=-\sum_{m=1}^{M}q_{\hat{V},m}\log p_{\hat{V},m} eq. 5

Other objectives §3.2.3. Masked language modeling masks 15% of text tokens §3.2.3. Video-text matching is a binary classifier on the multimodal [CLS] token with hard negatives from the batch, following ALBEF §3.2.3.

Total loss §3.2.3. The four terms are summed with weight 1 each.

\mathcal{L}=\mathcal{L}_\mathrm{vtc}+\mathcal{L}_\mathrm{pem}+\mathcal{L}_\mathrm{mlm}+\mathcal{L}_\mathrm{vtm} eq. 6

Pre-training data §3.3. WebVid-2M (2.5M video-text pairs) plus CC-3M image-text pairs, for 5.5M pairs in total. Each image is repeated to make a static video §3.3. The paper notes this is about an order of magnitude less data than HowTo100M §3.3.

The pseudo-code below shows the prompt bank (once, after the prompter is trained) and one pre-training step with the VTC and PEM terms. MLM and VTM are left out. Symbols follow §3.2 and the equations above.

# Build the prompt bank once, from the frozen prompter  [§3.2.2] [app. A.1]
# Input:  entities (M names, top 1k nouns) [§3.4]; 12 video templates
# Output: prompt_feats (M, 256), one averaged embedding per entity

def build_prompt_features(entities, templates, prompter):
    # 1. Instantiate every template with every entity  [app. A.1]
    prompts = [t.format(e) for t in templates for e in entities]
    # 2. Encode the prompts with the frozen text encoder and projection
    embeds = prompter.encode_text(prompts)   # (len(templates) * M, 256)
    # 3. Average the template embeddings of each entity  [§4.5] [app. A.1]
    prompt_feats = mean_over_templates(embeds)   # (M, 256)
    return prompt_feats

# One pre-training step, VTC and PEM terms only  [§3.2] [eq. 1-6]
# Input:  frames (B, 4, 3, 224, 224); captions; prompt_feats (M, 256)
# Output: loss_vtc + loss_pem  (MLM and VTM are added per [eq. 6])

def pretrain_step(frames, captions, prompt_feats, prompter):
    # 1. Encode video and text separately  [§3.1]
    video_tokens = video_encoder(frames)   # (B, 1 + K, d)
    text_tokens = text_encoder(captions)   # (B, 1 + N_t, d)

    # 2. Project the [CLS] tokens to a normalised 256-d space  [§3.2.1]
    video_feat = normalise(project_video(video_tokens[:, 0]))
    text_feat = normalise(project_text(text_tokens[:, 0]))

    # 3. Contrastive loss over the batch, both directions  [§3.2.1] [eq. 2-3]
    sim = video_feat @ text_feat.T / tau   # (B, B)
    labels = arange(B)
    loss_vtc = (
        cross_entropy(sim, labels)
        + cross_entropy(sim.T, labels)
    ) / 2

    # 4. Sample a random crop covering 30%-50% of the frame area  [§3.4]
    crop = random_crop(frames, area_fraction=(0.3, 0.5))

    # 5. Frozen prompter gives the soft entity pseudo-label  [§3.2.2] [eq. 4]
    crop_feat = normalise(prompter.project_video(prompter.video(crop)))
    pseudo = softmax(crop_feat @ prompt_feats.T / tau)   # (B, M)

    # 6. Drop crops whose top pseudo-label score is below 0.2  [§3.4]
    keep = max(pseudo, axis=1) >= 0.2   # (B,)

    # 7. Mean-pool multimodal tokens at the crop's positions  [§3.2.2]
    fused = multimodal_encoder(video_tokens, text_tokens)
    crop_tokens = mean_over_crop_positions(fused, crop)   # (B, d)

    # 8. Predict the entity distribution with an MLP  [§3.2.2]
    log_probs = log_softmax(mlp(crop_tokens))   # (B, M)

    # 9. Soft cross-entropy against the pseudo-label  [§3.2.2] [eq. 5]
    loss_pem = -sum(pseudo * log_probs, axis=1)[keep].mean()
    return loss_vtc + loss_pem

Hyperparameters

Pre-training values come from the paper §3.4 unless marked "code". Where the paper is silent, the code's value is given with its file. Fine-tuning values are in tab. 9 and tab. 10.

Fine-tuning configurations tab. 9 tab. 10

Further fine-tuning details app. A.2: frames are resized to 224 x 224; for N frames, N/2 are drawn from each half of the video; the same augmentation applies to all frames of a video; inference uses uniform sampling without augmentation; seed 42.

Results

Author-reported. Setup: pre-training on WebVid-2M and CC-3M (5.5M pairs) §3.3; fine-tuning end to end on 8 A100 GPUs for one to five hours per dataset §4.1. Retrieval uses 8 frames and QA 16 frames §4.1. Baselines are the methods listed in each table. The paper compares against each baseline's own pre-training data, so pre-training sets are not matched tab. 2 tab. 4.

Text-to-video retrieval, MSRVTT 1k test tab. 2. Greyed-out rows in the source use 9k training videos (SupportSet and FiT, fine-tuned).

Source: tab. 2. The text says the paper does not include CLIP4Clip, which uses CLIP encoders pre-trained on 400M image-text pairs §4.3.

Text-to-video retrieval, DiDeMo tab. 3. Paragraph-to-video retrieval, with sentences concatenated per video §4.1. Greyed rows are not present in this table.

Open-ended video QA, top-1 accuracy % tab. 4. The paper states that VQA-T (greyed in the table) uses 69M QA-specific pairs, while ALPRO uses 5.5M generic web pairs tab. 4 §4.4.

Ablations

Pre-training objectives tab. 1. Each row is a model pre-trained with the listed objectives, then fine-tuned for retrieval and QA. Retrieval values are R@1, R@5, R@10 and median rank (MdR); QA values are accuracy.

The paper reads this table as: VTC helps retrieval most, PEM helps QA most (especially MSVD-QA), and the two together help most datasets, with a slight drop on MSVD-QA §4.2.

Prompt ensembling tab. 5. Fine-tuned (FT) and zero-shot (ZS) MSRVTT, and MSVD-QA.

Number of entities tab. 6. The first row is MLM + VTM + VTC (no PEM).

Number of frames tab. 7. Both tasks use the same frame count in this table.

The paper states that gains saturate beyond 8 frames for retrieval §4.5.

Pseudo-label examples fig. 3 fig. 4. The paper shows pseudo-labels for video crops, with a more varied vocabulary than detector classes (examples include "towers", "summit", "yoga") fig. 3 fig. 4. These are qualitative examples; no quantitative pseudo-label quality is reported.

Paper vs code

Code: official repository salesforce/ALPRO at commit d21173f55a73b922d8ce2b06a05f78623b344fe7 (cloned 2026-10-18), BSD-3-Clause. Discrepancies found:

• Pseudo-label discard rule. Paper: discard a pseudo-label when the most likely entity's normalised similarity is below 0.2 §3.4. Code: the threshold is applied to the argmax index, not to the score, because torch.max(...)[1] returns indices, and it is applied to the scaled similarities before the softmax alpro_models.py alpro_models.py. In effect only crops whose top entity has index 0 are discarded, and the 0.2 threshold is never applied to a score.
• Frame sampling in pre-training. Paper: 4 frames sampled randomly, order kept §3.4. Code: the pre-training config sets headtail sampling [code: config_release/pretrain_alpro.json#L58-L59]. That sampler takes two sorted frames from the first half of the video and two from the second half [code: src/datasets/dataset_base.py#L169-L172].
• Temperature. Paper: learnable, value not given §3.2.1. Code: initial value 0.07 alpro_models.py, clamped to [0.001, 0.5] during training alpro_models.py. The prompter's temperature is loaded from its own checkpoint, so the same value applies to pseudo-labels alpro_models.py.
• Prompter training. Paper: VTC pre-training, then frozen §3.2.2. Code: the prompter config uses VTC only pretrain_prompter.json. Pseudo-labels are computed under torch.no_grad() alpro_models.py. I did not confirm that the prompter parameters are excluded from the optimizer, so the paper's "frozen" claim is supported by the no-grad call but not by an explicit requires_grad=False.
• Loss weights. Paper: equal weights eq. 6. Code: the four losses are added without weights run_pretrain_sparse.py. Consistent.
• Retrieval training and inference. Paper: fine-tuning sums VTC and VTM, and inference uses the VTM head §4.1. Code: fine-tuning sums both losses run_video_retrieval.py; inference takes the softmax of the two-class VTM logits run_video_retrieval.py. Consistent.
• Entity list. Paper: top 1k nouns from a POS tagger §3.4. Code: takes the first token of the first 1000 lines of data/unigrams.txt [code: src/pretrain/run_pretrain_sparse.py#L369-L370]. The POS filter is not in the repository, and the data file is absent, so the vocabulary cannot be checked.
• Image branch. Paper: each CC-3M image is duplicated to make a static video §3.3. Code: repeats the image num_frms times [code: src/datasets/dataset_pretrain_sparse.py#L167]. Consistent. The code also applies a RandomResizedCrop with scale (0.2, 1.0) to images [code: src/datasets/dataset_pretrain_sparse.py#L137]; the paper does not mention it.
• Templates. Paper: 12 video templates app. A.1. Code: the same 12 templates run_pretrain_sparse.py. Consistent.

Pitfalls

• Pseudo-label threshold. The implementation differs from the text (see Paper vs code). An implementer who follows the paper will keep crops that the released code discards, and vice versa. Pick one rule and document it.
• Temperature for pseudo-labels. The softmax temperature τ in eq. 4 controls how sharp the targets are. The paper gives no value. The code uses the prompter's learned temperature, which started at 0.07. Pseudo-label sharpness depends on this choice, and the paper reports no sensitivity study.
• Prompt embeddings are not renormalised in the code. The code averages the normalised template embeddings and keeps the mean as is [code: src/modeling/alpro_models.py#L470-L475] alpro_models.py. Averaging shrinks the norm when templates disagree, which scales the logits for each entity. The paper does not say whether to renormalise.
• Crop geometry beyond the area. The paper gives only the 30%-50% area §3.4. The code also samples an aspect ratio between 0.3 and 1/0.3 and snaps the crop to a 16-pixel patch grid [code: src/datasets/dataset_pretrain_sparse.py#L277-L291]. Patch snapping also defines the mask that pools the multimodal tokens.
• PEM normalisation. Eq. 5 is a per-crop sum and gives no batch-level normalisation. The code divides by the number of non-discarded crops [code: src/modeling/alpro_models.py#L230].
• Unused context branch. The code builds a context input with the crop erased, but uses it only when use_mask_prob is above zero. The default is 0, so the branch is inactive alpro_models.py [code: src/modeling/alpro_models.py#L94].
• Negatives span all GPUs. VTC gathers features across all GPUs, so the negatives are the global batch of 256 and not one GPU's batch [code: src/modeling/alpro_models.py#L110-L111]. This matters for anyone who runs with a smaller world size.
• Non-square inputs. Frames are resized to 224 x 224 and aspect ratios are not kept. The paper reports no significant drop §4.1.
• Cost. Pre-training used 16 A100 GPUs for 100k iterations §3.4. Fine-tuning takes one to five hours per dataset on 8 A100 GPUs §4.1. Each PEM step adds a forward pass through the prompter.
• Data. The pre-training data is scraped from the web and may contain unsuitable videos, harmful text and private information. The authors say the model needs further analysis before deployment §5.
• Comparisons. Several baselines use other pre-training data or more training videos (greyed in tab. 2 and tab. 4). Compare the rows with their own training sets in mind.
• Frame count. In tab. 7 the MSVD-QA score at 8 frames is 45.4, while tab. 1 and tab. 4 report 45.9 for QA at 16 frames. The retrieval setting uses 8 frames.
• Entity count. Going from 1000 to 2000 entities lowers zero-shot R@1 from 24.1 to 22.4 tab. 6. The paper attributes this to noisy low-frequency entities §4.5.

Lineage

• Builds on: ActBERT's detector-based regional labels, which ALPRO replaces with prompter pseudo-labels §1 §3.2.2. CLIP's contrastive objective and prompt ensembling §3.2.2 §4.5. ALBEF's hard-negative mining for VTM §3.2.3. ClipBERT and FiT for sparse sampling and image-text pre-training §1 §3.3.
• Compared against: the methods listed in tab. 2, tab. 3 and tab. 4.
• Superseded by: not stated in this card.