MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

In plain words

Many video models learn from the pictures, or from the spoken words that come with them. This one learns from all three at once: the frames, the subtitles and the sound. It was pretrained on about 20 million YouTube videos.

The training task works like a fill-in-the-blank test. Parts of the subtitles or of the sound are hidden, and the model must pick out which hidden piece belongs in each gap from a set of candidates. To do that, it has to understand what is happening on screen and what would be said or heard there. Because the sound is both a hint and a target, the model learns about sound as well as about speech and pictures.

After extra training on one task, the model answers questions about movie scenes, TV episodes and activity clips. It can also fill in a blank with a short phrase without any task training. The authors report that audio helps even on tasks that have no sound.

TL;DR

• Self-supervised video encoder over 5-second segments: a ViT for frames, an audio transformer for sound, BPE embeddings for subtitles, and one joint transformer over all segments and modalities §3.1.
• Pretraining hides 25% of subtitle and audio spans. A contrastive loss matches each hidden span to its own independent encoding among in-batch candidates §3.2 eq. 1.
• Pretrained on 20 million YouTube videos §3.3. Pretraining took 5 days for the Base model and 16 days for Large on TPU v3-512 §3.2.
• Author-reported: state of the art after finetuning on VCR, TVQA and Kinetics-600, and competitive zero-shot results on four video tasks §4.
• Code is MIT-licensed and the repository links checkpoints. Several paper and code details disagree; read Paper vs code before reimplementing.

Problem

• Video-language models usually learn from frames with subtitles. Models that use audio are rare, and before this paper no VCR model used audio in pretraining §1 §4.1.
• Subtitles from YouTube are automatic and time-aligned poorly. Each word has one timestamp that often appears about 0.1 s before the word is spoken, and pauses make word intervals loose app. C.1. If text and audio are misaligned, a model can cheat by reading the masked word from the neighbouring audio app. C.
• Running a new speech recogniser over all videos was judged too expensive; the authors used a timing regressor and conservative masking instead app. C.
• Predicting a perceptual signal from the same signal (audio from audio, frames from frames) tends to learn recording artefacts; the authors avoid this by never giving a modality as input when it is also the target §3.2.
• Context limit: the conclusion says the model learns from 40-second videos §6. In the method, a video is split into two groups of 8 segments of 5 s, each encoded separately app. B.5.

Method

Each video is cut into segments of 5 s by default. Each segment has a frame from its middle, the subtitle words spoken in it, and its audio §3.1. Each segment gives the joint encoder exactly one of text or audio, never both §3.1.

1. Segmenting. A 5-second sliding window runs over the video. Where two segments in a row have fewer than 8 subtitle tokens, they are merged 90% of the time, at most twice, so the segment can span up to 15 s app. C (footnote).
2. Audio subsegments. Each 5-second audio segment is split into three equal subsegments of about 1.6 s, each aligned with the subtitle text for that span app. B.2 app. C.
3. Independent encoders.
   • Frames: a ViT-B/16 (Base) or ViT-L/16 (Large), then 2×2 query-key-value attention pooling. A 192×320 frame gives a 6×10 grid of 60 tokens §3.1 app. B.2.
   • Audio: each subsegment is a 1-D sequence of 60 spectrogram hops, encoded by a 12-layer audio transformer and pooled by a factor of 5 to 6 tokens app. B.2 tab. 8.
   • Text: a BPE embedding table. Spans of at most 15 tokens go through a separate 4-layer span encoder with a CLS token, which produces the independent targets for masked text §3.1 app. B.2.
4. Joint encoder. A bidirectional transformer (12 layers in Base, 24 in Large) reads all segments together, with the text and pooled audio of each video segment and its pooled frames §3.1 tab. 8. Positions use a rotary encoding over (height, width, text position, segment index) app. B.1.
5. Two masked views per video. Each 5-second segment has three subsegments, and 25% of them are masked §3.2 app. C.4.
   • View VT→TA: frames and subtitles are input. Masked subtitle spans get a MASK token, and masked audio gets a MASKAUDIO token. The model predicts both text and audio app. C.4.
   • View VTA→T: frames plus either subtitles or audio per subsegment are input. Masked subtitles get MASK tokens, and the model predicts only text. An audio subsegment next to a masked one is turned into subtitles 80% of the time, so the masked words cannot leak through misaligned audio app. C.3 app. C.4.
6. Contrastive span objective. The representation at each MASK or MASKAUDIO position must match the independent encoding of its own span, with every other candidate in the batch as a negative. Text and audio use the same loss form §3.2 eq. 1. The learned scale is clipped at 100 §3.2.
7. Candidates. Each video gives 48 text spans, chosen from 38 spans of web text and 24 spans of its own subtitles. Empty spans are penalised, and video spans are chosen four times as often app. C.4. All 48 audio spans of each video are candidates app. C.4. With 1024 videos per batch, that is 49,152 text candidates and 49,152 audio candidates tab. 8.
8. Frame loss. Each frame must match the joint encoder's contextualised representation of the transcript at that segment, with the same symmetric contrastive form §3.2. The paper describes the input as the transcript; the code can substitute audio (see Paper vs code).
9. Total loss. The three losses are summed §3.2 eq. 2.
10. Resolution adaptation (FixRes). After pretraining, the model is trained for a short extra stage at larger frames: 288×512 widescreen and 384×384 square. The joint sequence grows to 1312 tokens app. B.2 tab. 8.

Pseudocode of the contrastive term (the paper gives equations, not code; this follows the official implementation, which the paper does not describe in full):

# Contrastive span loss for one direction  [§3.2] [eq. 1]
# Input:  queries (M, d) at MASK positions; targets (K, d) span encodings
# Output: scalar loss

def span_contrastive_loss(queries, targets, log_scale):
    # Learned scale, clipped at 100 and split over both sides  [§3.2]
    # [code: pretrain/pretrain_model.py#L239-L249]
    scale = exp(min(log_scale, log(100)) / 2)
    # Unit-normalise both sides, then apply the scale  [eq. 1]
    queries = scale * unit_normalize(queries)            # (M, d)
    targets = scale * unit_normalize(targets)            # (K, d)
    # Score of every query against every candidate  [eq. 1]
    logits = queries @ transpose(targets)                # (M, K)
    # The true target of query i sits at index match_index[i]  [§3.2]
    positive = logits[arange(M), match_index]
    # Cross entropy over all candidates in the batch  [eq. 1]
    loss_forward = mean(logsumexp(logits, axis=1) - positive)
    # Reverse direction: each matched target against all queries  [eq. 1]
    loss_reverse = mean(logsumexp(logits, axis=0)[match_index] - positive)
    # The code averages the two directions; eq. 1 writes their sum
    # [code: pretrain/pretrain_model.py#L295-L300]
    return (loss_forward + loss_reverse) / 2

# One pretraining step on a batch of videos  [§3.1] [§3.2] [app. C.4]
# Input:  frames (B, 16, 3, 192, 320); audio and text per subsegment
# Output: scalar loss

def pretraining_step(batch):
    # Encode each frame with the ViT and 2x2 attention pooling  [app. B.2]
    frame_tokens = vit_attention_pool(batch.frames)     # (B, 16, 60, d)
    # Encode each 1.6 s audio subsegment on its own  [§3.1] [app. B.2]
    audio_tokens = audio_encoder(batch.audio)           # (B, 48, 6, d)
    # Independent CLS targets for frames and audio  [§3.2]
    frame_targets = vit_cls(batch.frames)
    audio_targets = audio_cls(batch.audio)
    # Encode every candidate text span with the span encoder  [§3.2]
    span_targets = span_encoder(batch.candidate_spans)
    # View VT -> TA: frames and text in, masked text and audio out  [app. C.4]
    out_ta = joint_encoder(frame_tokens, batch.view_vt_ta)
    # View VTA -> T: audio as input where unmasked, masked text out  [app. C.4]
    out_t = joint_encoder(frame_tokens, batch.view_vta_t)
    # Web-text sequence, which adds text candidates  [app. C.4]
    out_web = joint_encoder(batch.web_text)
    # Transcript-frame matching over the segments  [§3.2]
    out_frames = joint_encoder(frame_tokens, batch.transcript)
    # Read each output at the positions the loss needs  [§3.2]
    text_queries = gather_masks(out_t, out_web, batch.text_mask_pos)
    audio_queries = gather_masks(out_ta, batch.audio_mask_pos)
    frame_queries = gather_segments(out_frames, batch.segment_pos)
    # Three contrastive losses, summed  [§3.2] [eq. 2]
    loss_text = span_contrastive_loss(text_queries, span_targets, log_scales[0])
    loss_audio = span_contrastive_loss(
        audio_queries, audio_targets, log_scales[1])
    loss_frame = span_contrastive_loss(
        frame_queries, frame_targets, log_scales[2])
    return loss_text + loss_audio + loss_frame

Finetuning (appendix D.1 and Table 9). Each candidate is scored as a separate sequence, and a linear head is applied to the representation at a MASK token placed after the input app. D.1.

• VCR: the four answers and then the four rationales are each scored independently, with softmax cross entropy. Detection tags are drawn on the image, with a box colour set by hashing each tag's identifier. Random left-right flips are used unless the words "left" or "right" occur in the text, and images are randomly resized to 100–110% app. D.1.
• TVQA: five candidates, each in two versions, subtitles and audio. Training duplicates each sequence; at test time the two softmax outputs are averaged for "both" §4.2 app. D.1. Each question uses a 35-second window around the annotated time, with the annotated region given as two integers scaled by 100 app. D.1.
• Kinetics-600: four frames and six audio subsegments (10 s) per clip. The 600-way head is initialised from the span encoder's activations for the class names app. D.1.

Zero-shot (appendix D.2, §4.4). The answer options are encoded as phrases, and the model picks the closest phrase to the MASK representation §4.4.

• STAR: questions are converted into statements with the answer masked, and the four options form the label set app. D.2.
• EPIC-Kitchens: one MASK token and no text prompt. The label set is every verb and noun pairing in the vocabulary, with no in-domain training app. D.2.
• LSMDC: the blank is replaced by MASK, and the vocabulary of the LSMDC dataset is the label set app. D.2.
• MSR-VTT QA: questions are rewritten into statements by GPT-3 with a dozen examples in the prompt (text in appendix D.2). The candidate answers are the most common answers, 2000 in appendix D.2 and 1000 in §4.4; see Paper vs code app. D.2 §4.4.

Hyperparameters

Values from the paper unless the "Where" column names the code. Table 8 collects the architecture and pretraining settings. Finetuning values are in Table 9.

Results

Author-reported. Each result uses the setup in the cited table or section.

Visual Commonsense Reasoning, test set tab. 2 §4.1. Single models, with image size 288 × 512 for finetuning tab. 9.

Efficiency for one VCR candidate tab. 7: total GFLOPs for UNITER-Base 1794, UNITER-Large 1867, MERLOT 303, This paper Base 146, and This paper Large 341. Parameter counts for comparison: MERLOT 223M, UNITER-Base 154M, and UNITER-Large 378M app. B.3.

TVQA, validation and test accuracy tab. 3 §4.2. "Both" averages the subtitle and audio predictions.

Kinetics-600, validation top-1 and top-5 tab. 4 §4.3.

Zero-shot tasks tab. 5 tab. 10 §4.4. Supervised reference numbers are shown where the table gives them.

Per-question STAR accuracy is in the four STAR columns of tab. 5. The EK100 test-set numbers are in tab. 10: the Large model with audio scores verb 24.0, noun 25.5, action 5.8, against AVT+ at 25.6, 28.8 and 12.6 tab. 10.

Ablations tab. 1 §4.1.1. VCR Q→A validation accuracy after one epoch of pretraining, with the Base architecture unless noted.

The paper says audio pretraining helps VCR, which has no audio input §4.1.

Pretraining progress fig. 4 §4.1. Finetuned VCR accuracy rises with pretraining epochs. The caption gives a gain of 5% for Base and 8% for Large from nine more epochs. The paper says a longer run might help further.

Qualitative audio analysis fig. 5 §5. Examples show the masked audio being predicted from visual cues such as cooking, and the predictions improve during pretraining fig. 5. This is illustrative, not a measured result.

Paper vs code

Code: github.com/rowanz/merlot_reserve at commit d22c50f53172813687b795e1269634570308a5c6, MIT License, linked from the first author's project page. The repository includes pretraining, finetuning (VCR and TVQA only), data preparation and demo scripts. The README lists checkpoints for the pretrained Base and Large models, with and without FixRes, and for finetuned VCR and TVQA models, all in Google Cloud Storage [code: README.md]. The licence of the checkpoints is not stated; this card does not claim one.

Discrepancies, each with both sources:

• Audio sample rate. Appendix B.2 says 22500 Hz app. B.2. Table 8 and the config say 22050 Hz tab. 8 base.yaml. Use 22050.
• Spectrogram width. Appendix B.2 gives 192 windows per 5 s, with 12 held out app. B.2. The config gives spec_size 188 base.yaml. At 22050 Hz and hop 588, 5 s gives about 187.5 windows, so the code's 188 is consistent with the stated hop. Use 188 and 8 spare frames.
• Playback speed input. The code appends a playback-speed value to each audio frame, making 65 features per frame dataloader.py. The paper describes no such input.
• Spectrogram scaling. Spectrograms are stored as images and divided by a per-clip magic number dataloader.py [code: pretrain/dataloader.py#L112]. The paper does not describe the stored format or its normalisation.
• Frame-matching input. The paper says the frame-matching sequence uses the transcript §3.2. The code selects audio spans instead of text for about half of the videos, controlled by use_audio_token_prob = 0.5 [code: pretrain/configs/base.yaml#L10] dataloader.py [code: pretrain/dataloader.py#L619].
• Loss direction. Equation 1 writes the two directions as a sum eq. 1. The code averages them pretrain_model.py [code: pretrain/pretrain_model.py#L300]. The total loss is a sum, as in equation 2 pretrain_model.py eq. 2.
• Temperature. The paper says σ is learned and clipped at 100 §3.2. The code learns one log-scale per loss (three in total), initialised to 0, so σ starts at 1 modeling.py. The clip is on the log-scale at log 100 pretrain_model.py. The scale is split as the square root on each side pretrain_model.py.
• MSR-VTT label set. §4.4 says the label space is the top 1k options §4.4. Appendix D.2 says the top 2000 most common answers app. D.2.
• Extra denominator candidates. The code adds extra audio spans to the candidates in one direction, beyond the matched audio spans [code: pretrain/pretrain_model.py#L288-L290]. The paper describes all 48 audio spans per video as candidates app. C.4.
• Rotary encoding. The paper describes rotation of pairs app. B.1. The code pairs the dimensions differently from standard RoPE, and its comments say this may be a bug modeling.py [code: mreserve/modeling.py#L138-L140]. Match the code if you need the released checkpoints to behave identically.
• Attention augmentation. The code sometimes cuts the joint attention at a random point between the two groups, with probability 0.1 by default [code: pretrain/pretrain_model.py#L27] pretrain_model.py. The paper does not describe it.
• FixRes length. Appendix B.2 says one epoch for Base and half an epoch for Large app. B.2. Table 8 says one extra epoch for both tab. 8. The script runs 75,000 steps for whichever config is passed [code: pretrain/train_fixres.py#L141].
• TVQA epochs. Table 9 gives 3 epochs tab. 9. The TVQA script's default is 5 tvqa_finetune.py.
• VCR learning rate. Table 9 gives 2e-5 for Base tab. 9. The VCR README's example uses 1e-5 with 5 epochs README.md.
• VCR Large score. Table 2 gives Q→AR 72.0 for Large tab. 2. The efficiency table gives 71.5 for the same model tab. 7.
• Kinetics finetuning. The repository has no Kinetics finetuning code (finetune/ contains only vcr and tvqa), so the Kinetics settings are unchecked.
• Pretraining data pipeline. The repository has a data/offset_model directory with model.py and model_params.json. This card did not check it against the regressor in app. C.2.

No other discrepancies were found after checking the warmup, the cosine schedule, the 0.02 final scale, the masking rate, the 0.125 s threshold, the 4× multimodal span preference, the 750k steps, and the optimizer constants.

Pitfalls

• What an epoch is. The config defines an epoch over 77,410,304 sequences of 16 segments, about 75,596 steps at batch 1024 base.yaml. The paper says 10 epochs over 20 million videos §3.3 app. B.4. Use the step counts, not "epochs over videos".
• The 40-second limit. The conclusion says the model learns from 40-second videos §6. The joint encoder sees groups of 8 segments of 5 s app. B.5. A training video has two such groups. Long videos are not covered by a single pass.
• Timing noise. YouTube word timestamps run early, and the interval for each word is loose app. C.1. The MLP offset correction is trained on 2000 videos aligned with Google Cloud ASR, and it is bounded to about 0.3 s app. C.2. Errors larger than that remain. The masking rules are the main protection app. C.3.
• Alignment rules. When audio is an input, a masked span's neighbour can be converted to text, so the masked words are not visible in the audio app. C.3. Get the 0.125 s token donation rule and the 80% conversion right, or the task becomes a speech-recognition task app. C.3 dataloader.py dataloader.py.
• Subsegment length is stated two ways. Appendix C says 1.66 s, and Appendix B.2 says about 1.6 s app. C app. B.2. The code uses 60 hops of 588 samples at 22050 Hz, which is about 1.6 s base.yaml.
• Batch size and memory. Attention cost grows with the square of the sequence length, so the authors split videos into groups of 8 segments app. B.5. They also pack six sequences per video into one call, which keeps the TPU busy app. C.4. A reimplementation must keep both to match the throughput.
• Mixed precision. Training uses bfloat16 throughout, including the Adam moments app. B.4 base.yaml. Gradients with NaN values are set to zero, which the authors say fixed the symptom of spikes. They say they do not know why the spikes happen app. B.4.
• Learned scale starts at 1. The paper does not give the initial σ or how it is stored. The code starts it at 1 and clips it modeling.py pretrain_model.py.
• Weight decay masking. Pretraining applies decay only to tensors with more than one dimension optimization.py. The paper does not say so. Finetuning uses decay toward the pretrained weights and also masks small tensors optimization.py optimization.py.
• Finetuning regularisation. There is no dropout. The regularisation is L2 toward the pretrained weights, with a weight of 0.1 app. D.1. Plain weight decay toward zero would behave differently.
• Zero-shot prompts are fragile. The paper says it could not find a better text prompt for EPIC-Kitchens than a single MASK, and suspects one exists app. D.2. The STAR templates and the GPT-3 rewrites for MSR-VTT add errors, and the MSR-VTT rewrites were made with GPT-3, an external model the paper used at the time app. D.2. Reproduce with the same prompts or results will differ.
• TVQA gap. The text says Large beats prior work by 7.6% §4.2. From Table 3, the gap to MERLOT is 7.2 points for val subtitles, 7.8 for val both, 7.2 for test subtitles and 7.7 for test both. The 7.6% figure cannot be reproduced exactly from the table tab. 3.
• STAR margin. The text says Base beats supervised prior work by 3.7% §4.4. Table 5 gives 39.4 without audio and 40.5 with it, against 36.7 for ClipBERT, which is 2.7 and 3.8 points tab. 5.
• Compute. Base pretraining took 5 days on TPU v3-512 and Large 16 days §3.2. Efficiency numbers are FLOPs for one VCR candidate, and UNITER's count leaves out non-max suppression, so it is a lower bound app. B.3.
• Data. The project page lists Google Cloud Storage paths for the YT-Temporal metadata. This card does not check whether they are public or what licence the data carries README.md. Data collection is described in appendix E, which this card does not summarise.
• Numbering. Table and figure numbers were derived from the order of captions in the LaTeX source, not from the rendered PDF. Check them against the PDF before citing a number in another page.

Lineage

• Builds on: MERLOT (arXiv:2106.02636) for the frame-to-transcript matching and the VCR finetuning recipe §3.2 §4.1. Vision Transformer (arXiv:2010.11929) for the frame encoder §3.1. Audio Spectrogram Transformer (arXiv:2104.01778) for the audio encoder §3.1. CLIP (arXiv:2103.00020) for the contrastive form §3.2.
• Compared against: VATT (arXiv:2104.11178) for Kinetics-600 §4.3. UNITER, ERNIE-ViL, VilBERT and VisualBERT for VCR tab. 2. ClipBERT for STAR tab. 5. CLIP as a zero-shot baseline §4.4.
• Superseded by: not known from content in this wiki.