Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

In plain words

Most models that connect video with text were trained on small or low-quality clips. This work collects a much larger set of sharp, high-resolution videos on many topics, and pairs each video with the words that are spoken in it, taken from its subtitles.

Sharp video is expensive to process. The model handles this by looking at one sharp frame and several blurry frames from around it. The sharp frame keeps the detail, and the blurry frames show how things move. The model learns to combine the two views, then learns to match each clip with its text.

After this pre-training, the same model can be reused for searching videos, answering questions about them, and guiding edits to face images from a text prompt. The authors report that it beats earlier methods on these tasks.

TL;DR

The paper builds HD-VILA-100M, a 720p video-transcript dataset, and pre-trains a model on it in two stages: contrastive clip-sentence matching, then masked language modeling §3, §5.1. Its hybrid encoder takes one randomly chosen high-resolution frame and low-resolution frames from around it per segment §4.1. The authors report new results on 10 video-language understanding tasks and 2 text-to-visual tasks §1. Pre-training is compute-heavy, and the dataset's own datasheet says it is for research rather than commercial use app. E.

Problem

Video-language models either read frames at low resolution or use heavy spatiotemporal encoders that cannot be trained end to end within memory limits §1. Human-annotated video-text sets such as MSR-VTT and DiDeMo are small and expensive to label §1. Transcript-based sets such as HowTo100M are large but narrow, mostly instructional, and mostly at 240p §1, tab. 1. The paper's premise is that jointly raising resolution, diversity and scale, and training the video encoder with the language model, improves transfer §1.

Method

Dataset: HD-VILA-100M §3

1. Build a candidate list of about 14 million videos from popular YouTube channels, YouTube-8M and YT-Temporal-180M §3.1.
2. Keep only videos that have subtitles, are 720p and have English transcripts. Cap each category at 30K hours, which leaves 3.3 million videos in 15 balanced categories §3.1.
3. Split the ASR subtitles into sentences with an off-the-shelf punctuation tool. The paper reports 75.7% accuracy for it on its own test set §3.2.
4. Align each sentence to its clip with Dynamic Time Warping over the subtitle timestamps §3.2.
5. Result: 103M clip-sentence pairs. A clip lasts 13.4 s on average and a sentence has 32.5 words on average §3.3, tab. 1. The intro states 100M pairs and 371.5K hours §1.

Most sentences are 15 to 50 words long app. C.1. The pairs are weakly aligned: the authors call most of them weakly related app. C.3.

Hybrid video encoder §4.1

For each segment s_i of a clip, sample one high-resolution frame X_t at a random time t, plus 2N low-resolution frames at t + k r for k in {-N, ..., -1, 1, ..., N}, where r is the LR sampling rate §4.1. The HR frame is 3 × H × W and each LR frame is 3 × H/4 × W/4 §4.1.

• F_hr: a 4-stage ResNet. Its stage-3 output is the HR input to the hybrid Transformer, resized by an interpolation φ §4.1, eq. 1.
• F_lr: a 3-stage ResNet for the LR frames §4.1.
• T_hy: the hybrid Transformer, which uses divided space-time attention §4.1, eq. 1.
• D_hr: a convolution that changes channels, then a 2×2 max-pool. It turns the HR frame into the spatial feature V_hr §4.1, eq. 2.
• A linear layer fuses V_hr and V_hy into the segment feature V [§4.1, eq. 3].

\mathcal{V}_{hy} = T_{hy}\big(F_{lr}(X_{t-Nr}), \dots, \phi(F^3_{hr}(X_t)), \dots\big) eq. 1

\mathcal{V}_{hr} = D_{hr}\big(F_{hr}(X_t)\big) eq. 2

\mathcal{V} = \mathrm{Linear}\big([\mathcal{V}_{hr}, \mathcal{V}_{hy}]\big) eq. 3

Language encoder and joint Transformer §4.2

• The text and joint model is a 24-layer, 1024-dimensional Transformer that starts from BERT-large weights §4.2.
• Layers 1 to 12 process words alone. Layers 13 to 24 process the concatenation of word tokens and segment video tokens §4.2.
• Text tokens get learnable 1D position embeddings and vision tokens get learnable 2D embeddings §4.2.
• The word-only output is reused as the text embedding in downstream tasks, and video-text similarity is computed without joint attention, which the authors say makes retrieval linear in cost §4.2.

Pre-training objectives §4.3

Stage one uses contrastive matching. Within a batch of size B, a video and its own sentence are the positive pair and all other pairings are negatives. The video embedding is the mean of its segment embeddings §4.3. With normalized embeddings v_i and t_i and a temperature τ:

\mathcal{L}_{v2t} = -\frac{1}{B}\sum_{i=1}^{B} \log\frac{\exp(v_i^\top t_i/\tau)}{\sum_{j=1}^{B}\exp(v_i^\top t_j/\tau)}, \qquad \mathcal{L}_{t2v} = -\frac{1}{B}\sum_{i=1}^{B} \log\frac{\exp(t_i^\top v_i/\tau)}{\sum_{j=1}^{B}\exp(t_i^\top v_j/\tau)} eq. 4

Stage two uses masked language modeling. Word tokens are masked with BERT's strategy, and the masked token is predicted from the other words and the video tokens through an MLP head §4.3. Logits from the segments are aggregated into one video-level prediction §4.3:

\mathcal{L}_{\mathrm{MLM}} = -\mathbb{E}_{(\mathcal{W},\mathcal{V})} \log p\big(w_i \mid \mathcal{W}_{\setminus i}, \mathcal{V}\big) eq. 5

# Hybrid video encoder for one segment  [§4.1] [eq. 1] [eq. 2] [eq. 3]
# Input:  clip frames; t = index of the sampled HR frame; N, r
# Output: segment feature V

def encode_segment(frames, t, N, r):
    # Sample one HR frame at t and 2N LR frames around it  [§4.1]
    hr_frame = frames[t]                              # (3, H, W)
    lr_before = [frames[t - k * r] for k in range(N, 0, -1)]
    lr_after = [frames[t + k * r] for k in range(1, N + 1)]
    # Encode each LR frame with the LR ResNet F_lr  [§4.1]
    lr_tokens = [F_lr(f) for f in lr_before + lr_after]
    # Take the stage-3 features of the HR frame from F_hr  [§4.1] [eq. 1]
    hr_stage3 = F_hr.stage3(hr_frame)
    # Resize to the LR feature size; interpolation mode not stated  [eq. 1]
    hr_tokens = phi(hr_stage3)
    # Hybrid Transformer over LR, HR, LR in temporal order  [§4.1] [eq. 1]
    v_hy = T_hy(lr_tokens[:N] + [hr_tokens] + lr_tokens[N:])
    # Spatial path: full F_hr, then adapter D_hr  [§4.1] [eq. 2]
    v_hr = D_hr(F_hr(hr_frame))
    # Fuse both paths with one linear layer  [§4.1] [eq. 3]
    return Linear(concat(v_hr, v_hy))

# Stage one: contrastive video-sentence matching  [§4.3] [eq. 4] [§5.1]
# Input:  videos (B, d) and sentences (B, d), both L2-normalised
# Output: the two loss terms, L_v2t and L_t2v

def contrastive_losses(videos, sentences, tau):
    # Score every video against every sentence in the batch  [eq. 4]
    scores = matmul(videos, transpose(sentences)) / tau   # (B, B)
    # Matched pairs sit on the diagonal and are the positives  [§4.3]
    targets = arange(len(videos))
    # Video-to-text term  [eq. 4]
    loss_v2t = mean(cross_entropy(scores, targets))
    # Text-to-video term: same scores, rows and columns swapped  [eq. 4]
    loss_t2v = mean(cross_entropy(transpose(scores), targets))
    # Combination of the two terms is not stated  [§4.3]
    return loss_v2t, loss_t2v

# Stage two: masked language modeling  [§4.3] [§4.2] [§5.1]
# Input:  word tokens (L,) and the video tokens of each segment
# Output: scalar loss

def mlm_loss(word_tokens, segment_tokens):
    # Mask word tokens with BERT's strategy; masking rates not restated  [§4.3]
    masked, labels = bert_style_mask(word_tokens)
    # Layers 1-12 read the masked words alone  [§4.2]
    text_states = language_layers_1_to_12(masked)
    # Layers 13-24 read words and video tokens together  [§4.2]
    states = multimodal_layers_13_to_24(
        concat(text_states, segment_tokens))
    # MLP head gives vocabulary logits for each segment  [§4.3]
    segment_logits = [mlm_head(s) for s in segment_states(states)]
    # Merge segment logits into one video score; rule not stated  [§4.3]
    video_logits = aggregate(segment_logits)
    return cross_entropy(video_logits[masked], labels)

Downstream heads

• Video QA §5.2, app. D.1. On TGIF Action and Transition, each of five candidate answers is concatenated with the question. A two-layer MLP on the [CLS] token scores each candidate, and training uses cross-entropy. On MSRVTT-QA and TGIF FrameQA, answers are one-hot over a candidate set, and a two-layer MLP classifies over all candidates. On MSRVTT multiple choice, the retrieval model picks the caption with the highest similarity, with no QA-specific training §5.2, app. D.1.
• Text-to-video retrieval §5.3, app. D.2. The stage-one model is used §5.3.
• Text-to-visual generation §5.4, app. D.3. The pre-trained video and language encoders map a clip and a sentence to 1024-dimensional embeddings v and t. Two MLP mappers, f_v and f_t, turn them into StyleGAN W+ codes of size 18 × 512. A pre-trained StyleGAN G is kept fixed app. D.3.1.

\mathbf{w}^+_{\mathbf{v}} = f_v(\mathbf{v}), \qquad \mathbf{w}^+_{\mathbf{t}} = f_t(\mathbf{t}), \qquad \mathbf{I}_{rec} = G(\mathbf{w}^+_{\mathbf{v}}), \qquad \mathbf{I}_{edit} = G(\mathbf{w}^+_{\mathbf{v}} + \mathbf{w}^+_{\mathbf{t}}) app. D.3.1

Losses for the two outputs, against the middle HR frame I and the sentence T app. D.3.2:

\ell_2(\mathbf{I},\hat{\mathbf{I}}) = \|\mathbf{I}-\hat{\mathbf{I}}\|_2, \qquad \ell_{lpips}(\mathbf{I},\hat{\mathbf{I}}) = \|\mathcal{F}(\mathbf{I})-\mathcal{F}(\hat{\mathbf{I}})\|_2 app. D.3.2

\ell_{id}(\mathbf{I},\hat{\mathbf{I}}) = 1 - \langle \mathcal{R}(\mathbf{I}), \mathcal{R}(\hat{\mathbf{I}}) \rangle, \qquad \ell_{clip}(\mathbf{T},\hat{\mathbf{I}}) = 1 - \langle \mathcal{C}(\mathbf{T}), \mathcal{C}(\hat{\mathbf{I}}) \rangle / \gamma app. D.3.2

\ell = \lambda_1 \ell_2(\mathbf{I},\mathbf{I}_{rec}) + \lambda_2 \ell_{lpips}(\mathbf{I},\mathbf{I}_{rec}) + \lambda_3 \ell_{id}(\mathbf{I},\mathbf{I}_{rec}) + \lambda_4 \ell_2(\mathbf{I},\mathbf{I}_{edit}) + \lambda_5 \ell_{lpips}(\mathbf{I},\mathbf{I}_{edit}) + \lambda_6 \ell_{id}(\mathbf{I},\mathbf{I}_{edit}) + \lambda_7 \ell_{clip}(\mathbf{T},\mathbf{I}_{edit}) app. D.3.2

# Generation loss for one update, frozen StyleGAN  [§5.4] [app. D.3.1-D.3.2]
# Input:  middle HR frame I (1024 x 1024), sentence T, weights (7 values)
# Output: scalar loss

def generation_loss(frame, clip, sentence, weights):
    # Pre-trained encoders give a visual and a text embedding  [app. D.3.1]
    visual = visual_embedding(clip)                   # (1024,)
    text = text_embedding(sentence)                   # (1024,)
    # Map both into StyleGAN's W+ space  [app. D.3.1]
    w_visual = f_v(visual)                            # (18, 512)
    w_text = f_t(text)                                # (18, 512)
    # Reconstruction uses the visual code only  [app. D.3.1]
    recon = G(w_visual)
    # Edit adds the text code to the visual code  [app. D.3.1]
    edit = G(w_visual + w_text)
    # L2, LPIPS-style and identity terms for both outputs  [app. D.3.2]
    recon_terms = [l2(frame, recon), lpips(frame, recon),
                   identity(frame, recon)]
    edit_terms = [l2(frame, edit), lpips(frame, edit),
                  identity(frame, edit)]
    loss = dot(weights[0:3], recon_terms) + dot(weights[3:6], edit_terms)
    # Text-image term, with gamma = 100  [app. D.3.2]
    return loss + weights[6] * clip_term(sentence, edit, gamma=100)

Hyperparameters

Pre-training §5.1 unless noted

Video QA app. D.1

Text-to-video retrieval app. D.2

• Temperature 0.08 app. D.2. Warm-up and multi-step decay are used, with their values not stated app. D.2.
• Stop words are removed from MSR-VTT text, following the cited MIL-NCE work app. D.2.

Text-to-visual generation app. D.3.2

Results

Author-reported. Pre-trained on HD-VILA-100M, then fine-tuned per task. Baselines are the authors' choices, and the source tables gray out MMT and Support Set as not fully comparable §5.3. Each row's source table is listed.

Video question answering tab. 2. Accuracy. Strongest listed baseline in brackets.

Text-to-video retrieval, fine-tuned. R@1 / R@5 / R@10 / MedR.

Full baseline rows are in those tables. Grayed rows (MMT, Support Set) have higher R@1 than HD-VILA on ActivityNet, at 28.7 and 29.2 tab. 6.

Text-to-video retrieval, MSR-VTT zero-shot tab. 3, lower block. R@1 / R@5 / R@10 / MedR.

The authors report a 40.4% relative R@1 gain over VideoCLIP in this setting §5.3. The LSMDC gain of 55.4% over CE is also reported §5.3, tab. 5.

Text-to-visual generation. No quantitative metrics are reported in the text. The comparisons with StyleCLIP, TediGAN, pSp and SR3 are qualitative, shown in fig. 4 and fig. 5, and in the supplementary figures app. D.3.4. The super-resolution input is 16 × 16 and the output is 1024 × 1024 fig. 5.

Ablations, author-reported. Frame counts and data subsets are fine-tuned on MSR-VTT unless stated.

The authors report 66.7% relative R@1 gain for the full data over the HowTo100M subset §5.5, which matches tab. 7. They report a 29.1% relative drop when the data is downsampled to 360p §5.5.

Paper vs code

The paper's footnote links microsoft/XPretrain, in the hd-vila directory, at commit 902d083e. The repository's LICENSE is the Microsoft Research License Agreement for XPretrain. It permits non-commercial research use only and does not allow distribution [code licence §1.a, §2.a]. Because wikillm carries ads, this card uses none of the code: nothing is cited from it and no paper-vs-code comparison was made. The code field is recorded for reference only, as in the FLARE and DONeRF cards.

Pitfalls

• Frame budget is inconsistent. Pre-training samples seven frames per segment §5.1. Retrieval fine-tuning uses 11 frames per segment app. D.2, and the best setting in tab. 9 is 1 HR plus 10 LR frames. Whether the seven pre-training frames include the HR frame, and what r is, are not stated. Do not assume that fine-tuning frame counts apply to pre-training.
• Temperature is missing for pre-training. τ appears in eq. 4 without a value §4.3. The only value given is 0.08, for retrieval fine-tuning app. D.2.
• Loss details are missing. The text does not say how L_v2t and L_t2v combine §4.3. The segment-logit aggregator for MLM is not named §4.3. The MLM masking rates are not restated §4.3.
• Encoder depth is ambiguous. §4.1 gives F_lr three stages, while §5.1 uses ResNet-50 for both branches. Also not stated: the interpolation method φ, the output channels of D_hr, the input and output sizes of the linear fusion layer, and the warm-up and decay schedule for pre-training.
• Stage two freezing is vague. §5.1 says "the encoders" are frozen during stage two. Which modules count as encoders is not stated, so the trainable set must be inferred.
• Heavy compute. Stage one used 128 V100 GPUs and stage two used 32, with a total batch of 1024 and contrastive negatives gathered across all GPUs §5.1. Negatives must be gathered across devices, or the contrastive loss will differ from the paper's.
• Data is noisy. Pairs come from ASR transcripts split by a punctuation tool with 75.7% reported accuracy, aligned by DTW §3.2. The authors call most pairs weakly related app. C.3.
• Dataset terms are restrictive. The datasheet says the dataset is made for research rather than commercial use, and is under the Open Use of Data Agreement (O-UDA). It is distributed as video URLs and clip times, and videos may disappear over time app. E. The O-UDA terms were not read for this card. Whether the release happened is not verified.
• Zero-shot is not like-for-like. MSR-VTT zero-shot uses 224 × 320 crops with 4× up-sampling of middle frames, four segments of 11 frames, and stop word removal, while other downstream runs use 720p HR frames app. D.2.
• Checkpoint choice. The retrieval results use the last saved checkpoint, not the best one on validation app. D.2. The checkpoint used for video QA is not stated.
• Grayed baselines. On ActivityNet, HD-VILA's R@1 (28.5) is below the grayed MMT (28.7) and Support Set (29.2) rows tab. 6. The paper's "better performance" claim on ActivityNet therefore rests on R@5, R@50 and MedR, or on the non-grayed baselines.
• In-domain claim is not shown by the table. App. D.4 concludes that in-domain pre-training helps a lot. Its own table shows the HowTo100M-pretrained model ahead at 145K steps (15.7 against 6.6 R@1). The HD-VILA-100M model reaches 9.1 only at 504K steps. The HowTo100M row cites the original paper and is not a re-run app. D.4.
• HowTo100M baseline is partly reconstructed. For the data comparison, 31% of HowTo100M videos are originally 240p, and the authors up-sampled them to 720p by bicubic interpolation §5.5.
• Generation has no numbers. Editing and super-resolution are judged only by figures fig. 4, fig. 5, app. D.3.4.
• Generation terms are underspecified.
  • The StyleGAN checkpoint is not named; the text cites three StyleGAN works app. D.3.1.
  • The identity network R and the image-text network C are not named app. D.3.2.
  • The printed ℓ_lpips is an L2 distance between features of F. LPIPS normally weights channels with learned weights, and the paper does not say whether it does app. D.3.2.
  • App. D.3.1 has a naming slip. The sentence describing the text-driven edit names w_v where w_t is meant, as the I_edit formula shows app. D.3.1.
• FDVD is small and synthetic in its text. The face-description set has 613 videos at 1024 × 1024. Its descriptions are generated by a rule-based grammar from predicted face attributes. The authors say it will be released [app. D.3.3].
• Video QA on MSRVTT multiple choice is zero-shot for QA. It is answered by the retrieval model with no QA training §5.2, app. D.1.

Lineage

• Builds on: ClipBERT, for sparse frame sampling and end-to-end video training §2, §4.1. BERT-large for initialization §4.2. Divided space-time attention from the cited Transformer video work §4.1. Align-before-fuse pre-training §5.1. For generation, StyleGAN and the StyleCLIP and TediGAN approach §5.4, app. D.3.
• Compared against: ClipBERT, VideoCLIP, HowTo100M, CE, MMT, Support Set, HT MIL-NCE, DECEMBERT, VLM, JSFusion, ActBERT and others in tab. 2 to 6 §5.2 to §5.3.
• Data sources: YouTube-8M and YT-Temporal-180M provide video candidates §3.1.