Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives

In plain words

A video-language model takes a video and some text and does something with both. Examples are finding the clip that matches a sentence, writing a caption for a video, or answering a question about what happens in it. A video is harder than a single picture because the order and timing of events matter.

This survey organises the field around three questions: how the model is built, how it is trained, and where its training data comes from. It also places the reported scores of many methods side by side on three common benchmarks, which helps a reader see which design families have done well so far. The scores are listed against the papers that they cite, so the tables summarise other people's results and do not add new experiments.

TL;DR

• A survey of video-language understanding for text-video retrieval, video captioning and video question answering §1 §2.
• It sorts methods into architecture (pre-Transformer, Transformer-based, LLM-augmented), training (pre-training, fine-tuning) and data (curation, label annotation) §1 fig. 1.
• Its tables show the best reported text-video retrieval score as CLIP-ViP at 49.6 R@1 on MSRVTT and the best VideoQA score as LLaMA-Vid at 58.9 accuracy on MSRVTT tab. 1 tab. 3.
• Relevant if you want a map of the field. Not useful as an implementation spec, because it has no method of its own.

Problem

Earlier reviews of this area each cover one part: one architecture family, one training strategy, data augmentation, or one task §1. The authors argue that a real system needs all three parts together, and that a broad capability is needed rather than one task §1. The survey aims to cover tasks, architecture, training and data in one document §1.

Three challenges set the frame §3:

• Intra-modal and cross-modal interaction. Video needs both spatial and temporal modelling, and the gap between video and language is larger than between image and language §3.
• Cross-domain adaptation. Models must work on data unlike their training data, and fine-tuning large models is costly §3.
• Data preparation. Video-text data is larger, costlier to store and harder to annotate than image-text data §3 §6.2.

Method

This is a survey, so there is no training method to reimplement. The "method" below is the taxonomy the paper uses, with the main design choices stated in the text.

Tasks §2 fig. 2 fig. 3 app. A fig. 5 fig. 6:

1. Text-video retrieval: find a video for a text query (text-to-video), or a text for a video (video-to-text). Video moment retrieval is the finer variant that returns a moment inside one video §2.
2. Video captioning: write a sentence, or sometimes a paragraph, for a whole video. A transcript can be an optional input §2.
3. Video question answering: multi-choice, or open-ended as classification, generation or regression (for example counting) §2.
4. Other tasks named in the taxonomy include dense captioning, video chapter generation and action recognition fig. 1 app. A.

Architecture §4 fig. 4:

1. Pre-transformer §4.1. Unimodal encoders: CNNs for frame appearance, RNNs or 3D CNNs for temporal features, graphs for relations between objects or segments. Language uses pre-trained word vectors (word2vec or GloVe) with LSTM or GRU. Cross-modal fusion uses element-wise multiplication, attention, or graph networks.
2. Transformer-based §4.2:
   • Shared Transformer: one encoder receives visual patches and language tokens together §4.2.
   • Stacked Transformer: unimodal encoders first, then a cross-modal Transformer. Captioning adds a causal decoder §4.2.
   • Dual Transformer: separate video and text encoders produce global vectors compared with a simple operation such as cosine similarity. This keeps pairwise cost low §4.2.
3. LLM-augmented §4.3. The survey treats the LLM as the output generator, which it says dominates over the alternative of an LLM used as a controller that calls tools §4.3. The pipeline has a visual encoder, a semantic translator and an LLM. The translator is a Q-Former with learnable queries §4.3, or a simple linear projection of visual features §4.3.

# LLM as output generator  [§4.3]
# Input:  video frames, instruction text
# Output: answer tokens

def answer_video_question(frames, instruction):
    # 1. Encode the frames with a visual encoder (ViT or CNN)  [§4.3]
    visual_features = visual_encoder(frames)
    # 2. Translate visual features into the LLM input space  [§4.3]
    #    Q-Former variant: learned queries attend to the features
    #    linear variant: a projection of the visual features
    video_tokens = semantic_translator(visual_features)
    # 3. Put video tokens in front of the instruction and decode  [§4.3]
    prompt = concatenate(video_tokens, embed(instruction))
    return llm_generate(prompt)

Training §5:

1. Pre-training §5.1:
   • Language: masked language modelling. Masking the whole language input helped captioning in two cited works §5.1.
   • Video: masked video modelling predicts masked frame patches or frames. It gives a smaller gain than masked language modelling §5.1.
   • Video-text: contrastive alignment of separate embeddings §5.1, or a joint encoder with a single token that predicts whether a pair matches §5.1.
2. Fine-tuning §5.2:
   • Full fine-tuning updates all parameters.
   • Low-rank adapters or learnable prompt vectors update fewer parameters when compute or data is limited §5.2.
   • LLM-augmented models use two-stage instruction tuning. The first stage trains only the semantic translator to align visual features with the LLM's language space. The second stage tunes the translator on small, manually collected video-text pairs §5.2.

Data §6:

1. Curation §6.1: manual collection of online videos for pre-training (for example HowTo100M, YT-Temporal-180M), reuse of existing datasets, or recording by annotators. Video-specific augmentation is TubeTokenMix and temporal shift §6.1.
2. Label annotation §6.2: manual labels are of high quality but costly. Transcripts from YouTube are cheap but often grammatically wrong and misaligned in time §6.2. Model-generated captions and summaries are another source §6.2.

Hyperparameters

Not applicable. The paper is a survey and states no training setup of its own §1. Each row in the Results tables comes from a cited paper, and the survey does not say whether the scores were re-run tab. 1 tab. 2 tab. 3.

Results

Compiled by the survey from the cited papers. The setup differs per row: backbone, pre-training data and evaluation protocol tab. 1 tab. 2 tab. 3. Values are as printed in the survey, not re-run here, and the survey gives no hardware or training budget.

Text-video retrieval, MSRVTT, recall (%) tab. 1. The caption does not say which direction the recall is for. Columns: video encoder and text encoder as listed in the table.

Source: tab. 1. The caption names the architecture abbreviations and MSRVTT as the main dataset tab. 1.

Video captioning, MSRVTT (%) tab. 2. The caption lists BLEU-4 and METEOR. The table also has a CIDEr column, shown here.

Source: tab. 2.

Video question answering, accuracy (%) tab. 3. Two datasets, MSRVTT and MSVD.

Source: tab. 3.

Datasets tab. 4 app. B. The table lists about 50 datasets with their video source, whether annotation is manual or automatic, the tasks they serve, and a size. Pre-training corpora are automatic: HowTo100M (1.2M), YT-Temporal-180M (6M), WebVid-10M (10.7M) tab. 4. Downstream QA sets are mostly manual: NExT-QA (5.4K), CausalVid-QA (26K) tab. 4. The column header reads "#Videos/#Routes", which the paper does not explain tab. 4.

Survey claims about these tables §4.4. The text says dual Transformers are the most effective for text-video retrieval and that LLM-augmented models have begun to outperform Transformer-based models on VideoQA §4.4. Both match the rows above, but see Pitfalls for the limits of that comparison.

Paper vs code

No official code.

What was searched: the abstract, the arXiv metadata and the paper text were checked for code links. The only repository link is in the Limitations section: https://github.com/nguyentthong/video-language-understanding §9. The survey says it contains "open-source implementations" §9. A WebFetch read of the repository describes an awesome-style curated list of leaderboards, datasets and papers, with no LICENSE shown. It was not cloned, because it holds no implementation of a method from the paper. No project page is linked. No GitHub search was run, because the paper has no method to search for.

Pitfalls

• Mixed sources. The scores in the tables come from papers with different backbones (ViT, ResNet, EVA-G, BERT, Vicuna), different pre-training data and different protocols. The survey does not normalise them tab. 1 tab. 2 tab. 3. Whether the numbers were re-run or copied is not stated.
• Retrieval direction. The caption of tab. 1 does not say whether the recall is text-to-video or video-to-text. The text defines both directions §2. Check the source paper before comparing numbers.
• Caption mismatch. tab. 2 has a CIDEr column, but its caption mentions only BLEU-4 and METEOR tab. 2.
• Model size is a confound. The LLM-augmented row LLaMA-Vid uses EVA-G with Vicuna tab. 3. The best Transformer-based VideoQA rows use BERT with ViT or Video Swin backbones tab. 3. The survey's claim that LLM-augmented models lead on VideoQA §4.4 is therefore not a like-for-like comparison.
• Cost figures are the survey's own. The annotation costs given, about $16,000 and three months for 10K QVHighlights videos, and 100 students over a year for 5K NExT-QA videos §6.2, are quoted from the authors and not checked.
• Masked video modelling. The survey says it gives a smaller gain than masked language modelling §5.1, but no numbers back this in the text.
• Version. The source used is arXiv v4 (last updated 2026-05-12). The published date above is from v1 (2024-06-09). The licence is taken from the fetch record.
• Numbering. Figure, table and section numbers come from the LaTeX source, not a rendered PDF. Commented-out paragraphs in the source (for example the LLM text-based approach with VideoChat-Text and Retrieving-to-Answer) are not in the compiled paper and are not cited here.
• The Methods section of the paper has no equations or algorithm blocks, so no eq. or alg. citations apply.

Lineage

• Builds on: none stated. The introduction cites earlier reviews that each cover one part of the field: Transformer-based and LLM-augmented architectures §1, self-supervised learning and pre-training §1, data augmentation §1, and single tasks such as VideoQA, retrieval and captioning §1.
• Compared against: those reviews, named in §1 but not linked here, since no cards exist for them.
• Superseded by: none stated.