EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

In plain words

Most video AI tests use short clips, or ask questions that can be answered from a few seconds of footage. EgoSchema is a test built to find out whether a model can follow a long video. Each question is about a three-minute clip of first-person video, and the model picks one of five answers.

The authors also ask how much of the video you must watch to be sure of the right answer. For EgoSchema that is about 100 seconds, much longer than for other video datasets. The questions are written by a language model from human-written descriptions of the clip, then checked by people.

Today's video-language models score only a little above guessing (20% for five options), while people score about three in four. The benchmark is meant to show where long-video understanding still falls short.

TL;DR

• A zero-shot multiple-choice video QA benchmark: about 5,000 questions on 3-minute Ego4D clips, five options each §3.1 §4.2.
• The authors' measure of difficulty is temporal certificate length, with a median of about 100 seconds §1 §3.2 §4.1.
• Published video-language models score 19.6-32.1% (author-reported); humans reach about 76% §4.2.
• Build it yourself only if you have Ego4D access and can run the human curation; the answer key is mostly withheld.

Problem

Long-form video tasks often turn out to be short-term tasks in disguise: the answer depends on a few seconds, even if the clip is long §1. The authors measure this with the temporal certificate, the smallest set of sub-clips a human needs to watch to verify an annotation §3.2. Earlier datasets either use narrow label sets, such as like-ratio prediction in LVU §2, or have certificates of a few seconds, such as MAD §2.

Free-form answers are hard to score with BLEU or ROUGE, so the benchmark uses multiple-choice accuracy instead §1.

The construction assumes that a capable LLM can write hard questions and wrong answers from human narrations, and that people can then verify them §3.1.

Method

The benchmark has two parts: a data pipeline §3.1 and an evaluation protocol §4.2.

Data pipeline

The pipeline has four stages §3.1.

1. Clip selection §3.1.1. Take Ego4D RGB video and its timestamped narrations. Cut non-overlapping three-minute clips and keep those with at least 30 narrations, which is 10 per minute app. Clip length.
2. Question generation §3.1.2. Ask an LLM for N = 3 questions per clip. Then, in a second call, ask for one correct answer and M = 4 wrong answers per question. The authors call this Q(AW)-shot; it has two chained calls and is about 30% cheaper than the three-call QAW-shot variant §3.1.2.
3. Automatic filtering §3.1.3. Drop items with prompt keywords leaking into the output or with malformed format; regenerate those. Then drop any question an LLM can answer blindly, without the narrations §3.1.3.
4. Human curation §3.1.4. Two rounds. In round one, a question must be well formed, its correct answer must be right, each wrong answer must be wrong, and its certificate must be at least 30 seconds. Round one removes about 4-5x of the questions; round two keeps more than 97% of those that pass round one §3.1.4.

# EgoSchema construction pipeline  [§3.1]
# Input:  Ego4D RGB videos with timestamped narrations
# Output: curated 5-option questions, one per clip item

def build_benchmark(videos):
    # Stage I: cut 3-minute clips with at least 30 narrations  [§3.1.1]
    clips = [
        clip
        for clip in cut_three_minute_clips(videos)
        if len(clip.narrations) >= 30
    ]
    # Stage II: questions first, then answers, both from an LLM  [§3.1.2]
    # 3 questions per clip; each gets 1 correct and 4 wrong answers
    items = []
    for clip in clips:
        questions = llm_questions(clip.narrations, n=3)
        answers = llm_answers(questions, clip.narrations, m=4)
        items.extend(pair_up(questions, answers))
    # Stage III: rule filter, then drop items answerable without the video
    # [§3.1.3]
    items = [item for item in items if passes_rules(item)]
    items = [item for item in items if not answerable_blind(item)]
    # Stage IV: two human curation rounds  [§3.1.4]
    items = [item for item in items if passes_round_one(item)]
    items = [item for item in items if passes_round_two(item)]
    return items

Temporal certificate, the quantity the authors use to show difficulty §3.2:

L(C) = \sum_{i} \ell(s_i) §3.2

Here C is the certificate set of sub-clips s_i, and \ell is the duration of a sub-clip. The paper defines certificate length in words, as the summed duration of the sub-clips in the minimal necessary and sufficient set §3.2.

Certificate conventions §3.2: a certificate is at least 0.1 s; two non-contiguous certificates are merged when their closest ends are less than 5 s apart; annotators apply dataset-wide "meta-rules", such as label exclusivity, when they judge the certificate.

Evaluation protocol

Models are evaluated zero-shot, with frames sampled uniformly from each clip. Each model runs in two settings: the frame count it was trained on, and the most frames that fit on one 80 GB A100 §4.2. The authors evaluate four model families §4.2 app. Benchmarking details:

• FrozenBiLM (How2QA model), 10 and 90 frames §4.2.
• VIOLET, 5 and 75 frames; the authors report the best of several checkpoints. §4.2 says four fine-tuned models; the appendix lists one pre-trained and three fine-tuned §4.2 app. Benchmarking details.
• mPLUG-Owl, 1 to 30 frames. Each option is put into a yes/no question, and the option with the highest "Yes" probability wins §4.2.
• InternVideo (MSRVTT-fine-tuned), 10 to 90 frames. Each question-option pair is scored and the highest score wins §4.2.

For the generative mPLUG-Owl setup, the scoring logic is:

# Zero-shot scoring by "Yes" probability  [§4.2]
# Used for mPLUG-Owl; FrozenBiLM and InternVideo score differently
def predict(frames, question, options):
    scores = []
    for option in options:
        # ask the model whether this option answers the question  [§4.2]
        prompt = yes_no_prompt(question, option)
        # probability of the token "Yes"; 0 if it is not generated  [§4.2]
        scores.append(yes_probability(frames, prompt))
    # highest score wins; ties are broken uniformly at random  [§4.2]
    best_score = max(scores)
    tied = [i for i, s in enumerate(scores) if s == best_score]
    return random_choice(tied)

run_mplug.py

Hyperparameters

Results

Author-reported. Setup: zero-shot on EgoSchema; model sample size not stated §4.2; human accuracy is estimated on 5 hours of video, or 9 hours for the no-constraint setting §4.2.

Model accuracy on EgoSchema (multiple choice, chance = 20.0%) §4.2:

Human accuracy §4.2:

Difficulty: median temporal certificate about 100 seconds; the paper says it is 5.7x longer than the second-longest dataset and 10x to 100x longer than the rest §1 §4.1. Certificate estimates use 100 randomly chosen clips §4.1.

Dataset size: 5,063 instances, each with one video, one question and five options app. Datasheet.

Paper vs code

Official code: github.com/egoschema/EgoSchema, commit 505c787376b5e066d0ae406d0e0d41245cebba15 (last commit 2024-12-30). The paper links it in its datasheet app. Datasheet. It contains the download scripts, the released questions, 500 public answers and the baseline runners README.md. The baseline scripts are for mPLUG-Owl run_mplug.py, InternVideo run_internvideo.py, FrozenBiLM run_frozenbilm.py and VIOLET README.md. Scoring submissions go to a validation server validate.py.

Discrepancies:

• Dataset size. The paper says 5,063 instances app. Datasheet. The README says 5,031 questions for the full evaluation README.md.
• Generation code. The datasheet says the generation, filtering and curation code is in the supplementary material app. Datasheet. The repository has no generation, filtering or curation code.
• VIOLET checkpoint. The paper says the best of four fine-tuned checkpoints was reported (lsmdc-mc, msrvtt-qa, msrvtt-retrieval, plus the pre-trained model) app. Benchmarking details. The repository's default is ckpt_violet_msrvtt-cap.pt, a checkpoint the paper does not name args_egoSchema.json. The README asks the user to set path_ckpt themselves README.md.
• mPLUG-Owl decoding. The paper scores the "Yes" token probability and does not mention sampling. The code samples with do_sample=True and top_k=5 run_mplug.py, so repeated runs can differ.
• InternVideo backbone. The paper gives 478M parameters and names no backbone §4.2. The config says vit_base_patch16_224 run_internvideo.py, while the clip checkpoint path is ViT-L-14.pt run_internvideo.py. Which backbone runs is not clear from the config alone.
• FrozenBiLM size. The paper gives 1.2B parameters §4.2. The code loads microsoft/deberta-v2-xlarge run_frozenbilm.py. The parameter count is not checked against this model.

No discrepancies were checked for the human benchmark, because no code exists for it.

Pitfalls

• Licences. The paper is CC BY 4.0. The benchmark data and code are under the Ego4D licence, which the paper says allows research and commercial use §2 app. Datasheet. The repository has no LICENSE file, so the code licence is not stated. Do not redistribute videos or answer keys from this card, and do not copy code without a licence.
• Answer key is mostly private. Only 500 correct answers are public README.md. The rest is scored through a submission server validate.py, so local scores on the full set cannot be reproduced.
• Inconsistent multipliers. The abstract and §4.1 say 5.7x longer than the next dataset and 10x to 100x longer than the others §4.1. The introduction and contributions say 5x and 25x to 100x §1. Use the §4.1 figures.
• Certificates need human judgement. The certificate length depends on annotators and the dataset-wide meta-rules they are told about §3.2. The paper gives no inter-annotator agreement figures.
• Blind filter drops good questions. It removes any question that a model can guess without the narrations, even when the guess is only luck. The authors accept that cost to favour precision §3.1.3.
• Generation is hard to reproduce. The LLMs are proprietary and their versions and dates are not stated §3.1.2. About 85 prompts were tuned by hand §3.1.2. Expect different questions if you rerun the generation.
• No variance reported. Model scores come from single runs. The paper gives no confidence intervals or repeated runs §4.2, so gaps of one or two points between settings are not established.
• Non-monotonic frame effects. mPLUG-Owl drops from 31.1% at 5 frames to 20.0% at 30 frames §4.2. Do not assume more frames helps for every model.
• Human sample. Ten annotators at one company were used [app. Benchmarking details]; their agreement with each other is not reported.
• Random ties. When two options tie, the baselines pick one at random run_mplug.py. A tie-breaking rule is part of the protocol; report it if you compare numbers.

Lineage

• Builds on: Ego4D, which supplies the video and narrations §3.1.1. Ego4D has no card in this wiki.
• Compared against: LVU and MAD for long-form video §2; NExT-QA, AGQA, How2QA and MSRVTT-QA among video QA sets §4.1. None has a card in this wiki.
• Evaluated models: FrozenBiLM, VIOLET, mPLUG-Owl and InternVideo §4.2. None has a card in this wiki.
• Superseded by: not known from this wiki.