RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video

In plain words

Video assistants are often tested on a single question about a clip that is already finished. In real use, a camera keeps running, and the right answer to the same question changes as new things happen. RTV-Bench is a test built around that situation. The same kind of question is asked at several points in one video, and the correct answer may differ from one point to the next.

Each question comes in a small group. Two simpler questions come first, and a harder one follows. The score only gives credit for the hard question when the simpler ones in its group were answered correctly, so a model cannot do well by guessing on the hard part alone.

The authors tested ten models: open models that read a clip after it ends, open models built for live video, and two commercial models. The best result is around half of the questions answered correctly. The live-video models did better than the offline ones. Making a model larger, or feeding it more frames, did not reliably help.

TL;DR

RTV-Bench has 552 videos and 4,608 multiple-choice entries. Each entry asks a question at an end time, and the same question can appear at several end times with different correct answers §2.2 §2.3. Accuracy counts correct entries. Score counts a group's advanced (level-2) entries only when its basic entries pass §3.1. Author-reported overall accuracy is 50.02% for GPT-4o and 47.33% for IXC2.5-OL, the best open model tab. 2. Scaling and frame counts give small or inconsistent gains §3.2. The official code is an evaluation harness for Qwen2.5-VL and two online wrappers; it has no license README.md.

Problem

Most video benchmarks ask about a whole video once. They do not check whether a model can keep its understanding current while a stream unfolds, or whether it can recall earlier events when a later question depends on them §1 §2.1. The authors name two failure modes: memory loss and attention shift §2.1. They also say that existing streaming benchmarks often do not test whether the answer to a repeated question updates over time §1 §4.

Method

The paper is a benchmark plus an evaluation protocol. It has no model.

Benchmark design

• Multi-timestamp QA. The same underlying question is posed at several points in a video, and only the correct answer changes §2.2 §2.3.
• Hierarchy. Each group has three questions: two basic questions (levels 0 and 1) and one advanced question (level 2) that depends on them §2.3. The released data has 1,027 level-0, 1,029 level-1 and 2,552 level-2 entries, in 1,011 groups QA.json.
• Eight dimensions. Each entry carries one of eight labels: TP (timing and duration), SP (overall scene and layout), VP (fine visual detail), FP (what comes next), PU (what surface events mean), IA (why actors act), GU (the video's overall context) and SR (deductions from observations) §2.2 app. B. They group under perception, understanding and reasoning. The paper also crosses these with an object, action and event view app. A.
• Domains. Driving camera (1,773 entries), sports camera (1,654) and embodied (egocentric) perspective (1,181) QA.json §2.3 fig. 2.
• Annotation. Videos come from EgoSchema and public online videos. Annotators wrote questions from LLM-generated templates, then set the earliest timestamp at which each answer becomes valid §2.3.

Evaluation

Accuracy is the share of correct answers over the QA entries §3.1.

Score is the group-aware ratio below. It is an unnumbered display in §3.1. Here N is the number of groups that contain a level-2 entry, B_i is 1 when all level-0 and level-1 entries of group i are correct, and the counts are per group §3.1:

Score = sum_i (B_i * correct level-2 entries in group i)
        / sum_i (level-2 entries in group i)

The released scorer implements this rule compute_score.py:

# Group-aware Score  [§3.1] (unnumbered display)
# Input:  items with group id, level (0, 1 or 2) and correct (bool)
# Output: score in [0, 1]

def group_aware_score(items):
    # Group items by the id in questionID  [code: scripts/eval/compute_score.py#L75-L78]
    groups = group_items(items)
    points = 0
    total_q2 = 0
    for group in groups:
        # Groups with no level-2 item are skipped  [code: compute_score.py#L325-L327]
        if len(group.q2) == 0:
            continue
        # Basic check: all level-0 and level-1 items correct.  [code: compute_score.py#L321-L323]
        # With no such items the check passes.
        basic_passed = all(item.correct for item in group.q0 + group.q1)
        # Credit correct level-2 items only if basic_passed  [code: compute_score.py#L334]
        if basic_passed:
            points = points + sum(item.correct for item in group.q2)
        total_q2 = total_q2 + len(group.q2)
    return points / total_q2

Offline models are not given a stream. For each entry, the released harness gives the model a clip that runs from time 0 to the entry's end time, samples a fixed number of uniform frames, and asks for one option letter §3.1 infer_model_rtv.py:

# Offline answer for one entry  [§3.1]
# Input:  video, end_time, question, options
# Output: one option letter, or "Unknown"

def answer_offline(video, end_time, question, options):
    # Clip from time 0 to end_time; start_time is 0 in every entry
    # [code: scripts/data_preparation/prepare_rtv_clips.py#L84-L93]
    clip = cut(video, start=0.0, end=end_time)
    # Uniform frames, first and last included; n in {8, 16, 32, 64}
    # [§3.1] [code: scripts/utils/video_utils.py#L48-L53]
    frames = uniform_frames(clip, n)
    prompt = question + options + "answer with one option letter"
    # Greedy decoding, at most 64 new tokens
    # [code: scripts/infer/qwen25vl_wrapper.py#L90]
    text = model.generate(frames, prompt, max_new_tokens=64)
    # First standalone valid letter in the text, else "Unknown"
    # [code: scripts/utils/prompt_utils.py#L16-L36]
    return extract_letter(text, options)

Accuracy then counts an entry as correct when the predicted letter equals the answer. An exception during inference is recorded as "ERROR" and counted as wrong infer_model_rtv.py compute_acc.py.

Hyperparameters

Results

Author-reported. Setup: 552 videos, 4,608 entries, offline models given the prefix clip, online models queried at their timestamps §3.1. Columns are Acc (%) and Score, except FQA and MTQA, which have only Acc tab. 2.

tab. 2

Per-category tables give the same ranking in more detail. In the Perception table, GPT-4o has the highest Score on TP, VP and SP, and IXC2.5-OL has the highest TP accuracy tab. 3. In the Understanding table, GPT-4o is first on GU, PU and overall tab. 4. In the Reasoning table, GPT-4o is first on FP and SR tab. 5. The text gives the online-versus-offline gap on MTQA as 38.21% against "33–35%" §3.2.

Scale and frames, from the text of §3.2 and fig. 3:

• Qwen2.5-VL, 3B to 72B: gains of roughly 2 to 3 points, with the 72B model at 40.78% at 64 frames.
• More frames: no consistent gain across sizes. IXC2.5-OL drops as frames increase.

Paper vs code

The official repository is https://github.com/LJungang/RTV-Bench, at commit a928cbe82b11fd7be7c6f7c0a40c662427ec40d7. It is linked from the paper's project page and owned by the project leader's GitHub account [title block]. It has no LICENSE file, so all rights are reserved by default: cite it only, and quote no code README.md.

• Online models. §3.1 says online models keep a state and answer at query time. The shared loop gives every model the same prefix clip file and calls predict infer_model_rtv.py. The repository has wrappers only for Qwen2.5-VL, VideoChat-Online and DiSpider infer_model_rtv.py. I did not read the online wrappers in detail, so the streaming behaviour is not checked here.
• Missing models. IXC2.5-OL, VITA-1.5, GPT-4o and Gemini have no evaluation code in the repository infer. Their rows in tab. 2 cannot be reproduced from the repository.
• Offline clip. §3.1 says the offline model sees "the relevant video segment". The code clip runs from time 0 to the end time, because every entry has start_time 0 prepare_rtv_clips.py infer_model_rtv.py. The paper's wording suggests a shorter window than the code uses.
• Score rules. The paper says groups are counted when they contain a level-2 item §3.1. The code agrees compute_score.py. The paper does not say what happens when level 0 or level 1 is absent. The code treats that as a pass compute_score.py.
• Error handling. The paper does not say how failed or unparseable answers are scored. The code counts exceptions and "Unknown" letters as wrong infer_model_rtv.py prompt_utils.py prompt_utils.py.
• FQA and MTQA. The table caption defines these columns tab. 2. No script in the repository computes them.
• Default model. README gives a 7B download, but the example script runs Qwen2.5-VL-32B with 8 frames infer_offline_qwen2_5_vl.sh README.md.
• Dataset location. The README's download command names RTVBench/RTV-Bench on Hugging Face. Its header and project page link xunsh/RTV-Bench README.md.

No dataset license is stated in the paper or repository. The card does not redistribute videos or the QA file.

Pitfalls

• Frame choice. The paper reports, for each model, the best of the frame settings it ran §3.1. The frame count per row is not shown in tab. 2. Reported numbers are therefore best-of-sweep on the test set.
• Denominator of Score. Score divides by level-2 entries (2,552), not by groups (1,009 contain a level-2 entry) QA.json. Each group has about 2.5 level-2 entries because the same advanced question is asked at several end times. Each of those counts separately.
• Frame resolution. The Qwen wrapper sets a per-frame area of about 1,024 pixels, the same for 8 to 64 frames video_utils.py video_utils.py. The processor then resizes again with its own limits, which the repository does not set. The effective resolution was not checked and should be treated as unknown.
• Online-vs-offline MTQA claim. §3.2 says online models reach 38.21% on MTQA against "33–35%" for offline models. tab. 2 shows Qwen2.5-VL at 37.46% and VITA-1.5 at 36.32%. VideoChat-Online ties IXC2.5-OL at 38.21%.
• VideoLLaMA3 numbers differ. Understanding is 35.29 / 5.73 and Reasoning 35.78 / 6.80 in tab. 2. The detail tables give 35.35 / 5.33 tab. 4 and 35.95 / 5.40 tab. 5.
• Entry count. The abstract, introduction and conclusion, and the README give 4,608 entries, which matches the released QA file QA.json. §2.3 gives 4,631. The average duration is 18.2 min in §2.3 and 18.00 min in tab. 1.
• Appendix table. The caption of the table on preceding-question performance says "accuracy", but the cells are counts app. A. The four rows total 361, 580, 491 and 503 entries. The text cites 1,527 triples app. A. No code for this analysis is in the repository.
• Sizes. VITA-1.5 is "7B" in tab. 2 and "--" in tab. 3. GPT-4o and Gemini sizes are not stated.
• Video licences. Some source videos come from EgoSchema and others from the internet, and the paper states no licence for them §2.3. Any reuse needs its own check.
• Overall accuracy. The repository reports accuracy over every entry, including level-0 and level-1 compute_acc.py. The paper does not say which entries enter the Overall column of tab. 2.
• Figure-only numbers. The 72B value and the 2 to 3 point scaling gain come from fig. 3 and its text, not from a table.

Lineage

• Builds on: nothing stated as a method. Benchmarks it names as related: VStream, StreamingBench and OVO-Bench §1 §4.
• Compared against: VideoLLaMA2, VideoLLaMA3, Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video, VITA-1.5, IXC2.5-OL, VideoChat-Online, GPT-4o and Gemini 2.0 Flash §3.1 tab. 2.
• Superseded by: none known from this wiki.