Keyframe selection

Overview

A video-language model has a fixed input budget, so a long video must be cut down to a few frames before the model sees it. Which frames are kept decides which evidence the model can use. Uniform sampling is the usual default, and the paper reports that it often misses frames relevant to the query 2508.04369 §1.

The shared problem is to score frames by how useful they are for the question and keep a small subset. The shared constraint is that the answering model sees a fixed number of frames; the paper's main table uses 64 2508.04369 §4.1. The paper frames the choice as a decision that can itself be trained, as part of a joint policy over frame selection and answering, while the video model stays frozen 2508.04369 §3.1.

Two obstacles make learning the choice hard. Frame-level labels are usually missing from video question-answering data. And choosing a subset of frame indices is discrete, so supervised fine-tuning does not pass gradients through the choice 2508.04369 §1.

Variants

Uniform sampling

Evenly spaced frames up to the model's budget. No scoring and no training. The paper uses it as the reference row for each base model at 64 frames: 58.9 on LongVideoBench and 70.3 on MLVU for LLaVA-Video-7B, in the authors' own reproduction 2508.04369 tab. 1.

Pre-trained selectors without training

The paper groups earlier keyframe methods as training-free, using pre-trained selectors whose cross-modal understanding is limited and which cannot be optimised for the answer 2508.04369 §1 2508.04369 §2. Its comparison table lists selectors built on other models, for example a 13B MLLM for CoS, a 0.5B BLIP for AKS, and DINOv2-1B for LongVU 2508.04369 tab. 1.

Learned selector, supervised

A selector trained on labelled frame or sample data. The paper's comparison is a supervised fine-tuned selector trained on 30K samples. It scores 62.8 on LongVideoBench and 64.8 on Video-MME average, against 63.9 and 65.5 for the reward-trained selector with 10K samples 2508.04369 tab. 5. The supervised row is the authors' own run.

Learned selector, reward-based (TSPO)

TSPO trains a small temporal agent on frozen CLIP features. The agent scores each frame by the cosine between an event-aware feature and the question, plus the CLIP frame-text cosine 2508.04369 eq. 3. It draws frame subsets with Gumbel-noise top-k 2508.04369 eq. 4. It is updated with group-normalised advantages, without a learned critic 2508.04369 §3.3. The reward is answer correctness 2508.04369 eq. 10. On needle-style training clips, it also rewards the share of chosen frames that fall inside the target segment 2508.04369 eq. 11.

With LLaVA-Video-7B at 64 frames, the authors report 58.9 to 63.9 on LongVideoBench, 70.3 to 76.3 on MLVU, 64.4 to 65.5 on Video-MME average and 40.2 to 45.3 on LVBench 2508.04369 tab. 1. In the ablation, the answer reward alone on comprehensive data gives 62.8 and 65.5, and the localisation reward alone on needle data gives 63.4 and 64.6. The full mix gives 63.9 and 65.5 2508.04369 tab. 4. The paper leaves the columns of that table unlabelled; the two values are read as LongVideoBench and Video-MME average.

When to use which

Implementation sketch

The sketch follows the paper's equations. Frame scores and selection are described in §3.2, and the update in §3.3. Each step has a comment that cites the paper.

# Learned keyframe selector, one update for one question  [arxiv:2508.04369 §3.3]
# Inputs: frames, question, answer, is_needle, selector (trainable),
#         mllm (frozen), G (group size), k (frames kept)
# Output: scalar loss for the selector parameters only

def selector_update(frames, question, answer, is_needle, selector, mllm, G, k):
    # Frozen CLIP features and frame-text similarity, once per question
    # [arxiv:2508.04369 §3.2]
    feats, text_sim = clip_encode(frames, question)
    rewards, logps = [], []
    for _ in range(G):
        # Gumbel-noise top-k over frame scores, with its log-probability
        # [arxiv:2508.04369 eq. 3] [arxiv:2508.04369 eq. 4]
        idx, logp = selector.sample_topk(feats, text_sim, k)
        # Frozen MLLM answers from the chosen frames only  [arxiv:2508.04369 §3.1]
        reply = mllm.answer(frames[idx], question)
        r = float(reply == answer)                    # [arxiv:2508.04369 eq. 10]
        if is_needle:
            # Share of chosen frames inside the target segment
            # [arxiv:2508.04369 eq. 11]
            r += mean(in_target_segment(idx))
        rewards.append(r)
        logps.append(logp)
    # Group-normalised advantage, in place of a learned critic
    # [arxiv:2508.04369 eq. 6]
    adv = normalise_within_group(rewards)
    # Policy-gradient loss; the ratio to the old policy is 1 at the start
    # of the step [arxiv:2508.04369 eq. 9]
    loss = -sum(a * lp for a, lp in zip(adv, logps)) / G
    return loss

Notes on the sketch:

• The paper's selection probability is a product over the chosen frames 2508.04369 eq. 5. A released implementation averages the per-frame ratios instead, which changes the gradient scale. The gap is recorded on the TSPO card.
• At inference the paper uses deterministic top-k with no Gumbel noise 2508.04369 §4.1.
• The sketch uses the simplified objective with no KL term 2508.04369 eq. 9. The full objective includes a KL term to a reference model 2508.04369 eq. 7.

Contradictions

• Within the TSPO paper. The comparison text gives +4.9 on LongVideoBench and +11.2 on MLVU over Qwen2.5-VL. Table 1 gives +5.2 and +9.2 2508.04369 §4.1 2508.04369 tab. 1. The text also gives +6.0 on LongVideoBench for Chain-of-Shot, where Table 1 gives +5.0 2508.04369 §4.1 2508.04369 tab. 1. Use the table values.
• Across papers. None known. Only one paper card lists this concept so far.

Papers

• TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding: a learned keyframe selector for a frozen Video-MLLM, trained with answer-based and localisation rewards.