--- kind: concept slug: keyframe-selection title: Keyframe selection summary: >- Choosing a small set of frames from a long video so that a frozen video-language model can answer a question within its input budget. Options range from uniform sampling to a learned frame picker. TSPO trains one with answer-based rewards and reports gains of 1.1 to 6.0 points across four benchmarks for LLaVA-Video-7B at 64 frames. tags: [video-understanding, frame-sampling] papers: ["2508.04369"] related: [multimodal-language-model, policy-gradient, reinforcement-learning-verifiable-rewards] status: draft updated_at: 2026-11-06 --- ## Overview A video-language model has a fixed input budget, so a long video must be cut down to a few frames before the model sees it. Which frames are kept decides which evidence the model can use. Uniform sampling is the usual default, and the paper reports that it often misses frames relevant to the query [arxiv:2508.04369 §1]. The shared problem is to score frames by how useful they are for the question and keep a small subset. The shared constraint is that the answering model sees a fixed number of frames; the paper's main table uses 64 [arxiv:2508.04369 §4.1]. The paper frames the choice as a decision that can itself be trained, as part of a joint policy over frame selection and answering, while the video model stays frozen [arxiv:2508.04369 §3.1]. Two obstacles make learning the choice hard. Frame-level labels are usually missing from video question-answering data. And choosing a subset of frame indices is discrete, so supervised fine-tuning does not pass gradients through the choice [arxiv:2508.04369 §1]. ## Variants ### Uniform sampling Evenly spaced frames up to the model's budget. No scoring and no training. The paper uses it as the reference row for each base model at 64 frames: 58.9 on LongVideoBench and 70.3 on MLVU for LLaVA-Video-7B, in the authors' own reproduction [arxiv:2508.04369 tab. 1]. ### Pre-trained selectors without training The paper groups earlier keyframe methods as training-free, using pre-trained selectors whose cross-modal understanding is limited and which cannot be optimised for the answer [arxiv:2508.04369 §1] [arxiv:2508.04369 §2]. Its comparison table lists selectors built on other models, for example a 13B MLLM for CoS, a 0.5B BLIP for AKS, and DINOv2-1B for LongVU [arxiv:2508.04369 tab. 1]. ### Learned selector, supervised A selector trained on labelled frame or sample data. The paper's comparison is a supervised fine-tuned selector trained on 30K samples. It scores 62.8 on LongVideoBench and 64.8 on Video-MME average, against 63.9 and 65.5 for the reward-trained selector with 10K samples [arxiv:2508.04369 tab. 5]. The supervised row is the authors' own run. ### Learned selector, reward-based (TSPO) [TSPO](/papers/2508.04369) trains a small temporal agent on frozen CLIP features. The agent scores each frame by the cosine between an event-aware feature and the question, plus the CLIP frame-text cosine [arxiv:2508.04369 eq. 3]. It draws frame subsets with Gumbel-noise top-k [arxiv:2508.04369 eq. 4]. It is updated with group-normalised advantages, without a learned critic [arxiv:2508.04369 §3.3]. The reward is answer correctness [arxiv:2508.04369 eq. 10]. On needle-style training clips, it also rewards the share of chosen frames that fall inside the target segment [arxiv:2508.04369 eq. 11]. With LLaVA-Video-7B at 64 frames, the authors report 58.9 to 63.9 on LongVideoBench, 70.3 to 76.3 on MLVU, 64.4 to 65.5 on Video-MME average and 40.2 to 45.3 on LVBench [arxiv:2508.04369 tab. 1]. In the ablation, the answer reward alone on comprehensive data gives 62.8 and 65.5, and the localisation reward alone on needle data gives 63.4 and 64.6. The full mix gives 63.9 and 65.5 [arxiv:2508.04369 tab. 4]. The paper leaves the columns of that table unlabelled; the two values are read as LongVideoBench and Video-MME average. ## When to use which | Situation | Use | Why | |---|---|---| | No training budget, or no question-answer labels, and only the frozen model is available | Uniform sampling | Needs no training. It is the baseline the trained selector improves on at 64 frames [arxiv:2508.04369 tab. 1] | | Question-answer pairs are available, and the frozen model can run rollouts during training | Reward-trained selector, as in TSPO | Rewards come from answer correctness. Trained on 10K samples, it outperforms the supervised selector trained on 30K samples [arxiv:2508.04369 tab. 5] | | The answer sits in a short segment of a long video, and training clips with known target segments can be built | Add a localisation reward | On needle-style data, the localisation reward alone raises LongVideoBench to 63.4 against 58.9 for the base model [arxiv:2508.04369 tab. 4]. Building such clips is the authors' own pipeline [arxiv:2508.04369 §3.4] | | Inference latency matters, and the selector is added on top of an existing model | A small trained selector | Frame-selection time is 1.2 against 28.4 for CoS, units not stated, and the LLM time is the same, 2.7 [arxiv:2508.04369 tab. 6] | | A selector trained for one model should serve others without retraining | Reward-trained selector, trained on LLaVA-Video | Transfers without retraining to Qwen2-VL 7B, Qwen2.5-VL 7B and LLaVA-Video 72B on LongVideoBench and MLVU [arxiv:2508.04369 tab. 2]. The paper does not test beyond these four benchmarks | ## Implementation sketch The sketch follows the paper's equations. Frame scores and selection are described in §3.2, and the update in §3.3. Each step has a comment that cites the paper. ```pseudocode # Learned keyframe selector, one update for one question [arxiv:2508.04369 §3.3] # Inputs: frames, question, answer, is_needle, selector (trainable), # mllm (frozen), G (group size), k (frames kept) # Output: scalar loss for the selector parameters only def selector_update(frames, question, answer, is_needle, selector, mllm, G, k): # Frozen CLIP features and frame-text similarity, once per question # [arxiv:2508.04369 §3.2] feats, text_sim = clip_encode(frames, question) rewards, logps = [], [] for _ in range(G): # Gumbel-noise top-k over frame scores, with its log-probability # [arxiv:2508.04369 eq. 3] [arxiv:2508.04369 eq. 4] idx, logp = selector.sample_topk(feats, text_sim, k) # Frozen MLLM answers from the chosen frames only [arxiv:2508.04369 §3.1] reply = mllm.answer(frames[idx], question) r = float(reply == answer) # [arxiv:2508.04369 eq. 10] if is_needle: # Share of chosen frames inside the target segment # [arxiv:2508.04369 eq. 11] r += mean(in_target_segment(idx)) rewards.append(r) logps.append(logp) # Group-normalised advantage, in place of a learned critic # [arxiv:2508.04369 eq. 6] adv = normalise_within_group(rewards) # Policy-gradient loss; the ratio to the old policy is 1 at the start # of the step [arxiv:2508.04369 eq. 9] loss = -sum(a * lp for a, lp in zip(adv, logps)) / G return loss ``` Notes on the sketch: - The paper's selection probability is a product over the chosen frames [arxiv:2508.04369 eq. 5]. A released implementation averages the per-frame ratios instead, which changes the gradient scale. The gap is recorded on the [TSPO card](/papers/2508.04369). - At inference the paper uses deterministic top-k with no Gumbel noise [arxiv:2508.04369 §4.1]. - The sketch uses the simplified objective with no KL term [arxiv:2508.04369 eq. 9]. The full objective includes a KL term to a reference model [arxiv:2508.04369 eq. 7]. ## Contradictions - **Within the TSPO paper.** The comparison text gives +4.9 on LongVideoBench and +11.2 on MLVU over Qwen2.5-VL. Table 1 gives +5.2 and +9.2 [arxiv:2508.04369 §4.1] [arxiv:2508.04369 tab. 1]. The text also gives +6.0 on LongVideoBench for Chain-of-Shot, where Table 1 gives +5.0 [arxiv:2508.04369 §4.1] [arxiv:2508.04369 tab. 1]. Use the table values. - **Across papers.** None known. Only one paper card lists this concept so far. ## Papers - [TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding](/papers/2508.04369): a learned keyframe selector for a frozen Video-MLLM, trained with answer-based and localisation rewards.