TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding

In plain words

Video models that read language and pictures can only take in a limited amount of input at once. A long video therefore has to be shown to them as a few chosen frames, and most systems pick those frames evenly spaced or with a scoring rule that is never trained.

This paper trains the frame picker itself. A small add-on scores each frame for how well it matches the question and then keeps the best ones. The picker learns from the answers: if the video model answers correctly from the frames it was given, the choice was good. For training clips built to hide one relevant segment in unrelated footage, the picker also gets credit for choosing frames that fall inside that segment. The video model stays frozen, so only the small picker is trained.

TL;DR

• Adds a trainable temporal agent (about 3.5M learnable parameters on top of a frozen CLIP-Large) that selects keyframes for a frozen LLaVA-Video-7B or Qwen2.5-VL.
• Trains the agent with a GRPO-style objective: 8 sampled frame selections per question, group-normalised advantages, and rule-based rewards for answer accuracy and temporal localisation.
• Author-reported: LLaVA-Video-7B at 64 frames goes from 58.9 to 63.9 on LongVideoBench and from 70.3 to 76.3 on MLVU tab. 1.
• Code is public and mostly matches the paper, but it departs in several places that change the gradient and the training frame budget. See Paper vs code.

Problem

Video-MLLMs have a limited context window and expensive training, so a long video must be reduced to sparse frames before it reaches the language model §1. Uniform sampling often misses frames relevant to the query §1, fig. 1. Training-free keyframe methods use pre-trained selectors whose cross-modal understanding is limited, and they cannot be optimised §1, §2.

Two obstacles block a trainable sampler: frame-level labels are usually unavailable in video QA data, and choosing a subset of frame indices is discrete, so it does not pass gradients through supervised fine-tuning §1.

Method

Numbering note: the LaTeX source sets secnumdepth to 0, so the compiled paper prints no section numbers. The § numbers on this card are counted in source order: 1 Introduction, 2 Related Work, 3 Method (3.1 to 3.4), 4 Experiments (4.1 Settings, 4.2 Ablation), 5 Conclusion.

Policy factorisation §3.1. A standard Video-MLLM models the answer likelihood given the query and the sparse frames eq. 1. TSPO makes the frame choice part of the same decision process. It starts from candidate frames at a fixed rate, selects a subset with the temporal policy, and factorises the joint policy as the answer policy times the sampling policy eq. 2. The MLLM is kept frozen.

Event-aware temporal agent §3.2. The agent takes frozen CLIP frame features and a text feature.

1. Frame features get sinusoidal position encodings and pass through local window attention, with a window of length w centred on each frame. A residual MLP follows §3.2, fig. 2.
2. The event-text cosine similarity is computed from the refined features. The frame-text CLIP cosine is added to it. The fused score per frame is the sum of the two eq. 3.
3. A Gumbel-noise top-k over the score divided by a temperature gives the selected indices and their probabilities eq. 4. The sampling probability of a selection is the product of the per-frame probabilities eq. 5.
4. The temperature starts high for exploration and is annealed towards a low value §3.2.

Optimisation §3.3. The paper first recalls GRPO, which estimates the baseline from a group of rewards instead of a critic eq. 6. TSPO applies the same group idea to the sampler: for each question, it draws a group of frame selections, answers with the frozen MLLM, and scores each result. The full objective eq. 7 contains a KL term to a reference model. Because the MLLM is frozen, its likelihood ratio is 1 eq. 8. The simplified objective keeps only the sampling ratio times the advantage, with no KL term eq. 9.

Data and rewards §3.4. The paper uses two training data sources.

• Comprehensive temporal data: multiple-choice QA items longer than one minute from LLaVA-Video-178K. Items answerable from 4 uniform frames are dropped as too easy, and items unsolvable with 64 frames of a 1–3 minute video are dropped as too hard §3.4, fig. 4.
• Video needle-in-a-haystack data: target clips from LLaVA-Video-178K, with Qwen2.5-VL-generated event descriptions turned into multiple-choice questions. The target clips are concatenated and shuffled with distractor clips at segment level, giving 10 to 60 minute videos §3.4, fig. 4.

Together these form TSPO-10K, 10,000 samples §3.4.

Rewards §3.4:

• Accuracy: R_A = 1 if the predicted option equals the ground truth, else 0 eq. 10.
• Temporal localisation (needle data only): R_T = the number of selected frames inside the target clip divided by the number of selected frames eq. 11.
• Comprehensive data gets R_A + 1; needle data gets R_A + R_T.

# Event-aware temporal agent: frame scores  [§3.2] [eq. 3]
# Inputs: feats (T, D) CLIP frame features; text_feat (D,) CLIP text feature
#         clip_sim (T,) CLIP frame-text cosine; w window length; tau temperature
# Output: scores (T,) divided by tau

def selector_scores(feats, text_feat, clip_sim, w, tau):
    # Positions are the frame index divided by the number of frames
    #    [§3.2] [code: model/temporal_agent.py#L15-L18]
    positions = arange(T) / T                       # (T,)
    x = feats + sinusoidal(positions, D)            # (T, D)
    # Local attention over a window of w frames around each frame;
    # global attention switched off  [§3.2]
    #    [code: model/temporal_agent.py#L97-L104]
    attn = local_window_attention(x, window=w)      # (T, D)
    # Residual MLP; the residual is the input without position encoding
    #    [code: model/temporal_agent.py#L131]
    event = mlp(attn) + feats                       # (T, D)
    # Event-text cosine plus frame-text CLIP cosine, then divide by tau
    #    [eq. 3] [code: model/temporal_agent.py#L136-L141]
    scores = cosine(event, text_feat) + clip_sim    # (T,)
    return scores / tau

# TSPO update for one video-question pair  [§3.3] [eq. 9]
# Inputs: video, question, answer, is_needle, G rollouts, T_s frames
# Output: scalar loss to minimise

def tspo_loss(video, question, answer, is_needle, G, T_s, step, max_steps):
    # Candidate frames from the video; the loader caps them at 128
    #    [§3.2] [code: src/open_tspo/trainer/tspo_trainer.py#L456-L457]
    candidates = sample_candidates(video, max_frames=128)
    # CLIP features and CLIP similarity, computed once per pair
    #    [code: src/open_tspo/trainer/tspo_trainer.py#L497-L498]
    feats, text_feat, clip_sim = clip_features(candidates, question)
    # Temperature falls linearly from 0.025 to 0.01  [§4.1]
    #    [code: src/open_tspo/trainer/tspo_trainer.py#L496]
    tau = 0.025 - 0.015 * step / max_steps
    rewards, log_probs = [], []
    for g in range(G):
        # Gumbel-noise top-k selection of T_s frames  [§3.2] [eq. 4]
        #    [code: model/utils.py#L69-L80]
        idx, logp = gumbel_topk(selector_scores(...), T_s)
        # Frozen MLLM answers from the selected frames, greedy decoding
        #    [code: src/open_tspo/trainer/tspo_trainer.py#L527-L533]
        reply = mllm_answer(candidates[idx], question)
        r_acc = 1.0 if parse_option(reply) == answer else 0.0   # [eq. 10]
        if is_needle:
            # Share of selected frames inside the target clip  [eq. 11]
            r_loc = count_in_target(idx) / T_s
            rewards.append(r_acc + r_loc)
        else:
            # Constant offset; cancels in the group normalisation below
            rewards.append(r_acc + 1.0)
        log_probs.append(logp)
    # Group-normalised advantages  [§3.3] [code: tspo_trainer.py#L587-L592]
    advantages = normalise_within_group(rewards)
    loss = 0.0
    for g in range(G):
        # Ratio to the old policy, equal to 1 in the forward pass; the
        # paper's product over frames is replaced by a mean in the code
        #    [eq. 9] [code: src/open_tspo/trainer/tspo_trainer.py#L597-L603]
        ratio = mean(exp(log_probs[g] - stop_grad(log_probs[g])))
        loss = loss - ratio * advantages[g] / G
    return loss

Simplified objective as written eq. 9:

\mathcal{J}^*_{tspo}(\theta)=\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{\pi_{ts}(V_s\mid q,V_c)}{\pi_{ts_{old}}(V_s\mid q,V_c)}A_i\right] eq. 9

Hyperparameters

Values from the paper unless the Where column names only code. "not stated" means neither the paper nor the code gives a value.

Results

Author-reported. Setup: frozen MLLM with 64 sampled frames; starred rows are the authors' reproductions under 64 frames; the first three benchmarks use lmms-eval, LVBench uses its own protocol tab. 1 caption.

Table 1: main results tab. 1

The paper labels the selector "TSPO-0.4B"; 0.4B refers to the frozen CLIP-Large, not to the 3.5M learnable agent §4.1.

Table 2: transfer of the LLaVA-Video-trained selector to other Video-MLLMs, without extra training, 64 frames tab. 2

Table 3: comparison with other keyframe training methods, same settings as reported by those papers tab. 3, §4.2

Table 4: ablation of training data and rewards, read as LongVideoBench / Video-MME average tab. 4, §4.2. The column is unlabelled in the paper; the values are read this way because the first row matches the base model in Table 1.

Table 5: TSPO against supervised fine-tuning of the same selector tab. 5, §4.2

Table 6: inference efficiency with 128 candidate frames tab. 6, §4.2. The time units are not stated. These rows use a different setup from Table 1: the TSPO 128 to 64 row scores 60.6 / 65.3, not 63.9 / 65.5.

Headline claims in the text: average gains of 4.3 points on LLaVA-Video and 6.1 on Qwen2.5-VL across the four benchmarks §1. These match Table 1: the LLaVA-Video figures are +5.0, +6.0, +1.1 and +5.1, and the Qwen2.5-VL figures are +5.2, +9.2, +1.8 and +8.1. The paper gives no other headline numbers beyond the tables above.

Paper vs code

Repository searched: the link in the abstract and the paper's links point to github.com/Hui-design/TSPO, which we cloned at commit 19f4ecb2331bd1914fe06e96d35d0408bc866512. It is the official code. The repository root has no LICENSE file; src/open_tspo/ files carry an Apache-2.0 header and the bundled lmms-eval/ directory has its own licence file. No verbatim code is quoted here.

Discrepancies, each with both citations:

• Policy loss. The paper's sampling probability is a product over the selected frames eq. 5. The code averages the per-frame ratios instead tspo_trainer.py. The gradient therefore scales as 1/T_s, not as a sum.
• Log-probability source. The paper defines the selection probabilities through the Gumbel-perturbed top-k eq. 4. The code takes the log-probabilities from a noise-free softmax over the scores utils.py, then indexes them at the selected frames tspo_trainer.py.
• KL term. The paper's objective includes a KL term weighted by beta eq. 7. The code sets the KL to zero tspo_trainer.py and comments out the beta term tspo_trainer.py. The script still passes a beta of 0.04 train_deepspeed.sh. In effect the code implements the no-KL objective of eq. 9.
• Training frame count. The paper sets T_s = 16 during training §4.1. For comprehensive data the code uses half of that, 8 tspo_trainer.py.
• Candidate frame rate. The paper uses 1 FPS candidates §4.1. The training loader requests at most 128 frames at 1 FPS tspo_trainer.py. For videos longer than 128 s, the loader spaces the 128 frames uniformly across the whole video utils.py.
• Window default and shape. The paper's window is 12 §4.1, and the script passes 12 train_deepspeed.sh. The trainer and script defaults are 8 tspo_trainer.py tspo.py. The mask for window 12 covers offsets -6 to +5 and clamps at the ends, so it is not centred temporal_agent.py.
• Positional encoding. The paper says sinusoidal embeddings are added §3.2. The code uses the frame index divided by the number of frames temporal_agent.py, so the encodings depend on the candidate count.
• Global attention. The code computes a global attention branch and then sets its weight to zero, so only the windowed branch contributes temporal_agent.py. The output projection ffn_o is created but never used in the forward pass temporal_agent.py temporal_agent.py.
• Parameter count. The 3.5M figure §4.1 matches only if the unused ffn_o layer is counted. This is our calculation at dimension 768, the default in the code.
• Rollout decoding. The MLLM answers greedily during rollouts tspo_trainer.py. The paper does not state the decoding mode. Within a group, the only source of variation is the sampled frames.
• Text query. The code uses only the question stem, with the options removed tspo_trainer.py. The paper does not say what the text query contains.
• Accuracy parser. The rule-based accuracy reward first tries symbolic matching, then falls back to the first standalone letter a to e in the lower-cased reply tspo.py tspo.py.
• Comprehensive reward offset. The code adds 1 to the comprehensive reward for all rollouts tspo_trainer.py. Group normalisation removes a constant offset, so this has no effect on the advantages.
• Temperature schedule. The code anneals the temperature linearly with the training step tspo_trainer.py, matching the paper's endpoints §4.1.
• Inference. Inference uses deterministic top-k on the scores with no Gumbel noise temporal_agent.py, as the paper states §4.1. The evaluation scripts read precomputed frame indices from evaluation/jsons_idx/ TSPO_llava_video.sh. The README says these come from get_frame_idx.sh [README.md#L134]; we did not read that script.
• Training data. TSPO-10K is on Hugging Face [README.md#L82], not in the repository. The distractor clips are drawn from the same training file tspo.py. We found no code for the Qwen2.5-VL description step or the concatenation step (searched for "needle" in the repository).
• Paper text vs its own table. The comparison paragraph says +4.9 on LongVideoBench and +11.2 on MLVU over Qwen2.5-VL §4.1. Table 1 gives +5.2 and +9.2 tab. 1. The paragraph on Chain-of-Shot says +6.0 on LongVideoBench §4.1. Table 1 gives 63.9 against 58.9, which is +5.0.

Pitfalls

• Gradient scale. Because the code averages over the selected frames, the policy gradient is about T_s times smaller than the paper's product form. Reproducing the paper's loss requires a choice the paper does not spell out.
• No KL and no clipping. The loss uses only the current policy with a stop-gradient ratio. It has no reference model and no clipping. Do not assume the beta in the script does anything.
• Zero advantages. Greedy answers and a group of 8 can give identical rewards, which gives zero advantage and no gradient for that pair. The paper does not report how often this happens.
• Frame budget at training time. The 128-frame cap changes what R_T measures on long needle videos, because the candidate grid is no longer 1 FPS. Check the frame count before comparing with the paper.
• Lenient answer parsing. A reply that begins with the article "A" can be read as option A, because the parser takes the first standalone letter a to e tspo.py. Use the same parser for evaluation and training, or the numbers are not comparable.
• Cost. Each training pair costs one CLIP pass over up to 128 frames plus 8 MLLM generations of up to 256 tokens tspo_trainer.py tspo_trainer.py. Inference adds only the small selector.
• Transfer. The selector is trained against LLaVA-Video. Table 2 reports transfer to other models without retraining tab. 2. The paper does not say how well transfer holds beyond the four benchmarks.
• Unstated values. Optimiser, warm-up, effective batch size and the number of rollout groups in the paper are not stated. Do not fill them from other papers.
• Mixed numbers. The main table, Table 6 and the text use different frame settings and different numbers for the same model. Compare like with like.

Lineage

• Builds on: GRPO eq. 6, §3.3 and DeepSeek-R1, which is cited for GRPO §2, §3.3; Gumbel-softmax sampling §3.2; CLIP for frame and text features §3.2; LLaVA-Video, which provides the frozen MLLM §4.1. No cards are linked here, so these are named only.
• Compared against: LongVU, Chain-of-Shot, AKS, FrameVoyager, MLLM-VFS, TPO, and SFT* on the authors' own training data tab. 1 tab. 3 tab. 5.
• Superseded by: not stated.