---
kind: concept
slug: video-summarization
title: Video summarization
summary: >-
  Shortening a long video into still frames, a short clip or text that a person can review fast. The main choices are generic or question-driven selection, and the output form. One card so far, a 30-person study on 40-minute robot videos, with author-reported results.
tags: [video-understanding, summarization, human-robot-interaction, intermittent-supervision]
papers: ["2411.10016"]
related: [text-summarization, video-generation, human-in-the-loop, foundation-model, multimodal-language-model]
status: draft
updated_at: 2026-10-25
---

## Overview

Video summarization picks a smaller part of a long video for review: a set of
still frames (a storyboard), a shorter clip, or a text description. The case
studied so far is a robot that works unsupervised for about 40 minutes, after
which an operator must catch up on what happened [arxiv:2411.10016 §1]. Two
choices shape any design. The first is whether selection is generic or driven
by a question from the reviewer. The second is which output form the reviewer
sees [arxiv:2411.10016 §3]. The paper uses pretrained video and vision models
without fine-tuning [arxiv:2411.10016 §3.4.3].

The shared core is to score each unit of the video (a frame or a segment) for
relevance, then pick units within a size limit and show them in time order
[arxiv:2411.10016 §3.1] [§3.2].

## Variants

### Generic summary

No question is needed. An importance model scores every frame. The paper uses
STVT, trained on human-made TVSum summaries [arxiv:2411.10016 §3.1]. For still
frames, a greedy pass keeps important frames that are not too similar to frames
already kept. For a clip, change-point segmentation splits the video, each
segment gets the sum of its frame scores, and a 0/1 knapsack keeps segments
within 15 % of the video length [arxiv:2411.10016 §3.1]. Card:
[What did the Robot do in my Absence?](/papers/2411.10016).

### Query-driven summary

The reviewer's question is embedded in the same space as short video segments
(8 s, read at 1 fps). Relevance is the dot product between the question and
each segment [arxiv:2411.10016 §3.2] [tab. 1]. The top six segments form the
clip, joined in time order. The storyboard uses the same greedy rule with four
images [arxiv:2411.10016 §3.2] [tab. 1]. The paper does not state whether the
embeddings are normalised, so the scores can shift with embedding norms
[arxiv:2411.10016 tab. 1]. Card:
[What did the Robot do in my Absence?](/papers/2411.10016).

### Output form: storyboard, clip, text

The three forms come from the same selected units. The text form is generated by
a video-language model, from the summary clip rather than the full video
[arxiv:2411.10016 §3]. That model reads only 100 frames of the summary clip,
so the text covers much less than the clip it describes [arxiv:2411.10016 §3.4.4].

## When to use which

The evidence comes from one study with 30 participants and four videos
[arxiv:2411.10016 §3.4.1]. Read every row as a finding from that setting.

| Situation | Use | Why |
|---|---|---|
| The reviewer has no specific question yet | Generic summary | Needs no question. The paper found no significant accuracy gain for generic summaries over raw video (coef. -0.30, p = 0.130) [arxiv:2411.10016 tab. 2]. |
| Reviewer will ask specific questions and accuracy matters most | Query-driven summary | Accuracy rose over raw video (coef. 0.549, p = 0.005), but task time rose too (coef. 156.3, p = 0.002; units not stated) [arxiv:2411.10016 tab. 2]. |
| Review time is the tighter limit | Generic summary, or storyboard | Query-driven summaries added time. Storyboards cut time against video (coef. -124.7, p = 0.028) [arxiv:2411.10016 tab. 2]. |
| Questions are mostly about objects (existence, counts) | Storyboard | Storyboards were the most accurate form on object questions. Against video, the gain was 0.91 (p = 0.004) [arxiv:2411.10016 tab. 2]. |
| Questions are mostly about events (action order, motion, state change) | Clip | Storyboards were significantly weaker on event questions (interaction coef. -2.45, p < 0.001) [arxiv:2411.10016 tab. 2]. |
| Summary must be ready before the reviewer arrives | Pre-generated generic summary | Query summaries ran live and their generation time counted toward task time. Generic summaries were generated in advance [arxiv:2411.10016 §3.4.3] [§4.2]. |

## Implementation sketch

The core is the same for every variant. Only the score and the selection rule
change. Pseudocode in our own words, following the paper's steps.

```pseudocode
# Common core: score units, keep the best within a budget, show in time order
# Input:  scores (U,), costs (U,) in seconds, budget in seconds
# Output: selected unit indices, in time order

def select_units(scores, costs, budget):
    # Rank units by score, best first  [arxiv:2411.10016 §3.1] [§3.2]
    order = argsort_descending(scores)
    chosen, used = [], 0.0
    for u in order:
        # Skip units that no longer fit the budget  [arxiv:2411.10016 §3.2]
        if used + costs[u] > budget:
            continue
        chosen.append(u)
        used += costs[u]
    # Show the chosen units in the order they occur  [arxiv:2411.10016 §3.2]
    return sorted(chosen)
```

```pseudocode
# Generic clip: score segments by summed frame importance  [arxiv:2411.10016 §3.1]
# Input:  frame_importance (N,), segment_bounds [(start, end)], fps
# Output: (start, end) pairs, in time order

def generic_clip(frame_importance, segment_bounds, fps, keep_fraction):
    # A segment's score is the sum of its frame scores  [arxiv:2411.10016 §3.1]
    scores = [sum(frame_importance[a:b]) for a, b in segment_bounds]
    # Cost is the segment duration in seconds  [arxiv:2411.10016 §3.1]
    costs = [(b - a) / fps for a, b in segment_bounds]
    # Budget is a share of the video length  [arxiv:2411.10016 §3.4.4]
    budget = keep_fraction * len(frame_importance) / fps
    return select_units(scores, costs, budget)
```

Where the variants differ: a query-driven clip replaces `scores` with the
question's dot product and uses a top-k budget instead of a time budget
[arxiv:2411.10016 §3.2]. A storyboard also skips any frame too similar to one
already kept. The paper does not state the similarity measure or the visit
order [arxiv:2411.10016 §3.1].

## Contradictions

Only one paper so far, so none across papers. Within `arxiv:2411.10016`:

- §4.1 says H1a is partly supported, with only the generic-summary score
  above control. Table 2 shows that generic coefficient is not significant
  (p = 0.130). Read the supported effect as query-driven only
  [arxiv:2411.10016 §4.1] [tab. 2].
- The query frame rate is given as "original or lower" in §3.2 and as 1 fps in
  Table 1 and §3.4.3. The table value is used here [arxiv:2411.10016 §3.2]
  [tab. 1].

## Papers

- [What did the Robot do in my Absence?](/papers/2411.10016): compares generic and question-driven summaries of 40-minute robot videos, in storyboard, clip and text form, with a 30-person study.