Video summarization

Overview

Video summarization picks a smaller part of a long video for review: a set of still frames (a storyboard), a shorter clip, or a text description. The case studied so far is a robot that works unsupervised for about 40 minutes, after which an operator must catch up on what happened 2411.10016 §1. Two choices shape any design. The first is whether selection is generic or driven by a question from the reviewer. The second is which output form the reviewer sees 2411.10016 §3. The paper uses pretrained video and vision models without fine-tuning 2411.10016 §3.4.3.

The shared core is to score each unit of the video (a frame or a segment) for relevance, then pick units within a size limit and show them in time order 2411.10016 §3.1 §3.2.

Variants

Generic summary

No question is needed. An importance model scores every frame. The paper uses STVT, trained on human-made TVSum summaries 2411.10016 §3.1. For still frames, a greedy pass keeps important frames that are not too similar to frames already kept. For a clip, change-point segmentation splits the video, each segment gets the sum of its frame scores, and a 0/1 knapsack keeps segments within 15 % of the video length 2411.10016 §3.1. Card: What did the Robot do in my Absence?.

Query-driven summary

The reviewer's question is embedded in the same space as short video segments (8 s, read at 1 fps). Relevance is the dot product between the question and each segment 2411.10016 §3.2 tab. 1. The top six segments form the clip, joined in time order. The storyboard uses the same greedy rule with four images 2411.10016 §3.2 tab. 1. The paper does not state whether the embeddings are normalised, so the scores can shift with embedding norms 2411.10016 tab. 1. Card: What did the Robot do in my Absence?.

Output form: storyboard, clip, text

The three forms come from the same selected units. The text form is generated by a video-language model, from the summary clip rather than the full video 2411.10016 §3. That model reads only 100 frames of the summary clip, so the text covers much less than the clip it describes 2411.10016 §3.4.4.

When to use which

The evidence comes from one study with 30 participants and four videos 2411.10016 §3.4.1. Read every row as a finding from that setting.

Implementation sketch

The core is the same for every variant. Only the score and the selection rule change. Pseudocode in our own words, following the paper's steps.

# Common core: score units, keep the best within a budget, show in time order
# Input:  scores (U,), costs (U,) in seconds, budget in seconds
# Output: selected unit indices, in time order

def select_units(scores, costs, budget):
    # Rank units by score, best first  [arxiv:2411.10016 §3.1] [§3.2]
    order = argsort_descending(scores)
    chosen, used = [], 0.0
    for u in order:
        # Skip units that no longer fit the budget  [arxiv:2411.10016 §3.2]
        if used + costs[u] > budget:
            continue
        chosen.append(u)
        used += costs[u]
    # Show the chosen units in the order they occur  [arxiv:2411.10016 §3.2]
    return sorted(chosen)

# Generic clip: score segments by summed frame importance  [arxiv:2411.10016 §3.1]
# Input:  frame_importance (N,), segment_bounds [(start, end)], fps
# Output: (start, end) pairs, in time order

def generic_clip(frame_importance, segment_bounds, fps, keep_fraction):
    # A segment's score is the sum of its frame scores  [arxiv:2411.10016 §3.1]
    scores = [sum(frame_importance[a:b]) for a, b in segment_bounds]
    # Cost is the segment duration in seconds  [arxiv:2411.10016 §3.1]
    costs = [(b - a) / fps for a, b in segment_bounds]
    # Budget is a share of the video length  [arxiv:2411.10016 §3.4.4]
    budget = keep_fraction * len(frame_importance) / fps
    return select_units(scores, costs, budget)

Where the variants differ: a query-driven clip replaces scores with the question's dot product and uses a top-k budget instead of a time budget 2411.10016 §3.2. A storyboard also skips any frame too similar to one already kept. The paper does not state the similarity measure or the visit order 2411.10016 §3.1.

Contradictions

Only one paper so far, so none across papers. Within arxiv:2411.10016:

• §4.1 says H1a is partly supported, with only the generic-summary score above control. Table 2 shows that generic coefficient is not significant (p = 0.130). Read the supported effect as query-driven only 2411.10016 §4.1 tab. 2.
• The query frame rate is given as "original or lower" in §3.2 and as 1 fps in Table 1 and §3.4.3. The table value is used here 2411.10016 §3.2 tab. 1.

Papers

• What did the Robot do in my Absence?: compares generic and question-driven summaries of 40-minute robot videos, in storyboard, clip and text form, with a 30-person study.