'What did the Robot do in my Absence?' Video Foundation Models to Enhance Intermittent Supervision

In plain words

A team of robots can work for a long time without anyone watching. When one of them needs help, the operator has to catch up on what happened, and the raw camera footage of a 40-minute run is slow and tiring to scrub through. This paper asks whether existing video AI models can squeeze that footage into something a person can review quickly.

The system offers two kinds of summary. A generic one picks out the important moments with no input from the operator. A query-driven one first takes a question from the operator, such as a question about a particular object, and then picks the moments that match it. Each summary can be shown as a set of still pictures, as a short video, or as a paragraph of text.

The authors tested this with 30 people who had to answer questions about robot footage. Asking a question first helped people answer correctly, but it made the task take longer. The still-picture form worked best overall, though it was weaker for questions about events.

TL;DR

• Generic and query-driven summaries of 40-minute robot videos, in three modalities: storyboard, short video and text §3.
• All models are pretrained and used without fine-tuning §3.4.3.
• Query-driven summaries raised accuracy over raw video (logistic coef. 0.549, p = 0.005) but added task time (OLS coef. 156.3 s, p = 0.002) tab. 2.
• Storyboards were the most accurate modality on object questions; on event questions they were significantly weaker tab. 2.
• Single 30-person study with four videos, author-reported. No official code.

Problem

Operators supervising robots intermittently must learn what the robot did while unsupervised. Reviewing raw video takes time, puts high cognitive load on the operator and needs expertise §1. Prior LLM work on robot-to-human summaries was limited to a single study with a custom-trained model §1 §2. The authors target long egocentric videos (40 min at 15 fps) of a search-and-rescue mission §3.4.1. Two challenges are named: generic summaries may miss what a task needs, and generated summaries may be wrong in ways the presentation hides §1.

Method

The framework has four steps: preprocessing, embedding, relevance generation and selection fig. 2 tab. 1. The generic and query-driven pipelines use different models at each step tab. 1. In both pipelines, the text summary is generated from the summary video, not from the original video, to respect the memory limits of VideoChat2 §3.

Generic summaries §3.1

1. Frames are taken at the original 15 fps and embedded with ResNet18 tab. 1.
2. STVT gives a per-frame importance score, trained on TVSum human summaries §3.1 tab. 1.
3. Storyboard: a greedy pass keeps frames that are important but not too similar to frames already kept §3.1. Similarity uses the top L principal components of the ResNet18 features §3.1 tab. 1.
4. Video: a cosine-kernel change point method (KTS) splits the frames into segments; each segment's score is the sum of its frame scores; a 0/1 knapsack picks segments within a budget of K = 15 % of the video length §3.1 tab. 1.
5. Text: VideoChat2 describes the generic summary video from a fixed prompt, given context about the robot's environment and task §3.1.

# Generic storyboard  [§3.1] [tab. 1]
# Input:  frame_features (N, 512), importance (N,); M = 24, delta = 0.5
# Output: selected frame indices, in time order

def generic_storyboard(frame_features, importance, budget, delta):
    # Reduce ResNet18 features to their top L = 100 components  [§3.1]
    reduced = pca(frame_features, n_components=100)     # (N, 100)
    # Visit frames in an order the paper does not state  [§3.1]
    selected = []
    for frame in frames_in_visit_order(importance):
        # Skip a frame that is too similar to one already kept  [§3.1]
        if is_similar_to_any(reduced[frame], reduced[selected], delta):
            continue
        selected.append(frame)
        if len(selected) == budget:
            break
    # Show the chosen frames in time order, without their scores  [§3.1]
    return sorted(selected)

# Generic summary video  [§3.1] [tab. 1] [§3.4.4]
# Input:  frame_features (N, 512), importance (N,), fps = 15
# Output: list of (start, end) frame ranges, in time order

def generic_video(frame_features, importance, fps, keep_fraction):
    # Find segment boundaries; their number is not fixed  [§3.1]
    segments = kts_cosine(frame_features)
    # Only segments of at least D = 1 s may be chosen  [§3.1] [tab. 1]
    candidates = [s for s in segments if duration(s, fps) >= 1.0]
    # Score each segment by the sum of its frame importances  [§3.1]
    scores = [sum(importance[a:b]) for a, b in candidates]
    # Pick segments under a budget of K = 15 % of the video  [§3.1] [tab. 1]
    budget = keep_fraction * len(importance) / fps
    chosen = knapsack_01(candidates, scores, budget)
    return sorted(chosen)

Query-driven summaries §3.2

1. The video is downsampled to 1 fps. For video and text it is cut into fixed 8 s segments, the length LanguageBind uses §3.2 tab. 1.
2. The text query and the frames or segments are embedded with LanguageBind (768-d), so both live in one space §3.2 tab. 1.
3. Relevance is the dot product between the query and each frame or segment §3.2 tab. 1.
4. Storyboard: the same greedy rule as generic, on LanguageBind embeddings, with m = 4 images §3.2 tab. 1.
5. Video: the top k = 6 segments are joined in time order §3.2 tab. 1.
6. Text: VideoChat2 gets the query-driven summary video and the user query §3.2.

# Query summary video  [§3.2] [tab. 1] [§3.4.4]
# Input:  segment_embeddings (S, 768), query_embedding (768,); k = 6
# Output: segment indices, in time order

def query_video(segment_embeddings, query_embedding, k):
    # Dot product of query and segment embeddings  [§3.2] [tab. 1]
    relevance = segment_embeddings @ query_embedding    # (S,)
    # Keep the k most relevant segments  [§3.2] [tab. 1]
    top = argsort_descending(relevance)[:k]
    # Join the chosen segments in time order  [§3.2]
    return sorted(top)

Information equalisation §3.4.4. The authors tuned the parameters so the two summary types carry similar amounts of information. The generic video is about 6 min (15 % of 40 min). The query video, assuming six queries for six questions, is 4 min 48 s with k = 6. The query storyboard has 4 images and the generic storyboard 24. VideoChat2 reads only 100 frames of the summary video, and its length penalty was set so the generic text is about six sentences and each query answer about one sentence §3.4.4.

User study §3.3 §3.4. Each of 30 participants was assigned to one of two groups, generic (G) or query-driven (Q). Everyone also did a control task with the raw video §3.3. Within each group, the three summary modalities (video, storyboard, text) plus control were four tasks in randomised order §3.3. Each task had six questions: three about objects (existence and counts) and three about events (action sequence, moving direction, state transitions) §3.4.2. Questions were multiple choice with a "Not sure" option §3.4.2. Participants could play the long video at 2x, 4x or 8x §3.3. The four videos were different, and the same videos and questions were used for all participants in G and Q §3.3. The interface is shown in fig. 3.

Data: two robots, the quadruped Spot and an all-terrain robot (ATR), front camera, 40 min each at 15 fps §3.4.1 fig. 4.

No equations or algorithm environments are numbered in the paper. The pseudocode above is ours.

Hyperparameters

Results

Author-reported. Setup: 30 participants, four 40-minute robot videos §3.4.1; the query-driven models ran live on an H100 GPU §3.4.3; control is raw video. Coefficients are from the regression models in tab. 2. Score is modelled with logistic regression, time with ordinary least squares §4.

The paper states that raw per-condition accuracy percentages are not given. Headline claims: query-driven summaries improve accuracy over raw data at the cost of time; storyboards improve accuracy and cut time against video and text; storyboards are weaker for event questions §4.4 §5.

Paper vs code

No official code.

The paper gives no code link, and the arXiv record carries no code link in the metadata we store. A GitHub repository search for the title's keywords, for the authors' surname, and for the method's key words ("intermittent supervision", "storyboard video summary robot", "video foundation models robot summaries") found no repository from the authors. A search for "katuwandeniya" returned one unrelated personal repository, which we did not use.

Pitfalls

• Contradictory H1a sentence. §4.1 says H1a is partly supported with "only score of G > Control". Table 2 and the same section show the G coefficient is not significant (p = 0.130); the supported effect is Q only. Read H1a as Q-only.
• Video-to-condition mapping not stated. Four videos were used for four tasks §3.3. Which video went with which modality is not stated, so video and modality effects may be confounded.
• Query frame rate differs between sections. §3.2 says the query pipeline reads the video at its original or a lower rate; Table I and §3.4.3 say 1 fps. Implement 1 fps, as the table states.
• Greedy selection is underspecified. The visit order, the similarity measure and the tie-breaking are not stated. The threshold delta = 0.5 is given once for ResNet18 PCA features and once for LanguageBind embeddings, with no statement that it transfers between them.
• Dot product without normalisation. Relevance is the raw dot product tab. 1. Whether embeddings are normalised is not stated, so scores can change with embedding norms.
• KTS details. The paper uses a cosine kernel and an unspecified number of change points §3.1. Whether the 1 s minimum applies to every segment or only to selected ones is ambiguous; §3.1 reads as selected ones.
• Live latency counts toward query tasks. Task time includes model loading and summary generation §4.2. Generic summaries were pre-generated §3.4.3, so the two conditions differ in compute before the participant starts.
• Questions per query not reported. The equalisation assumes six queries per participant §3.4.4. Actual query counts are not given, except that storyboard users made 30 % more queries than the other modalities §4.2.
• Text uses only 100 frames. VideoChat2 reads 100 frames of the summary video §3.4.4, so the text summary covers much less than the video it describes.
• Statistical procedure. Models were reduced stepwise, dropping insignificant interactions until only significant ones remained §4. p-values after this selection are not adjusted, and no multiple-comparison correction is stated.
• Time units. The time coefficients are given without units tab. 2. The intercepts (about 470 to 550) suggest seconds, but this is not stated.
• Null results read as findings. No significant familiarity effect is reported as suggesting that novices can use the tool equally §4.3.3 §5. This absence of evidence is weak with 30 participants.
• Sample. One study, 30 participants, four videos from one mission type §3.4.1. The post-study comparison is 11/15 in G against unanimous in Q §4.3.5, so the groups were small.
• Model versions. Model checkpoints, prompts and decoding settings are not stated beyond the length penalty §3.4.4.
• Preference is subjective. The authors say preference is subjective, with storyboards and video equal in Q §4.4. Use the accuracy and time results for design decisions, not the preference scores.

Lineage

• Builds on: VideoChat2 / MVBench for the text model §3.2; LanguageBind (arXiv:2310.01852) for query embeddings tab. 1; STVT (hsu2023video in the paper, no arXiv id in its bibliography) for generic importance §3.1.
• Compared against: Dechant et al., the single prior LLM work on robot action summaries, which trained one model to summarise and answer §2. The authors say they use no dataset-specific fine-tuning, unlike that work §2.
• Superseded by: not stated.