Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

In plain words

Language models that chat well can read text but cannot watch a video. This paper gives one such model a view of the video. A standard image model looks at each frame, its output is condensed into a short list of vectors, and a single simple layer translates those vectors into the chatbot's own input format. The chatbot is then trained on a set of about 100,000 questions and answers about videos. Some were written by people and some were generated with other AI tools.

The authors also propose a way to grade video chat answers. Another AI model acts as the judge and scores answers on aspects such as correctness, level of detail and consistency.

TL;DR

• Video-ChatGPT = frozen CLIP ViT-L/14 frame encoder, spatiotemporal pooling, one trainable linear projection, Vicuna-7B language model §3.1 fig. 1.
• Only the projection is trained, on 100K video-instruction pairs §3.2 §4 §5.1.
• The paper also proposes a GPT-3.5-judged benchmark with five aspects, plus zero-shot video QA §5.2.
• All numbers are author-reported. The judge is a closed API model.

Problem

Image-based vision-language models are the usual base for multimodal chat, but they generally do not consider video input §1. Training a video model from scratch needs video-caption data that is scarce, and substantial compute §3. The paper also notes that no quantitative benchmark existed for judging video conversation models §1.

Method

Overview

The model starts from LLaVA, which joins a CLIP image encoder to Vicuna and is trained on generated instruction data §3 fig. 1. Video-ChatGPT adapts that design to video: the encoder runs on every frame, and the frame features are pooled along two axes. The result is a fixed-layout token sequence with a temporal part and a spatial part. The encoder and language model stay frozen. Only the linear projection learns §3.1 §3.2.

Steps

1. Frame encoding. Each of the T frames is encoded independently by CLIP ViT-L/14 into an h x w grid of D-dimensional patch features, with patch size p = 14 and h = H/p, w = W/p §3.1. The grid has N = h x w patches.
2. Temporal tokens. Average over the spatial grid, giving one vector per frame (T vectors of size D) §3.1.
3. Spatial tokens. Average over frames, giving one vector per patch position (N vectors of size D) §3.1.
4. Concatenation. Stack temporal and spatial tokens into the video feature matrix of shape (T+N) x D §3.1 eq. 1.
5. Projection. A single trainable linear layer g maps each row into the language model's embedding space of width K §3.1 eq. 2. The paper calls g an adapter and says more complex designs are possible §3.1.
6. Concatenation with text. The projected video tokens are concatenated with the embedded text query, whose length is L, and fed to the language model §3.1.
7. Instruction tuning. The prompt has the form USER: <Instruction> <video tokens> Assistant:. The loss is the autoregressive likelihood of the answer tokens. Only g is updated §3.2.

# Build video tokens for one clip  [§3.1] [eq. 1] [eq. 2]
# Input:  frames (T, H, W, 3)
# Output: video tokens (T+N, K) in the language model's embedding space

def encode_video(frames):
    # Frozen CLIP ViT-L/14 patch features per frame  [§3.1]
    frame_features = clip_vit_l14(frames)            # (T, h, w, D)
    # Average over the spatial grid: one vector per frame  [§3.1]
    temporal_tokens = mean(frame_features, over=(h, w))  # (T, D)
    # Average over frames: one vector per patch position  [§3.1]
    spatial_tokens = mean(frame_features, over=T)        # (N, D)
    # Stack both kinds of tokens  [§3.1] [eq. 1]
    video_features = concat(temporal_tokens, spatial_tokens)  # (T+N, D)
    # Single trainable linear layer g  [§3.1] [eq. 2]
    return linear_g(video_features)                  # (T+N, K)

# One instruction-tuning step  [§3.2]
# Frozen: CLIP encoder, language model. Trained: linear layer g only.
# Input:  frames, instruction text, answer text

def training_step(frames, instruction, answer):
    # Prompt template: USER: <instruction> <video tokens> Assistant:  [§3.2]
    text_tokens = tokenize(instruction)
    video_tokens = encode_video(frames)
    prompt = concat(text_tokens, video_tokens)
    # Next-token loss on the answer tokens  [§3.2]
    return cross_entropy(language_model(prompt, answer), answer)

Data: video instruction pairs

The dataset has 100,000 video-instruction pairs §4. Two annotation routes are used, and they are mixed in the final set §5.3.

• Human-assisted. Annotators enrich ActivityNet-200 ground-truth captions with appearance, spatial and temporal details §4.1 fig. 2. The paper describes the annotators as being told to neutralise tone and bias §10.
• Semi-automatic. BLIP-2 produces frame captions, GRiT produces dense captions for objects, and Tag2Text produces tags for each key frame §4.2 fig. 3. Dense captions are kept only when their vocabulary words agree with the Tag2Text tags. GPT-3.5 then merges the frame captions into one video-level caption and drops details that are inconsistent across frames §4.2 fig. 4.
• GPT-assisted post-processing. GPT-3.5, prompted with in-context examples, writes question-answer pairs covering detailed descriptions, summaries, creative tasks and conversation §4.3.

# Keep a dense frame caption only when it agrees with the frame's tags  [§4.2]
# Input:  dense_caption (text), tags (set of strings)
# Output: filtered caption text, or nothing when it is dropped

def keep_dense_caption(dense_caption, tags):
    # Only the words before the first colon are checked
    # [code: data/generate_instruction_qa_semi_automatic.py#L33]
    caption_words = split_words(before_first_colon(dense_caption))
    # Keep only the words that belong to the tag vocabulary
    # [code: data/generate_instruction_qa_semi_automatic.py#L35]
    vocab_words = [w for w in caption_words if w in tag_vocabulary]
    # Drop the caption if any kept word is missing from the tags
    # [code: data/generate_instruction_qa_semi_automatic.py#L37]
    if all(w in tags for w in vocab_words):
        return join(vocab_words)
    return nothing

Hyperparameters

Results

Author-reported. Setup: a test set based on ActivityNet-200, with dense captions and QA pairs from human annotations §5.2. Judge: GPT-3.5 §5.2. All models are 7B variants tab. 1 tab. 2.

Video-based generative benchmark (1-5 scale, higher is better) tab. 1

Zero-shot question answering (accuracy in %, and score on a 1-5 scale) tab. 2

Ablation: training data mix (same benchmark as tab. 1) tab. 3

Ablation: open-source judge (Vicuna-1.5 13B as the judge) tab. 4

The paper reports no variance, seeds or significance tests for these numbers.

Qualitative examples are described in §11 and shown in fig. 5 to fig. 11. The paper does not give them numerical results.

Paper vs code

Code: official repository at commit 8d54ebf README.md. The paper links the repository in its abstract.

• Frame count and sampling. Paper does not state how many frames are used §3.1. Code samples at most 100 frames, at the midpoints of equal segments save_spatio_temporal_clip_features.py save_spatio_temporal_clip_features.py. Inference uses the same rule model_utils.py model_utils.py.
• Temporal tokens are padded to 100. The paper's token count is T+N §3.1 eq. 1. Code zero-pads the temporal tokens to 100 before the projection save_spatio_temporal_clip_features.py single_video_inference.py. The padded slots pass through the projection, so they become the projection's bias vector rather than zeros. Training fixes the total at 356 train.py.
• Start and end tokens. The paper's prompt template has no boundary tokens §3.2. Code wraps the visual tokens in <vid_start> and <vid_end> single_video_inference.py train_video_chatgpt.md. Their embeddings are initialised to the mean of the existing embeddings video_chatgpt.py.
• Which parameters train. The paper says only the linear layer is trained §3.2. With --tune_mm_mlp_adapter True and --mm_use_vid_start_end, the code also marks the whole input embedding matrix as trainable video_chatgpt.py train_video_chatgpt.md. The forward pass detaches most text embeddings in the video path video_chatgpt.py, so the gradient in practice seems to reach only the boundary-token rows. This is inferred from reading the code and was not run.
• Question placement in training data. The paper's template puts the instruction before the video tokens §3.2. The conversion script alternates the question before and after <video> by example index convert_instruction_json_to_training_format.py. Inference puts the video tokens after the question single_video_inference.py.
• Conversation template. Training uses vicuna_v1_1 train.py with the system prompt in video_conversation.py. Inference defaults to video-chatgpt_v1 with a different system prompt single_video_inference.py video_conversation.py.
• Scoring scale. Paper says 1-5 §5.2. The correctness prompt asks for an integer from 0 to 5 evaluate_benchmark_1_correctness.py. The ActivityNet-QA prompt asks for a yes/no and a 0-5 score evaluate_activitynet_qa.py.
• Semi-automatic filter. Paper says captions containing words outside the tag set are removed §4.2. Code returns only the tag-vocabulary words from the part of the caption before the first colon generate_instruction_qa_semi_automatic.py. It does not return the original caption.
• Dataset and prompts. Paper says the dataset and prompts "will be made publicly available" §4.2. The repository does not contain them; the docs link external download locations train_video_chatgpt.md.
• Evaluator model. Paper says GPT-3.5 §5.2. Code calls gpt-3.5-turbo evaluate_benchmark_1_correctness.py. No dated snapshot is pinned.

No discrepancies found in: the projection formula, the mean pooling directions, the learning rate, the epoch count, or the overall batch size of 32.

Pitfalls

• Frame budget is not in the paper. The 100-frame cap, the segment-midpoint sampling and the 224 x 224 resize all come from code. Implement them from the code, or your features will not match the released weights.
• Padding changes the layout. Zero-padding the temporal tokens to 100 happens before the projection. Implementations that pad after projection get different vectors.
• Layer choice matters. The features come from the penultimate CLIP layer with the CLS token dropped save_spatio_temporal_clip_features.py. Using the final layer or keeping CLS is a silent change.
• Boundary tokens and template. Training and inference use different system prompts and different question placement (see Paper vs code). Pick one convention and apply it at both ends.
• Trainable parameters. The paper's claim that only the projection trains does not hold for the code's configuration (see Paper vs code). Decide explicitly whether the embedding matrix is trainable.
• The judge is a closed model. Tables 1 and 2 rely on GPT-3.5 §5.2. The ablation with Vicuna-1.5 13B shows the same ordering tab. 4, but absolute scores move (for example Video-ChatGPT correctness is 2.40 in tab. 1 and 2.49 in tab. 4). Compare numbers only under the same judge.
• Small score gaps. Differences of a few hundredths on a 1-5 judge scale are reported without variance. For example the temporal-understanding row in tab. 1 is a tie at 1.98 between Video-ChatGPT and LLaMA Adapter.
• Blind-test result is near chance. The human-versus-automatic QA test used 50 videos and reached 52% §5.3. That is close to 50%, the chance level for a two-way choice. The paper reads it as evidence the automatic data is reliable.
• Inference setup is unstated in the paper. Sampling at temperature 0.2, FP16 and up to 1024 new tokens come from code only single_video_inference.py.
• Long videos and small objects. The authors report difficulty with subtle temporal relations in videos longer than two minutes and with small-object details §7.
• Cost. Training took about 3 hours on 8 A100 40GB GPUs §5.1. Data generation depends on GPT-3.5 and three off-the-shelf captioning or tagging models §4.2.
• Data licence. The paper says the dataset is built on ActivityNet, which it describes as MIT-licensed, and that the authors will release their data under MIT §9. We have not verified those terms, and the videos are not in this repository.

Lineage

• Builds on: LLaVA, "Visual Instruction Tuning" [arXiv:2304.08485], which supplies the CLIP-plus-Vicuna design and the initial weights §3 fig. 1.
• Compared against: Video Chat, LLaMA Adapter and Video-LLaMA, and FrozenBiLM for zero-shot QA §5.2 tab. 1 tab. 2.