Otter: A Multi-Modal Model with In-Context Instruction Tuning

In plain words

Most chat-style vision models learn from one picture and one question at a time. Otter's authors asked whether a model would learn better if each training example also came with a few worked examples, the way a student sees solved problems before attempting one. Each example is a short bundle of pictures (or video frames) plus an instruction and its answer, and the model reads these before it answers the real question.

To train this, the authors built MIMIC-IT, a dataset of about 3 million instruction-response pairs over images and videos. They used a large language model to write many of the questions and answers, and they group related examples so that each training item carries its own context. Otter is a vision-language model built on an existing open re-implementation of Flamingo, and only a small part of it is trained. The authors report that the context helps training converge faster and generalise better, and that the model does well on questions about several images and on video.

TL;DR

Otter (OpenFlamingo-9B base: MPT-7B decoder, CLIP ViT-L/14 encoder) is instruction-tuned with multi-modal in-context examples taken from MIMIC-IT (about 3M pairs, 2.2M unique instructions) §4 tab. 1. Only the perceiver resampler and the gated cross-attention layers train, about 1.4B parameters §3.1. The paper reports better convergence and benchmark numbers with in-context examples §6.3 and strong video-QA results tab. 8. All numbers are author-reported on author-chosen setups.

Problem

Instruction tuning for large multimodal models usually uses single image-instruction-response pairs, or earlier dialogue turns as context. The authors argue this discards useful signal: pairing images with text examples during instruction tuning could help the model follow complex instructions and keep its in-context learning ability §1 §2. They also report that Otter trained without in-context examples does well at zero shots but loses in-context gains as shots increase §6.2 fig. 11.

Existing visual instruction datasets are mostly single images with one turn. The paper frames the need as multi-image and video inputs with in-context context per example §1 §4.

Method

Model. Otter is built on OpenFlamingo-9B, a re-implementation of Flamingo. The vision encoder is CLIP ViT-L/14 and the language decoder is MPT-7B §3.1. A perceiver resampler turns the image or video features into a fixed set of visual tokens. Gated cross-attention layers inside the language decoder let text tokens attend to those visual tokens. The paper says the gating keeps the language layers unchanged at initialisation §3.1.

Trainable parts. Only the perceiver resampler and the gated cross-attention modules train; the rest of OpenFlamingo-9B is frozen §3.1. The code also unfreezes the input embeddings, see Paper vs code.

Objective. Training is standard token-level supervised fine-tuning. The probability of response tokens is the product of next-token probabilities, conditioned on the visual tokens and the instruction eq. 1 §3.2.

Data format. Each MIMIC-IT item is a query (instruction, response, images or video) plus a set of in-context examples. Videos are treated as ordered sequences of images §4 eq. 2. Otter-Image uses the image count dimension for the context images; Otter-Video uses the frame dimension, with added time embeddings §3.2 fig. 2.

Context construction. In the LLaVA-derived subsets (LA-I2I and LA-T2T), in-context examples are the top-K most similar training items, by CLIP image-to-image similarity or by instruction text-to-text similarity §4.1. The other subsets pair sequential images or video clips with several instruction-response sets that serve as mutual in-context examples §4.1. The paper reports 2-shot and 4-shot contexts for its ablations §6.3 tab. 10.

Task descriptions. Each MIMIC-IT subset also gets a short task description written by GPT-4 from 20 to 50 samples; it is prepended before each instruction §6.5. The authors' ablation adds the three description parts (coverage, ability, answer format) one at a time tab. 9.

Data generation (Syphus). ChatGPT generates the instruction-response pairs from images with captions, boxes or other annotations, using a system message, visual annotations and in-context examples as the prompt. A cold-start stage picks the system message and examples; the pairs are then translated into seven more languages §5.1 fig. 4. Dataset cleaning used regular expressions for fixable errors and manual checks by two people §5.3.

Pseudocode for one training step, following §3.1 and §3.2:

# Otter training step: visual tokens condition gated cross-attention  [§3.1]
# Input:  images (B, N, T, 3, 224, 224); context and query tokens
# Output: scalar loss on response tokens  [eq. 1]

def training_step(images, instruction_tokens, response_tokens):
    # Encode frames with the frozen CLIP ViT-L/14  [§3.1]
    patch_feats = vision_encoder(images)                 # (B, N, T, P, d_v)
    # Resample patch features into a fixed set of visual tokens  [§3.1]
    visual_tokens = perceiver_resampler(patch_feats)     # (B, N, T, K, d)
    # Decode with the frozen MPT-7B; gated cross-attention is trainable  [§3.1]
    logits = decoder(instruction_tokens, response_tokens, visual_tokens)
    # Token-level negative log-likelihood of the response  [eq. 1]
    return cross_entropy(logits, response_tokens)

Note: the paper does not state K (the number of visual tokens per frame). The code default is 64 modeling_otter.py. The loss masking of instruction tokens is not traced here; check the code before copying it.

Hyperparameters

Values from the paper's hyperparameter table tab. 11 and text §6.5, with the code default where the paper is silent. "not stated" means the paper and the code examined here do not give a value.

Dataset figures from the paper: 3M instances, 2.2M unique instructions, 502K clips or 8.1M images across eight subsets tab. 1. Otter-Image draws on academic sets (VG, VQAv2, GQA, OK-VQA, OCR-VQA, A-OKVQA, TextVQA, RefCOCO, COCO-ITM, ImageNet, ImageNet-Paragraphs) with only the answer-format part of the task description added as prefix §6.1. The per-source pair counts appear only in tables that are commented out in the LaTeX source, so this card does not give them. The paper's mixture ablation is in tab. 12.

Results

Author-reported. Setup: Otter-Image-9B against open LMMs on single-image benchmarks; the paper does not describe the evaluation protocol per benchmark in the text read here §6.2 tab. 2. The baseline column shows the strongest single listed baseline per benchmark, not every baseline.

Video question answering, zero-shot, scored with a ChatGPT-based evaluator (GPT-4 is used for "ChatGPT" throughout) §6.2 §6.5:

Captioning (ChatGPT score G, plus BLEU-1 / METEOR / ROUGE-L): Otter-Video scores G = 54.3 on MSVD-Cap (mPLUG-Owl-V 51.5) and G = 67.9 on MSRVTT-Cap (mPLUG-Owl-V 39.0) §6.5.

Ablations (author-reported, LLaVA-Instruct-158K subsets for task descriptions):

In-context examples at training and evaluation (MME-Cog):

Training convergence curves with and without task descriptions and with 2-shot in-context examples are in fig. 9 and fig. 10. The few-shot COCO caption curves are in fig. 11. The loss gap is reported as appearing after about 20K steps §6.3.

Paper vs code

Official code: EvolvingLMMs-Lab/Otter at commit 1e7eb9a6fb12ef410082e796c463b99495637b85 (MIT). The repository's README links this paper and cites all five named authors from its own citation block, so it is treated as the authors' repository. The arXiv comments and the paper text do not link it. The commit is later than the paper version and mixes in other models (Fuyu, Idefics, OtterHD), so only the Otter path is compared here.

Discrepancies found:

• Trainable size: paper says 1.4B trainable §3.1; Table 2 says 1.3B for Otter-Image tab. 2.
• Trainable set: paper trains only the perceiver and the gated cross-attention §3.1. The code also unfreezes the input embeddings (and the LM head for LLaMA-style decoders) modeling_otter.py modeling_otter.py. The second freezing path sets the gated cross-attention and perceiver trainable modeling_otter.py.
• Epochs: paper says 6 tab. 11; the repo's scripts use 3 README.md.
• Learning rate and warmup: paper 1e-5 and 0.03 tab. 11; the repo's MPT example uses 2e-5 and 0.01 README.md. The OtterHD demo script matches the paper's 1e-5 and 0.03 Demo_OtterHD.sh.
• Video frames: paper says 64 frames sampled for DC/TVC training §6.1 and 8 frames in the table tab. 11. The code default resample_frames is 32 train_args.py.
• Parallelism: the paper's table says DeepSpeed Zero2 tab. 11; the README example uses the ZeRO-3 accelerate config README.md.
• Weight decay: the paper does not state it; the code default is 0.1 train_args.py. The grouping applies decay only to gated cross-attention weights, excluding the gates, norms and biases train_utils.py.
• Instruction format: the code's simple format produces User: {instruction} GPT:<answer>{response}<|endofchunk|> mimicit_dataset.py. Figure 2 shows the same User and GPT labels fig. 2.

Pitfalls

• Trainable count. 1.4B in the text and 1.3B in Table 2 for the same model §3.1 tab. 2. Budget memory for the unfrozen embeddings too.
• Video frame count. The text says 64 frames for training, the table says 8 for Otter-Video, and the code default is 32. Check which one a training run uses before comparing numbers.
• Otter-Image and Otter-Video differ. Batch size per GPU (8 vs 4) and sample frames (– vs 8) differ tab. 11. The paper says the video model needs a smaller batch because of multiple frames §6.5.
• Evaluator. Video and caption scores use a ChatGPT (GPT-4) judge with a fixed prompt, shown in the paper §6.5. Judge-version drift changes scores. The paper gives no variance or repeated runs.
• Author-reported setups. MMBench numbers are on the test set §6.2. The baseline numbers in Table 2 are as the authors report them tab. 2.
• Data generation. The dataset depends on GPT-4 or ChatGPT generations and their safety policy §5.1 §4.3. The paper reports some remaining wrong dialogues (e.g. "the user (you)") and fixes them with regular expressions and manual checks §5.3 fig. 4.
• Sensitive content. The paper states that faces in some sources were classified for race and gender on a 5% sample, and reports the distribution §4.3 tab. 3. It also says some sources (TV clips, web images) may contain personal information §4.3.
• Unstated details. Number of GPUs, total training steps, seed, weight decay and max sequence length are not stated in the paper.
• Resolution. Inputs are 224 × 224 and the authors name fine-grained OCR as a weak spot §3.2 §7.

Lineage

• Builds on: Flamingo (gated cross-attention and perceiver resampler), and OpenFlamingo, the open reproduction used as the base §3.
• Uses data ideas from: self-instruct for generating instructions §4; LLaVA for the single-image base data §6.1 §6.3.
• Compared against: LLaVA-1.5, Qwen-VL, InstructBLIP, Idefics, mPLUG-Owl, VideoChat and Video-ChatGPT (author-chosen set) tab. 2 tab. 8.
• Superseded by: not known from content in this wiki.