OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

In plain words

Some AI models read pictures and text in the same prompt and answer in text. You can put a few example pictures with their answers into the prompt, and the model picks up the task from those examples without any retraining. DeepMind's Flamingo does this well, but its weights and training data are closed. OpenFlamingo is an open attempt to build the same kind of model.

The recipe reuses parts that already exist. A strong image encoder turns each picture into a set of numbers. A text model that already knows language is kept frozen. New layers are inserted into the text model so that its words can look at the pictures, and a small module compresses the image features first. Most of the model is therefore not retrained. The new parts learn from openly available collections of web pictures with text: one of single image-caption pairs and one of interleaved web documents with pictures in them.

The result comes close to Flamingo on seven benchmarks that cover captioning, question answering and hateful-meme classification. It is weaker on some of them, especially reading text in images and questions that need a precise short answer. The authors call the project an ongoing effort. The code and the trained models are public.

TL;DR

• The model predicts the next text token from earlier text and the most recent image. The image enters through gated cross-attention layers added to a frozen language model §3.1 tab. 1.
• Only the image compression module, the cross-attention layers and (in most variants) the image and end-of-chunk embeddings are trained; CLIP ViT-L/14 and the language model stay frozen §3.1.
• Training uses LAION-2B image-caption pairs and Multimodal C4 interleaved web documents, with a next-token loss and AdamW at learning rate 1e-4 §3.2 §3.3.
• Author-reported: OF-3B and OF-9B reach 85% and 89% of Flamingo's average over seven benchmarks when demonstrations are chosen by retrieval (RICES) §1 fig. 1. With random demonstrations at 0 shots, the largest deficits are on OK-VQA and TextVQA tab. 11.
• Read the Pitfalls before implementing: several training details the paper leaves open are fixed in the code, and the code and paper differ in a few evaluation settings.

Problem

Most open vision-language models accept one image and a question, and many are trained on curated caption data rather than web data §1. Flamingo accepts an arbitrary interleaving of images and text, so a prompt can carry demonstrations for a new task, which enables few-shot in-context learning §1. Flamingo's weights, training data, code and hyperparameters are not public, which limits research on how web-scraped data affects these models §1. The paper aims to build an open model in this family, using public components §1 §2.

Method

Architecture. The model matches the Flamingo design §3.1. An interleaved sequence of images and text is turned into text predictions: each next text token is conditioned on all previous text tokens and on the last image before it §3.1. Text tokens reach their image through dense cross-attention modules attached to the layers of a frozen autoregressive language model §3.1. Each image is first encoded by a frozen CLIP ViT-L/14 vision encoder, and its patch features pass through a trainable Perceiver resampler §3.1. Preprocessing places an image marker at each image position and inserts an end-of-chunk marker after the text that follows an image §3.1. Video input is not supported §3.1.

The paper has no numbered equations or algorithms. The mechanism is described in prose, and the pseudocode below follows the code (see Paper vs code).

Model variants tab. 1. All five models use CLIP ViT-L/14. The cross-attention interval says how often a cross-attention module is inserted: every n-th language-model layer. The image and end-of-chunk embeddings are trainable in most models; for the OF-4B models all embeddings are frozen, including the randomly initialised image and end-of-chunk embeddings §3.1.

"Instruct" means the language model was instruction-tuned on language-only tasks; the OpenFlamingo models themselves were not tuned on vision-language tasks tab. 1.

Training data §3.2. Training samples shards with replacement, in WebDataset format §3.2.

• LAION-2B: the English subset of image-caption pairs, with captions truncated to 32 tokens. Every pair has CLIP ViT-B/32 cosine similarity of at least 0.28. This replaces ALIGN, which Flamingo used and which is not public §3.2.
• Multimodal C4 (MMC4): interleaved web documents with 101M samples. CLIP ViT-L/14 soft-aligns images to sentences. An image is dropped when its cosine similarity with the following text is below 0.24. Sequences contain 1 to 6 images (median 2). Single-image sequences are rejected with probability 0.5 §3.2 app. B. This replaces M3W, which Flamingo used and which is not public §3.2.
• ChatGPT-generated sequences (OF-4B only, see the footnote in §3.3): 417K interleaved sequences. ChatGPT wrote text with alt-text placeholders, and LAION-5B retrieval filled in the images. Ten candidates per caption were retrieved with an aesthetic filter (score 9, weight 0.5) and re-ranked with CLIP ViT-L/16@336px §3.2 app. C.

Table 2 gives median image and token counts per sequence, from 1,000 sampled sequences each tab. 2. Figure 4 shows the MMC4 histograms fig. 4.

Training objective and optimiser §3.3. All models are trained with next-token prediction and AdamW. Each optimiser step uses gradients from one LAION batch and one interleaved batch. The LAION batch is twice the size of the interleaved batch. The loss weights are 1.0 for the interleaved data and 0.2 for LAION. Gradients are accumulated over both datasets between optimiser steps. The learning rate rises linearly at the start and is then held at 1e-4. Weight decay of 0.1 applies to the dense cross-attention layers §3.3.

# One optimiser step over both data streams  [§3.3]
# laion_batch: 2x the size of mmc4_batch; both are (images, token ids)

def training_step(model, optimizer, laion_batch, mmc4_batch):
    # Caption loss on image-caption pairs, weight 0.2  [§3.3]
    laion_loss = next_token_loss(model, laion_batch)
    (0.2 * laion_loss).backward()
    # Caption loss on interleaved sequences, weight 1.0  [§3.3]
    mmc4_loss = next_token_loss(model, mmc4_batch)
    (1.0 * mmc4_loss).backward()
    # Gradients from both streams are summed before the update  [§3.3]
    # Learning rate is 1e-4 after a linear warm-up  [§3.3]
    optimizer.step()
    optimizer.zero_grad()

# Conditioning of text on images  [§3.1] [tab. 1]
# text_ids: (B, L) tokens with image markers; images: (B, T, 3, H, W)

def forward(model, text_ids, images):
    # Frozen CLIP ViT-L/14 gives patch features per image  [§3.1]
    patches = clip_vit_l14(images)                   # (B, T, N, d_vis)
    # Trainable Perceiver resampler compresses each image  [§3.1]
    visual = perceiver(patches)                      # (B, T, K, d_vis)
    hidden = embed(text_ids)                         # (B, L, d)
    for index, layer in enumerate(lm_layers):
        # Every n-th LM layer gets a gated cross-attention block  [tab. 1]
        if (index + 1) % n == 0:
            # Text attends to the last image before it  [§3.1]
            hidden = gated_xattn[index](hidden, visual)
        hidden = layer(hidden)                       # frozen decoder layer
    return lm_head(hidden)

Distributed training §3.3 tab. 3. All models were trained on 64 GPUs across 8 nodes on Stability AI's cluster. The OF-4B models used Fully Sharded Data Parallel; the others used data parallel. Table 3 lists the GPU type and precision per model.

Training length §3.3. The models were trained for 60M interleaved (MMC4) examples and 120M LAION-2B examples. OF-4B models also saw about 240K ChatGPT-generated sequences within the 60M interleaved examples; the other models saw only MMC4 §3.3, footnote.

Loss curves §3.3 fig. 5. After an initial improvement, MMC4 loss decreases very slowly. The authors suggest that most MMC4 text can be predicted without the image, because MMC4 has long paragraphs between images §3.3.

Evaluation §3.4. The seven benchmarks are COCO captions and Flickr-30K (CIDEr), VQAv2, OK-VQA, TextVQA and VizWiz (VQA accuracy), and HatefulMemes (ROC AUC). Each runs at 0, 4, 8, 16 and 32 in-context examples, on the splits used by Flamingo. Random demonstrations come from the training split. Each setting is run over three seeds, and the results are averaged §3.4. In Appendix A.2, demonstrations are instead picked by RICES: the top-k training examples by cosine similarity of CLIP ViT-L/14 image features app. A.2.

• Zero-shot uses two text-only examples with their images removed §3.4.
• Prompts: captions use "Output:" followed by the caption; VQA uses "Question:" and "Short answer:"; HatefulMemes asks whether an image with the given text written on it is hateful, and answers "yes" or "no" §3.4.
• Decoding: beam search with 3 beams. Captions stop at 20 tokens and VQA at 5 tokens, or earlier at an end-of-chunk marker §3.4.
• Classification (HatefulMemes): score the completions "yes" and "no", and answer with the more likely one. Prompt ensembling averages logits over 6 orderings of the demonstrations §3.4.
• Metrics: CIDEr for captions; VQA accuracy, meaning exact match against the set of ground-truth answers; ROC AUC for HatefulMemes §3.4.

Hyperparameters

Results

All numbers are author-reported. Setup: seven benchmarks, five shot settings, random demonstrations, three seeds (mean with standard deviation over seeds). Baselines for Flamingo are taken from Alayrac et al. and not re-run [tab. 4 caption]. Full table, verbatim values:

Source: tab. 11. Flamingo columns: tab. 4 caption.

Headline and summary claims (author-reported):

• Averaged over the seven benchmarks and five shot settings, with RICES demonstrations, OF-3B and OF-9B reach 85% and 89% of their Flamingo counterparts. The abstract gives a range of 80 to 89% §1 fig. 1.
• The text states that OF-3B, OF-3B (I), OF-4B (I) and OF-9B each average more than 86% of their Flamingo counterparts §4.
• With 32 RICES-selected shots, OF-9B averages more than 62% of fine-tuned state of the art, against 72% for Flamingo-9B §4 app. A.1. The state-of-the-art numbers were taken from Papers With Code on 19 June 2023 tab. 8 app. A.1.
• OF-9B beats Flamingo-9B at 0 shots on VQAv2 (52.7 vs 51.8) and COCO (79.5 vs 79.4) with random demonstrations tab. 11 §4.
• OF-9B is clearly behind Flamingo-9B at 0 shots on OK-VQA (37.8 vs 44.7) and TextVQA (24.2 vs 31.8) with random demonstrations tab. 11.
• OF-3B is behind Flamingo-3B at 0 shots on VQAv2 (44.6 vs 49.2) tab. 11.
• Instruction-tuned language models on average beat their base versions at 3B and 4B §4 fig. 6.
• The 4B models degrade after 4 or 8 shots on several tasks, where the 3B and 9B models generally improve §4 tab. 11.

RICES versus random demonstrations for OF-9B tab. 9:

RICES helps on most settings, but on Flickr-30K it hurts badly at 0 and 4 shots. The authors suggest that the retrieved examples are so similar to the test image that the model copies their captions, including wrong details app. A.2. Full RICES tables with deviations are in tab. 12.

Frozen versus trainable embeddings (small OPT-125M models, 20M samples, not the released models) tab. 5:

Language model choice (VQAv2 validation, 20M samples) tab. 7:

Fine-tuned state of the art used as reference tab. 8: mPLUG (COCO, 155.1), Unified VLP (Flickr, 67.4), PaLI-17B (VQAv2 84.3, OK-VQA 64.5, TextVQA 73.1, VizWiz 73.3), and Hate-CLIPper (HatefulMemes 85.8). These use different protocols from the in-context setting and are reference points only.

Paper vs code

Official repository: mlfoundations/open_flamingo, MIT license, at commit 655f693fbfa04cd6e9a987d960654624d48917cf. The paper links it §1.

1. TextVQA generation length. The paper uses 5 tokens for VQA §3.4. The evaluation code uses 10 tokens for TextVQA evaluate.py, with the 5-token default for other VQA sets evaluate.py.
2. Length penalty. The paper does not state one. The code's signature defaults to 0.0 evaluate.py, evaluate.py, while the docstrings say the default is -2.0 evaluate.py, evaluate.py. The code runs with 0.0.
3. HatefulMemes scoring. The paper scores the completions "yes" and "no" by log-likelihood and picks the more likely one §3.4. The code uses rank classification with log-likelihood normalised by length evaluate.py.
4. Prompt ensembling. The paper says classification tasks average over 6 orderings §3.4. The code only does this when --classification_prompt_ensembling is passed evaluate.py. The HatefulMemes call does not pass it evaluate.py, and the example evaluation script does not either run_eval.sh. The number of orderings is min(6, k!) evaluate.py. For zero-shot (two examples) that gives 2.
5. Seeds. The paper averages three seeds §3.4. The defaults are one trial with seed 42 evaluate.py. The example evaluation script sets neither flag run_eval.sh.
6. Weight decay under FSDP. The paper applies weight decay 0.1 to the cross-attention layers only §3.3. The code does this for DDP and for FSDP with original parameters train.py. Without original parameters, the FSDP branch applies weight decay 0.1 to every trainable parameter, including the Perceiver and any trainable embeddings train.py. The paper says OF-4B used FSDP tab. 3. Which branch was used for the released models is not stated.
7. Warm-up. The paper says the learning rate rises linearly but gives no length §3.3. The released 3B script uses 1875 steps run_train.sh. The code default is 5000 train.py.
8. Gradient clipping. The code clips the global gradient norm to 1.0 train_utils.py. The paper does not mention clipping.
9. Augmentation. The code applies a random horizontal flip with probability 0.5 to training images data.py. The paper does not mention it. For LAION the transform is applied after batching data.py, so one flip decision covers a whole 64-image batch. For MMC4 each sequence gets its own decision data.py.
10. Loss masking and non-finite losses. Tokens before the first image get no loss train_utils.py. A non-finite MMC4 loss skips the batch train_utils.py. The paper mentions neither.
11. Embedding gradients. When the image and end-of-chunk embeddings are trainable, the code zeroes gradients for all other rows of the input embedding matrix train_utils.py. This matches the paper's statement §3.1.
12. Perceiver sizes and gate initialisation. The paper gives neither. The code uses depth 6, 64 latents, 8 heads, head dimension 64 and an FF multiplier of 4 helpers.py. The gated blocks start with both gates at 0 helpers.py.
13. Frozen embeddings. The code freezes the input embeddings only with freeze_lm_embeddings factory.py. The default is off factory.py. The released 3B script does not set it run_train.sh. The OF-4B configuration is not in the repository.
14. Compute. The paper trains on 64 GPUs over 8 nodes §3.3. The released script requests one node with eight tasks run_train.sh.
15. Headline average. fig. 1 states that its averages use RICES. The main text and tab. 11 use random demonstrations §3.4 §4. The 85% and 89% figures therefore come from the RICES setting.
16. MMC4 filter. The paper describes a CLIP ViT-L/14 similarity threshold of 0.24 app. B. The released 3B script passes this value as mmc4_textsim_threshold run_train.sh. I did not trace where the similarity is computed, so the scoring model is unverified against code.

No discrepancies were found for the LAION caption length of 32 tokens data.py, the MMC4 image bounds train.py, the captioning settings of 3 beams and 20 tokens evaluate.py, the cross-attention interval flamingo_lm.py, or the 480-epoch sample counts run_train.sh.

Searched: the paper's linked repository only. The paper links it, so no other search was needed.

Pitfalls

• Headline versus tables. The 85% and 89% averages use RICES fig. 1, while the main tables use random demonstrations §3.4 tab. 11. Random demonstrations give lower results on OK-VQA and TextVQA and at 0 shots on Flickr-30K. Choose the setting that matches your use.
• Numbers in the prose that do not match the tables. The text gives 7.8 points for OF-9B's TextVQA deficit at 0 shots §4; tab. 11 gives 31.8 minus 24.2, which is 7.6. For RICES, the text gives 19.2 ROC AUC on HatefulMemes at 32 shots, while tab. 9 gives 73.6 minus 53.8, which is 19.8. The text gives 20.4 CIDEr for Flickr-30K at 0 shots and 13.1 at 4 shots; tab. 9 gives 20.3 and 13.6 app. A.2. Use the tables, not the prose.
• Zero-shot is not zero examples. The zero-shot setting places two text-only examples in the prompt, with their images removed §3.4. Comparisons with methods that see no examples are therefore not like-for-like.
• Baselines are copied. The Flamingo numbers come from Alayrac et al. and were not re-run tab. 4 caption. The fine-tuned state-of-the-art numbers were gathered from Papers With Code in June 2023 and use other protocols app. A.1.
• Few-shot scaling is uneven. OF-3B (I) and OF-9B generally improve with more shots, but the gap to Flamingo widens at higher shot counts §4. The 4B models fall after 4 or 8 shots. They differ from the other models in backbone (RedPajama) and frozen image and end-of-chunk embeddings §4. The paper tests the embeddings only in small OPT-125M runs §5.1 tab. 5, which do not settle the cause.
• The data are mostly single-image. MMC4 sequences have a median of 2 images tab. 2. The authors suggest this limits how well the models use many demonstrations §4 app. B. Raising the MMC4 threshold to 0.32 cuts the data sharply: 88.7% of sequences are dropped, compared with 42.7% at 0.24 app. B.
• Language model dominates VQA. Switching the language model moves VQAv2 validation accuracy far more than extra training does. Going from OPT-1.3B to MPT-1B (Instruct) adds about 9 points at 0 shots tab. 7 §5.2.
• VQA failure modes. On the VQAv2 validation split, OF-9B scores 30.5% on numeric questions and 70.6% on yes/no questions. It also gives verbose answers, which fail exact match, and sometimes answers about the central object instead of the one the question asks about §5.2 tab. 6. Table 6 is a figure-like table with pictures; this card describes it in words only.
• Data licences. The paper does not state the licences of LAION-2B or MMC4 content, or of the ChatGPT-generated sequences it releases §3.2. Check them before commercial reuse of data.
• Safety. The models were not fine-tuned for safety, and they inherit the risks of their language models §5.4.
• Training length. The reported runs stop at 60M MMC4 and 120M LAION samples §3.3. The paper reports no results beyond that point.
• Resources. The paper gives the GPU types and sharding per model tab. 3 but no total compute or memory budget for a single-GPU setup.

Lineage

• Builds on: Flamingo (Alayrac et al., 2022; bibliography key alayrac2022flamingo). OpenFlamingo keeps its cross-attention design and Perceiver resampler, and swaps ALIGN for LAION-2B and M3W for MMC4 §3.1 §3.2. Flamingo is not a card on this wiki.
• Data: LAION (arXiv:2210.08402, LAION card), cited in the paper as LAION-2B §3.2. Multimodal C4 (arXiv:2304.06939) is not a card on this wiki, so nothing from its content is used here §3.2.
• Vision encoder: CLIP ViT-L/14 §3.1.
• Compared against: Flamingo-3B and Flamingo-9B numbers tab. 4, and fine-tuned state of the art tab. 8.
• Derived work: Otter, a multimodal assistant fine-tuned from OpenFlamingo on MIMIC-IT, and Multimodal-GPT, instruction-tuned from OpenFlamingo §5.3. Both are named in the paper only.
• Superseded by: not known from content on this wiki.