FLAVA: A Foundational Language And Vision Alignment Model

In plain words

Most vision models are trained for pictures, most language models for text, and most image-text models either match a whole image to a caption or mix the two inside one network. FLAVA tries to do all three with one recipe. It has a picture part, a text part, and a part that combines the two.

It learns from three kinds of data: image-caption pairs, plain pictures and plain text. On pairs it learns to line up matching images and captions, and to fill in hidden words and hidden picture pieces using the other half. On pictures alone and text alone it learns to fill in hidden pieces too. It also learns to tell a real pair from a caption swapped in from elsewhere.

The authors use only openly available data, and report results on 35 tasks across pictures, text and combined picture-and-text problems. Their point is that one model can be good at all three, not just the best at one.

TL;DR

FLAVA is a single transformer family: an image encoder, a text encoder, and a fusion encoder that reads both. It is pretrained on 70M public image-text pairs plus ImageNet-1k images and CCNews/BookCorpus text, using five losses. The author-reported macro average over 35 tasks is 75.85, above CLIP ViT-B/16 trained on 400M pairs (66.78) in the paper's own table, though CLIP is better on vision alone. Pretraining details are partly in the code, not the paper.

Problem

Contrastive models such as CLIP and ALIGN transfer well to images but are weak on tasks that need both modalities at once §1. Fusion models, which mix the two inputs early in one transformer, handle combined tasks but often ignore unimodal performance §1, §2. The authors also point out that CLIP and ALIGN were trained on large corpora that are not public §1. The goal is one model that is good at vision, language and vision-and-language tasks, trained only on openly available data §1, §5.

Method

FLAVA has three transformer encoders §3.1:

• Image encoder: ViT-B/16 applied to resized patches, with a learned classification token CLS_I §3.1. Pre-norm (layer norm before attention) §3.4 image_encoder.py.
• Text encoder: same ViT architecture as the image encoder but separate weights, applied to BERT-tokenized text, with CLS_T §3.1. The paper stresses that this differs from prior work §3.1.
• Multimodal encoder: a 6-layer transformer over the concatenation of linearly projected image and text hidden states, plus CLS_M §3.1. Two learned linear projections map each unimodal width into the multimodal width model.py.

For downstream tasks, a classifier head sits on CLS_I for vision tasks, on CLS_T for language tasks, and on CLS_M for multimodal tasks. One pretrained model is evaluated per task §3.1 fig. 2.

Multimodal objectives on image-text pairs §3.2:

1. Global contrastive loss L_GC. Like CLIP's, but gradients flow through the gathered embeddings of all GPUs, not only the local ones. The paper reports this as a gain, measured in the supplement §3.2 app. C.
2. Masked multimodal modeling L_MMM. Image patches are masked in rectangular blocks and tokenized by a pretrained dVAE (BEiT-style). 15% of text tokens are masked (BERT-style). The multimodal encoder predicts the masked codebook ids and the masked words §3.2.
3. Image-text matching L_ITM. A classifier on CLS_M decides whether an image and a text match §3.2.

Unimodal objectives §3.3:

• Masked image modeling (MIM) on unpaired images, reconstructing dVAE tokens of block-masked patches §3.3.
• Masked language modeling (MLM) on unpaired text, 15% masked §3.3.

Initialisation and schedule §3.3 app. B.1:

• Text encoder: pretrained with MLM on CCNews and BookCorpus first.
• Image encoder: initialised from a DINO ViT-B/16 trained on ImageNet-1k. The paper reports DINO beat BEiT §3.3 app. C.
• Then the whole model is trained jointly with round-robin sampling. Each step draws one dataset type with a fixed probability and computes that type's losses §3.3 tab. A.2.

# One joint pretraining step: draw one dataset type per step  [§3.3]
# Probabilities 0.70 image-text, 0.15 image, 0.15 text  [tab. A.2]
def pretraining_step(model, batch_type, batch):
    if batch_type == "image_text":
        # Un-masked encoders feed the global contrastive loss  [§3.1] [§3.2]
        image_seq = model.image_encoder(batch.image)
        text_seq = model.text_encoder(batch.text)
        # Project CLS tokens, L2-normalise, softmax over the gathered batch
        # with a learned temperature  [§3.2]
        loss_gc = global_contrastive(
            model.image_proj(image_seq[:, 0]),
            model.text_proj(text_seq[:, 0]),
        )
        # Mask image blocks and 15% of text tokens  [§3.2]
        # Block sizes  [code: torchmultimodal/transforms/flava_transform.py]
        image_masked, image_targets = mask_image_blocks(batch.image)
        text_masked, text_targets = mask_text_tokens(batch.text, rate=0.15)
        image_masked_seq = model.image_encoder(image_masked)
        text_masked_seq = model.text_encoder(text_masked)
        # Fuse the masked streams  [§3.1]
        fused = model.mm_encoder(image_masked_seq, text_masked_seq)
        # ITM: label 1 for true pairs, 0 for swapped captions  [§3.2]
        loss_itm = itm_head(fused[:, 0], batch.itm_labels)
        # Code applies MMM only to matched pairs  [code: flava.py#L418-L437]
        matched = batch.itm_labels == 1
        loss_mmm = (
            mmm_text_head(fused[matched], text_targets[matched])
            + mmm_image_head(fused[matched], image_targets[matched])
        )
        return loss_gc + loss_itm + loss_mmm
    if batch_type == "image":
        # MIM: predict dVAE ids of block-masked patches  [§3.3]
        masked, targets = mask_image_blocks(batch.image)
        seq = model.image_encoder(masked)
        return mim_head(seq[masked_positions(targets)], targets)
    # MLM: predict 15% masked tokens from unimodal text  [§3.3]
    masked, targets = mask_text_tokens(batch.text, rate=0.15)
    seq = model.text_encoder(masked)
    return mlm_head(seq[masked_positions(targets)], targets)

Key equations: the paper gives none numbered. The contrastive loss is described in words §3.2; the code's form is in flava.py.

Hyperparameters

Values from the paper's appendix table (pretraining) and the official code where the paper is silent. Code values are the repository defaults or the example configs, not the paper's own run, which the repository does not contain.

Fine-tuning and evaluation app. B.2:

Results

Author-reported. Setup: 35 tasks (22 vision, 8 GLUE, 5 multimodal), pretrained on 70M pairs (PMD) plus unimodal data for the full model §4 tab. 2.

Macro average and ablations (the macro value equals the mean of the three averages, with a missing average counted as zero; inferred from the table) tab. 3 tab. 4:

The column order in tab. 3 is wrong for four rows. See Pitfalls.

Zero-shot retrieval (Flickr30K and COCO, recall at 1) tab. 4:

FLAVA trails CLIP-400M on text retrieval and leads on image retrieval in this table.

Comparison with other models tab. 5 (VQAv2 is test-dev; GLUE is dev; other entries accuracy; ImageNet is linear evaluation):

GLUE dev results for FLAVA are in tab. 5 (for example MNLI 80.3, QNLI 87.3, SST-2 90.9, STS-B 85.7). The paper says FLAVA "approaches" BERT-base on several of them §4.1. It also says FLAVA is about 6× smaller in data than CLIP §4.1 (70M vs 400M). The relative differences to CLIP-400M are in fig. 4.

Paper vs code

Official code: facebookresearch/multimodal, folder examples/flava and torchmultimodal/ [project page: https://flava-model.github.io/]. The code is pinned at code_commit. Citations below use paths relative to that repository.

• Projection dimension. Paper: 512 tab. A.1. Code: default text_and_image_proj_size is 768, and the loss projects to the hidden size model.py flava.py.
• Precision. Paper: FP16 with layer norm in FP32 §3.4. Code: AMP with bfloat16 by default in the native training config pretrain_debug.yaml train.py.
• Batch size. Paper: 8192 tab. A.1. Code: 8 per process pretrain_debug.yaml; the global batch depends on world size, not stated.
• Optimizer values. Learning rate 1e-3, weight decay 0.1, warm-up 10,000 and β values match the paper pretrain_debug.yaml. Code also uses 1e-8 for ε, which the paper does not state.
• MMM applied only to matched pairs. Code drops rows whose ITM label is not 1 before the MMM losses, and before the contrastive loss, which masks rows flava.py contrastive_loss_with_temperature.py. The paper describes MMM as running on paired data without mentioning this filter §3.2.
• Separate MMM heads. Code gives MMM its own text and image heads, not the MLM or MIM heads flava.py. The paper says a multilayer perceptron predicts the masked targets §3.2.
• Masking numbers. The paper names BEiT-style block masking with no numbers §3.2 §3.3. Code: 14×14 grid, 75 patches, minimum 16 per block, aspect ratio between 0.3 and 1/0.3 flava_transform.py flava_transform.py.
• ITM negatives. Code swaps a random other caption with probability 0.1 and labels true pairs 1 and swapped pairs 0 transforms.py. Not in the paper.
• Temperature. Code: learned logit scale, initialised at ln(1/0.07), clamped to ≤ ln 100 flava.py flava.py. Not in the paper.
• Weight decay scope. Code passes all model parameters to AdamW, so biases and layer norms also decay model.py. Not stated in the paper.
• Dataset sampling. The 0.70 / 0.15 / 0.15 ratios tab. A.2 were not located in the example training or data files checked.
• Configs. Example configs are named 900m through 10b and a debug config . None is labelled as the paper's 70M PMD run, and the repository does not hold that config. The paper's reported run cannot be reproduced from the repo alone.
• Fine-tuning. Repository fine-tuning configs cover QNLI and rendered SST-2 only qnli.yaml. The VQAv2, SNLI-VE and Hateful Memes settings app. B.2 were not checked against code.
• Architecture, consistent. Pre-norm encoders §3.4 match image_encoder.py and text_encoder.py.

Pitfalls

• Table 3 column order. The header reads Vision, NLP, Multimodal, Macro, but for FLAVA_C, FLAVA_MM, the no-init row and the full row, the first two numbers are NLP then Vision. Table 4 gives the correct order (for the full model: vision 79.44, NLP 78.19). Macro values agree with Table 4, so use Table 4 for the per-modality numbers tab. 3 tab. 4.
• Appendix macro row. In the supplementary ablation table, the macro row is not the mean of its own modality averages: the full model shows 78.82, versus 75.85 from the main-paper means. For MIM it shows 28.73, versus 19.15. Use the main-paper macro values tab. C.1 tab. 3.
• Table 5 vs Table 4 for the same model. FLAVA's VQAv2, SNLI-VE and Hateful Memes scores differ between Table 5 (72.8, 79.0, 76.7) and the ablation table (72.49, 78.89, 76.09). The paper does not explain the gap. Table 5 is labelled test-dev for VQAv2 tab. 5 tab. 4.
• Batch size and scale. The 8192 batch and the 150K-iteration ablation schedule app. B.1 imply several GPUs and long runs. Compute and wall-clock time are not stated. The debug and scale configs use different step counts.
• Multiple seeds. The paper does not report seeds, variance or repeats. Treat differences of about one point as unconfirmed.
• Layer norm epsilon. Encoders use 1e-12 in code model.py (not stated in the paper). The MLM and MIM heads use 1e-5 flava.py. Match both to the code if you reproduce it.
• Masked head shape. Each MLM and MIM head has a dense layer, GELU, layer norm and a decoder tied to the vocabulary bias flava.py. The paper does not describe it.
• Global contrastive gradients. The paper reports a gain from backpropagating through all-GPU gathered embeddings §3.2 app. C. Its local-gradient variant (the open-source CLIP style) scores lower on the vision and NLP averages tab. C.1, columns 3 vs 4. Implementing the global version needs a cross-GPU gather that keeps gradients.
• Mixed-modality ablation costs. The paper's ablation rows that use DINO instead of BEiT init need an ImageNet-pretrained DINO encoder first §3.3 app. B.1. The init is an extra training run to budget for.
• Scene text. The paper says FLAVA does poorly on SST rendered as image text, because PMD has little scene text app. C. Expect weak OCR-style vision tasks.
• Architecture gains vs data. Part of FLAVA's gain over the CLIP-PMD baseline comes from architecture changes (BERT vocabulary, width 768, ViT text encoder), not only from the objectives. The paper reports +6.99 macro from these app. D tab. D.1.
• Figures. No image is reused here. Describe fig. 1 (overview), fig. 2 (architecture) and fig. 3 (PMD examples) in words only.

Lineage

• Builds on: CLIP-style contrastive loss §3.2; BEiT block masking and dVAE tokens §3.2; DINO encoder initialisation §3.3; BERT and ViT architectures §3.1; RoBERTa training recipe for the text encoder app. B.1.
• Compared against: CLIP ViT-B/16 (released 400M, and a PMD-trained variant), SimVLM base, VisualBERT, UNITER base, ViLBERT, LXMERT, UniT, VL-BERT tab. 5 tab. 1.
• Superseded by: not stated in content on this wiki.