PaLI: A Jointly-Scaled Multilingual Language-Image Model

In plain words

Many image-and-text models handle one kind of job, such as captioning a photo or answering a question about it, and were built mainly for English. PaLI takes a different route: one model reads an image plus a text prompt and writes text back. Captioning, question answering and reading text in a photo all use the same interface; only the prompt changes.

The model reuses two large pretrained parts instead of learning everything from scratch: a language model that already knows many languages, and a large image encoder. The authors argue that earlier large vision-language models scaled the language side far more than the vision side, so they scaled both and trained them together. The training data covers more than 100 languages. The card's results section lists state-of-the-art claims on several captioning and VQA benchmarks, all author-reported.

TL;DR

• PaLI is an encoder-decoder Transformer. Image patches go through a ViT, and the text side is initialised from mT5. The output is generated text, with no task-specific heads §3.1.
• It is trained on a mix of eight pretraining tasks over a WebLI-based corpus in more than 100 languages §3.2 app. A.2.
• The largest model, PaLI-17B, uses a new 4B-parameter ViT (ViT-e) and mT5-XXL. The authors report state of the art on several captioning and VQA benchmarks §4.
• Relevant if you want a multilingual, generative vision-language model with an open-vocabulary output. Retraining is costly: the pretraining data (WebLI) is not released, and the largest run used 1,024 TPUv4 chips for 7 days app. A.1.

Problem

Vision-language models were often English-only, and many relied on classification heads over a fixed answer set or on external detectors §1 §2. Prior large vision-language models also tended to scale the language side far more than the vision side, which the authors argue leaves vision under-capacity §1 §3.1. Reusing unimodal checkpoints reduces training cost but needs a single interface for many tasks §3.1.

Method

Interface. Every task is cast as "image and text in, text out". There are no task heads; the task is selected by a text prompt §3.1. Prompts for each pretraining task are listed in app. A.2.

Architecture §3.1 fig. 1:

1. The image goes through a plain Vision Transformer. Its output patch features are not pooled §3.1.
2. The visual tokens are passed to a text encoder-decoder, which is an mT5 model. The text says the visual tokens are fed to the encoder, and also that they are passed to the encoder-decoder via cross-attention §3.1. See Pitfalls.
3. Decoding produces text tokens over the 250k mT5 vocabulary §4.2.

Model variants app. A.1:

• PaLI-3B: ViT-G (1.8B) and mT5-Large (1B). Total 3.0B.
• PaLI-15B: ViT-G (1.8B) and mT5-XXL (13B). Total 14.8B.
• PaLI-17B: ViT-e (4B in text; 3.9B in the size table) and mT5-XXL. Total 16.9B.

ViT-e §3.1 app. A.1: the authors train a 4B-parameter vanilla ViT with the same recipe as ViT-G, with two changes. First, the final learning-rate decay phase is run two times: one run uses inception-crop augmentation and the other does not. The two resulting weight sets are then combined by averaging, which the paper calls "souping" §3.1. Second, ViT-e is wider and deeper than ViT-G (see the hyperparameter table below).

Pretraining data (WebLI) §3.2 app. B: about 10B images and 12B alt-texts in 109 languages. Additional OCR annotations were extracted for all images. The authors keep the top 10% of image-text pairs by cross-modal similarity, about 1B examples. Near-duplicate images are removed against the train, validation and test splits of 68 common vision datasets.

Pretraining mixture §3.2 app. A.2 tab. 6 for the ablation of each part: eight tasks, with training examples per task listed in the mixture table below. Each task uses a template prompt and teacher-forced cross-entropy over the output tokens §3.2.

1. Span corruption on text-only data, after mT5. 15% of tokens are corrupted with sentinel spans.
2. Split-captioning on WebLI alt-text. Each alt-text is split at a random point into two parts; the first is input and the second is the target.
3. Captioning on CC3M-35L: CC3M translated into 35 languages (English plus 34), target is the alt-text in the given language.
4. OCR on WebLI OCR text: the target is the concatenated OCR text in the language of the alt-text.
5. English and cross-lingual VQA on VQ²A-CC3M-35L, with English answers only.
6. English and cross-lingual visual question generation on the same triplets.
7. English-only object-aware VQA, with questions built from Open Images object labels.
8. Object detection as text. Targets are box coordinates in the order y_min, x_min, y_max, x_max over 0–999, plus class labels app. A.2.

Training schedule §3.3 app. A.1:

• Phase 1: one epoch over the 1.6B-example mixture at 224 px. The vision encoder is frozen; only the language component is updated.
• Phase 2 (PaLI-17B only): 10k steps at 588 px with batch 1024, covering 10M examples, on a reduced mixture of OCR, CC3M-35L and VQ²A, weighted equally. All parameters are updated. The patch size stays the same and the position embeddings are upsampled bilinearly app. A.1.

# PaLI inference for one image and one text prompt  [§3.1] [§4.2]
# Output is text; the decoding strategy is not stated in the paper.
def answer(image, prompt):
    # Encode the image with the ViT, with no pooling on its output  [§3.1]
    visual_tokens = vit(image)                  # (N_patches, d_vit)
    # Prompt is tokenized with the mT5 vocabulary  [§4.2]
    prompt_tokens = tokenize(prompt)            # 250k mT5 vocabulary
    # Combining visual and text tokens in the encoder input is ambiguous
    # in the paper  [§3.1]; this sketch assumes concatenation.
    encoder_input = concat(visual_tokens, prompt_tokens)
    encoder_states = encoder(encoder_input)
    output = []
    for step in range(max_length):
        # Next-token choice (greedy, beam, etc.) is not stated  [§4.2]
        next_token = decoder(encoder_states, output)
        if next_token == end_of_sequence:
            break
        output.append(next_token)
    return detokenize(output)

Hyperparameters

Values for the main models unless stated. "Not stated" means the paper does not give it.

Pretraining mixture app. A.2. Examples per task, in millions:

Fine-tuning per task app. A.3:

Fine-tuning resolution: 490 px for PaLI-3B and PaLI-15B; 588 px for PaLI-17B §4. Crossmodal-3600 captioning used 224 px checkpoints fine-tuned on COCO-35L at 224 px §4.1.

Results

Author-reported. Setup: fine-tuned on each benchmark's train split, evaluated on the listed split. Baselines are the models the paper compares against at the time of writing §4.

Image captioning (CIDEr) tab. 1:

PaLI-17B is reported to be the best cross-entropy-trained result on COCO without CIDEr optimisation §4.1.

Multilingual captioning, Crossmodal-3600 (CIDEr, 35-language average) tab. 2: PaLI-17B 53.6; PaLI-3B 47.0; prior reference model (0.8B) 28.9.

Visual question answering (accuracy) tab. 3:

Cross-lingual and multilingual VQA tab. 4: on xGQA (English answers) PaLI-17B scores 54.2 in English against 41.5 for the MPT baseline; on MaXM it scores 56.4 in English against 36.6. The paper reports per-language values for both benchmarks tab. 4.

Zero-shot ImageNet (top-1, generative scoring) tab. 5: PaLI-17B 72.11 ImageNet, 81.97 ImageNet-R, 44.70 ImageNet-A, 63.83 ImageNet-Sketch, 64.46 ImageNet-v2, 42.62 ObjectNet. Flamingo-80B 1-shot is 71.9 on ImageNet tab. 5 §4.4.

Language-only app. C.1: PaLI-17B vs mT5-XXL (our setting). SuperGLUE 88.2 vs 89.3; XNLI 84.9 vs 84.5; XQuAD 81.8 / 66.0 vs 82.6 / 66.6; TyDiQA-GoldP 81.2 / 66.5 vs 81.6 / 66.3.

Scaling §4.5 fig. 2: moving from ViT-G to ViT-e adds +3.2 average points over seven V&L benchmarks, with about +2B vision parameters. Moving from mT5-Large to mT5-XXL adds +3.1 with about +12B language parameters. Phase 2 at 588 px adds +2.0.

Ablation of pretraining mixture (PaLI-3B, 200M data coverage) tab. 6: the full mixture gives COCO 141.4, TextVQA 41.6, VQAv2 76.0. Removing split-captioning drops COCO by 1.0 and TextVQA by 2.8. Removing OCR drops TextVQA by 1.7.

Headline comparisons in text §4: VQAv2 84.3 is +2.2 over Flamingo under open-vocabulary generation and +0.3 over BEiT-3 under closed-vocabulary classification §4.2.

Paper vs code

The paper names big_vision (Apache-2.0) as the codebase used to pretrain ViT-e app. A.1. The code field above points to it, but that repository is a general vision codebase, not the PaLI model or its training pipeline. Findings:

• ViT-e variant: big_vision defines the "e" size as width 1792, depth 56, MLP 15360, 16 heads vit.py. This matches the ViT-e row of the architecture table app. A.1.
• PaLI status in big_vision: the README lists PaLI as "(partial)" with no code or config links README.md.
• A search for PaLI-specific model code, training configs, the mT5 integration, or the WebLI data pipeline found none. Unverified against code: all PaLI hyperparameters, the mixture, and the phase-2 schedule. The paper is the only source for these.
• Model size: §3.1 says ViT-e has 4B parameters; the size table says 3.9B image encoder and the ViT table says 3926M app. A.1.

Pitfalls

• Visual token interface is ambiguous. §3.1 says visual tokens are fed to the encoder and also that they reach the encoder-decoder through cross-attention. The paper does not say whether they are concatenated with the text in the encoder input or are only attended by the decoder. No projection layer between ViT output and the mT5 width is stated. Test the ambiguity before committing to one design.
• Decoding is not stated. Beam width, length penalty and maximum length are not given §4.2. VQA results are exact-match on generated text, so decoding choices change the score §4.2.
• VQA scoring. Only exact match counts as correct §4.2. The paper does not state answer normalisation (case, punctuation, whitespace). Limitations also note synonyms and paraphrases are counted as wrong app. E.
• Crossmodal-3600 normalisation. The paper normalises Unicode, tokenises and removes punctuation before CIDEr, and uses an unnamed neural segmenter for Chinese, Japanese, Korean and Thai §4.1. Reproducing the numbers needs those tools.
• OCR input changes results a lot. TextCaps reaches 160.4 CIDEr with detected OCR tokens and 135.4 without them app. C.5. TextVQA shows the same effect: 73.06 with OCR and 58.80 without app. C.5. The OCR service is "publicly available" and not named §4.1 app. A.2.
• Phase 2 mixture is described with a macro. The text says "\qsq" in the high-res phase, which resolves to VQ²A. Whether the CC3M-35L and VQ²A parts use the same 35 languages as in phase 1 is not stated.
• Detection output format. Coordinates are integers 0–999 in y_min x_min y_max x_max order. Quantisation details beyond that are not stated app. A.2.
• Zero-shot classification is expensive. It scores all 1,000 class strings per image through the decoder app. A.3, so cost grows with the number of classes.
• Data is not public. WebLI is not released. The top-10% threshold is tuned on cross-modal similarity, but the threshold value is not stated app. B. Near-duplicate removal against 68 datasets is described but not reproducible from the paper alone §3.2 app. B.
• Pretraining batch size for PaLI-3B and PaLI-15B is not stated app. A.1.
• Scale of compute. The 17B run used 1,024 TPUv4 chips for 7 days, plus 512 chips for 3 days in phase 2 app. A.1. Few groups can reproduce it.
• Fine-tuning search was limited. The paper says it did not sweep all combinations app. A.3, so the per-task learning rates in the table are the ones it reports.
• No variance is reported. Seeds, repeats and confidence intervals are not given.
• Language quality varies by language. The authors note output quality differs across languages, and fine-tuning on English-only data can erode multilingual ability [ethics statement] app. E.

Lineage

• Builds on: mT5 (the language component, §3.1), ViT-G/ScalingViT (the 1.8B vision component, §3.1), and ViT (the base architecture, §3.1). None of these have cards on this wiki yet.
• Uses LiT for the zero-shot evaluation of ViT-e app. C.4.
• Compared against: Flamingo, CoCa, GIT and GIT2, SimVLM, BEiT-3, OFA, LEMON, and Unified-IO §2 tab. 1 tab. 3.
• Superseded by: not stated in content already on this wiki.