Generative Multimodal Models are In-Context Learners

In plain words

Emu2 is a very large model, 37 billion parameters, that treats pictures and text as one stream. Each picture is turned into a small fixed set of numbers that sit in the sequence next to ordinary words. The model is trained to guess what comes next: a word, or the next picture.

Because it learns from long mixed sequences, you can put a few example pairs of pictures and answers into the prompt, and the model picks up the pattern for a new question without being retrained. This is the "in-context learning" in the title. The authors also show it can follow instructions about pictures and answer questions about them.

The same design lets it draw. The picture slots it predicts can be handed to an image decoder, so it can produce images, including images composed from reference pictures and location hints. The authors present it as a general base model for many multimodal tasks.

TL;DR

• Emu2 is a 37B generative multimodal model. Each image becomes 64 visual embeddings, interleaved with text, and one autoregressive model predicts the next text token or the next image embedding §2.1 §2.2.2.
• Large-scale interleaved pretraining gives few-shot in-context learning on VQA-style tasks, without a task-specific architecture §1 §3.1 tab. 1.
• Two instruction-tuned variants: Emu2-Chat for dialogue and grounding, and Emu2-Gen for controllable image generation §2.3.
• Author-reported headline results: 68.8 VQAv2 at 16 shots tab. 1; 84.9 VQAv2 and 48.5 MM-Vet for Emu2-Chat tab. 2.
• The code is Apache-2.0, but the repository's README lists the model weights as non-commercial. See Pitfalls.

Problem

Multimodal systems usually need a task-specific architecture and a large supervised training set for each new task, which is hard to scale §1. The paper asks whether a scaled generative multimodal model can solve new tasks from a few demonstrations or simple instructions, as large language models do §1.

The authors identify model scale and large-scale multimodal sequences as the main levers §1 §5. They also state that in-context ability may fail on complex scenes, for example counting in a crowd, and that it should improve with more and better training data and larger models app. A.

Method

Emu2 has three parts: a visual encoder (initialised from EVA-02-CLIP-E-plus), a multimodal model (initialised from LLaMA-33B), and a visual decoder (initialised from SDXL-base). Video decoding uses a diffusion decoder initialised from Stable Diffusion 2.1 §2.1 §2.2.3 fig. 2. Compared with its predecessor Emu [arXiv:2307.05222], Emu2 connects the encoder to the language model by pooling and a linear projection, without the extra C-Former module §2.1.

1. Encode each image into a fixed set of embeddings. Patch features from the visual encoder are average-pooled to an 8×8 grid, giving 64 embeddings per image. A linear projection maps them to the language model's width §2.1 §2.2.2. The widths used by the code are 1792 for the encoder and 6656 for the language model emu_conf.py emu.py.
2. Interleave the sequence. Each image's embeddings sit between [IMG] and [/IMG]. Sequences start and end with <s> and </s>. For video, a [VIDEO] token starts the sequence of frame embeddings app. B.1.
3. Predict the next element. Text positions use a cross-entropy loss on the next token. Image positions use a regression loss on the next visual embedding §2.2.2. The paper gives no formula for either loss and does not state how they are weighted §2.2.2.
4. Pretrain in two stages. Stage 1 trains on image-text and video-text pairs at 224 px with only the captioning (text) loss, then continues at 448 px. Stage 2 freezes the visual encoder and trains the projection and language model on image-text, video-text, interleaved, grounded and language-only data §2.2.2 app. B.1.
5. Pretraining data. Image-text pairs from LAION-2B and CapsFusion-120M, video-text pairs from WebVid-10M, interleaved data from Multimodal-C4 and YT-Storyboard-1B, grounded pairs from GRIT-20M and a CapsFusion-based grounded set, and language-only text from The Pile §2.2.1 app. B.1.
6. Visual decoder. The SDXL-base U-Net is trained to decode the 64 embeddings back to a 1024×1024 image. The visual encoder and the SDXL VAE are frozen, and only the U-Net is updated. The decoder runs without the language model during training. It is trained with 10% of image embeddings dropped, for classifier-free guidance §2.2.3 app. B.3. The video decoder inserts 1-D temporal convolutions after each 2-D convolution and extends spatial attention to spatial-temporal attention §2.2.3.
7. Chat instruction tuning (Emu2-Chat). The sequence has the form <system> [USER]: <instruction> [ASSISTANT]: <answer>. Only answer tokens are supervised §2.3.1. Academic-task data and multimodal chat data use different system messages §2.3.1 app. C.1. During this stage each image is pooled to 16×16 = 256 embeddings instead of 8×8 §2.3.1 chat.py.
8. Controllable generation (Emu2-Gen). Training sequences contain text phrases, bounding boxes drawn as images on a black background, and object images. The regression loss applies only to the visual embeddings of the last image. The visual encoder is frozen. Tokens of entities and boxes are randomly dropped, and object images get background and crop augmentation §2.3.2.

Pseudocode for step 1, following the released inference code:

# Turn one image into 64 visual embeddings  [§2.1] [code: emu.py#L78-L90]
# Input:  image (B, 3, 448, 448)
# Output: embeddings (B, 64, 1792)
def encode_image(image):
    # Run the visual encoder and drop the class token  [code: emu.py#L81-L82]
    patches = visual_encoder(image)[:, 1:, :]   # (B, 1024, 1792)
    # Lay the 32x32 patch tokens out as a grid  [code: emu.py#L83-L85]
    grid = reshape_to_grid(patches)             # (B, 1792, 32, 32)
    # Average-pool to 8x8 with stride 4  [§2.1] [code: emu.py#L87-L88]
    pooled = avg_pool(grid, window=4, stride=4)  # (B, 1792, 8, 8)
    # Flatten the 8x8 grid into a sequence of 64  [code: emu.py#L89]
    return flatten_to_sequence(pooled)          # (B, 64, 1792)

Pseudocode for step 3, from the paper's description (no code is released for training):

# Unified next-element objective over an interleaved sequence  [§2.2.2]
# Input:  logits and predicted visual embeddings at each position
# Output: two loss terms; their combination is not stated
def pretraining_losses(logits, pred_visual, target_visual, tokens):
    # Text positions: cross-entropy on the next token  [§2.2.2]
    text_loss = cross_entropy(logits[text_pos], tokens[text_pos])
    # Image positions: regression on the next visual embedding  [§2.2.2]
    image_loss = regression(pred_visual[image_pos], target_visual[image_pos])
    # Stage 1 uses only the text term  [§2.2.2]
    return text_loss, image_loss

Pseudocode for the image-generation path, following the released inference code. Each pass re-runs the full language model over the sequence:

# Decode 64 image embeddings with the LM  [§2.1]
# Input:  prompt ending in the image-start token
# Output: 64 embeddings, later mapped to the visual decoder's input
# [code: emu.py#L109-L153]
def generate_image_embeddings(prompt):
    # Nothing has been decoded yet  [code: emu.py#L109]
    decoded = []
    # One full pass per slot; no key/value cache  [code: emu.py#L133]
    for step in range(64):
        # Feed decoded slots back in as inputs  [code: emu.py#L129-L131]
        hidden = language_model(prompt, decoded)
        # Read hidden states of the slots so far  [code: emu.py#L140-L147]
        decoded = project_down(hidden_at_slots(hidden, step + 1))
    return decoded                              # (B, 64, 1792)

Key equations: none. The paper has no numbered equations, and the losses are described only in prose §2.2.2.

Hyperparameters

Values in the tables are as stated in the paper. "Code" rows come from the released inference code, which contains no training code.

Pretraining (base model)

Visual decoder

Instruction tuning (Emu2-Chat)

Controllable generation (Emu2-Gen)

Inference

Prompt templates, as quoted in app. E.1. Few-shot: "[image] based on the picture, [question] short answer:". Zero-shot: "[image] based on the picture, answer in one word or phrase. [question] short answer:".

Results

All numbers are author-reported. Setups: few-shot evaluation with the RICES selection app. E.1; instruction-tuned evaluation with beam size 5 app. E.2; generation evaluated on MS-COCO and DreamBench §3.3 app. E.3. Benchmark splits and metrics are listed in app. E (summary table).

Few-shot, zero-shot to 16 shots tab. 1. Baselines marked with * are not Emu2. Shot 0* means two text shots and zero image shots, following Flamingo.

The text reports that Emu2 beats Flamingo-80B and IDEFICS-80B on VQAv2, VizWiz and TextVQA at every shot count from 4 upward, with 37B versus 80B parameters §3.1. At 0 shots Emu2 is below both on VQAv2 tab. 1.

Instruction-tuned: VQA and LMM benchmarks tab. 2. * marks benchmarks whose training split the model was trained on. Emu2-Chat is the last row.

Appendix A names GPT-4V at 67.7 on MM-Vet, above Emu2's 48.5, and describes Emu2 as state of the art among public models app. A.

Referring expression comprehension tab. 3. Metric: accuracy, per the benchmark summary app. E. CogVLM is greyed in the table because it is a specialist grounding model.

Zero-shot text-to-image generation on MS-COCO tab. 4. CLIP-I uses CLIP-ViT-L and compares generated and reference images; CLIP-T uses CLIP-ViT-B and compares image and prompt. The authors evaluated 30k sampled validation prompts. DALL-E 2 and DALL-E 3 CLIP-T is on 4,096 samples.

Zero-shot subject-driven generation on DreamBench tab. 5. Each prompt generates four images, 3,000 in total §3.3. Rows marked * are zero-shot methods.

Autoencoding: on MS-COCO, the visual encoder plus visual decoder reach CLIP-I 0.907 §3.3. No table is given for this number, and the paper compares it only qualitatively with SEED and Emu fig. 3.

Paper vs code

The repository is github.com/baaivision/Emu (the paper's author-footnote link). The Emu2 code is under Emu2/ at commit d9f12f431a4ea8219bd80ed2aa3173b7fa26fb12, and the repository's LICENSE file is Apache-2.0. The code contains inference only: model definition, chat wrapper, diffusion decoder and demos. No training loop, optimiser, loss, data pipeline or evaluation script was found in the Emu2 directory.

• Consistent: 8×8 pooling to 64 embeddings for pretraining §2.1 matches the default n_query = 64 emu_conf.py and the pooling in emu.py.
• Consistent: chat-stage 16×16 pooling §2.3.1 matches n_query=256 in the instruct configuration chat.py.
• Consistent: the system message texts app. C.1 match the constants constants.py.
• Consistent: the visual decoder defaults to 50 Euler steps and guidance scale 3.0 app. E.3 diffusion.py, and outputs 1024×1024 diffusion.py.
• Discrepancy: the paper says the grounding coordinates are 224 discrete tokens app. B.1. The code adds patch_index_0000 to patch_index_0256, which is 257 tokens, by default lm.py lm.py. Which one the released weights use cannot be settled from the code alone.
• Discrepancy: the encoder is "EVA-02-CLIP-E-plus" in the paper §2.1, and the code names it eva-clip-4b-14-x emu_conf.py. The code configuration (width 1792, 64 layers, 448 px) is what the code builds; it is not verified that the two names refer to the same weights.
• Discrepancy: the paper says the language model is LLaMA-33B §2.1. The shipped configuration has hidden size 6656, 60 layers and 52 heads config.json, which matches the 33B-class size and the 6656 projection width emu.py.
• Not verifiable from code: every pretraining, visual-decoder and fine-tuning hyperparameter in the Hyperparameters section, because no training code is present.
• Not verifiable from code: the few-shot RICES selection and benchmark numbers. A grep for RICES and few-shot in Emu2/ found only the README and a tokenizer file.
• Not present: a standalone video decoder module. emu/diffusion.py is the image decoder.

Model weights are not part of the license of this code. The Emu2 README lists the model type and license as "Non-commercial license" and says the weights are initialised from LLaMA README.md. Do not treat the published weights as usable for commercial work based on this card.

Pitfalls

• Weights license. The code is Apache-2.0, but the README lists the model as non-commercial README.md. Any commercial use of the weights needs a separate check. This card cannot clear it.
• Conflicting LLM token counts. Stage 2 language-only data is "3.8B tokens" in the main text §2.2.2 and "3.6 billion tokens" in the appendix app. B.1. Pick one before budgeting the run.
• Conflicting warm-up for the visual decoder. The text gives 2,000 warm-up steps and 6,000 decay steps §2.2.3. The appendix table gives 2,500 warm-up steps out of 8,000 app. B.3.2.
• Conflicting warm-up for Emu2-Gen. The text warms up over the first 100 steps §2.3.2. The appendix table gives a warm-up ratio of 0.0 app. D.2.
• Unstated losses. The regression loss form and the weighting between the text and image terms are not stated §2.2.2. The paper gives no further detail on the regression target. A re-implementation has to pick these.
• Unstated pooling for other resolutions. The pooling stride is derived from the grid size, so 448 px inputs give a 4×4 window and 224 px inputs a 2×2 window emu.py. This is inferred from code, not stated in the paper.
• Dropped class token. The code removes the first token of the encoder output before pooling emu.py. The paper does not say this.
• Long sequences fill quickly. Chat-stage images use 256 tokens each chat.py, and training sequences are capped at 2,048 tokens with the excess truncated §2.3.1. A single 2,048-token sequence can therefore hold at most eight images before any text.
• Generation cost. The released code runs the full language model 64 times to decode one image, once per output slot, with no key/value cache emu.py. Budget latency accordingly.
• Short default answers. The chat generation default is max_new_tokens = 10 chat.py. Longer answers are cut off unless the caller raises it. The paper does not state an answer-length limit.
• Zero-shot prompt matters. The paper says the plain prompt made the model write sentences, so a modified prompt was used for zero-shot app. E.1. Reproduce that prompt exactly to match the reported zero-shot numbers.
• Few-shot results depend on example selection. Results use RICES app. E.1. Random examples are not what the reported numbers measure.
• Chat-data truncation. Excess tokens beyond 2,048 are truncated without a stated rule for which part is kept §2.3.1.
• Padding side. The model sets padding and truncation to the left emu.py. Batched inputs must use the same convention.
• Figures and numbering. Section, table and figure numbers follow the fetched arXiv LaTeX source (v2, dated 2024-05-08). Appendix table numbers are not cited because the numbering was not verified.

Lineage

• Builds on: Emu (arXiv:2307.05222), which used a C-Former between encoder and language model. Emu2 drops the C-Former and uses mean-pooling plus a linear projection §2.1. Emu2 also initialises its components from EVA-02-CLIP-E-plus, LLaMA-33B and SDXL-base §2.1.
• Compared against: Flamingo (9B, 80B), IDEFICS (9B, 80B), Kosmos-1, Kosmos-2, BLIP-2, InstructBLIP, Shikra, Qwen-VL, LLaVA-1.5, CogVLM and Emu-I for understanding tab. 1 tab. 2 tab. 3; MUSE, Imagen, DALL-E 2, DALL-E 3, SDv1.5, SDXL, GILL, SEED and Emu for generation tab. 4; and Kosmos-G, BLIP-Diffusion, SuTI and DreamBooth for subject-driven generation tab. 5.
• Superseded by: not stated in this wiki.