Paper · 2024-03-08 · cs.CL, cs.AI
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini 1.5 Pro (a sparse mixture-of-experts Transformer) and Flash (a dense model distilled from Pro) take text, video, audio and code in one context of up to millions of tokens. Author-reported: near-perfect needle recall to 1M tokens and 99.2% at 10M text tokens; Pro matches or beats Gemini 1.0 Ultra on most core benchmarks. Architecture and training details are largely not stated.
Mixture of experts (MoE)Multimodal language modelIn-context learningKnowledge distillationlong-contextmultimodalmixture-of-expertsevaluationmodel-report
AI-compiled by claude-haiku-5-5 on 2026-10-24, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Gemini 1.5 is a family of AI models from Google that can read a very large input in one go: text, video, audio and code, mixed together. The report describes two models. Pro is the bigger one. It is built so that each input uses only part of the model's parameters, which lets the total size grow while the work done per input stays the same. Flash is the lighter one: a smaller model trained to imitate Pro, built for lower latency when serving.
The authors care most about how much the model can hold at once. Their tests hide a short fact somewhere in a huge pile of text, or in a multi-hour video, and ask the model to find it. Pro finds it almost every time, up to about ten million tokens. They also give the model a grammar book for a language with almost no presence on the web, and check whether it can translate using only that book.
The report is mostly a list of results. It says little about how the models were built or trained, so treat the numbers as the authors' own measurements rather than as a recipe.
TL;DR
- Gemini 1.5 Pro is a sparse mixture-of-experts Transformer; Flash is a dense Transformer distilled from Pro §3.1 §3.2 app. model card.
- Context: tested to 10M tokens for text and code, 10.5 hours of video and up to 22 hours of audio input §5.2.1 app. model card.
- Author-reported: 99.2% text needle recall at 10M tokens; near-human in-context translation of Kalamang from one book §5.2.1 §5.2.2.
- Pro matches or beats Gemini 1.0 Ultra on most core text, vision and audio benchmarks; it trails Ultra on a few §6.1 §6.2 §6.3.
- Not open: no weights, no code, and no parameter count, data mixture or training hyperparameters.
Problem
The paper compares against models with 128k (GPT-4 Turbo) and 200k (Claude 2.1) context windows, and notes Claude 3 offers up to 1M §1. Gemini 1.5 aims to handle "multiple millions" of tokens while keeping recall and reasoning quality, across modalities, at practical latency §1 §5. The report argues that efficiency at all context lengths matters because output time per character dominates request latency for chained or agentic prompts §3.3. The motivating cases are: long documents, hours of audio or video, and large codebases.
Method
The report does not give a training algorithm, so this section covers what it states. Pseudocode for the evaluation procedure is given after the overview.
Gemini 1.5 Pro is a sparse mixture-of-experts (MoE) Transformer §3.1 app. model card. A learned routing function sends each input to a subset of the parameters, so total parameter count can grow while the active count per input stays fixed §3.1. The paper says architecture changes allow inputs up to 10 million tokens "without degrading performance" §3.1. This is the authors' claim; the changes themselves are not described §3.1.
Gemini 1.5 Flash is a dense Transformer decoder app. model card with the same long-context and multimodal interface as Pro §3.2. The report states three differences from Pro:
- it computes attention and the feed-forward block in parallel §3.2;
- it is online-distilled from Pro, meaning it learns from Pro during training rather than from a fixed dataset alone §3.2 app. model card;
- it is trained with higher-order preconditioned optimisers §3.2. Which optimiser, and its settings, are not stated §3.2.
Serving. The authors measure the average time per output character for English, Japanese, Chinese and French, using 10,000-character inputs §3.3, table on time per output character. Each figure is the mean over 32 queries, and the minimum across those means is reported §3.3.
Training. Both models were trained on multiple 4096-chip pods of TPUv4, spread over several datacentres, on multimodal and multilingual data §4. Training used JAX and ML Pathways app. model card. Instruction tuning used paired instructions and responses, then tuning on human preference data §4.
# Text needle-in-a-haystack recall [§5.2.1]
# Input: essays (repeated to fill context), context_length, depths
# Output: recall, the fraction of correct magic-number answers
def needle_recall(model, essays, context_length, depths):
# Fill the context by concatenating and repeating the essays
haystack = fill_with_essays(essays, context_length)
hits = []
for depth in depths:
# Insert the needle at a linearly spaced depth; city and number
# are varied per query [§5.2.1]
prompt = insert_needle(haystack, depth, city, number)
# Ask the model for the magic number of the chosen city [§5.2.1]
answer = model.generate(prompt + question_for(city))
hits.append(number in answer)
return mean(hits)Long-context diagnostics. Next-token negative log-likelihood is recorded at each token position on held-out long documents (up to 1M tokens) and on concatenated code repositories (up to 10M tokens) §5.2.1. A power law L(x) = \alpha x^\beta + \gamma is fitted to the cumulative curve §5.2.1.
Long-context QA. The Les Misérables book (1,462 pages, 710K tokens) is given in full as context. A 100-question set is judged by a side-by-side auto-rater, and ranked with a Bradley-Terry model §5.2.2. The win probability is P(M_A \text{ better than } M_B) = e^{\beta_A} / (e^{\beta_A} + e^{\beta_B}) §5.2.2.
Hyperparameters
The report gives almost no hyperparameters. The model card states that the parameter count, model stats and compute requirements are "not reported" app. model card. The table lists every number that is stated.
| Name | Value | Where | Notes |
|---|---|---|---|
| Pod size | 4096 TPUv4 chips per pod | §4 | several pods across datacentres |
| Parameter count (Pro, Flash) | not stated | app. model card | "Model Stats: not reported" |
| Layers, experts, width, routing top-k | not stated | §3.1 | MoE routing described only in general terms |
| Flash distillation loss and schedule | not stated | §3.2 | "online distilled" from Pro |
| Flash optimiser | higher-order preconditioned, unnamed | §3.2 | no settings given |
| Learning rate, batch size, warmup, tokens seen | not stated | §4 app. model card | |
| Pre-training data mixture | not stated beyond domains | §4 | web, code, image, audio, video |
| Context length (product) | up to 1M (Pro, Flash); 2M reported for Flash in tests | §5.2.1 | the paper's claim is 10M for research tests |
| Max input per modality | video up to 2 h, audio up to 22 h, text strings | app. model card | from the model card |
| Serving latency measurement | 10,000-character inputs; mean over 32 queries; minimum taken | §3.3 | May 2024 API measurements |
| Needle depths | linearly spaced; city and number varied per query | §5.2.1 | Paul Graham essays as haystack |
| Video haystack | 1 frame per second; 37,994 frames; about 9.9M tokens | §5.2.1 | seven back-to-back copies of the AlphaGo documentary |
| MTOB resources | about 250k tokens total; half-book setting about 100k | §5.2.2 | ~500-page grammar, ~2000-entry wordlist, ~400 parallel sentences |
| Les Misérables QA | 100 questions; 710K-token full context vs RAG with 4k tokens | §5.2.2 | RAG: TF-IDF index, top passages by cosine similarity |
| Instruction-following sets | 1326 long prompts (307 words, ~8 instructions); 406 short (66 words, ~5) | §6.1 | human-rated, internal |
| Function-calling splits | BFCL, excluding Java, JavaScript and execution-based splits | §6.1 | annotations and metrics partly corrected by authors |
Results
Author-reported throughout. The setup in each row is the one the paper gives. Where the paper shows a 1.0 Ultra or 1.0 Pro column, it is the baseline for that row. Numbers in the core-capability table come from the paper's table §6.1; the paper says all 1.5 numbers are after instruction tuning, except those marked * (pretrained models) §6.1.
Long-context recall and use
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| Text needle, 1M tokens | recall | Claude 2.1: 98% at 200k; GPT-4 Turbo tested to 128k | Pro: 100% to 530k, >99.7% at 1M, 99.2% at 10M; Flash: 100% to 2M | §5 §5.2.1 |
| Video needle, 10.5 h (~9.9M tokens) | retrieval across all depths | GPT-4V: only the first ~3 minutes | Pro: retrieves at all tested depths; Flash: >99.8% to 2M | §5.2.1 |
| Audio needle | recall | not compared | Flash: 99.1% to 2M | §5 |
| Text NLL, long documents | power-law fit | Gemini 1.0 Pro improves only to 32K | Pro: decreasing to 1M; Flash: to 1M | §5.2.1 |
| Code NLL, repositories | power-law fit | not compared (no logits for other APIs) | Pro: decreasing to 10M, with a deviation near 10M | §5.2.1 |
| Les Misérables, full book vs RAG | side-by-side preference | Pro with 4k-token RAG; GPT-4 Turbo with 4k RAG | Pro full context preferred in 78% of cases (P=0.7795) and 83% vs GPT-4 Turbo RAG | §5.2.2 |
| Kalamang to English, half grammar | human score 0-6 (BLEURT) | GPT-4 Turbo 2.04; Claude 3 Opus 3.74 | Pro 4.14 (BLEURT 63.9) | §5.2.2 table |
| Kalamang to English, full book | human score 0-6 (BLEURT) | human learner 5.52 (70.3) | Pro 4.00 (64.6); Flash 3.14 (57.4) | §5.2.2 table |
| English to Kalamang, full book | human score 0-6 (chrF) | human learner 5.60 (57.0) | Pro 5.46 (59.0); Flash 4.66 (52.0) | §5.2.2 table |
The Kalamang "0-shot" setting, with no book in context, is near random for all models tested, including Pro (0.18 on kgv to eng; 0.00 on eng to kgv) §5.2.2.
Core text
| Benchmark (setting) | 1.0 Ultra | 1.5 Flash | 1.5 Pro | Source |
|---|---|---|---|---|
| GPQA (Pro 0-shot) | 35.7% (4-shot) | 39.5% (0-shot) | 46.2% (0-shot) | §6.1 table |
| MATH (4-shot Minerva) | 53.2% | 54.9% | 67.7%; 77.9%* maj@64 | §6.1 table |
| AMC 2022-23 (4-shot) | 30% | 34.8% | 46.4% | §6.1 table |
| GSM8K (11-shot) | 88.9% | 86.2% | 90.8% | §6.1 table |
| MMLU (5-shot) | 83.7%; 90.0%* maj@32 | 78.9% | 85.9%; 91.7%* maj@32 | §6.1 table |
| BIG-Bench Hard (3-shot) | 83.6% | 85.5% | 89.2% | §6.1 table |
| DROP F1 (variable shots) | 82.4 | 78.4 | 74.9 | §6.1 table |
| HellaSwag (10-shot) | 87.8% | 86.5% | 93.3% | §6.1 table |
| HumanEval (0-shot) | 74.4% (PT) | 74.3% | 84.1% | §6.1 table |
| Natural2Code (0-shot) | 74.9% | 77.2% | 82.6% | §6.1 table |
| WMT23, BLEURT (1-shot) | 74.4 (PT) | 74.1 | 75.3 | §6.1 table |
| MGSM (8-shot) | 79.0% (PT) | 82.6% | 87.5% | §6.1 table |
| PhysicsFinals, 61 problems (internal) | 41.0% (25 correct) | 57.4% | 63.9% (39 correct) | §6.1 table, §7 |
| HiddenMath, 179 problems (internal) | 11.2% | 6.7% | 20.1% | §6.1 table, §7 |
| Functional MATH (December snapshot) | 55.8% | 53.6% | 64.6% | §6.1 table, §7 |
* pretrained model, not instruction-tuned §6.1.
Pro scores below Ultra on DROP, so "beats Ultra" is not universal §6.1. The paper reports the Functional MATH reasoning gap as 20.3% for Pro (81.1% on the original problems, 64.6% on the modified ones), against 25.1% for 1.0 Ultra §7.
Function calling (BFCL, excluding Java, JavaScript and execution splits; §6.1 table): weighted average 67.8% (1.0 Pro), 81.8% (Flash), 88.4% (Pro). Parallel functions: 38.5%, 73.5%, 88.5%.
Instruction following (human-rated, §6.1 table):
| Set | Metric | 1.0 Pro | 1.0 Ultra | 1.5 Flash | 1.5 Pro |
|---|---|---|---|---|---|
| 1326 long prompts | per-instruction | 82.6 ± 0.8 | 88.2 ± 0.6 | 87.9 ± 0.6 | 90.2 ± 0.6 |
| 1326 long prompts | full-response | 44.4 ± 2.7 | 53.9 ± 2.7 | 55.0 ± 2.7 | 58.6 ± 2.7 |
| 406 short prompts | per-instruction | 87.8 ± 1.5 | 90.7 ± 1.3 | 87.1 ± 1.5 | 85.9 ± 1.6 |
| 406 short prompts | full-response | 63.4 ± 4.7 | 68.4 ± 4.6 | 66.8 ± 4.6 | 64.0 ± 4.7 |
Vision (author-reported, 0-shot unless noted)
| Benchmark | 1.0 Ultra | 1.5 Flash | 1.5 Pro | Source |
|---|---|---|---|---|
| MMMU (val), CoT | 59.4% | 56.1% | 62.2% | §6.2 table |
| MathVista (testmini), CoT | 53.0% | 58.4% | 63.9% | §6.2 table |
| AI2D (test) | 87.7% | 91.7% | 94.4% | §6.2 table |
| ChemicalDiagramQA (internal) | 55.2% | 56.6% | 69.7% | §6.2 table |
| ChartQA (test), CoT | 80.8% | 85.4% | 87.2% | §6.2 table |
| BetterChartQA (internal) | 47.9% | 59.0% | 65.8% | §6.2 table |
| DocVQA (test), ANLS, with Google Cloud OCR | 92.4% | 89.9% | 93.1% | §6.2 table |
| DUDE (test), ANLS | 44% | 48% | 46% | §6.2 table |
| TAT-DQA (test), ANLS | 13.2% | 23.6% | 37.8% | §6.2 table |
| InfographicVQA (test), ANLS | 80.3% | 75.3% | 81.0% | §6.2 table |
| TextVQA (val) | 82.3% | 78.7% | 78.7% | §6.2 table |
| VQAv2 (test-dev) | 77.8% | 80.1% | 80.2% | §6.2 table |
| V* bench (high-res) | 62.8% | 72.8% | 71.7% | §6.2 table |
| BLINK (val) | 51.7% | 56.5% | 61.4% | §6.2 table |
| RealWorldQA (test) | 64.7% | 67.5% | 70.4% | §6.2 table |
The DUDE column above lists Flash as the best of the three where the table bolds it; the text says Pro outperforms both 1.0 models on DUDE §6.2.
Video (author-reported, 0-shot unless noted)
| Benchmark | 1.0 Ultra | 1.5 Flash | 1.5 Pro | Source |
|---|---|---|---|---|
| VATEX (test), CIDEr, 4-shot | 62.7 | 57.1 | 64.6 | §6.2 table |
| VATEX ZH (val), CIDEr, 4-shot | 50.8 | 48.1 | 55.3 | §6.2 table |
| YouCook2 (val), CIDEr, 4-shot | 135.4 | 80.9 | 106.5 | §6.2 table |
| ActivityNet-QA (test) | 52.2% | 55.3% | 57.5% | §6.2 table |
| EgoSchema (test), CoT | 61.5% | 65.7% | 72.2% | §6.2 table |
| OpenEQA (val), LM-judged | 61.3 | 63.1 | 57.9 | §6.2 table |
The text says Pro is slightly behind on OpenEQA, and attributes this to refusals to answer §6.2.
Audio (author-reported; pre-instruction-tuning model per footnote)
| Task (metric) | USM | Whisper | 1.0 Pro | 1.0 Ultra | 1.5 Flash | 1.5 Pro | Source |
|---|---|---|---|---|---|---|---|
| YouTube en-us, WER ↓ | 5.8% | 6.5% | 4.8% | 4.7% | 4.9% | 4.8% | §6.3 table |
| YouTube 52 languages, WER ↓ | 22.8% | 41.4% | 22.5% | 21.0% | 23.8% | 22.6% | §6.3 table |
| Multilingual LibriSpeech en-us, WER ↓ | 7.0% | 6.2% | 4.8% | 4.4% | 5.2% | 4.2% | §6.3 table |
| FLEURS 55 languages, WER ↓ | 11.2% | 16.6% | 6.4% | 6.0% | 9.8% | 6.5% | §6.3 table |
| CoVoST 2, 20 languages to English, BLEU ↑ | 31.5 | 29.4 | 40.0 | 41.0 | 36.1 | 39.4 | §6.3 table |
Serving latency (time per output character, ms; inputs of 10,000 characters)
| Language | 1.5 Flash | 1.5 Pro | GPT-3.5 | GPT-4 | Claude-3 Haiku | Claude-3 Sonnet | Claude-3 Opus | Source |
|---|---|---|---|---|---|---|---|---|
| English | 1.5 | 4.3 | 2.6 | 6.8 | 2.2 | 6.2 | 10.5 | §3.3 table |
| Japanese | 4.3 | 10.9 | 12.9 | 35.4 | 10.7 | 23.9 | 46.6 | §3.3 table |
| Chinese | 5.2 | 14.1 | 18.4 | 53.4 | 12.8 | 30.0 | 55.3 | §3.3 table |
| French | 1.9 | 4.7 | 4.1 | 10.7 | 3.0 | 6.8 | 13.4 | §3.3 table |
The paper notes Flash generates over 650 characters per second in English, more than 30% faster than Claude 3 Haiku §3.3.
Mathematics and Flash-8B
The math section reports a Math-specialised Pro variant §7. Flash-8B is described separately and is "under active development", with preliminary results §8. Neither is covered in detail here.
Paper vs code
No official code.
No link to code appears in the paper's abstract or comments in the raw arXiv metadata, and the paper names no project page. A GitHub repository search for "gemini 1.5" under google-deepmind returned no results. An unscoped search for "gemini-1.5 technical report" returned two unrelated third-party repositories, which are not used. The models are closed: the report releases no weights.
Pitfalls
- Architecture is not specified. Pro is called a sparse MoE with "architecture changes" for long context §3.1, but the paper gives no parameter count, number of experts, routing function, layer count or attention variant. An implementer cannot reproduce the model from this card.
- Flash's distillation is described in one phrase. "Online distilled" from Pro §3.2 is not defined: no loss, no teacher-student schedule, no sampling rule.
- Training data and compute are not reported. The model card says compute requirements and model stats are "not reported" app. model card. The data is described only by domain §4. No token count is given.
- Model names hide several snapshots. The arXiv version used here (v5, updated 2024-12-16) reports an updated Pro. Section 2 gives the change from the February release: MATH from 58.5% to 67.7%, GPQA from 41.5% to 46.2%, MathVista from 52.1% to 63.9%, InfographicVQA from 72.7% to 81.0%, and EgoSchema from 65.1% to 72.2% §2. Numbers from the February model should not be mixed with these.
- Benchmark settings differ by row. Some numbers are 0-shot, some 4-shot, some 11-shot, some maj@k (majority vote over k samples) and some use chain of thought. The 1.0 Ultra entries are sometimes pretrained-only (PT) or use a different shot count. Compare only within a row.
- Contamination. The authors say n-gram decontamination is imperfect §6.1. They report HumanEval leakage: one epoch on the HumanEval test split raised 1.0 Ultra from 74.4% to 89.0% §6.1. Their internal benchmarks (PhysicsFinals, HiddenMath, Natural2Code) are the less contaminated part.
- Scoring can be inflated by the setup. DocVQA gets Google Cloud OCR annotations as extra input §6.2 table caption. MMMU, MathVista, ChartQA and EgoSchema allow chain-of-thought before the answer §6.2 table captions.
- Long-context retrieval is a simple test. Needle recall asks for one inserted fact; the paper itself calls the task simple §5.2.1. It adds NLL curves and realistic tasks alongside it §5.2. The paper also notes that repetition in code can give outsized gains, which may explain the deviation near 10M §5.2.1.
- The video haystack repeats one video. The 10.5 h input is seven copies of one documentary §5.2.1. Repeated content may make retrieval easier than novel content; this is an inference from the setup, not a result the paper tests.
- Auto-rater is the model itself. The Les Misérables comparison uses Gemini 1.5 Pro as the auto-rater §5.2.2. Judge and candidate share a model family, which can bias the preference scores. The paper does not measure this.
- Single human rater for MTOB. The Kalamang scores come from one non-native rater who can recognise their own translations §5.2.2. The text gives the eng-to-kgv full-book human score as 5.58, but the table shows 5.60 §5.2.2.
- Best setting is not the full book. On Kalamang to English, Pro's best score is in the half-book setting (4.14), not the full book (4.00) §5.2.2. The authors give no explanation.
- Latency is a snapshot. Serving times are API measurements against other vendors' APIs in May 2024 §3.3. They depend on load and provider configuration, and cannot be reproduced later.
- Audio results use a different model state. The audio evaluations use the pre-instruction-tuning model "for fair comparison" §6.3, footnote.
- Cost is not in the paper. The paper does not give prices or serving cost, so the card makes no cost claims.
- Paper says Pro is sparse but the model card says Flash is dense. This is consistent with §3.2 and app. model card, but do not infer Flash's architecture from Pro's.
Lineage
- Builds on: Gemini 1.0 (the authors' earlier report) §3.1; a long history of MoE work at Google, cited in §3.1.
- Compared against: GPT-4 Turbo, Claude 3 Haiku, Sonnet and Opus, Claude 2.1, GPT-4V and Gemini 1.0 Pro and Ultra §5 §6 §3.3.
- Superseded by: not stated in the paper.
Source and license
Adapted from “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” by Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Hennigan, George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross McIlroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safranek-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićević, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, Iain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilic, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe Chai Sim, Sergi Caelles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo Li, Zizhao Zhang, Mariko Iinuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Cobon-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Arnold, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo-yiin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean-baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwanicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Galilee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ionel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Santamaria-Fernandez, Wojciech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Tsendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquette-Choo, James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplanis, XiangHai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrini, DongHyun Choi, Ivan Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kępa, François-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd Al Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, Oriol Vinyals (arXiv:2403.05530), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.