Emergent Abilities of Large Language Models

In plain words

When you make a language model bigger, most benchmark scores rise smoothly, and you can predict the big model's score from the small ones. This paper collects cases where that prediction fails: small and medium models score at chance, and then, past some size, the score jumps well above chance. The authors call these "emergent abilities" and borrow the physics slogan "more is different".

Examples include multi-digit arithmetic, a broad exam-style benchmark covering 57 subjects, and a word-sense task where an early large model stayed at chance and only a much larger later model got past it. Some prompting tricks behave the same way: asking the model to reason step by step, or fine-tuning it to follow instructions, does nothing or even hurts on small models and only helps on large ones.

The paper is a survey, not a new method. Its practical lesson for engineers is caution: a technique or task that fails on your small model may still work on a bigger one, and the reverse cannot be read off a scaling curve either. The authors also show that part of the "jump" depends on how you score: a smoother measure such as the model's log-likelihood of the right answer keeps improving even while accuracy sits at chance. The size at which an ability appears is not fixed; better data or architecture can move it down.

TL;DR

Defines emergence at the level of scaling curves: an ability is emergent if performance is near random up to some scale and then rises well above random, so it is not predictable by extrapolating smaller models §2. Surveys 8 few-shot examples across GPT-3, LaMDA, Gopher, Chinchilla and PaLM fig. 2 and 4 augmented-prompting examples (chain of thought, instruction tuning, scratchpad, P(True) calibration) fig. 3, with a full list of emergence thresholds in tab. 1. Its own analyses show that cross-entropy loss improves smoothly before the downstream metric jumps app. A.1, that partial-credit metrics do not remove emergence on generative tasks app. A.2, and that scale alone is not the only factor (PaLM 62B is emergent on 14 BIG-Bench tasks where GPT-3 175B and LaMDA 137B are not) §5.2 app. F. Read it to decide how much to trust small-model ablations of prompting methods.

Problem

Scaling laws predict cross-entropy loss well across many orders of magnitude, but some downstream tasks do not improve continuously with scale and cannot be forecast in advance §1. Practitioners who evaluate techniques on small models or extrapolate from scaling curves can miss abilities that only appear at large scale §2. The paper asks which abilities behave this way, at what scale, and why §1.

Method

This is a survey; there is no algorithm to implement. The "method" is a definition plus a way of reading scaling curves, applied to results from prior work and to the authors' own re-analysis of BIG-Bench.

1. Definition. "An ability is emergent if it is not present in smaller models but is present in larger models." §2 On a scaling curve (x: model scale, y: performance) this shows as near-random performance up to a critical scale, then a rise to substantially above random, a phase transition §2.
2. Scale axis. Primary x-axis is training compute in FLOPs; parameter-count versions of every plot are in the appendix §2 app. D fig. 11 fig. 12. Dataset size is not used as an axis because many model families train every size on the same number of tokens §2. WikiText103 perplexity is offered as an alternative axis §5.3 fig. 4.
3. Few-shot emergence. A task is emergent under few-shot prompting if the model is at random until some scale and then well above random §3.
4. Augmented-prompting emergence. A prompting or finetuning technique is counted as emergent if it gives no improvement, or hurts, relative to the plain baseline until the model is large enough §4.
5. BIG-Bench classification. All 210 BIG-Bench tasks were manually labelled as smoothly increasing, emergent with GPT-3 or LaMDA, emergent with PaLM, flat (no model above random), or other app. A.3 app. E. Because "near-random" is subjective, two co-authors labelled jointly and only kept emergent labels they agreed on with confidence app. A.3. Each task appears in one category only; tasks already emergent with GPT-3/LaMDA are excluded from the PaLM category app. E. 40 tasks fell into no main category (noisy, uncorrelated with scale, etc.) app. A.3.
6. Metric-dependence check. For the six BIG-Bench tasks emergent for LaMDA (three generative scored by exact match or BLEU, three classification scored by accuracy), plot cross-entropy loss of the target next to the downstream metric app. A.1. Two possible outcomes are spelled out: loss also flat (model truly not improving) or loss improving while the metric is flat (improvement hidden by the metric) app. A.1. For generative tasks, error rate is shown with greedy decoding (T=0) and sampling (T=1) fig. 5. For multiple-choice tasks, log-probabilities of correct and incorrect options are also plotted fig. 6.
7. Partial-credit metrics. Three emergent generative BIG-Bench tasks are re-plotted with every metric BIG-Bench provides (BLEU, ROUGE, BLEURT, ...) app. A.2 fig. 7. Word unscramble and repeat copy logic are excluded because only exact match makes sense for them app. A.2.

Procedure: classify a task's scaling curve  [§2] [§3] [app. A.3]
Input: (scale_i, score_i) for models of one family, random-baseline score r
1. Order models by training FLOPs (or parameters).
2. If every score_i is approximately r                -> "flat"
3. Else if scores rise predictably with scale          -> "smoothly increasing"
4. Else if score_i ~ r for all models below some scale s*
        and score_i substantially above r beyond s*   -> "emergent at s*"
5. Else                                               -> "other"
   ("approximately r" and "substantially above" are judged by hand; no
    numeric threshold is given)

Hyperparameters

No training or model of its own. The relevant settings are the evaluation setups and the scale figures of the surveyed models.

Training scale of the surveyed models, reproduced from the paper tab. 2:

Results

All results are from prior work, as compiled by the authors, except the BIG-Bench and MMLU re-analyses in the appendices.

Emergence thresholds tab. 1, reproduced from the paper:

Highlights from the text (as described by the authors):

• BIG-Bench arithmetic (3-digit add/sub, 2-digit multiplication): near zero for several orders of magnitude, then above random at 2·10^22 FLOPs (13B) for GPT-3 and 10^23 FLOPs (68B) for LaMDA §3 fig. 2A. IPA transliteration, word unscrambling and Persian QA emerge at similar scales §3 fig. 2B–D; Persian QA needed both PaLM's training data and 62B scale §5.6.
• TruthfulQA: GPT-3 never above random; Gopher only at 280B (5·10^23 FLOPs), then more than 20% above random §3 fig. 2E.
• MMLU: GPT-3, Gopher, Chinchilla at ~10^22 FLOPs (~10B) or less are at chance; 3–5·10^23 FLOPs (70B–280B) is well above chance §3 fig. 2G.
• WiC (one-shot): GPT-3 and Chinchilla stay at random up to ~5·10^23 FLOPs; PaLM 540B (2.5·10^24 FLOPs) is above random §3 fig. 2H.
• Chain of thought beats standard prompting only from ~10^23 FLOPs (~100B) §4 fig. 3A. Instruction finetuning (FLAN) hurts at ≤ 7·10^21 FLOPs (8B) and helps from ~10^23 FLOPs (~100B) §4 fig. 3B. Scratchpad helps 8-digit addition from ~9·10^19 FLOPs (40M) §4 fig. 3C. P(True) calibration beats standard calibration only at the largest model, ~3·10^23 FLOPs (52B) §4 fig. 3D.
• Cross-entropy loss improves for all six LaMDA-emergent BIG-Bench tasks even where EM/BLEU/accuracy is random (≤ 10^22 FLOPs, ≤ 27B), i.e. all are "Outcome 2" app. A.1. Generative tasks show an elbow in loss at the point of emergence app. A.1 fig. 5. On multiple-choice tasks, average log-probability of both correct and incorrect answers rises with scale, because larger models give less extreme probabilities, and the two curves separate at the scale where accuracy jumps app. A.1 fig. 6.
• Partial-credit metrics (BLEU, ROUGE, BLEURT) still show emergence on the three generative tasks app. A.2 fig. 7.
• BIG-Bench keywords with the highest share of emergent tasks: analogical reasoning, word sense disambiguation, truthfulness, social reasoning, emotional understanding; arithmetic and mathematics have a relatively low share; visual reasoning has the largest share of flat tasks (8/13) app. A.3 fig. 8. The authors find no clear trend in which task types emerge app. A.3.
• MMLU by category, second-largest (7B) vs largest model, accuracy % fig. 9:

• Scale is not the only factor: PaLM 62B is above random on 14 BIG-Bench tasks where LaMDA 137B and GPT-3 175B are near random (anachronisms, ascii word recognition, conceptual combinations, cryptonite, disambiguation qa, emoji movie, goal step wikihow, gre reading comprehension, linguistics puzzles, logic grid puzzle, metaphor boolean, metaphor understanding, odd one out, parsinlu qa) §5.2 app. F. The authors suggest data quality and architecture (e.g. split digit encodings) as possible causes, untested §5.2.

Paper vs code

No official code. The paper is a survey; its plots are drawn from published results of other papers and BIG-Bench fig. 2 fig. 3.

Pitfalls

• Emergence is not a property of the task alone. The scale where an ability appears depends on data quality, architecture, objective and training procedure; the paper explicitly does not claim any fixed scale is required §2 §5.2. Do not treat tab. 1 numbers as thresholds that transfer to a different model family.
• Metric choice partly creates the jump. Exact match on long targets and final-answer-only scoring hide incremental progress §5.1; cross-entropy of the target improves smoothly below the jump app. A.1. When evaluating small models, also log the target log-likelihood. The authors note this does not fully explain emergence: classification tasks fig. 2D–H and partial-credit metrics app. A.2 still show it, and loss curves do not predict where the jump happens §5.1.
• Log-probabilities of wrong answers rise too. On multiple choice, both correct and incorrect options gain log-probability with scale because predictions become less extreme app. A.1. Comparing average log-prob of the correct answer alone across scales can mislead; compare against the incorrect options.
• Labels are manual. "Near-random" and "substantially above random" have no numeric threshold; the BIG-Bench categories reflect two co-authors' joint judgement app. A.3. Logical arguments has only 32 examples fig. 6.
• Small-model ablations can reverse sign. Instruction finetuning hurts at 8B and below in FLAN §4; chain of thought does not help below ~100B in the cited setting §4. Conversely, later work induced instruction following in an 11B encoder-decoder model §5.2, so "emergent" can also become "available earlier".
• Inconsistent scale figures across the paper. Check which number you use:
  • Anthropic 52B: 2.6E+23 FLOPs in tab. 1 but 2.65E+22 in tab. 2; the tab. 2 value is below the 12B model's (6.12E+22), so it is likely a typo (our inference, not stated).
  • Scratchpad: 40M parameters at 8.9E+19 FLOPs §4 tab. 1, but tab. 2 lists the LaMDA model at 8.90E+19 FLOPs as 57M.
  • LaMDA sizes: 68B in §3 tab. 1 vs 69B in tab. 2; 27B in app. A.1 vs 29B in tab. 2.
  • Gopher 280B and Chinchilla 70B: 5.0E+23 FLOPs in tab. 1 vs 5.46E+23 and 5.63E+23 in tab. 2.
  • PaLM 540B: 2.5·10^24 FLOPs in §3 tab. 1 vs 3·10^24 in §5.
  • Chinchilla 7.1B token count is marked "[sic] 199B" by the authors tab. 2.
• FLOPs vs parameters. Curves look alike on both axes only because dense model families scale compute roughly with parameters §2. Chinchilla has a quarter of Gopher's parameters at similar compute, and sparse MoE models break the link §2; pick the axis deliberately.
• WikiText103 perplexity as an axis tracks FLOPs only for Gopher/Chinchilla and is confounded across families by training-data differences §5.3.
• Risks can scale too. Bias on ambiguous BBQ contexts, toxicity and training-data memorisation are reported to increase with scale in cited work §5.4.

Lineage

• Builds on: scaling laws (Kaplan et al., arXiv:2001.08361; Hoffmann et al., arXiv:2203.15556) and BIG-Bench (arXiv:2206.04615); collects results from GPT-3 (Brown et al. 2020), Gopher (arXiv:2112.11446), PaLM, chain-of-thought prompting (arXiv:2201.11903), FLAN (Wei et al., ICLR 2022), scratchpad (arXiv:2112.00114) and P(True) calibration (arXiv:2207.05221) §3 §4.
• Compared against: not applicable (survey).
• Superseded by: not known from content in this wiki.