Emergent abilities

Overview

An ability is emergent when it is absent in smaller models and present in larger ones, so it cannot be read off the curve of smaller models 2206.07682 §2. On a plot of score against scale, the score sits near chance up to a critical size and then rises well above chance 2206.07682 §2. For anyone who picks a model size or a technique, the question is whether a capability will appear at the size they can afford.

Every "emergent" label depends on three choices: the scale axis, the metric and the baseline it is compared against 2206.07682 §2, app. A.1. The variants below are these choices, and each can move the verdict.

The field has no shared definition of emergence. A 2025 survey lists the competing meanings and argues that an abrupt curve alone does not establish emergence 2503.05788 §1, §2.

This page rests on nine cards. Two surveys carry most of the claims: the 2022 survey 2206.07682 and the 2025 survey 2503.05788, which also reviews the evidence on metrics, compression, prediction and mechanisms. A distillation study reports smooth scaling for its small models 2301.12726 §4.2. The chain-of-thought paper gives the most detailed per-size results for one prompting technique 2201.11903. A study of in-context learning with flipped and unrelated labels reports that the override of priors appears with scale in the families it tests 2303.03846 §3 §4. A sampling study argues that greedy evaluation hides capability; its emergence-related results are per model rather than a size sweep, and several of its tables are simulated 2601.07239 §2 tab. reasoning-brittleness. A training suite adds a view over training time 2304.01373. The foundation-model report frames the topic and has no experiments 2108.07258 §1.1. A zero-shot analogy study reports one large model's results against human samples, with a preliminary check on a second model for one task, and records no sweep over sizes 2212.09196 §2 §2.4. The survey's thresholds are compiled from other papers' results and from the authors' own labelling of BIG-Bench tasks 2206.07682 app. A.3, tab. 1. Those other papers have no cards on wikillm, so their content is not used here.

Variants

These are ways to test for emergence, not algorithms.

Few-shot emergence on a task

A task is emergent under few-shot prompting when the model is at random up to some scale and then clearly above random 2206.07682 §3. Examples from the survey: 3-digit addition moves above chance at about 2·10^22 training FLOPs (13B parameters, GPT-3) 2206.07682 §3 fig. 2A. TruthfulQA stays at chance for GPT-3 and only rises for the 280B Gopher model 2206.07682 §3 fig. 2E. WiC is at chance for models up to about 5·10^23 FLOPs and above chance for PaLM 540B 2206.07682 §3 fig. 2H. The paper does not set a numeric threshold for "at random" or "well above". It labelled these by hand, with two co-authors agreeing jointly 2206.07682 app. A.3.

Emergence of an augmented prompt or finetune

A technique is emergent when it gives no gain, or hurts, relative to the plain baseline until the model is large enough 2206.07682 §4. The survey's examples:

• Chain-of-thought prompting beats standard prompting only from about 10^23 FLOPs (~100B parameters) 2206.07682 §4 fig. 3A. The chain-of-thought paper reports the same scale range in its own runs. PaLM 540B with eight exemplars reaches 56.9% on GSM8K against 17.9% for standard prompting, and LaMDA 137B goes from 6.5% to 14.3% (mean of five seeds), both author-reported 2201.11903 app. B. Most models below about 10B parameters do not benefit 2201.11903 app. A. See chain-of-thought prompting.
• Instruction finetuning (FLAN) hurts at 7·10^21 FLOPs and below (8B) and helps from about 10^23 FLOPs 2206.07682 §4 fig. 3B.
• Scratchpad helps 8-digit addition from about 9·10^19 FLOPs (40M parameters) 2206.07682 §4 fig. 3C.
• P(True) calibration beats standard calibration only at the largest model tested, about 3·10^23 FLOPs (52B) 2206.07682 §4 fig. 3D.
• Distilling chain-of-thought maths solutions from a large teacher into T5 and FlanT5 students (250M to 11B parameters) gives a log-linear curve, not a flat one, in the authors' description 2301.12726 §4.2 fig. 2. The same students score 0.0 on BIG-Bench Hard chain of thought at 11B and 3B, and 0.3 and 0.1 at 760M and 250M 2301.12726 tab. 1. Here the technique is training, not prompting. See Specializing Smaller Language Models towards Multi-Step Reasoning and knowledge distillation.

Emergence of overriding a prior: flipped or unrelated labels

Another test asks whether a model learns the input-label mapping from the prompt, or only recalls the task it already knows. Flipping the exemplar labels of a binary task, or replacing the label words with meaningless ones such as "Foo" and "Bar", separates the two. The paper reports that large models follow the flipped exemplars and still learn the unrelated-label mapping, while small models keep their prior behaviour 2303.03846 §3 §4.

• With flipped labels, text-davinci-002 falls from 90.3% at 0% flipped to 22.5% at 100% flipped 2303.03846 §3. Small models stay near their prior accuracy, and large ones drop well below chance 2303.03846 §3 fig. 2. The paper reports the override as emerging with scale.
• For PaLM, accuracy at 100% flipped goes as low as 31%. Flan-PaLM does not fall below chance 2303.03846 §5 fig. 7.
• With unrelated targets, Flan-PaLM-8B scores 9.6 points above PaLM-8B 2303.03846 §5 fig. 6.
• In a linear-classification task with N=16, the largest Codex model is about 19 points above chance, against about 9 for the smaller Codex models 2303.03846 §6 fig. 8. The paper finds code-davinci-002 working up to N=64, and finds no scale effect for PaLM in that range 2303.03846 app. C.4.

The card has caveats on this variant. Each dataset uses 100 evaluation examples, with no confidence intervals or repeated runs 2303.03846 §2.3. The sizes of InstructGPT and Codex are not public, and the scale claim for those families assumes they grow in size 2303.03846 §2.2. The paper also says that davinci behaves like a small model in the unrelated-label setting, so the threshold depends on family and task 2303.03846 §3 app. C.1 app. C.2. At 50% flipped, binary labels are at chance, so that point gives no information 2303.03846 §3.

Sampling budget: a lever that is not a size axis

A sampling study argues that a single greedy completion can understate a model's capability, and that the spread across samples is worth measuring 2601.07239 §2. It compares greedy decoding (temperature 0) with temperature 0.7 and top-p 0.9 sampling, and with majority vote over k samples 2601.07239 §3 §4. Its gains are per model, with weights held fixed, and the card records no size sweep for them 2601.07239 §4.

• On BESSTIE, greedy accuracy for many model and task pairs lies between 40% and 65%. Best-of-16 typically adds 10 to 20 points, and best-of-64 adds 2 to 5 more 2601.07239 §4. These are author-reported. The paper does not state a tie rule for even k 2601.07239 §4.
• On InstruSum, the style gain saturates near k=8, reaching about 10 to 13 points for LLaMA-3, Gemma-2 and Mixtral-8x22B 2601.07239 §4.

Two parts of this card are not measured results. The chain-of-thought table is captioned "simulated", and the safety table is captioned "our simulations" 2601.07239 tab. reasoning-brittleness tab. safety-concealed-risk. The card says not to present them as measured results. The sampling gains compare decoders at fixed sizes, so they show how a verdict can change with the decoding protocol, not whether a size threshold exists.

Scale axis: compute, parameters, data or training time

The main x-axis is training FLOPs. Parameter-count versions of the plots are in the appendix 2206.07682 §2 app. D. Dataset size is not used, because many model families train every size on the same number of tokens 2206.07682 §2. WikiText103 perplexity is offered as an alternative axis 2206.07682 §5.3 fig. 4.

The two main axes agree only when compute grows with parameters, as in dense model families. Chinchilla has about a quarter of Gopher's parameters at similar compute, and sparse mixture-of-experts models break the link 2206.07682 §2. The perplexity axis tracks FLOPs only for Gopher and Chinchilla, and is confounded across families by training data 2206.07682 §5.3.

A third axis is the training step. Pythia keeps 154 checkpoints per model and a fixed data order, so the data seen by each checkpoint can be recovered 2304.01373 §2.4 §2.7. Its term-frequency study reports that a correlation between answer accuracy and how often the relevant facts appear in the training data shows up only after about step 65,000 (about 45% of training), for models of 2.8B parameters and up 2304.01373 §1 §3.3 fig. 4. A jump found this way is only located to the gap between saved checkpoints.

Compression is a further axis. The 2025 survey reports LLaMA models from 7B to 65B keeping most in-context learning, chain-of-thought and instruction-following ability at 4 bits, and falling near random at 2 bits 2503.05788 §3.4. In a study of about 40 methods, a 70B model at 2 bits kept about 90% of its MMLU accuracy, while models below 10B lost 25% or more 2503.05788 §3.4. The collapse points differ by method: unstructured pruning near 55 to 65% sparsity, quantization below 3 bits 2503.05788 §3.4.

Pre-training loss is a candidate predictor, not a size axis. The survey reports one two-stage loss-based prediction with relative error of 5% at 7B and 10% at 13B 2503.05788 §3.7. The same survey says the loss-threshold account is not causal, and that a common confounder is plausible 2503.05788 §3.3.

Metric: discrete score or smooth loss

Exact match, accuracy and similar scores can sit at chance while the loss on the target keeps improving. For the six BIG-Bench tasks that the paper calls emergent for LaMDA, the cross-entropy loss improved even where the score was at random, at up to about 10^22 FLOPs 2206.07682 app. A.1. For generative tasks the loss shows an elbow at the point where the score jumps 2206.07682 app. A.1 fig. 5.

Partial-credit metrics (BLEU, ROUGE, BLEURT) still show the jump on the three generative tasks the authors re-plotted, so the smoothness of the metric does not remove emergence on its own 2206.07682 app. A.2 fig. 7. On multiple-choice tasks, the average log-probability of correct and incorrect answers both rise with scale, because larger models give less extreme probabilities 2206.07682 app. A.1 fig. 6.

A threshold on a score is another way to label a jump. The 2025 survey reports a meta-analysis that calls a metric-task pair emergent when a spread-to-gap score exceeds 100, and says that 20 to 100 may still mark a discontinuous jump. The "median difference" in that score is not defined precisely in the text 2503.05788 §3.1.

Mechanistic evidence

The 2025 survey's test asks three things: whether a low-dimensional internal structure tracks the ability, whether changing that structure changes the behaviour, and whether the structure screens off the individual weights 2503.05788 §2. It reports that few abilities pass this test in the cited work 2503.05788 §2. One example is a set of symbolic abstraction heads, which a linear decoder reads with above 98% accuracy on disjoint token sets 2503.05788 §3.6. That result comes from one study.

When to use which

Implementation sketch

The survey has no algorithm of its own. The closest thing is its procedure for labelling a task's scaling curve, which it applied by hand to 210 BIG-Bench tasks 2206.07682 app. A.3. A rewrite for your own evaluation runs, with a log-likelihood column added:

# Order one model family by training compute (or parameters) [arxiv:2206.07682 §2]
models = sorted(family, key=train_flops)

# Record the task score and the target log-likelihood at each size
# (the second is the metric check in [arxiv:2206.07682 app. A.1])
for m in models:
    score[m] = accuracy_or_exact_match(m, task)
    ll[m]    = mean_log_prob_of_target(m, task)

# Label the curve; "near random" and "well above" are judged, not set
if all(score[m] is near chance r for m in models):
    label = "flat"
elif score rises steadily with scale:
    label = "smoothly increasing"
elif score is near r below some size s*, and well above r beyond s*:
    label = "emergent at s*"
else:
    label = "other"

# If ll keeps rising while score is flat, the jump may be
# partly a metric effect, not a change in the model

Where variants differ:

• Augmented prompts: run the same loop with the technique on and off at each size, and label the gap rather than the score 2206.07682 §4.
• Partial credit: replace the score with a graded metric such as BLEU or ROUGE, and keep the exact-match score next to it 2206.07682 app. A.2.
• Training time: if checkpoints are saved, run the loop over steps as well as sizes, with the same data order at every step 2304.01373 §2.4.
• Compression: run the loop over bit widths or sparsity levels in place of sizes, since collapse points differ by method 2503.05788 §3.4.
• Prior check: replace the score with the flipped-label or unrelated-label score at each size, and label the curve on that score 2303.03846 §3 §4.
• Sampling: at each size, add a best-of-k column next to the greedy score, with the decoder and k stated 2601.07239 §4.

Contradictions

Within the survey, the same model is given different sizes or FLOPs in different tables and sections. The paper does not reconcile them. Both sides are listed:

• Scratchpad model. The 8-digit addition emergence point is 40M parameters at 8.9·10^19 FLOPs in the text and table 1 2206.07682 §4 tab. 1. Table 2 lists a LaMDA model at 8.90E+19 FLOPs with 57M parameters 2206.07682 tab. 2.

• Anthropic 52B model. Table 1 gives 2.6E+23 FLOPs for the P(True) model. Table 2 gives 2.65E+22 for the same 52B model, a value below the 12B model's 6.12E+22. The card reads the table 2 value as likely a typo; this is an inference, not stated in the paper 2206.07682 tab. 1 tab. 2.

• LaMDA size. 68B in the text and table 1 2206.07682 §3 tab. 1; 69B in table 2 2206.07682 tab. 2. The appendix gives 27B for the same family in one place and table 2 gives 29B 2206.07682 app. A.1 tab. 2.

• Gopher 280B and Chinchilla 70B. Table 1 gives 5.0E+23 FLOPs for both 2206.07682 tab. 1. Table 2 gives 5.46E+23 and 5.63E+23 2206.07682 tab. 2.

• PaLM 540B compute. 2.5·10^24 FLOPs in the text and table 1 2206.07682 §3 tab. 1; 3·10^24 in the discussion section 2206.07682 §5.

• Chain-of-thought scale. The survey places the chain-of-thought gain at about 10^23 FLOPs (~100B) 2206.07682 §4 fig. 3A, and lists chain of thought on math word problems as emergent at LaMDA 68B, 1.3·10^23 FLOPs 2206.07682 tab. 1. The chain-of-thought paper states its gains at about 100B parameters 2201.11903 §3.2 and gives LaMDA 137B results 2201.11903 app. B. Its card does not transcribe a 68B number, so the two cannot be checked against each other from wikillm's sources. The 10B figure for "most models do not benefit" comes from the chain-of-thought paper's appendix, and no source here reports the 10B to 100B range.

• Smooth scaling of distilled chain of thought. The survey places the chain-of-thought gain at about 10^23 FLOPs (~100B) for prompted models 2206.07682 §4 fig. 3A. The distillation study says its specialized FlanT5 students scale log-linearly rather than flat 2301.12726 §4.2 fig. 2. The two settings differ: one prompts models of different sizes, the other trains each student on teacher outputs. The sources do not reconcile them, and no reading is given here.

• Prior use in few-shot prompting. The flipped-label study cites earlier random-label work that reads small effects of label noise as a sign that models rely on priors 2303.03846 §1 §7.1. The same study reports that large models follow flipped exemplars 2303.03846 §3. Both are cited here. The two studies use different tasks and label settings, and this page does not reconcile them.

• Metric artifacts. The 2025 survey reports that Schaeffer et al. argue jumps vanish under linear metrics, while Du et al. and Zhao et al. argue they survive 2503.05788 §3.1. Gu et al. report transitions under validation loss 2503.05788 §3.1. The 2022 survey says partial-credit metrics still show the jump on three generative tasks 2206.07682 app. A.2. This page does not pick a side. Check the metric and the seed count before choosing one.

• Predictability. The 2022 survey defines an emergent ability as one that a smaller model's curve does not predict 2206.07682 §2. The 2025 survey reports a loss-based two-stage prediction with 5% to 10% relative error 2503.05788 §3.7. The two predictors differ, since one uses scale and the other uses loss, so the sources do not settle whether jumps can be forecast.

For thresholds, use the values from the table you are comparing against, and note which one.

Papers

• Emergent Abilities of Large Language Models: defines emergence on scaling curves, lists emergent few-shot tasks and augmented prompts from prior work, and re-analyses BIG-Bench with loss and partial-credit metrics.
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: shows chain-of-thought gains that appear only at about 100B parameters, with per-size results and the small-model cost. Results are author-reported.
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling: a suite of models with checkpoints and a fixed data order, with a term-frequency study of when accuracy starts to track training frequency. Each case study is a single run per model.
• On the Opportunities and Risks of Foundation Models: names emergence as one of two central themes of foundation models. It has no experiments, so use it for framing only.
• Emergent Analogical Reasoning in Large Language Models: reports zero-shot analogy results for GPT-3 text-davinci-003 against undergraduate samples on four task families. One model variant, author-reported, with no size sweep.
• Specializing Smaller Language Models towards Multi-Step Reasoning: distils GSM8K chain-of-thought solutions into T5 and FlanT5 students, reports log-linear scaling for them, and shows a large loss of general reasoning.
• Emergent Abilities in Large Language Models: A Survey: reviews the competing definitions and the evidence on metrics, compression, prediction, mechanisms, reasoning models and misalignment. It reports no experiments of its own, so its numbers are quoted from cited studies.
• Larger language models do in-context learning differently: flips or unrelates the labels of few-shot prompts and reports that large models override priors and learn unrelated mappings, with a Flan-PaLM comparison. Author-reported, 100 examples per dataset.
• Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition: argues that greedy decoding hides capability and tail risk, and measures best-of-k gains. Several tables are simulated, and no code was found.