Neural scaling laws

Overview

A scaling law is a fitted power law that maps a training resource to a downstream error rate. The clearest form is E = β·C^α, where E is the error rate, C is compute and α is the exponent 2212.07143 §4.1. The controlled study varies model size, pre-training data size and samples seen, and counts compute in GMAC [arxiv:2212.07143 app. "Predictions derived from scaling laws"]. The meta-analysis instead uses compute estimated for published papers 2007.05558 §3.2.

The invariant the fitted variants share: over the scales measured, the error follows a power law in the resource, so one slope and one intercept summarise a set of runs 2212.07143 §4.1. The controlled study reports this for zero-shot classification, retrieval, linear probing and fine-tuning 2212.07143 §5. The meta-analysis fits one slope per benchmark 2007.05558 §3.2. Neither claims the law holds beyond the range it measured 2212.07143 §5 2007.05558 §3.3. A separate fit tracks training compute against publication date, not error against compute. It gives a doubling time per era 2202.05924 §1 tab. 2, so it describes the growth of the resource, not the error it buys.

Ten papers do not fit a curve of their own. Six sweep model size, data or task count and describe the trend without a fitted law: a protein model family 2206.13517 §4.1, a text-world simulator 2512.18832 §6.1, instruction finetuning across model sizes 2210.11416 §3, a scientific language model family 2211.09085 tab. 6, 3D training data for view synthesis 2307.05663 §4.1 and a transformer diffusion policy 2409.14411 §4.2. One argues that curated data can substitute for scale 2306.11644 §1. The other derives a schedule from a scaling law for the loss 2602.06797 §3.2. Two are surveys that describe the trend in words and run no experiments: one on scaling in general 2307.10169 §2.3, and one on emergence, which defines a jump against a curve fitted to small models 2503.05788 §2.

Variants

Power-law fit on the best run per compute bin

The forecasting method. Runs are plotted by compute and error, and the fit uses the Pareto frontier, the lowest-error run in each compute bin, because the total compute of different runs is not exactly aligned 2212.07143 app. "Predictions derived from scaling laws". A line is fitted in log-log space through those points 2212.07143 §4.1. The number of bins is not stated in the paper's text. Card: Reproducible scaling laws for contrastive language-image learning.

Scaling the pre-training data source

The same model family trained on two data sources, LAION (open) and OpenAI's private WIT-400M. The fitted exponents differ by task. For zero-shot ImageNet classification the reported exponents are −0.11 for OpenCLIP on LAION and −0.16 for OpenAI CLIP; for MS-COCO retrieval R@5 they are −0.08 and −0.05; for Flickr30K retrieval R@5 they are −0.19 and −0.10 2212.07143 §4.1 2212.07143 §4.2. A more negative exponent means faster improvement 2212.07143 §4.1. Card: Reproducible scaling laws for contrastive language-image learning.

Scaling samples seen with a schedule per run

The training-length axis. Each samples-seen scale gets its own cosine annealing schedule, so a short run is not read off the tail of a longer one 2212.07143 §3.2. The paper calls this a way to avoid suboptimal training for runs of different length 2212.07143 §3.2. Card: Reproducible scaling laws for contrastive language-image learning.

Literature meta-analysis: benchmark error against estimated compute

Instead of training runs, this method collects published deep-learning results for six benchmarks, estimates each model's training compute, and regresses log error on log compute per benchmark 2007.05558 §3.2 tab. 2a. For ImageNet, the simplest fit gives an exponent of 12.1 with a 95% interval of 10.6 to 14.1 2007.05558 tab. 2a. The headline reading is that halving ImageNet error needs about 5,000 times more compute, with a range of roughly 1,500 to 17,500 2007.05558 §3.2. A back-of-envelope argument gives a lower bound: computation grows at least as the fourth power of performance, where performance is 1/error 2007.05558 §2. Most papers do not report compute, so many values are derived. For models with only network operations, hardware burden comes from a fit on ImageNet alone 2007.05558 app. §8.1. Card: The Computational Limits of Deep Learning.

Data filtering rules measured at four scales

A benchmark for choosing image-text training data. The candidate pool holds 12.8B pairs, and four scales each multiply the samples seen and the pool size by 10, from 12.8M to 12.8B 2304.14108 §3.1. The best baseline, DataComp-1B with ViT-L/14, reaches 79.2% zero-shot ImageNet at the same compute as OpenAI's 75.5% 2304.14108 tab. 1. Filtering beats no filtering at every scale 2304.14108 §5.1. Random subsets keep improving with size, while CLIP-score filtering peaks at an intermediate size near the top 30% 2304.14108 §5.1 fig. 2. Card: DataComp: In search of the next generation of multimodal datasets.

Small model on curated text instead of more data

A 1.3B-parameter code model. Pre-training uses about 7B tokens of filtered code and generated textbook-style text, then about 180M tokens of generated exercises for fine-tuning 2306.11644 §2.3 tab. 1. The authors report 50.6% pass@1 on HumanEval and 55.5% on MBPP 2306.11644 tab. 1. Their comparison includes models trained on up to 1T tokens 2306.11644 tab. 1. The paper cites TinyStories as an earlier case where data quality changed the scaling picture 2306.11644 §1. That is the authors' argument, not a fitted law. Card: Textbooks Are All You Need.

Learning-rate schedule derived from a functional scaling law

The paper takes a functional scaling law from prior work as given and derives step-size schedules for one-pass SGD on a linear model with power-law features 2602.06797 §3.2. For easy tasks, power decay from the first step is optimal. For hard tasks, the rate holds at the stability limit for an initial stretch and then decays over the last fraction of training 2602.06797 thm. 4.1. The results are asymptotic, up to constants, and the paper gives no numeric peak rate 2602.06797 §4. Its neural-network runs are small, and the best decay ratio changes with peak rate and batch size 2602.06797 §7 2602.06797 app. D.2. Card: Learning-rate schedules under a functional scaling law.

Model-size sweep read from perplexity and zero-shot ranking

A family of four decoder-only protein models, from 151M to 6.4B parameters 2206.13517 §3.1 tab. 1. On the Test-max90 held-out set, perplexity falls from 12.9 to 9.9 across the family 2206.13517 tab. 2. On Test-max50 it does not fall at every step: 14.3 at 764M, 14.4 at 2.7B, then 13.9 at 6.4B 2206.13517 tab. 2. The paper fits no curve. For zero-shot ranking on narrow fitness landscapes, the best single size is 764M, with an average Spearman of 0.505, against 0.476 at 6.4B 2206.13517 tab. 3. Card: ProGen2: Exploring the Boundaries of Protein Language Models.

Robot diffusion-policy size sweep with a changed recipe

Diffusion Policy in a transformer (DP-T) is the baseline. The authors report that it does not improve with depth: 80.1% success at 8 layers, 78.4% at 12 and 74.6% at 14 2409.14411 §1 fig. 1. Their ScaleDP changes two things at once. It replaces cross-attention conditioning with adaptive layer norm, and it removes the causal mask from action self-attention 2409.14411 §3.2. Five sizes, from Ti at 10M to H at 1B parameters, are defined in 2409.14411 tab. 1. On a four-task Franka set the average success rises from 58.75% at S to 87.50% at H [arxiv:2409.14411 tab. 2]. On three bimanual UR5 tasks it rises from 53.33% at S to 98.33% at H 2409.14411 tab. 3. The MetaWorld curves by size and demonstration count appear only as figures, with no numbers in the text 2409.14411 §4.2 fig. 4. The results are author-reported, and each real-robot success rate uses 20 trials per task 2409.14411 §4.1. The scaling statements are qualitative, with no exponent to extrapolate. Card: Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation.

Data and model-size sweeps for a text-world simulator

A fine-tuned LLM that predicts the next observation of a text environment is trained on logged agent trajectories 2512.18832 §3.1 §4. The paper varies the amount of training data per environment. Structured environments saturate near 20K trajectories, WebShop keeps improving up to about 70K, and StableToolBench shows no saturation at 160K 2512.18832 §6.1 fig. 2. The paper describes these saturation points, not a fitted exponent. The model size sweep covers Qwen2.5 at 0.5B to 7B 2512.18832 app. A.1. A 1.5B model already captures the structured dynamics, while open-ended settings gain from more capacity 2512.18832 §6.2 fig. 3. The training sizes are caps in the released config, and the file sizes differ from the stated counts 2512.18832 app. A.1. Card: From Word to World: Can Large Language Models be Implicit Text-based World Models?.

Task count, model size and CoT share in instruction finetuning

No fitted law. The study finetunes pretrained checkpoints on a mixture of 1,836 instruction tasks, and varies the number of tasks, the model size and the share of chain-of-thought (CoT) data 2210.11416 §1. Every size gains over its own no-finetuning baseline: by 15.5 points of normalized average at 8B, 10.4 at 62B and 9.4 at 540B 2210.11416 §3 tab. 3. Most of that gain is in place by 282 tasks. For Flan-PaLM at 540B the normalized average is 57.5 at 282 tasks and 58.5 at 1,836 [arxiv:2210.11416 tab. 3]; the paper reads the later gains as small 2210.11416 §3 fig. 4. Without CoT examples in the mixture, held-out CoT scores fall below the no-finetuning baseline at 540B; adding the nine CoT datasets recovers them 2210.11416 §4.2 fig. 5. Finetuning costs between 0.2% and 1.6% of pretraining compute per model 2210.11416 §2.2 tab. 2. The checkpoint is chosen on the held-out tasks, so held-out gains are slightly optimistic 2210.11416 §2.2. Card: Scaling Instruction-Finetuned Language Models.

Model-size and repeated-epoch sweeps on a scientific corpus

A family of decoder-only models from 125M to 120B parameters, trained on a 106B-token corpus of scientific text 2211.09085 §4.1 tab. 2. Equation recall rises smoothly with size 2211.09085 tab. 6; perplexity on four holdout sets falls with size 2211.09085 tab. 16; protein function scores rise with scale 2211.09085 tab. 18. The paper fits no curve to these. The run covers about 4.25 epochs, 450B tokens over the corpus 2211.09085 §5.1. Validation loss keeps falling over four epochs for all sizes. The 120B model starts to overfit at the start of the fifth epoch, and its run is stopped there 2211.09085 §5.1 fig. 6. The gap between predicted and true citation counts shrinks with scale 2211.09085 §5.4.2 fig. 11. Card: Galactica: A Large Language Model for Science.

Qualitative scaling discussion in a survey

Not a method and not a fit. The survey states that scaling laws predict performance improving as a power law with compute, with diminishing returns 2307.10169 §2.3. It also notes that papers disagree on how to split a fixed compute budget between model size and data 2307.10169 §2.3. It gives no exponent and cites no fitted values for this topic. The paper is dated mid-2023, so later work is not covered. Card: Challenges and Applications of Large Language Models.

Data-size sweep for 3D view synthesis, curves only

Zero123-XL is trained on up to 10M objects and PixelNeRF on about 2M 2307.05663 §4.1 2307.05663 §4.2. Both improve as data grows 2307.05663 §4.1 2307.05663 §4.2 fig. 6. The curves appear only as a figure, with no table of values, and LPIPS is multiplied by 10 for display 2307.05663 app. A fig. 6. No exponent is fitted, so the trend beyond these sizes is unknown. Card: Objaverse-XL: A Universe of 10M+ 3D Objects.

Log-linear fit of training compute over time, per era

Not error against compute. The paper curates 123 milestone ML systems and estimates the training compute of each, then fits a least-squares line of log10 compute on publication date for each era 2202.05924 §1 app. A. Pre-2010 systems double about every 21.3 months, regular-scale systems from 2010 to 2022 about every 5.7 months, and large-scale systems from September 2015 about every 9.9 months, all as point estimates from table 2 2202.05924 tab. 2. The intervals are 95% bootstrap ranges, and the bootstrap also perturbs each compute estimate by a factor of up to two 2202.05924 app. A app. H. The large-scale group is chosen by a threshold set after looking at the data 2202.05924 §3.2. Card: Compute Trends Across Three Eras of Machine Learning.

Emergence as a jump against a curve fitted to small models

A survey definition, not a fit. An ability is called emergent when it is absent in smaller models, present in larger ones, and not predicted by a scaling curve fitted to the small models 2503.05788 §2. The survey gives one score for this: the spread of a metric across sizes, divided by a typical gap between sizes. The cited BIG-Bench meta-analysis treats a score above 100 as emergent, and the survey suggests 20 to 100 may also mark a jump 2503.05788 §3.1. The text does not define the typical gap precisely 2503.05788 §3.1. Whether a jump appears depends on the metric: some studies say it vanishes under linear metrics and others say it survives 2503.05788 §3.1. Card: Emergent Abilities in Large Language Models: A Survey.

When to use which

The paper's explanation for the data-source gap is a hypothesis about how the data was curated 2212.07143 §5. Do not assume it when you pick a data source for a different task.

Implementation sketch

The fit and forecast step, in plain form. The paper gives the fit in log-log space on the frontier points and the forecast from the fitted line.

# Fit E = beta * C**alpha on the frontier of measured runs  [arxiv:2212.07143 §4.1]
# Input:  runs as (compute C in GMAC, error rate E)
# [arxiv:2212.07143 app. "Predictions derived from scaling laws"]
# Output: exponent alpha and scale beta

def fit_frontier(runs, n_bins):
    # Divide compute into bins; keep the lowest-error run per bin
    # [arxiv:2212.07143 app. "Predictions derived from scaling laws"]
    frontier = best_run_per_bin(runs, n_bins)
    # Least-squares line through the frontier in log-log space  [arxiv:2212.07143 §4.1]
    alpha, log_beta = least_squares(log10(C(frontier)), log10(E(frontier)))
    return alpha, 10 ** log_beta

def forecast(C_new, alpha, beta):
    # Use only near the measured compute range  [arxiv:2212.07143 §5]
    return beta * C_new ** alpha

A schedule from the functional-scaling-law paper, in plain form. The paper states the power-decay shape for the one-pass SGD theorem; the peak rate must be tuned 2602.06797 thm. 6.3.

# Power decay over N steps  [arxiv:2602.06797 thm. 6.3]
# Input:  total steps N, decay power p, peak rate eta0 (tuned)
# Output: step size at step k

def power_decay(k, N, p, eta0):
    # Decay exponent must exceed q - 1, where q is the capacity  [arxiv:2602.06797 thm. 6.3]
    return eta0 * (1 - k / N) ** p

Contradictions

Between papers: the data-scale and data-quality papers point in different directions. Random subsets keep improving with size in the filtering study 2304.14108 §5.1, while the phi-1 paper argues that curated text lets a 1.3B model match much larger models trained on more tokens 2306.11644 §1 2306.11644 tab. 1. They test different tasks, image-text zero-shot and Python code, so this is not a direct conflict. Do not resolve it for either domain.

Inside the meta-analysis, the text and the table disagree. The text gives an ImageNet exponent of 12.5 2007.05558 §3.2, while the table's simplest fit gives 12.1 2007.05558 tab. 2a. For object detection, the text gives 17.1 2007.05558 §3.2 and the table gives 16.0 2007.05558 tab. 2b. The two projection forms also disagree by many orders of magnitude at a 1% top-1 error target 2007.05558 tab. 3, although the authors say the two fit the data about equally well 2007.05558 §3.3.

Inside the controlled study, the fit method is described two ways. The main text fits the Pareto frontier 2212.07143 §4.1. The appendix says the frontier is built by binning compute and taking the best run per bin 2212.07143 app. "Predictions derived from scaling laws". The paper does not state the bin count.

Inside the instruction-finetuning study, the 540B normalized average is 58.5 in table 3 2210.11416 tab. 3 and 58.4 in table 5 [arxiv:2210.11416 tab. 5]. The two tables use different mixture proportions [arxiv:2210.11416 app. E], so this is not a direct conflict, but the figures are not interchangeable. The main text gives 11B and 62B BBH numbers that differ from table 5 2210.11416 §7 tab. 5; use the table.

The survey says papers disagree on how to split a fixed compute budget between model size and data 2307.10169 §2.3. It does not name the papers, and the cards on this page do not settle the question, so treat it as open.

Between compute-trend analyses, the paper reports that earlier estimates disagree: one gives a 3.4-month doubling time for 2012 to 2018, another more than two years for 2018 to 2020 2202.05924 §2 tab. 1. The authors attribute the gap to small earlier datasets that assumed one trend for all models 2202.05924 §3.2 app. E. Their own large-scale trend is chosen after the fact 2202.05924 §3.2, so treat its slope as one reading.

Inside the emergence literature, the survey reports that some studies find jumps vanish under linear metrics and others find they survive, and that validation-loss studies report transitions 2503.05788 §3.1. It does not settle the dispute, and the card does not either.

Inside the protein-model sweep, held-out perplexity falls with size on Test-max90 2206.13517 tab. 2, while zero-shot ranking on narrow landscapes peaks at 764M 2206.13517 tab. 3. These are different measures, so this is not a direct conflict. The paper fits no law to either, and the cards do not resolve which measure to plan around.

The paper's text gives global batch sizes of 86 to 88K 2212.07143 §3.2, while its per-run table lists smaller global batches for several runs tab. 18. The appendix says training uses up to 1520 GPUs 2212.07143 app. "Experimental details", while the largest run in the table uses 1024 tab. 18. Use the table values for per-run settings.

Papers

• Reproducible scaling laws for contrastive language-image learning: fits error against compute for OpenCLIP on LAION data and OpenAI CLIP on WIT-400M, with exponents per task, and forecasts that are not measured.
• The Computational Limits of Deep Learning: regresses benchmark error on estimated training compute for published results, then projects the compute needed for lower error; the estimates and projections are extrapolations.
• DataComp: In search of the next generation of multimodal datasets: a benchmark for image-text data filtering at four compute scales, reporting how filter rankings hold across scales.
• Textbooks Are All You Need: a 1.3B code model trained on curated and generated text, with reported HumanEval and MBPP results and no fitted scaling law.
• Learning-rate schedules under a functional scaling law: derives step-size schedules for one-pass SGD on linear models under a functional scaling law, with small neural-network checks.
• ProGen2: Exploring the Boundaries of Protein Language Models: a size sweep of protein language models from 151M to 6.4B parameters, with perplexity and zero-shot ranking reported per size and no fitted law.
• Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation: five transformer diffusion-policy sizes from 10M to 1B, with adaptive layer norm and non-causal action attention, on author-run robot tasks.
• From Word to World: Can Large Language Models be Implicit Text-based World Models?: a text-world simulator with data and model-size sweeps, reported as saturation points and figures.
• Scaling Instruction-Finetuned Language Models: instruction finetuning on 1,836 tasks with and without CoT data, across T5, PaLM and U-PaLM up to 540B, with held-out scores per size and no fitted law.
• Galactica: A Large Language Model for Science: a 125M to 120B model family on a curated scientific corpus, with size sweeps in tables, repeated-epoch behaviour and no fitted law.
• Objaverse-XL: A Universe of 10M+ 3D Objects: a web-scale 3D object dataset, with Zero123-XL and PixelNeRF data-size trends shown only as a figure.
• Challenges and Applications of Large Language Models: a mid-2023 survey of open LLM challenges, with a qualitative scaling discussion and no experiments.
• Compute Trends Across Three Eras of Machine Learning: log-linear fits of training-compute doubling times for 123 milestone systems, with author-estimated compute and bootstrap intervals.
• Emergent Abilities in Large Language Models: A Survey: a survey of how emergence is defined and measured, covering metric artifacts and mechanisms, with no experiments of its own.