---
kind: concept
slug: neural-scaling-laws
title: Neural scaling laws
summary: >-
  How to fit a power law of error against a training resource, forecast larger runs, and see where the forecast breaks. Covers CLIP runs, published-compute meta-analysis, data filtering, a small model on curated data, a learning-rate schedule, and size, task or data sweeps with no fit, including a robot diffusion policy. Forecasts beyond the measured range are extrapolations.
tags: [scaling, evaluation, vision-language, data-quality, learning-rate, protein, agents, instruction-tuning, scientific-llm, 3d-vision, robotics, diffusion-policy]
papers: ["2007.05558", "2206.13517", "2210.11416", "2211.09085", "2212.07143", "2304.14108", "2306.11644", "2307.05663", "2307.10169", "2409.14411", "2512.18832", "2602.06797", "2202.05924", "2503.05788"]
related: [emergent-abilities, language-model-pretraining, vision-language-pretraining, contrastive-learning, zero-shot-learning, dataset-curation, learning-rate-schedule, code-completion, world-model, instruction-tuning, chain-of-thought-prompting, training-compute-estimation, novel-view-synthesis, diffusion-models, imitation-learning, adaptive-layer-normalization]
status: draft
updated_at: 2026-10-31
---

## Overview

A scaling law is a fitted power law that maps a training resource to a
downstream error rate. The clearest form is E = β·C^α, where E is the error
rate, C is compute and α is the exponent [arxiv:2212.07143 §4.1]. The
controlled study varies model size, pre-training data size and samples seen,
and counts compute in GMAC [arxiv:2212.07143 app. "Predictions derived from
scaling laws"]. The meta-analysis instead uses compute estimated for published
papers [arxiv:2007.05558 §3.2].

The invariant the fitted variants share: over the scales measured, the error
follows a power law in the resource, so one slope and one intercept summarise
a set of runs [arxiv:2212.07143 §4.1]. The controlled study reports this for
zero-shot classification, retrieval, linear probing and fine-tuning
[arxiv:2212.07143 §5]. The meta-analysis fits one slope per benchmark
[arxiv:2007.05558 §3.2]. Neither claims the law holds beyond the range it
measured [arxiv:2212.07143 §5] [arxiv:2007.05558 §3.3]. A separate fit
tracks training compute against publication date, not error against compute.
It gives a doubling time per era [arxiv:2202.05924 §1] [tab. 2], so it
describes the growth of the resource, not the error it buys.

Ten papers do not fit a curve of their own. Six sweep model size, data or task count and
describe the trend without a fitted law: a protein model family
[arxiv:2206.13517 §4.1], a text-world simulator [arxiv:2512.18832 §6.1],
instruction finetuning across model sizes [arxiv:2210.11416 §3], a scientific
language model family [arxiv:2211.09085 tab. 6], 3D training data for view
synthesis [arxiv:2307.05663 §4.1] and a transformer diffusion policy
[arxiv:2409.14411 §4.2]. One argues
that curated data can substitute for scale [arxiv:2306.11644 §1]. The other
derives a schedule from a scaling law for the loss [arxiv:2602.06797 §3.2].
Two are surveys that describe the trend in words and run no experiments:
one on scaling in general [arxiv:2307.10169 §2.3], and one on emergence, which
defines a jump against a curve fitted to small models [arxiv:2503.05788 §2].

## Variants

### Power-law fit on the best run per compute bin

The forecasting method. Runs are plotted by compute and error, and the fit
uses the Pareto frontier, the lowest-error run in each compute bin, because
the total compute of different runs is not exactly aligned
[arxiv:2212.07143 app. "Predictions derived from scaling laws"]. A line is
fitted in log-log space through those points [arxiv:2212.07143 §4.1]. The
number of bins is not stated in the paper's text. Card:
[Reproducible scaling laws for contrastive language-image learning](/papers/2212.07143).

### Scaling the pre-training data source

The same model family trained on two data sources, LAION (open) and OpenAI's
private WIT-400M. The fitted exponents differ by task. For zero-shot ImageNet
classification the reported exponents are −0.11 for OpenCLIP on LAION and
−0.16 for OpenAI CLIP; for MS-COCO retrieval R@5 they are −0.08 and −0.05;
for Flickr30K retrieval R@5 they are −0.19 and −0.10 [arxiv:2212.07143 §4.1]
[arxiv:2212.07143 §4.2]. A more negative exponent means faster improvement
[arxiv:2212.07143 §4.1]. Card:
[Reproducible scaling laws for contrastive language-image learning](/papers/2212.07143).

### Scaling samples seen with a schedule per run

The training-length axis. Each samples-seen scale gets its own cosine annealing
schedule, so a short run is not read off the tail of a longer one
[arxiv:2212.07143 §3.2]. The paper calls this a way to avoid suboptimal
training for runs of different length [arxiv:2212.07143 §3.2]. Card:
[Reproducible scaling laws for contrastive language-image learning](/papers/2212.07143).

### Literature meta-analysis: benchmark error against estimated compute

Instead of training runs, this method collects published deep-learning results
for six benchmarks, estimates each model's training compute, and regresses log
error on log compute per benchmark [arxiv:2007.05558 §3.2] [tab. 2a]. For
ImageNet, the simplest fit gives an exponent of 12.1 with a 95% interval of
10.6 to 14.1 [arxiv:2007.05558 tab. 2a]. The headline reading is that halving
ImageNet error needs about 5,000 times more compute, with a range of roughly
1,500 to 17,500 [arxiv:2007.05558 §3.2]. A back-of-envelope argument gives a
lower bound: computation grows at least as the fourth power of performance,
where performance is 1/error [arxiv:2007.05558 §2]. Most papers do not report
compute, so many values are derived. For models with only network operations,
hardware burden comes from a fit on ImageNet alone [arxiv:2007.05558 app. §8.1].
Card: [The Computational Limits of Deep Learning](/papers/2007.05558).

### Data filtering rules measured at four scales

A benchmark for choosing image-text training data. The candidate pool holds
12.8B pairs, and four scales each multiply the samples seen and the pool
size by 10, from 12.8M to 12.8B [arxiv:2304.14108 §3.1]. The best baseline,
DataComp-1B with ViT-L/14, reaches 79.2% zero-shot ImageNet at the same
compute as OpenAI's 75.5% [arxiv:2304.14108 tab. 1]. Filtering beats no
filtering at every scale [arxiv:2304.14108 §5.1]. Random subsets keep
improving with size, while CLIP-score filtering peaks at an intermediate size
near the top 30% [arxiv:2304.14108 §5.1] [fig. 2]. Card:
[DataComp: In search of the next generation of multimodal datasets](/papers/2304.14108).

### Small model on curated text instead of more data

A 1.3B-parameter code model. Pre-training uses about 7B tokens of filtered
code and generated textbook-style text, then about 180M tokens of generated
exercises for fine-tuning [arxiv:2306.11644 §2.3] [tab. 1]. The authors report
50.6% pass@1 on HumanEval and 55.5% on MBPP [arxiv:2306.11644 tab. 1]. Their
comparison includes models trained on up to 1T tokens [arxiv:2306.11644 tab. 1].
The paper cites TinyStories as an earlier case where data quality changed the
scaling picture [arxiv:2306.11644 §1]. That is the authors' argument, not a
fitted law. Card:
[Textbooks Are All You Need](/papers/2306.11644).

### Learning-rate schedule derived from a functional scaling law

The paper takes a functional scaling law from prior work as given and derives
step-size schedules for one-pass SGD on a linear model with power-law features
[arxiv:2602.06797 §3.2]. For easy tasks, power decay from the first step is
optimal. For hard tasks, the rate holds at the stability limit for an initial
stretch and then decays over the last fraction of training
[arxiv:2602.06797 thm. 4.1]. The results are asymptotic, up to constants, and
the paper gives no numeric peak rate [arxiv:2602.06797 §4]. Its neural-network
runs are small, and the best decay ratio changes with peak rate and batch size
[arxiv:2602.06797 §7] [arxiv:2602.06797 app. D.2]. Card:
[Learning-rate schedules under a functional scaling law](/papers/2602.06797).

### Model-size sweep read from perplexity and zero-shot ranking

A family of four decoder-only protein models, from 151M to 6.4B parameters
[arxiv:2206.13517 §3.1] [tab. 1]. On the Test-max90 held-out set, perplexity
falls from 12.9 to 9.9 across the family [arxiv:2206.13517 tab. 2]. On
Test-max50 it does not fall at every step: 14.3 at 764M, 14.4 at 2.7B, then
13.9 at 6.4B [arxiv:2206.13517 tab. 2]. The paper fits no curve. For zero-shot
ranking on narrow fitness landscapes, the best single size is 764M, with an
average Spearman of 0.505, against 0.476 at 6.4B [arxiv:2206.13517 tab. 3].
Card: [ProGen2: Exploring the Boundaries of Protein Language Models](/papers/2206.13517).

### Robot diffusion-policy size sweep with a changed recipe

Diffusion Policy in a transformer (DP-T) is the baseline. The authors report
that it does not improve with depth: 80.1% success at 8 layers, 78.4% at 12
and 74.6% at 14 [arxiv:2409.14411 §1] [fig. 1]. Their ScaleDP changes two
things at once. It replaces cross-attention conditioning with adaptive layer
norm, and it removes the causal mask from action self-attention
[arxiv:2409.14411 §3.2]. Five sizes, from Ti at 10M to H at 1B parameters,
are defined in [arxiv:2409.14411 tab. 1]. On a four-task Franka set the
average success rises from 58.75% at S to 87.50% at H [arxiv:2409.14411
tab. 2]. On three bimanual UR5 tasks it rises from 53.33% at S to 98.33% at H
[arxiv:2409.14411 tab. 3]. The MetaWorld curves by size and demonstration
count appear only as figures, with no numbers in the text
[arxiv:2409.14411 §4.2] [fig. 4]. The results are author-reported, and each
real-robot success rate uses 20 trials per task [arxiv:2409.14411 §4.1]. The
scaling statements are qualitative, with no exponent to extrapolate. Card:
[Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation](/papers/2409.14411).

### Data and model-size sweeps for a text-world simulator

A fine-tuned LLM that predicts the next observation of a text environment is
trained on logged agent trajectories [arxiv:2512.18832 §3.1] [§4]. The paper
varies the amount of training data per environment. Structured environments
saturate near 20K trajectories, WebShop keeps improving up to about 70K, and
StableToolBench shows no saturation at 160K [arxiv:2512.18832 §6.1] [fig. 2].
The paper describes these saturation points, not a fitted exponent. The model
size sweep covers Qwen2.5 at 0.5B to 7B [arxiv:2512.18832 app. A.1]. A 1.5B
model already captures the structured dynamics, while open-ended settings gain
from more capacity [arxiv:2512.18832 §6.2] [fig. 3]. The training sizes are
caps in the released config, and the file sizes differ from the stated counts
[arxiv:2512.18832 app. A.1]. Card:
[From Word to World: Can Large Language Models be Implicit Text-based World Models?](/papers/2512.18832).

### Task count, model size and CoT share in instruction finetuning

No fitted law. The study finetunes pretrained checkpoints on a mixture of
1,836 instruction tasks, and varies the number of tasks, the model size and
the share of chain-of-thought (CoT) data [arxiv:2210.11416 §1]. Every size
gains over its own no-finetuning baseline: by 15.5 points of normalized
average at 8B, 10.4 at 62B and 9.4 at 540B [arxiv:2210.11416 §3] [tab. 3].
Most of that gain is in place by 282 tasks. For Flan-PaLM at 540B the
normalized average is 57.5 at 282 tasks and 58.5 at 1,836 [arxiv:2210.11416
tab. 3]; the paper reads the later gains as small [arxiv:2210.11416 §3]
[fig. 4]. Without CoT examples in the mixture, held-out CoT scores fall below
the no-finetuning baseline at 540B; adding the nine CoT datasets recovers them
[arxiv:2210.11416 §4.2] [fig. 5]. Finetuning costs between 0.2% and 1.6% of
pretraining compute per model [arxiv:2210.11416 §2.2] [tab. 2]. The
checkpoint is chosen on the held-out tasks, so held-out gains are slightly
optimistic [arxiv:2210.11416 §2.2]. Card:
[Scaling Instruction-Finetuned Language Models](/papers/2210.11416).

### Model-size and repeated-epoch sweeps on a scientific corpus

A family of decoder-only models from 125M to 120B parameters, trained on a
106B-token corpus of scientific text [arxiv:2211.09085 §4.1] [tab. 2]. Equation
recall rises smoothly with size [arxiv:2211.09085 tab. 6]; perplexity on four
holdout sets falls with size [arxiv:2211.09085 tab. 16]; protein function
scores rise with scale [arxiv:2211.09085 tab. 18]. The paper fits no curve to
these. The run covers about 4.25 epochs, 450B tokens over the corpus
[arxiv:2211.09085 §5.1]. Validation loss keeps falling over four epochs for all
sizes. The 120B model starts to overfit at the start of the fifth epoch, and
its run is stopped there [arxiv:2211.09085 §5.1] [fig. 6]. The gap between
predicted and true citation counts shrinks with scale [arxiv:2211.09085 §5.4.2]
[fig. 11]. Card: [Galactica: A Large Language Model for Science](/papers/2211.09085).

### Qualitative scaling discussion in a survey

Not a method and not a fit. The survey states that scaling laws predict
performance improving as a power law with compute, with diminishing returns
[arxiv:2307.10169 §2.3]. It also notes that papers disagree on how to split a
fixed compute budget between model size and data [arxiv:2307.10169 §2.3]. It
gives no exponent and cites no fitted values for this topic. The paper is
dated mid-2023, so later work is not covered. Card:
[Challenges and Applications of Large Language Models](/papers/2307.10169).

### Data-size sweep for 3D view synthesis, curves only

Zero123-XL is trained on up to 10M objects and PixelNeRF on about 2M
[arxiv:2307.05663 §4.1] [arxiv:2307.05663 §4.2]. Both improve as data grows
[arxiv:2307.05663 §4.1] [arxiv:2307.05663 §4.2] [fig. 6]. The curves appear
only as a figure, with no table of values, and LPIPS is multiplied by 10 for
display [arxiv:2307.05663 app. A] [fig. 6]. No exponent is fitted, so the
trend beyond these sizes is unknown. Card:
[Objaverse-XL: A Universe of 10M+ 3D Objects](/papers/2307.05663).

### Log-linear fit of training compute over time, per era

Not error against compute. The paper curates 123 milestone ML systems and
estimates the training compute of each, then fits a least-squares line of
log10 compute on publication date for each era [arxiv:2202.05924 §1]
[app. A]. Pre-2010 systems double about every 21.3 months, regular-scale
systems from 2010 to 2022 about every 5.7 months, and large-scale systems from
September 2015 about every 9.9 months, all as point estimates from table 2
[arxiv:2202.05924 tab. 2]. The intervals are 95% bootstrap ranges, and the
bootstrap also perturbs each compute estimate by a factor of up to two
[arxiv:2202.05924 app. A] [app. H]. The large-scale group is chosen by a
threshold set after looking at the data [arxiv:2202.05924 §3.2]. Card:
[Compute Trends Across Three Eras of Machine Learning](/papers/2202.05924).

### Emergence as a jump against a curve fitted to small models

A survey definition, not a fit. An ability is called emergent when it is
absent in smaller models, present in larger ones, and not predicted by a
scaling curve fitted to the small models [arxiv:2503.05788 §2]. The survey
gives one score for this: the spread of a metric across sizes, divided by a
typical gap between sizes. The cited BIG-Bench meta-analysis treats a score
above 100 as emergent, and the survey suggests 20 to 100 may also mark a jump
[arxiv:2503.05788 §3.1]. The text does not define the typical gap precisely
[arxiv:2503.05788 §3.1]. Whether a jump appears depends on the metric: some
studies say it vanishes under linear metrics and others say it survives
[arxiv:2503.05788 §3.1]. Card:
[Emergent Abilities in Large Language Models: A Survey](/papers/2503.05788).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need to estimate a bigger controlled run before paying for it | Fit on smaller runs, forecast only close to the measured range | The paper's own forecasts are extrapolations; it advises extrapolating only to scales near those measured [arxiv:2212.07143 §5] |
| Target task is zero-shot ImageNet classification and your data is LAION-style open data | Expect a shallower slope than OpenAI CLIP on private data | Reported −0.11 vs −0.16 [arxiv:2212.07143 §4.1] |
| Target task is image-text retrieval and your data is LAION-style open data | Expect a steeper slope than OpenAI CLIP on private data | Reported −0.08 vs −0.05 on MS-COCO and −0.19 vs −0.10 on Flickr30K [arxiv:2212.07143 §4.2] |
| You plan to add more samples seen but keep the same data | Check for a data bottleneck first | ViT-L/14 on LAION-400M barely improved from 13B to 34B samples seen [arxiv:2212.07143 §4.2] |
| You compare recipes trained with different batch sizes | Expect a small effect, check with a control | Batch-size changes moved zero-shot ImageNet by 0.2 to 0.5% [arxiv:2212.07143 §3.2] |
| You can only afford small runs | Read the fit as a rough guide | The paper names limited sampling of the scale space as a limitation [arxiv:2212.07143 §5] |
| You have only published results and no runs of your own | Use the per-benchmark slope as an order-of-magnitude guide | Compute is estimated for most papers, and the text and table give different ImageNet exponents [arxiv:2007.05558 §3.2] [tab. 2a] |
| You are tempted to plan around a far error target from a projection | Treat the projection as a warning, not a plan | The polynomial and exponential fits describe the data about equally well but differ by orders of magnitude at 1% top-1 error [arxiv:2007.05558 §3.3] [tab. 3] |
| You pick a data filter for image-text data at a fixed budget | Re-check the ranking at your own scale | Filters beat no filtering at each scale, but random subsets and CLIP filtering grow differently with size [arxiv:2304.14108 §5.1]; rank correlation across scales is 0.71 to 0.90 [arxiv:2304.14108 §5.2] |
| You need a small model for a narrow code task and have a strong model to generate data | Treat the curated-text recipe as one data point | Reported on HumanEval and MBPP for a 1.3B model [arxiv:2306.11644 tab. 1]; the generation prompts are not published [arxiv:2306.11644 §2.2] and the pass@1 decoding setup is not stated [arxiv:2306.11644 tab. 1] |
| You train with one pass over the data and must choose a learning-rate decay | Use power decay for easy tasks, hold then decay for hard tasks, with a tuned peak | The shapes come from theory for linear models [arxiv:2602.06797 thm. 4.1]; the paper gives no finite-N rule for the decay point [arxiv:2602.06797 §4] |
| You pick a protein language model size for zero-shot variant ranking | Compare a few sizes on your own landscape; do not assume the largest ranks best | Narrow-landscape ranking peaks at 764M [arxiv:2206.13517 tab. 3]; perplexity and ranking did not move together across the family [arxiv:2206.13517 tab. 2] [tab. 3] |
| You need to know how many agent trajectories to collect for a text-world simulator | Find the saturation point in your own environment; the paper's points differ by environment | Structured tasks saturate near 20K, WebShop not until about 70K, StableToolBench not by 160K [arxiv:2512.18832 §6.1] [fig. 2] |
| You plan instruction finetuning and wonder whether more training tasks help | Add tasks, but expect the gain to flatten after a few hundred | Normalized average at 282 vs 1,836 tasks: 20.5 vs 21.9 at 8B, 37.9 vs 38.8 at 62B, 57.5 vs 58.5 at 540B [arxiv:2210.11416 tab. 3] |
| Your instruction mix has no chain-of-thought examples and you need reasoning on held-out tasks | Include CoT examples in the mix | Finetuning without them lowered held-out CoT scores below the baseline at 540B [arxiv:2210.11416 §4.2] [fig. 5] |
| You compare the finetuning gain across model sizes | Expect a smaller gain at larger size, with a higher absolute score | In this family the gain is 15.5, 10.4 and 9.4 points at 8B, 62B and 540B, while the scores rise [arxiv:2210.11416 §3] [tab. 3]; one model family only |
| You plan several epochs over a fixed text corpus | Watch validation loss for overfitting in the last epochs, and stop before it rises | In the scientific-corpus run, the largest model began to overfit at the start of its fifth epoch [arxiv:2211.09085 §5.1] [fig. 6]; the 30B model shows a per-epoch plateau-then-decline pattern [arxiv:2211.09085 fig. 7] |
| You want an overview of open scaling questions before planning a run | Read the survey for the open problems, not for numbers | It runs no experiments, and the figures it quotes come from other work [arxiv:2307.10169 §1] [§2.3] |
| You plan to grow a 3D object dataset to improve view synthesis | Expect gains with more data across the measured range; the paper reports no slope to extrapolate | Zero123-XL and PixelNeRF improve up to about 10M and 2M objects, shown only as curves [arxiv:2307.05663 §4.1] [fig. 6] |
| You plan to add depth to a transformer diffusion policy and expect it to keep improving | Change the conditioning and the attention mask along with the size, and compare at your own size | Depth alone fell from 80.1% at 8 layers to 74.6% at 14 [arxiv:2409.14411 §1] [fig. 1]; the paper's ScaleDP changes conditioning and the mask together [arxiv:2409.14411 §3.2] |
| You need a rough rate for how training compute of frontier systems grows | Use the era-matched doubling time as an order of magnitude, not a forecast | Regular-scale 2010 to 2022 is about 5.7 months and large-scale about 9.9 months [arxiv:2202.05924 tab. 2]; compute is estimated to about a factor of two [arxiv:2202.05924 app. H] |
| You need to predict whether a capability appears at a larger size | Do not rely on a curve fitted to small models alone; test the metric you will report | A jump is defined against such a curve, and the metric decides whether it shows [arxiv:2503.05788 §2] [§3.1] |
| You keep a causal mask in the action self-attention of a transformer policy | Try removing it; on the paper's bimanual tasks it helped at each size tested | Average rises 33.33 to 53.33 at S, 50.00 to 56.67 at B and 66.66 to 90.00 at L [arxiv:2409.14411 tab. 4]; 20 trials per task, one task set |

The paper's explanation for the data-source gap is a hypothesis about how the
data was curated [arxiv:2212.07143 §5]. Do not assume it when you pick a data
source for a different task.

## Implementation sketch

The fit and forecast step, in plain form. The paper gives the fit in log-log
space on the frontier points and the forecast from the fitted line.

```pseudocode
# Fit E = beta * C**alpha on the frontier of measured runs  [arxiv:2212.07143 §4.1]
# Input:  runs as (compute C in GMAC, error rate E)
# [arxiv:2212.07143 app. "Predictions derived from scaling laws"]
# Output: exponent alpha and scale beta

def fit_frontier(runs, n_bins):
    # Divide compute into bins; keep the lowest-error run per bin
    # [arxiv:2212.07143 app. "Predictions derived from scaling laws"]
    frontier = best_run_per_bin(runs, n_bins)
    # Least-squares line through the frontier in log-log space  [arxiv:2212.07143 §4.1]
    alpha, log_beta = least_squares(log10(C(frontier)), log10(E(frontier)))
    return alpha, 10 ** log_beta

def forecast(C_new, alpha, beta):
    # Use only near the measured compute range  [arxiv:2212.07143 §5]
    return beta * C_new ** alpha
```

A schedule from the functional-scaling-law paper, in plain form. The paper
states the power-decay shape for the one-pass SGD theorem; the peak rate must
be tuned [arxiv:2602.06797 thm. 6.3].

```pseudocode
# Power decay over N steps  [arxiv:2602.06797 thm. 6.3]
# Input:  total steps N, decay power p, peak rate eta0 (tuned)
# Output: step size at step k

def power_decay(k, N, p, eta0):
    # Decay exponent must exceed q - 1, where q is the capacity  [arxiv:2602.06797 thm. 6.3]
    return eta0 * (1 - k / N) ** p
```

## Contradictions

Between papers: the data-scale and data-quality papers point in different
directions. Random subsets keep improving with size in the filtering study
[arxiv:2304.14108 §5.1], while the phi-1 paper argues that curated text lets a
1.3B model match much larger models trained on more tokens
[arxiv:2306.11644 §1] [arxiv:2306.11644 tab. 1]. They test different tasks,
image-text zero-shot and Python code, so this is not a direct conflict. Do not
resolve it for either domain.

Inside the meta-analysis, the text and the table disagree. The text gives an
ImageNet exponent of 12.5 [arxiv:2007.05558 §3.2], while the table's simplest
fit gives 12.1 [arxiv:2007.05558 tab. 2a]. For object detection, the text gives
17.1 [arxiv:2007.05558 §3.2] and the table gives 16.0 [arxiv:2007.05558 tab. 2b].
The two projection forms also disagree by many orders of magnitude at a 1%
top-1 error target [arxiv:2007.05558 tab. 3], although the authors say the two
fit the data about equally well [arxiv:2007.05558 §3.3].

Inside the controlled study, the fit method is described two ways. The main
text fits the Pareto frontier [arxiv:2212.07143 §4.1]. The appendix says the
frontier is built by binning compute and taking the best run per bin
[arxiv:2212.07143 app. "Predictions derived from scaling laws"]. The paper does
not state the bin count.

Inside the instruction-finetuning study, the 540B normalized average is 58.5
in table 3 [arxiv:2210.11416 tab. 3] and 58.4 in table 5 [arxiv:2210.11416
tab. 5]. The two tables use different mixture proportions [arxiv:2210.11416
app. E], so this is not a direct conflict, but the figures are not
interchangeable. The main text gives 11B and 62B BBH numbers that differ from
table 5 [arxiv:2210.11416 §7] [tab. 5]; use the table.

The survey says papers disagree on how to split a fixed compute budget
between model size and data [arxiv:2307.10169 §2.3]. It does not name the
papers, and the cards on this page do not settle the question, so treat it as
open.

Between compute-trend analyses, the paper reports that earlier estimates
disagree: one gives a 3.4-month doubling time for 2012 to 2018, another more
than two years for 2018 to 2020 [arxiv:2202.05924 §2] [tab. 1]. The authors
attribute the gap to small earlier datasets that assumed one trend for all
models [arxiv:2202.05924 §3.2] [app. E]. Their own large-scale trend is chosen
after the fact [arxiv:2202.05924 §3.2], so treat its slope as one reading.

Inside the emergence literature, the survey reports that some studies find
jumps vanish under linear metrics and others find they survive, and that
validation-loss studies report transitions [arxiv:2503.05788 §3.1]. It does
not settle the dispute, and the card does not either.

Inside the protein-model sweep, held-out perplexity falls with size on
Test-max90 [arxiv:2206.13517 tab. 2], while zero-shot ranking on narrow
landscapes peaks at 764M [arxiv:2206.13517 tab. 3]. These are different
measures, so this is not a direct conflict. The paper fits no law to either,
and the cards do not resolve which measure to plan around.

The paper's text gives global batch sizes of 86 to 88K [arxiv:2212.07143 §3.2],
while its per-run table lists smaller global batches for several runs
[tab. 18]. The appendix says training uses up to 1520 GPUs
[arxiv:2212.07143 app. "Experimental details"], while the largest run in the
table uses 1024 [tab. 18]. Use the table values for per-run settings.

## Papers

- [Reproducible scaling laws for contrastive language-image learning](/papers/2212.07143): fits error against compute for OpenCLIP on LAION data and OpenAI CLIP on WIT-400M, with exponents per task, and forecasts that are not measured.
- [The Computational Limits of Deep Learning](/papers/2007.05558): regresses benchmark error on estimated training compute for published results, then projects the compute needed for lower error; the estimates and projections are extrapolations.
- [DataComp: In search of the next generation of multimodal datasets](/papers/2304.14108): a benchmark for image-text data filtering at four compute scales, reporting how filter rankings hold across scales.
- [Textbooks Are All You Need](/papers/2306.11644): a 1.3B code model trained on curated and generated text, with reported HumanEval and MBPP results and no fitted scaling law.
- [Learning-rate schedules under a functional scaling law](/papers/2602.06797): derives step-size schedules for one-pass SGD on linear models under a functional scaling law, with small neural-network checks.
- [ProGen2: Exploring the Boundaries of Protein Language Models](/papers/2206.13517): a size sweep of protein language models from 151M to 6.4B parameters, with perplexity and zero-shot ranking reported per size and no fitted law.
- [Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation](/papers/2409.14411): five transformer diffusion-policy sizes from 10M to 1B, with adaptive layer norm and non-causal action attention, on author-run robot tasks.
- [From Word to World: Can Large Language Models be Implicit Text-based World Models?](/papers/2512.18832): a text-world simulator with data and model-size sweeps, reported as saturation points and figures.
- [Scaling Instruction-Finetuned Language Models](/papers/2210.11416): instruction finetuning on 1,836 tasks with and without CoT data, across T5, PaLM and U-PaLM up to 540B, with held-out scores per size and no fitted law.
- [Galactica: A Large Language Model for Science](/papers/2211.09085): a 125M to 120B model family on a curated scientific corpus, with size sweeps in tables, repeated-epoch behaviour and no fitted law.
- [Objaverse-XL: A Universe of 10M+ 3D Objects](/papers/2307.05663): a web-scale 3D object dataset, with Zero123-XL and PixelNeRF data-size trends shown only as a figure.
- [Challenges and Applications of Large Language Models](/papers/2307.10169): a mid-2023 survey of open LLM challenges, with a qualitative scaling discussion and no experiments.
- [Compute Trends Across Three Eras of Machine Learning](/papers/2202.05924): log-linear fits of training-compute doubling times for 123 milestone systems, with author-estimated compute and bootstrap intervals.
- [Emergent Abilities in Large Language Models: A Survey](/papers/2503.05788): a survey of how emergence is defined and measured, covering metric artifacts and mechanisms, with no experiments of its own.