The Computational Limits of Deep Learning

In plain words

Many AI results of the last decade, such as image recognition, translation and game playing, came from training larger models on more hardware. This paper asks how much of the progress depends on that extra computing. The authors collected published training costs and benchmark scores, then fitted a curve for each benchmark that shows how much the error drops for each increase in compute.

The steepness of the curve is the key number. On the ImageNet image-classification benchmark, the fit implies that halving the error needs about 5,000 times more compute. If that trend continued, the cost in money and energy would become unsustainable, so the authors argue that progress will need much more efficient methods or different kinds of machine learning.

The numbers describe past trends across published papers, not a guarantee of the future. Most papers do not report how much compute they used, so the compute values are estimates. The curves are only as good as those estimates.

TL;DR

• Meta-analysis of about 1,526 deep-learning papers from arXiv and other sources, covering six benchmarks in the main analysis §3.2. The introduction gives 1,527 §1.
• For each benchmark it regresses log error on log compute, reporting an implied polynomial exponent tab. 2.
• ImageNet: compute scales about as performance to the power 12 (fitted value 12.1), so halving error needs about 5,000 times more compute §3.2 tab. 2a.
• Relevant if you need to estimate training compute from a paper's reported numbers, or want a worked scaling-fit recipe. It proposes no model.

Problem

Deep-learning progress is assumed to need more computing power, but the size of that need and whether it can continue were unclear. The authors argue that the cost is structural: a model with more parameters than training examples costs compute roughly in proportion to parameters times data, and error falls only as the inverse square root of the data size §2. Together these give a rough lower bound of computation growing at least as the fourth power of performance (1/RMSE) §2.

Existing scaling studies usually test one model family. The authors instead measure the field as a whole, which they argue captures innovation as well as scale §4. Their scope is limited to benchmarks with exact right or wrong answers, so generative language and audio models are excluded §3.2. Few papers report the compute they used, so most of the analysis rests on estimates §3.2.

Method

The study has three parts: collecting data, estimating compute, and fitting and projecting.

1. Data collection. Papers were found by searching arXiv and benchmark sites for specific tasks and benchmarks. A manual review recorded the benchmark, metric, network size, training details and hardware. Results were cross-checked against paperswithcode §3.2 app. §7.1.
2. Compute estimates. Two measures are used. Network operations sum, over pre-training, training and fine-tuning, the epochs times the operations per forward pass times the passes per epoch eq. 1. Hardware burden sums, over the same stages, the processors times their computation rate times the time used eq. 2. Hardware burden depends on implementation efficiency and is less precise §3.2.
3. Conversion. Where a model has only network operations, a log-log regression fitted on ImageNet models that report both measures converts them to hardware burden. The authors report R² of 0.84 for this fit app. §8.1 fig. 6. They apply the same ImageNet conversion to MS COCO app. §8.1.
4. Scaling fit. For each benchmark, ordinary least squares regresses log error on log computation §3.2 tab. 2. The fitted slope gives the exponent α in computation ∝ performance^α, with performance = 1/error §3.2.
5. Robustness checks on ImageNet tab. 2a:
   • spec 2 adds a time trend, measured in years, to capture algorithmic progress §3.2;
   • spec 3 uses an exponential form, regressing linear error on log computation §3.2;
   • spec 4 is a 10% quantile regression, which targets the best-performing models §3.2.
6. Projection. The fitted polynomial and exponential forms are extended to target error rates. The required compute is converted to CO₂ and dollars using cost methods cited to other work §3.3 tab. 3.

# Compute estimates per model  [eq. 1] [eq. 2]
# Input:  per stage (pre-training, training, fine-tuning):
#         epochs, flops per pass, passes per epoch
#         or processors, rate, time
# Output: network operations and hardware burden (FLOPs)

def network_operations(stages):
    # Sum epochs x flops per pass x passes per epoch  [eq. 1]
    total = 0
    for stage in stages:
        total += (
            stage.epochs
            * stage.flops_per_pass
            * stage.passes_per_epoch
        )
    return total

def hardware_burden(stages):
    # Sum processors x computation rate x time  [eq. 2]
    total = 0
    for stage in stages:
        total += stage.processors * stage.rate * stage.time
    return total

def fit_scaling(computation, error):
    # OLS of log10(error) on log10(computation)  [§3.2] [tab. 2a]
    x = log10(computation)
    y = log10(error)
    slope, intercept = ols(x, y)
    # Implied exponent: computation ~ performance^alpha  [§3.2]
    alpha = -1 / slope
    return alpha, intercept

Key relations used by the method:

\text{Computation} \propto \text{Performance}^{\alpha}, \quad \text{Performance} = 1/\text{error} §3.2

\log_{10}(\text{error}) = a + b\,\log_{10}(\text{computation}), \quad \alpha = -1/b §3.2

\text{Computation} = \Omega(\text{Performance}^{4}) §2 (back-of-envelope lower bound)

Hyperparameters

The paper trains no models. These are its analysis settings.

Results

Author-reported. Setup: published deep-learning papers per benchmark, compute estimated as above. The scaling exponent is the implied polynomial exponent. No independent replication is reported.

The paper's headline is that halving ImageNet error needs about 2^12.5, which is more than 5,000 times, as much compute §3.2. The 95% range for that factor is about 1,500 to 17,500 §3.2. The authors do not expect the projected targets to be reached §3.3. Rows for the other benchmarks in tab. 3 are not transcribed here, because the table's layout is hard to map reliably from the LaTeX source.

Paper vs code

The paper links the repository MIT-FutureTech/TheComputationalLimitsOfDeepLearning §Data and code availability. The MIT-FutureTech lab is the authors' own group, so it is treated as official. The repository's README is a title only; the code and data are in data/ and notebooks/. The repository carries a GPL-3.0 license, which is copyleft, so no code is quoted here. Its commit is 62f1e4d22762ede9503b7a10ac3027e499e42ab1.

• Data counts match. The ImageNet CSV has 208 rows, matching tab. 1 image-classification-on-imagenet.csv. The COCO, SQuAD and CoNLL CSVs have 116, 41 and 54 rows, also matching tab. 1.
• Spec 1 (network operations). The notebook regresses log10 top-1 error on log10 network operations normalised to AlexNet image-classification-on-imagenet.ipynb. Its intercept, −0.6285, matches the paper's −0.629 tab. 2a. Its output has n = 77 and R² 0.720 image-classification-on-imagenet.ipynb, against the paper's n = 80 and R² 0.712.
• Spec 2 (time trend). The code adds year − 2012 as a regressor image-classification-on-imagenet.ipynb. The output year coefficient is −0.0250 (SE 0.004), R² 0.831, n = 77 image-classification-on-imagenet.ipynb. The paper reports −0.022 (SE 0.003), R² 0.822, n = 80 tab. 2a.
• Spec 3 (linear error). The code regresses top-1 error on log10 normalised network operations image-classification-on-imagenet.ipynb. The output has R² 0.628 and n = 77 image-classification-on-imagenet.ipynb; the paper reports R² 0.622 and n = 80.
• Spec 4 (quantile). The code runs smf.quantreg(...).fit(q=.1) image-classification-on-imagenet.ipynb image-classification-on-imagenet.ipynb. The output has pseudo-R² 0.5792, n = 77, intercept −0.6726 image-classification-on-imagenet.ipynb. The paper reports pseudo-R² 0.557, n = 80, intercept −0.676.
• Hardware burden. When hardware burden is missing, the code fills it from network operations via the conversion image-classification-on-imagenet.ipynb. The consolidated regression has n = 102 and R² 0.694 image-classification-on-imagenet.ipynb image-classification-on-imagenet.ipynb. The paper reports n = 104, R² 0.702, intercept −1.111 tab. 2b.
• Conversion fit. The code's log-log fit has n = 11 and R² 0.851 image-classification-on-imagenet.ipynb image-classification-on-imagenet.ipynb. The text gives R² 0.84 app. §8.1.
• Dependent variable. The code uses log10(1/top1_error_percent) for the hardware-burden fits image-classification-on-imagenet.ipynb. This is the same as the appendix formula up to a constant shift, so slopes match and intercepts differ.

No discrepancies found in the data counts. The differences above are between the printed tables and the committed notebook outputs.

Pitfalls

• Inconsistent numbers in the paper. The paper gives 1,527 papers in §1 and 1,526 in §3.2. It gives an ImageNet exponent of 12.5 in the text (from a rounded slope of −0.08), 12.1 in tab. 2a (from −0.082) and 12.2 in §4. For object detection the text says 17.1 while tab. 2b says 16.0. Use the table values, and note the rounding.
• Printed tables and committed notebook outputs differ. The notebook's spec 1 has n = 77, spec 2 has year coefficient −0.0250, and the hardware-burden fit has n = 102 and R² 0.694. These don't match the printed n = 80, −0.022 and n = 104 with R² 0.702. The cause is not stated. Cross-check both before reusing either.
• Compute is estimated, not measured. Most papers report too little to compute it §3.2. Hardware burden assumes a computing precision the authors often must guess, and they note that some papers conflate multiply-adds with FLOPs app. §8.1.
• Hidden conversion. Models with only network operations are converted to hardware burden using an ImageNet-only fit. MS COCO uses that same conversion app. §8.1. Treat those hardware-burden values as derived.
• Dependent variable is ambiguous in the paper. The text writes log(1/error). Table 2b labels its column log10(TOP 1). The appendix uses log10(1/(1 − TOP1/100)). The notebook uses log10 of the error in percent. Slopes agree, intercepts depend on the convention. Error is a fraction in the appendix and a percentage in code.
• Exponent conversion. For a fit of log10(error) on log10(compute), α = −1/slope. For a fit of log10(1/error), α = 1/slope. Mixing these two forms gives the wrong sign.
• Small samples. The SQuAD and CoNLL fits use n = 12, and the COCO fit uses n = 20 tab. 2b. The CoNLL 95% interval runs from 20.4 to 200, and the authors call that result preliminary §3.2.
• Functional form drives the projection. The polynomial and exponential fits describe the data about equally well §3.3. Their projections differ enormously; for the 1% ImageNet target they give 10³³ and 10⁹² FLOPs tab. 3. Report both, not one.
• Projections are extrapolations. They assume the fitted curve holds beyond the observed range, and the authors say the targets will not be reached §3.3. Cost figures use constants cited to other works; they are not derived in this paper.
• Training compute only. Deployment cost is excluded, so the totals are lower bounds §5.
• The 10× claim is a derived result. The paper states that three years of algorithmic progress equal 10× compute §3.2. From spec 2's year coefficient (−0.022) and slope (−0.065), this is about 0.34 decades per year, or roughly 10× over three years. That is our arithmetic from the table, not a separate estimate in the paper.
• The 10× per year figure for the largest models (2012–2019) comes from another source §3.1. It is not this paper's fit.
• Bounding argument. The fourth-power lower bound is a back-of-envelope argument §2. It assumes that error falls as 1/√n and that model size grows in step with data.
• Simulation is an analogy. The regression analogy (1000 simulations, 10 nonzero parameters) illustrates the trade-off but does not prove it for deep networks app. §9.
• Box AP description is loose. The appendix says AP "sums IOUs between 0.5 and 0.95" app. §7.2. The benchmark's standard AP is defined differently, so use the benchmark's own definition rather than this description.
• Version. The source used is arXiv v2 (last updated 2022-07-27). Supplement section numbers (§7 to §9) and figure and table numbers are inferred from the LaTeX source order, since no compiled PDF was available.
• Figures are described, not embedded. fig. 2 shows ImageNet compute over time and error; fig. 3 shows other benchmarks against hardware burden; fig. 4 compares scaling studies.

Lineage

• Builds on: none within this wiki. It cites earlier work by the first author on hardware slowdown §3.1 and exponential compute growth in weather prediction §2.
• Compared against: the 2018 "AI and Compute" analysis §4 tab. 4; "Compute Trends Across Three Eras of Machine Learning" (2022) §4 tab. 4; a 2021 critique of the "AI and Compute" trend §4 tab. 4; a 2021 study asking how long more computing power can keep driving AI progress §4 tab. 4; and EfficientNet as a scaling reference §4.
• Superseded by: not known from content in this wiki.