Compute Trends Across Three Eras of Machine Learning

In plain words

Training a modern machine learning model takes an enormous amount of arithmetic, usually called compute. The authors ask how that amount has grown over time. They collect a list of about 120 well-known models, estimate how much compute each one needed to train, and plot the results against publication date.

The list does not rise in a single straight line. Up to about 2010 the amount grew at a slow, steady pace. After deep learning took off, growth sped up a lot. From about 2015 a second group of very large projects, run by big companies, pulled well ahead of the rest. The authors split the history into three eras to describe this.

The estimates are rough, because many papers do not report their compute and the authors had to work it out. The paper is most useful as a sense of scale: how fast the cost of the largest training runs has been climbing, and why the newest runs are so expensive.

TL;DR

The paper curates a dataset of 123 milestone ML systems with estimated training compute and fits log-linear trends to it §1 app. A. It reports compute doubling about every 21 months from 1952 to 2010 tab. 2, about every 6 months from 2010 to 2022 for regular-scale models tab. 2, and about every 10 months for a separate large-scale trend that starts in late 2015 tab. 2 §3.2. All numbers are the authors' own estimates, and the dataset is not in the repository, so use the paper for the trend shape and the sizes, not as exact measurements.

Problem

Forecasts of ML progress need a quantity that can be measured over decades. The authors use training compute, arguing it is the most readily quantified of compute, data and algorithms §1. Earlier analyses disagree: one reports a 3.4-month doubling time for 2012 to 2018, another a doubling time of more than two years for 2018 to 2020 §2 tab. 1. The authors argue that these conflicts arise because earlier datasets were small and assumed a single trend for all models §3.2 app. E.

The analysis rests on assumptions the paper states openly: that the chosen systems are a usable sample, although the authors call the dataset biased app. A, that compute estimates are accurate to about a factor of two app. H, and that the large-scale trend is a separate trend that the authors chose after the fact §3.2.

Method

The pipeline has five parts. The dataset of systems and the compute estimates are the expensive, manual part. The statistics are simple log-linear regressions with a bootstrap.

1. Select systems. Models come from papers with a learning component, experimental results and a state-of-the-art advance, plus at least one notability criterion. From 2020 on the authors fall back to subjective selection app. A. The systems are called milestone models app. A.
2. Estimate training compute. Where a paper does not report compute, the authors follow the method in the appendix of AI and Compute (forward-pass compute or GPU time). Only the compute of the final training run is counted app. A.
3. Remove low-compute outliers. For each system, a log10 Z-score is computed against systems published within a window around it. Systems with a Z-score below -2 are excluded app. A tab. 5.
4. Assign eras. Systems before 2010 form the Pre Deep Learning Era. From 2010 on, systems form the Deep Learning Era, which is split into regular-scale models and large-scale models. The large-scale group is systems whose Z-score exceeds 0.76 after the start of the large-scale era §3.2 app. A.
5. Fit trends. Each group gets a least-squares line of log10 training compute on time. The doubling time is the inverse of the slope converted to months. Confidence intervals come from a bootstrap that also perturbs each compute estimate app. A §3.1.

Pseudocode for step 5, the fit and its bootstrap:

# Trend fit for one group of systems
# [app. A] [code: Compute-Trends_visualization.ipynb#L362-L409]
# Input:  dates (N,) in days, compute (N,) in FLOPs, number of bootstrap draws
# Output: slope (OOMs/day), doubling time (months), 95% intervals

def fit_trend(dates, compute, n_bootstrap):
    # X is the publication date in days, not logged  [code: ...#L364-L368]
    x = dates
    # The y-axis is log10 of the training compute  [code: ...#L373-L374]
    y = log10(compute)
    # Point estimate from the unperturbed data  [code: ...#L381]
    point_slope = least_squares_slope(x, y)
    slopes = []
    doubling_times = []
    for _ in range(n_bootstrap):
        # Resample systems with replacement  [code: ...#L391]
        index = sample_with_replacement(len(x))
        # Skip draws with fewer than three distinct systems  [code: ...#L393]
        if count_distinct(index) <= 2:
            continue
        # Multiply each estimate by a factor log-uniform in [1/2, 2]  [app. A]
        # [code: ...#L399]
        noise = uniform(log10(1/2), log10(2), size=len(index))
        noisy_y = y[index] + noise
        boot_slope = least_squares_slope(x[index], noisy_y)
        slopes.append(boot_slope)
        # 30-day month, slope per day  [code: ...#L408-L409]
        doubling_times.append(log10(2) / boot_slope / 30)
    # 2.5%, 50% and 97.5% quantiles  [app. A]
    # [code: ...#L423-L445]
    slope_interval = quantiles(slopes, [0.025, 0.5, 0.975])
    doubling_interval = quantiles(doubling_times, [0.025, 0.5, 0.975])
    return point_slope, slope_interval, doubling_interval

Low-compute outlier test, step 3:

# Z-score filter within a window
# [app. A] [code: Compute-Trends_visualization.ipynb#L246-L261]
# Input:  systems with date and compute; window (years); z threshold
# Output: True if the system is a low-compute outlier

def is_low_outlier(system, systems, window_years, z_threshold):
    # Keep systems within half the window on each side of this one
    # (the code's window is 2 years in total  [code: ...#L247-L249])
    neighbours = [
        other for other in systems
        if abs(other.date - system.date) <= window_years / 2
    ]
    # Skip systems with fewer than two neighbours, self included
    # [code: ...#L253]
    if len(neighbours) < 2:
        return False
    # Z-score of this system's log10 compute among its neighbours
    # [code: ...#L257]
    logs = [log10(other.compute) for other in neighbours]
    z = (log10(system.compute) - mean(logs)) / standard_deviation(logs)
    # The paper uses -2  [app. A] [code: ...#L258]
    return z < z_threshold

Hyperparameters

The paper has no training hyperparameters, since it does not train models. The table lists the analysis settings that change the results.

Results

Author-reported. Setup: 123 milestone ML systems curated by the authors, estimated training compute, 1952 to 2022 §1 app. A. Doubling times are months from a log-linear fit; intervals are 95% bootstrap intervals written as [2.5%; median; 97.5%] app. A.

Summary of the three eras tab. 2:

Log-linear regressions for the Deep Learning period and its subsets tab. 3:

Regular-scale and large-scale trends from 2010 to 2022 tab. 4:

Per-domain trends, 2009 to 2022 tab. 7:

Comparison with the earlier analysis over its period tab. 8:

Record-setting models tab. 6 (author-reported; see Pitfalls for printed inconsistencies):

Sensitivity of the large-scale trend to its Z threshold tab. 9: the large-scale doubling time is 10.3 months at Z > 0.76 (n = 20), 10.7 months at Z > 0.6 (n = 26), and 11.6 months at Z > 0.54 (n = 32). The regular-scale doubling time is 6.0, 6.0 and 6.7 months for the same thresholds. The authors call the differences small app. F.

Headline claims, as stated in the abstract and conclusion: compute doubled roughly every 20 months before 2010 and roughly every 6 months after the advent of Deep Learning in the early 2010s (abstract). The large-scale trend started about 2 to 3 orders of magnitude above the previous trend and shows a 10-month doubling time §4.

Paper vs code

The official repository is ML-Progress/Compute-Trends, linked from the paper's introduction §1. It is at code_commit 758bcfcf. It contains one notebook with Colab form fields, Compute-Trends_visualization.ipynb, the figure PDFs and a README. It does not contain the dataset of systems. The notebook reads the dataset from a Google Sheet at run time Compute-Trends_visualization.ipynb. It does not contain the compute estimation. The repository has no LICENSE file, so cite it by path and line only, and do not copy code from it.

Discrepancies found:

1. Bootstrap samples. The paper uses B = 1000 app. A. The function signature default is 500 Compute-Trends_visualization.ipynb. The form default and the per-figure presets use 1000 Compute-Trends_visualization.ipynb Compute-Trends_visualization.ipynb. Fewer draws make the interval endpoints noisier, so check which value a run used.
2. Outlier window. The paper says the Z-score uses systems within 1.5 years app. A, while tab. 5 says 2 years. The code's window is outlier_window_size * 52 * 7 days, which is 2 years in total, or 1 year on each side Compute-Trends_visualization.ipynb. A code comment says "3-year window" Compute-Trends_visualization.ipynb. The comment does not match the arithmetic.
3. Large-scale cut-off. The paper says large-scale models are those with Z > 0.76 after 2016 app. A tab. 5 uses Z ≥ 0.76. The code requires the publication date to be after 2015-09-01 Compute-Trends_visualization.ipynb Compute-Trends_visualization.ipynb. The form default is 2012-09-30 Compute-Trends_visualization.ipynb. Systems published between 2015 and 2016 therefore depend on which setting was used.
4. Z threshold comparison. The text uses strict Z > 0.76 app. A. The code uses strict > Compute-Trends_visualization.ipynb. tab. 5 says ≥. This affects only a system with Z exactly 0.76, so it is small.
5. Start of the Deep Learning Era. The paper uses 2010 app. D. The code default is 2009-12-31 Compute-Trends_visualization.ipynb, which is effectively the same boundary.
6. Doubling-time conversion. The paper reports months but does not state the conversion from a daily slope. The code uses 30 days per month Compute-Trends_visualization.ipynb Compute-Trends_visualization.ipynb.
7. Noise. The paper says each estimate is multiplied by a log-uniform factor between 1/2 and 2 app. A. The code adds uniform noise in log10 space between log10(1/2) and log10(2) Compute-Trends_visualization.ipynb. These agree.
8. Consistent with the code. The Z-score cut at -2 app. A Compute-Trends_visualization.ipynb and the percentile intervals app. A Compute-Trends_visualization.ipynb match the paper.

The code reproduces the regressions and the plots from a dataset it downloads. It does not reproduce the compute estimates or the dataset compilation.

Pitfalls

• The printed counts do not reconcile. The paper curates 123 systems and says that 5 are excluded as low-compute outliers app. A, which leaves 118. The figure captions use n = 121 fig. 1 fig. 2. The large-scale group has n = 16 in tab. 2, n = 19 in tab. 4 and n = 20 in the outlier figure caption fig. 4. The low-compute outliers are n = 5 in app. A but n = 7 in fig. 4. Do not rely on any single count.
• A record-setting row has an impossible interval. The 2016 to 2022 row with AlphaGo Zero and Master excluded has interval [6.1; 7.6; 7.1] tab. 6. The upper bound is below the median, so one of the printed values is wrong. The paper does not explain it.
• Record-setting text and table disagree. The text says that including AlphaGo Zero and Master gives a one-year doubling time app. B. The table row for the 2016 to 2022 record-setting group with n = 7 gives 8.5 months tab. 6. The table does not show which row is the text's case.
• The large-scale group is chosen after the fact. The authors say the cut-off is an intuitive decision, justified post hoc §3.2. Other reasonable thresholds move the large-scale doubling time from 10.3 to 11.6 months tab. 9. The authors also list other systems that could be on either side, such as NASv3, Libratus and Megatron-LM app. F.
• The large-scale slope has little data. The paper says the apparent slowdown of the large-scale trend may be noise §3.2. The 2016 to 2022 all-model fit has R² 0.33 tab. 4.
• Only the final training run is counted. Hyperparameter search and failed runs are not in the compute figures app. A. Actual project cost is higher.
• Compute estimates are rough. The authors expect most estimates to be within a factor of about two app. H. Errors in the data are not corrected systematically; the authors say few corrections have been received since publication app. H.
• Selection bias raises doubling times. Notable systems are more likely to be large and recent, which the authors say makes the doubling-time estimate higher app. H.
• Games have no consistent trend. The Games fit has a confidence interval from -4.1 to 35.5 months tab. 7 app. C. Do not quote a games doubling time.
• The dataset is outside the repository. The notebook reads it from a Google Sheet at run time Compute-Trends_visualization.ipynb. The paper links a separate data page §1. Rerunning the notebook may give different numbers if the sheet changes. The dataset's own license is not stated in what this card reads, and this card does not reproduce its rows.
• The 2010 era start is a judgement. The authors give evidence for 2010 and 2012 app. D and say the results barely change between the two §3.1. A reader who wants 2012 can use tab. 3.
• The standard deviation convention is not stated. The paper does not say whether the Z-score uses the population or sample standard deviation. The code calls stats.zscore with its default arguments Compute-Trends_visualization.ipynb.
• Large-scale models are not a proven separate category. The authors say they are "not confident in the distinction" §4.

Lineage

• Builds on: none; this is a standalone analysis. Earlier compute trend work it compares with: AI and Compute, which reports a 3.4-month doubling time for 2012 to 2018 §2; a later addendum to it (sastry2019), which reports a doubling time of about 2 years from 1959 to 2012 §2; lyzhov2021trend, which reports a doubling time of more than 2 years for 2018 to 2020 §2. None of these are cards in this wiki yet, so they are not linked.
• Compared against: the regressions in tab. 8 reproduce the earlier analysis's period with the authors' large-scale split app. E.
• Superseded by: not stated in the paper.