Training compute estimation

Overview

Most papers do not report how much compute they used, so the value has to be estimated from what they do report 2007.05558 §3.2. The shared definition is the total work done over all training stages: pre-training, training and fine-tuning 2007.05558 eq. 1 2007.05558 eq. 2. The estimate covers training only. Deployment cost is excluded, so the total is a lower bound 2007.05558 §5.

Variants

Network operations

Sum, over stages, the epochs times the floating-point operations per forward pass times the passes per epoch 2007.05558 eq. 1. It needs the architecture and the training schedule. When a paper reports multiply-adds instead of FLOPs, the paper doubles the count, since one multiply-add is two arithmetic operations 2007.05558 §3.2. Card: The Computational Limits of Deep Learning.

Hardware burden

Sum, over stages, the number of processors times their computation rate times the time used 2007.05558 eq. 2. It needs the hardware and the wall-clock time. The authors note it is less precise, because it depends on how efficiently the implementation used the hardware 2007.05558 §3.2.

Converting between the two

When a model has only network operations, a log-log regression fitted on ImageNet models that report both measures gives hardware burden. The reported fit has R² of 0.84 2007.05558 app. §8.1. The authors reuse the same ImageNet fit for MS COCO 2007.05558 app. §8.1, so converted values for other benchmarks are a rough approximation, not a measured value.

Per-system estimates over many models

When the goal is a trend over many models rather than one number, each system gets its own estimate, and the uncertainty is carried through. The authors use the method of the earlier AI and Compute analysis: forward-pass compute, or GPU time where the paper gives it. Only the final training run is counted, so hyperparameter search and failed runs are left out 2202.05924 app. A. They expect most estimates to be within a factor of about two [arxiv:2202.05924 app. H]. To reflect that, each bootstrap draw multiplies every estimate by a log-uniform factor between 1/2 and 2 2202.05924 app. A. Systems whose log10 compute is more than two standard deviations below their neighbours in a time window are dropped as low-compute outliers 2202.05924 app. A 2202.05924 tab. 5. The authors also warn that notable systems skew large and recent, which pushes doubling-time estimates up [arxiv:2202.05924 app. H]. Card: Compute Trends Across Three Eras of Machine Learning.

When to use which

When a scaling fit is the goal, the paper regresses log error on log compute per benchmark and takes the implied exponent as minus one over the slope 2007.05558 §3.2. See neural-scaling-laws for how such fits are used to forecast.

Implementation sketch

Same two sums as the paper, plus the fit used for conversion and the noise model for per-system estimates. Each step is cited to its equation or section.

# Network operations per model  [arxiv:2007.05558 eq. 1]
# Input:  per stage (pre-training, training, fine-tuning):
#         epochs, flops per forward pass, passes per epoch
# Output: total FLOPs

def network_operations(stages):
    # Sum epochs x flops per pass x passes per epoch
    total = 0
    for stage in stages:
        total += (
            stage.epochs
            * stage.flops_per_pass
            * stage.passes_per_epoch
        )
    return total

# Hardware burden per model  [arxiv:2007.05558 eq. 2]
# Input:  per stage: processors, rate per processor, time used
# Output: total FLOPs

def hardware_burden(stages):
    # Sum processors x rate x time
    total = 0
    for stage in stages:
        total += stage.processors * stage.rate * stage.time
    return total

# Convert network operations to hardware burden  [arxiv:2007.05558 app. §8.1]
# Fit: log-log OLS on ImageNet models that report both measures

def to_hardware_burden(net_ops, fit):
    # Apply the ImageNet fit; the result is derived, not measured
    return fit.predict(net_ops)

# Multiply-add counts  [arxiv:2007.05558 §3.2]
def flops_from_multiply_adds(madds):
    # One multiply-add is two arithmetic operations
    return 2 * madds

# Noise for one bootstrap draw of per-system estimates
# [arxiv:2202.05924 app. A] [arxiv:2202.05924 app. H]
# Input:  log10 compute per system
# Output: log10 compute with each estimate perturbed

def perturbed_draw(log_compute):
    # Each estimate may be off by about a factor of two
    # Multiply by a log-uniform factor in [1/2, 2]
    noise = uniform(log10(1/2), log10(2), size=len(log_compute))
    return log_compute + noise

Contradictions

• Number of papers. The introduction gives 1,527 papers and the methodology gives 1,526 2007.05558 §1 2007.05558 §3.2.
• ImageNet exponent. The text gives 12.5 2007.05558 §3.2, the table gives 12.1 for the network-operations fit 2007.05558 tab. 2a, and the comparison section gives 12.2 for the field as a whole 2007.05558 §4. The text value comes from a rounded slope, so the table value is the one to cite.
• Object detection exponent. The text gives 17.1 and the hardware-burden table gives 16.0 2007.05558 §3.2 2007.05558 tab. 2b.
• Projected compute at the same target. The polynomial and exponential fits describe the data about equally well, but at a 1% ImageNet top-1 error they project 10³³ and 10⁹² FLOPs 2007.05558 tab. 3. Report both forms.
• Printed tables and released code. A check of the authors' released notebook, recorded on the paper card, finds different sample sizes and R² values from the printed tables. The paper does not state the cause (Paper card).
• Doubling times across studies. The 2202.05924 authors report that earlier analyses disagree: one gives a doubling time of 3.4 months for 2012 to 2018, another more than two years for 2018 to 2020 2202.05924 §2 2202.05924 tab. 1. They attribute the gap to small datasets and a single trend assumed for all models 2202.05924 §3.2 2202.05924 app. E. Name the dataset and the period whenever a doubling time is quoted.

Papers

• The Computational Limits of Deep Learning: defines network operations and hardware burden, the ImageNet conversion between them, and the scaling fits built on those estimates.
• Compute Trends Across Three Eras of Machine Learning: estimates training compute for a curated set of milestone systems and fits bootstrapped trends, with a factor-of-two noise model on each estimate.