---
kind: concept
slug: training-compute-estimation
title: Training compute estimation
summary: >-
  How to estimate the training compute of a model that reports none. Count network operations from the architecture, or hardware burden from processors, rate and time; a log-log conversion, fitted on ImageNet only, links them. Per-system estimates are expected within a factor of two. Operation counts are more precise; hardware burden is easier to find but depends on implementation efficiency.
tags: [scaling, compute, hardware, forecasting, meta-analysis]
papers: ["2007.05558", "2202.05924"]
related: [neural-scaling-laws, model-flops-utilization]
status: draft
updated_at: 2026-10-31
---

## Overview

Most papers do not report how much compute they used, so the value has to be
estimated from what they do report [arxiv:2007.05558 §3.2]. The shared
definition is the total work done over all training stages: pre-training,
training and fine-tuning [arxiv:2007.05558 eq. 1] [arxiv:2007.05558 eq. 2].
The estimate covers training only. Deployment cost is excluded, so the total
is a lower bound [arxiv:2007.05558 §5].

## Variants

### Network operations

Sum, over stages, the epochs times the floating-point operations per forward
pass times the passes per epoch [arxiv:2007.05558 eq. 1]. It needs the
architecture and the training schedule. When a paper reports multiply-adds
instead of FLOPs, the paper doubles the count, since one multiply-add is two
arithmetic operations [arxiv:2007.05558 §3.2]. Card:
[The Computational Limits of Deep Learning](/papers/2007.05558).

### Hardware burden

Sum, over stages, the number of processors times their computation rate times
the time used [arxiv:2007.05558 eq. 2]. It needs the hardware and the wall-clock
time. The authors note it is less precise, because it depends on how efficiently
the implementation used the hardware [arxiv:2007.05558 §3.2].

### Converting between the two

When a model has only network operations, a log-log regression fitted on
ImageNet models that report both measures gives hardware burden. The reported
fit has R² of 0.84 [arxiv:2007.05558 app. §8.1]. The authors reuse the same
ImageNet fit for MS COCO [arxiv:2007.05558 app. §8.1], so converted values for
other benchmarks are a rough approximation, not a measured value.

### Per-system estimates over many models

When the goal is a trend over many models rather than one number, each system
gets its own estimate, and the uncertainty is carried through. The authors use
the method of the earlier *AI and Compute* analysis: forward-pass compute, or
GPU time where the paper gives it. Only the final training run is counted, so
hyperparameter search and failed runs are left out [arxiv:2202.05924 app. A].
They expect most estimates to be within a factor of about two [arxiv:2202.05924
app. H]. To reflect that, each bootstrap draw multiplies every estimate by a
log-uniform factor between 1/2 and 2 [arxiv:2202.05924 app. A]. Systems whose
log10 compute is more than two standard deviations below their neighbours in a
time window are dropped as low-compute outliers [arxiv:2202.05924 app. A]
[arxiv:2202.05924 tab. 5]. The authors also warn that notable systems skew
large and recent, which pushes doubling-time estimates up [arxiv:2202.05924
app. H]. Card: [Compute Trends Across Three Eras of Machine Learning](/papers/2202.05924).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Paper gives layer sizes, epochs and passes per epoch | Network operations [arxiv:2007.05558 eq. 1] | Counts the work directly; the paper calls it the more precise measure for this reason [arxiv:2007.05558 §3.2] |
| Paper gives processor count, hardware speed and training time only | Hardware burden [arxiv:2007.05558 eq. 2] | Usually easier to find; it depends on implementation efficiency [arxiv:2007.05558 §3.2] |
| Only network operations are known and the task is image classification | Convert with the ImageNet fit [arxiv:2007.05558 app. §8.1] | The fit was built on ImageNet models that report both measures |
| Only network operations are known and the task is not image classification | Report it as network operations only | The paper's conversion was fitted on ImageNet and applied to MS COCO; it did not build a fit for other tasks [arxiv:2007.05558 app. §8.1] |
| Estimating many systems to study a trend | Per-system estimates with the factor-of-two noise and bootstrap [arxiv:2202.05924 app. A] | Makes the uncertainty of each estimate visible in the trend's interval, rather than treating the estimates as exact |
| Need the full project cost, including hyperparameter search | Neither method's total [arxiv:2202.05924 app. A] [arxiv:2007.05558 eq. 1] | The 2202.05924 estimates exclude hyperparameter search and failed runs; the 2007.05558 sum covers only the stages it names |
| Comparing across papers | Report which measure each value uses | The two measures differ, and converted values are derived rather than reported |

When a scaling fit is the goal, the paper regresses log error on log compute
per benchmark and takes the implied exponent as minus one over the slope
[arxiv:2007.05558 §3.2]. See [neural-scaling-laws](/concepts/neural-scaling-laws)
for how such fits are used to forecast.

## Implementation sketch

Same two sums as the paper, plus the fit used for conversion and the noise
model for per-system estimates. Each step is cited to its equation or section.

```pseudocode
# Network operations per model  [arxiv:2007.05558 eq. 1]
# Input:  per stage (pre-training, training, fine-tuning):
#         epochs, flops per forward pass, passes per epoch
# Output: total FLOPs

def network_operations(stages):
    # Sum epochs x flops per pass x passes per epoch
    total = 0
    for stage in stages:
        total += (
            stage.epochs
            * stage.flops_per_pass
            * stage.passes_per_epoch
        )
    return total

# Hardware burden per model  [arxiv:2007.05558 eq. 2]
# Input:  per stage: processors, rate per processor, time used
# Output: total FLOPs

def hardware_burden(stages):
    # Sum processors x rate x time
    total = 0
    for stage in stages:
        total += stage.processors * stage.rate * stage.time
    return total

# Convert network operations to hardware burden  [arxiv:2007.05558 app. §8.1]
# Fit: log-log OLS on ImageNet models that report both measures

def to_hardware_burden(net_ops, fit):
    # Apply the ImageNet fit; the result is derived, not measured
    return fit.predict(net_ops)

# Multiply-add counts  [arxiv:2007.05558 §3.2]
def flops_from_multiply_adds(madds):
    # One multiply-add is two arithmetic operations
    return 2 * madds

# Noise for one bootstrap draw of per-system estimates
# [arxiv:2202.05924 app. A] [arxiv:2202.05924 app. H]
# Input:  log10 compute per system
# Output: log10 compute with each estimate perturbed

def perturbed_draw(log_compute):
    # Each estimate may be off by about a factor of two
    # Multiply by a log-uniform factor in [1/2, 2]
    noise = uniform(log10(1/2), log10(2), size=len(log_compute))
    return log_compute + noise
```

## Contradictions

- **Number of papers.** The introduction gives 1,527 papers and the
  methodology gives 1,526 [arxiv:2007.05558 §1] [arxiv:2007.05558 §3.2].
- **ImageNet exponent.** The text gives 12.5 [arxiv:2007.05558 §3.2], the table
  gives 12.1 for the network-operations fit [arxiv:2007.05558 tab. 2a], and the
  comparison section gives 12.2 for the field as a whole
  [arxiv:2007.05558 §4]. The text value comes from a rounded slope, so the
  table value is the one to cite.
- **Object detection exponent.** The text gives 17.1 and the hardware-burden
  table gives 16.0 [arxiv:2007.05558 §3.2] [arxiv:2007.05558 tab. 2b].
- **Projected compute at the same target.** The polynomial and exponential
  fits describe the data about equally well, but at a 1% ImageNet top-1 error
  they project 10³³ and 10⁹² FLOPs [arxiv:2007.05558 tab. 3]. Report both forms.
- **Printed tables and released code.** A check of the authors' released
  notebook, recorded on the paper card, finds different sample sizes and R²
  values from the printed tables. The paper does not state the cause
  ([Paper card](/papers/2007.05558)).
- **Doubling times across studies.** The 2202.05924 authors report that
  earlier analyses disagree: one gives a doubling time of 3.4 months for
  2012 to 2018, another more than two years for 2018 to 2020
  [arxiv:2202.05924 §2] [arxiv:2202.05924 tab. 1]. They attribute the gap to
  small datasets and a single trend assumed for all models
  [arxiv:2202.05924 §3.2] [arxiv:2202.05924 app. E]. Name the dataset and the
  period whenever a doubling time is quoted.

## Papers

- [The Computational Limits of Deep Learning](/papers/2007.05558): defines
  network operations and hardware burden, the ImageNet conversion between
  them, and the scaling fits built on those estimates.
- [Compute Trends Across Three Eras of Machine Learning](/papers/2202.05924):
  estimates training compute for a curated set of milestone systems and fits
  bootstrapped trends, with a factor-of-two noise model on each estimate.