Compute and Energy Consumption Trends in Deep Learning Inference

In plain words

Bigger AI models keep getting bigger, so it is natural to ask whether the electricity they use per answer grows just as fast. This paper looks at the cost of running a trained model (inference), not training it, because a model is trained once but run millions of times.

The authors collect, for many image-classification and language models, how much arithmetic one prediction needs. They also collect, for Nvidia graphics cards, how much arithmetic each watt of power buys. Dividing one by the other gives an estimate of the energy one prediction uses, and they track that estimate over the years.

The main finding is that the most accurate new models do use much more energy per prediction, but models that come a year or two after a breakthrough often reach similar accuracy with far less arithmetic. Cheaper hardware and better software soften the growth further. The authors warn that the picture depends on how many predictions are made overall, which they could not estimate.

TL;DR

• Energy per inference is estimated as model FLOPs divided by GPU FLOPS per watt, using peak FLOPS and TDP eq. 1 eq. 2 § Methodology.
• For ImageNet, the best model per year needs exponentially more FLOPs, but the general trend across all models grows much more slowly § Computer Vision Analysis; the energy estimate for the best models rises steeply, while the trend across all models is nearly flat fig. 9.
• Results are author-reported estimates, not measured power draw.
• Relevant if you need a rough, order-of-magnitude energy figure for model inference, or to choose a model that is accurate enough at lower FLOPs.

Problem

Training cost is often reported, while inference cost is rarely reported, even though inference can dominate total compute because a deployed model runs repeatedly § Introduction. Energy depends on FLOPs, hardware efficiency and the number of uses, and these are seldom studied together § Introduction. The authors assume a constant number of uses per person when they discuss aggregate impact § Forecasting and Multiplicative Effect.

Method

The study is a meta-analysis of published numbers, not a benchmark campaign § Methodology. Four kinds of sources are used, in this order of preference: the original papers, model implementations (for example PyTorch model zoos), existing compilations, and a FLOP-counting tool when no other source is reliable § Methodology.

Step 1. Collect model data for computer vision. Image-classification models are chosen by popularity (citations and mentions in other papers) and accuracy on ImageNet, using an existing compilation filtered by year [§ CV Models Data Compilation]. For each model the authors record top-1 accuracy (one crop), parameters, forward-pass FLOPs, release date and training data [§ CV Models Data Compilation]. Accuracy is always for one crop, because multi-crop averaging multiplies the real compute app. Methodology Details for CV Models.

Step 2. Collect model data for NLP. All models from 2017 onward for which an inference-compute estimate exists are included, with GLUE score as the performance measure § NLP Models Data Compilation. FLOPs are reported for a single input of 128 tokens unless noted § Performance and Compute (NLP) tab. 7.

Step 3. Collect GPU data. Nvidia GPUs from 2010 to 2021 are used, with peak FLOPS, memory, TDP and launch date § Hardware Data Compilation. Efficiency is peak FLOPS divided by TDP, which assumes full utilisation [§ Hardware Data Compilation] tab. 8.

Step 4. Correct tensor-core FLOPS. Theoretical tensor-core FLOPS are far above what inference achieves, so the authors scale them with measured mixed-precision speed-ups for V100, A100 and T4 taken from Nvidia benchmark catalogue numbers, computed separately for CV and NLP tab. 4 [app. Hardware mixed precision speed-ups] [app. Hardware data compilation: floating point precision details]. The adjusted GPU table is tab. 9.

Step 5. Fit efficiency over time. Adjusted GFLOPS-per-watt values are plotted against launch date, with one fit for CV and one for NLP, on a logarithmic axis § Hardware Progress fig. 8. The fit is used to pick the efficiency for a model's release date § Energy Consumption Analysis.

Step 6. Convert FLOPs to energy. FLOPS per watt equals FLOPs per joule, so energy per forward pass is the model's FLOPs divided by that efficiency eq. 1 eq. 2 § Energy Consumption Analysis.

Step 7. Estimate missing FLOPs. For EfficientNet variants without a reported count, FLOPs are scaled from a base model by depth, width and resolution ratios eq. 3 eq. 4 [app. EfficientNet-Based Models FLOPs Estimation]. ViT-G/14 is estimated by scaling its reported resolution counts with the square of the resolution ratio app. ViT-G/14 FLOPs Estimation.

Step 8. Compare with humans. Energy per prediction is compared with a human body's energy use per second (about 100 J/s) and with average per-capita energy use including external consumption, using US figures [§ Energy Consumption Analysis].

Equations as the paper numbers them (the main-text scaling relation is an unnumbered display):

\text{Efficiency} = \frac{\text{HW perf.}}{\text{Power}} \quad \text{units } \frac{\text{FLOPs}}{\text{Joule}} \quad [\text{eq. 1}]

\text{Energy} = \frac{\text{Forward pass FLOPs}}{\text{Efficiency}} \quad \text{units Joule} \quad [\text{eq. 2}]

\text{FLOPs} \propto d + w^2 + r^2 \quad [\text{§ Number of Parameters and FLOPs}]

The main-text relation uses a sum. The EfficientNet estimate in the appendix uses a product instead eq. 4; see Pitfalls.

Pseudocode for the energy estimate (the paper's method, restated):

# Energy of one forward pass  [eq. 2] [§ Energy Consumption Analysis]
# model_flops: FLOPs of one forward pass, one input  [tab. 2] [tab. 7]
# gpu_flops_per_watt: peak FLOPS / TDP, GPU adjusted for precision  [tab. 9]

def energy_per_inference(model_flops, gpu_flops_per_watt):
    # FLOPs divided by FLOPs per joule gives joules  [eq. 1] [eq. 2]
    return model_flops / gpu_flops_per_watt

# Efficiency at a release date  [§ Hardware Progress] [fig. 8]
# Fit separately for CV and NLP rows of the adjusted GPU table  [tab. 9]
# Regression method and log base are not stated.

def efficiency_at(release_date, fit):
    # fit is a straight line in log10 space against launch date
    log_efficiency = fit.slope * release_date + fit.intercept
    return 10 ** log_efficiency

# Scaled-model FLOPs  [eq. 4] [app. EfficientNet-Based Models FLOPs Estimation]
# base_flops: reported FLOPs of the base model (EfficientNet-B7: 74 GFLOPs)
# depth_ratio, width_ratio, res_ratio: target over base (d, w, r)

def scaled_flops(base_flops, depth_ratio, width_ratio, res_ratio):
    # The paper multiplies d * w^2 * r^2 in eq. 4, not the sum in eq. 3
    return base_flops * depth_ratio * width_ratio ** 2 * res_ratio ** 2

Hyperparameters

This is an analysis, so the "hyperparameters" are the study's constants. Values are as stated; nothing below is tuned by the authors' training.

Results

Author-reported. Setup: a literature and benchmark-catalogue compilation of ImageNet models (computer vision) and GLUE scores (language), plus Nvidia GPU specifications § Methodology. No model is run by the authors. Energy values are estimates, not measurements § Energy Consumption Analysis.

Models with about the same FLOPs as AlexNet, from the paper's own table tab. 1:

Selected CV rows from the full model table tab. 2; the full table has about 110 rows, and the paper reports the values as listed there. Asterisked values are estimates in the paper; dagger values were checked with a tool.

Headline trend statements, as the paper states them:

• AlexNet reaches 56% top-1 at 1.42 GFLOPs; Meta Pseudo Labels (EfficientNet-L2) reaches 90.2% at an estimated 1040 GFLOPs § Performance and Compute.
• Looking only at the best model per year, FLOPs grow exponentially; over all models the trend is much flatter § Computer Vision Analysis fig. 3.
• For NLP, the best models' GFLOPs grow clearly exponentially while the trend over all models does not § Compute Trend fig. 6.
• Mixed-precision inference speed-ups relative to FP32 tab. 4:

• Adjusted GPU efficiency for CV and NLP tab. 9 (GFLOPS per watt): T4 mixed precision reaches 312.15 (CV) and 365.46 (NLP); V100 mixed reaches 119.03 (CV) and 138.13 (NLP); A100 mixed reaches 130.88 (CV) and 183.23 (NLP). The T4 point for 2018 stands out in the fit § Hardware Progress fig. 8.
• Energy per forward pass, the paper's conclusion: the best models' estimated Joules rise steeply, the trend over all models is nearly flat, and "general use" models keep systematic accuracy gains at roughly constant energy § Computer Vision Analysis § Discussion and Future Work fig. 9 fig. 11.
• Comparison with humans: the energy of one prediction from the best models approaches the body's energy per second, and is far below average external consumption per second § Energy Consumption Analysis fig. 13.
• The paper names the multiplicative factor (number of predictions) as the main caveat and does not estimate it § Forecasting and Multiplicative Effect.

Paper vs code

No official code.

Search performed:

1. Links in the paper: the footnote gives a bit.ly short link described as a "temporary copy" of data and code § Introduction. It redirects to a Google Drive folder titled "AI Inference Energy Consumption". The folder holds CSV and Excel data files and R plotting scripts (for example CV_plots.R, GPU_plots.R), with no README, no version history and no license visible. It is not a repository, its owner is not shown as an author, and it cannot be cloned, so it is not recorded as code.
2. Project page: none linked in the arXiv metadata or the paper.
3. GitHub search: 2109.05472, inference energy consumption trends, and energy inference hernandez-orallo returned no repositories; a search for Desislavov returned nothing; the user lookup for Desislavov returned no repositories.

The Drive folder's R scripts are plotting code over the tables; they do not include the estimation of FLOPs. Discrepancies between paper and this folder were not checked because the folder is not a verified author repository.

Pitfalls

• Peak, not measured. Efficiency is peak FLOPS divided by TDP, assuming full utilisation § Hardware Data Compilation. The paper states that real efficiency varies with workload and that measuring power per network would be needed for precision § Energy Consumption Analysis. Read the energy values as order-of-magnitude and as trend estimates, not as measurements.
• Tensor-core FLOPS are adjusted, but inputs are few. Mixed-precision factors come from a small set of benchmarked models per domain tab. 3 tab. 4. The paper does not state the formula for the adjusted FLOPS. The listed values appear to be the V100 FP32 (or T4 FP32) peak multiplied by the ratio, checked by hand, but the entries do not match that product exactly: V100 mixed CV is listed as 35.71 tab. 9, while 15.70 x 2.27 = 35.64 tab. 4.
• Mixed bases for speed-ups. A100 ratios are relative to V100 FP32 and T4 ratios to T4 FP32 tab. 4. Compare numbers only within the same base.
• Two scaling formulas. The main text and eq. 3 write FLOPs as a sum, d + w^2 + r^2 § Number of Parameters and FLOPs eq. 3. The NoisyStudent estimate in eq. 4 multiplies the same terms, d x w^2 x r^2. Use the product form for scaling estimates; the paper gives no citation for the sum.
• ViT-G/14 arithmetic. The appendix gives 2859.9 GFLOPs at 384 px and a resolution ratio of 518/384 = 1.3490 to get "approximately 5270" app. ViT-G/14 FLOPs Estimation. Squaring the ratio and multiplying gives about 5204. Treat 5270 as the paper's stated value, not as reproduced. The paper also assumes the test resolution equals the fine-tuning resolution.
• Estimated versus reported. Several GFLOPs values are estimates (asterisk) or tool counts (dagger) tab. 2. The text says CV-reported operation counts were doubled to FLOPs, but the table does not mark which entries were doubled.
• Date conflict. The text says Meta Pseudo Labels reached 90.2% "in 2021" § Performance and Compute, while the table lists 23/03/2020 tab. 2.
• BERT-Large is borrowed. BERT-Large's GFLOPs are taken from ELECTRA-Large, and its parameter count is estimated app. NLP data tab. 7. ELMo's score is the dev-set score. Some GLUE scores exclude the WNLI task tab. 7.
• Human comparison rounding. 2,000 kcal per day gives 96.85 J/s, which the text rounds to about 100 § Energy Consumption Analysis. The external figure of 79,897 kWh per year converts to about 9,115 J/s, which the text rounds to about 10,000. The conclusions depend on a one-second comparison.
• Extra training data. Models trained on JFT, Instagram or other large sets are in the tables, but the extra data does not change their inference FLOPs § Performance and Compute. This is true of the reported FLOPs, not of whether the model is worth its energy.
• Multiplicative factor. Per-capita inference rate is an assumption (one pass per second) and is not estimated [§ Forecasting and Multiplicative Effect]. Totals depend on it entirely.
• Minor table rounding. Some adjusted values differ slightly from the theoretical table, for example K20x 16.77 vs 16.74 and GTX 1080 49.28 vs 49.29 tab. 8 tab. 9. Negligible for the trends.
• Figure-only results. Several claims rest on plots (fig. 3, fig. 8, fig. 9, fig. 11); the paper gives no fit parameters or underlying numbers for the fitted lines beyond the tables.

Lineage

• Builds on: the EfficientNet scaling relation [§ Number of Parameters and FLOPs], the NoisyStudent and FixEfficientNet estimates, and the GLUE benchmark § NLP Analysis.
• Compared against: earlier inference studies of CNN power on embedded boards and GPUs, and energy-versus-FLOPs correlation work § Background; compute trends in training § Background.
• Superseded by: not stated.