Power Hungry Processing: Watts Driving the Cost of AI Deployment?

In plain words

Most public talk about the energy cost of AI is about training: the one-off work of building a model. A model that answers millions of requests, though, spends most of its life being used, and every use costs a little electricity and carbon. This paper asks how much one kind of use costs, for different kinds of models.

The authors ran 88 models on ten kinds of task, from labelling text and images to writing text and pictures. Some models were built for one job. Others are general-purpose models that are given instructions to do many jobs. For each model they measured the energy and carbon used to answer 1,000 requests, one request at a time, on the same hardware. Here a request is one inference: one run of the model on one input.

The main finding is that the cost depends mostly on the task. Choosing a label is cheap. Writing words or pictures is expensive, and a general-purpose text model costs much more than a small specialised model for the same labelling job. The authors also work out how many requests a model must serve before the energy of its use adds up to the energy spent training it. For the models they studied that number is in the hundreds of millions, so popular models can reach it quickly.

TL;DR

• Measures the energy and CO2eq of 1,000 inferences for 88 models across 10 tasks and 30 datasets. The run used CodeCarbon on a node of eight NVIDIA A100 GPUs on AWS §3.1 §3.2 tab. 1.
• Classification is cheap and generation is expensive. Mean energy per 1,000 inferences is 0.002 kWh for text classification and 2.907 kWh for image generation tab. 2 §4.1.
• Zero-shot multi-purpose models (Flan-T5, BLOOMz) cost more than task-specific models on the same task, most clearly for classification and question answering §4.2 fig. 3 fig. 4.
• Using the energy of BLOOMz training and fine-tuning as the baseline, inference reaches parity after about 204.6 million to 592.6 million queries tab. 5 §4.3.
• The carbon figures use about 139 g CO2eq/kWh in the authors' own data, not the 297.6 g/kWh the paper states. See Pitfalls.

Problem

Inference is often the largest share of deployed ML compute. The paper cites an AWS estimate of 80 to 90% of ML cloud demand, and two company figures: about a third of Meta's internal ML footprint, and 60% of Google's ML energy §1. Yet measured work on inference cost was limited, and most studies covered training §1 §2. Existing tools for estimating ML carbon also disagree with one another, so results from different studies are hard to compare §2.

The paper's setting is fixed deployment on one compute region and hardware type. It holds code, hardware, datasets, batch size and Python libraries steady across models so that the differences come from the models §5. It assumes inference runs sequentially, one input per call, without batching §3.2.

Method

The study has three parts: a set of tasks and datasets, a set of models, and a measurement procedure.

1. Tasks and datasets. Ten tasks from five modalities. Each task is tested on three of the most downloaded Hugging Face Hub datasets §3.1 tab. 1.
2. Task-specific models. For each task, the 8 most downloaded Hub models were taken, after discarding models in another language or incompatible with the chosen datasets. This gives 80 models §3.2 app. A (footnote in §3.2).
3. Multi-purpose models. Eight zero-shot models: Flan-T5 base, large, xl and xxl, and BLOOMz at 560M, 1B, 3B and 7B parameters. They were tested on three tasks only: text classification, extractive question answering and summarization §3.2 §4.2 tab. 3.
4. Prompting. Zero-shot models get a fixed prompt per task, the same for every model in the task. The paper's example is "Summarize the following text: [text]. Summary:" §3.2. The code's actual prompts differ (see Paper vs code).
5. Measurement. Each model is run on 1,000 inferences per dataset, one input at a time, and each set is repeated 10 times for statistical significance §3.2. Energy and emissions come from CodeCarbon §3.2.
6. Carbon conversion. Emissions equal energy times the grid carbon intensity. The paper uses one region, us-west-2 (Oregon), and reports both energy and carbon §3.2.
7. Training comparison. The training energy of the BLOOMz models is not published by the BLOOMz authors. The paper takes training energy from the BLOOM carbon study's logs, and fine-tuning energy from the BLOOMz training logs, estimated from GPU thermal design power §4.3 (footnote).

Quality check: task-specific models were scored with the evaluate library and zero-shot models with the LM Evaluation Harness §3.2 §4.2 app. B. The authors note that the two model types are hard to compare on the same metrics, and aim only for a standardized evaluation §4.2.

Tasks and datasets

Models

Task-specific models, as listed in app. A (Hugging Face Hub IDs as printed; see Pitfalls for one spelling):

Multi-purpose models (tab. 3): Flan-T5 base (222M), large (750M), xl (2.8B), xxl (11B); BLOOMz-560M (559M), BLOOMz-1B (1.7B), BLOOMz-3B (3B), BLOOMz-7B (7B). The code uses bigscience/bloomz-1b7 and bigscience/bloomz-7b1 for the paper's BLOOMz-1B and BLOOMz-7B zeroshot_sentiment_imdb.py.

Measurement procedure

No numbered equations or algorithms appear in the paper. The procedure below follows the released code. Pseudocode is written in our own words.

# Energy and emissions of one model on one dataset  [§3.2]
# Input:  model id, samples (1,000 inputs, one at a time)
# Output: energy (kWh) and emissions (kg CO2eq) of the query task  [§4.1]

def measure_model(model, samples):
    # Start a CodeCarbon tracker that samples power every second
    # [§3.2] [code: code/text_classif/text_classif_imdb.py#L40]
    tracker = start_tracker(power_interval_s=1)
    # Loading the model is recorded as its own task
    # [code: code/text_classif/text_classif_imdb.py#L42-L44]
    tracker.start_task("load model")
    pipe = load_pipeline(model)
    tracker.stop_task()
    # Send each input alone, with no batching  [§3.2]
    # [code: code/text_classif/text_classif_imdb.py#L47-L49]
    tracker.start_task("query model")
    for sample in samples:
        pipe(build_prompt(sample))
    query = tracker.stop_task()
    # Only the query task is used for the reported numbers
    # [jupyter/CO2-inference-paper.ipynb#L294-L308]
    return query.energy_consumed, query.emissions

Parity between training and inference energy:

# Inferences at which inference energy equals the energy of building
# the model, training plus fine-tuning  [§4.3] [tab. 5]
# Input:  training energy (kWh), fine-tuning energy (kWh),
#         energy of one inference (kWh)
# Output: number of inferences (float)

def cost_parity(training_kwh, finetuning_kwh, inference_kwh):
    # Total energy spent building the model  [tab. 5] [§4.3]
    lifecycle_kwh = training_kwh + finetuning_kwh
    # Divide by the energy of one inference  [§4.3]
    return lifecycle_kwh / inference_kwh

Hyperparameters

Results

Author-reported throughout. Each figure is the paper's value under the stated hardware, dataset and prompt setup; none was reproduced for this card.

The authors say parity can be reached quickly for popular models §5. They also say they see no convincing case for generative zero-shot models where the task is well defined, such as web search and navigation §5.

Full tables

Table 2: energy per 1,000 queries, ten tasks tab. 2 (kWh)

Table 3: zero-shot models, 1,000 inferences tab. 3 (averages; the caption does not say over which tasks)

Table 4: average input and output length in tokens tab. 4 (output columns: BLOOMz-560M, 1B, 3B, 7B, then Flan-T5 base, large, xl, xxl)

Table 5: BLOOMz training cost and parity tab. 5

Appendix B: performance of the 32 models checked for quality app. B

Scores as printed (accuracy for classification and QA; F1 for SQuAD; ROUGE for summarization). Blank cells are tasks the model was not run on.

Figures

No images are reproduced on this page. The paper's figures, described in words (figure numbers follow the LaTeX order; see Pitfalls):

• fig. 1: tasks against mean carbon per 1,000 queries, log scale. The paper cites it for generative tasks and image tasks being the most intensive §5.
• fig. 2: five modalities, model size (parameters) against carbon per 1,000 inferences, both axes log scale. The paper says task structure explains more of the spread than model size does §4.1.
• fig. 3: emissions per dataset, log scale, for task-specific and multi-purpose models. Dot size is model size. For the discriminative tasks the paper's QA example shows a gap of about 30×; the summarization gap is smaller §4.2.1.
• fig. 4: text classification accuracy against model size, with dot size for emissions. Multi-purpose models vary less in accuracy §4.2.1.
• fig. 5: total emissions of the zero-shot models per task, linear scale. Summarization is the most carbon-intensive of the three tasks; the spread between tasks is larger for decoder-only models §4.2.2.
• fig. 6: summarization output length against emissions. Decoder-only models show a steeper slope than seq2seq models §4.2.2.
• The two appendix figures (app. B) plot model size against summarization and QA accuracy, with emissions as dot size.

Paper vs code

The official repository is sashavor/co2_inference, cloned at commit 9ae9895facde2582cb5a981ab03e7bb5ddccafc9. It is MIT-licensed LICENSE. The paper links it §3.2 §5, and its owner is an author.

• Carbon intensity. The paper says the region's intensity is 297.6 g CO2eq/kWh §3.2. The shipped emissions data do not use that value: the row for text_classif_imdb gives 0.0005925 kg for 0.0042703 kWh, about 139 g/kWh [data/emissions/complete/text_classif_imdb.csv#L2]. The per-task base files give the same ratio [data/emissions/base/emissions_base_f9cd7c9f-423e-4759-949b-f1921a667434.csv#L2]. Table 3's ratios (about 138–141 g/kWh) match this figure, not 297.6 tab. 3.
• Which energy counts. The paper says "energy consumed during inference" §3.2. The notebook that produces the reported numbers reads only the query model task rows [jupyter/CO2-inference-paper.ipynb#L294] [jupyter/CO2-inference-paper.ipynb#L308]. The complete-run CSVs have no task column and include loading [data/emissions/complete/text_classif_imdb.csv#L1]. The paper does not state which the reported numbers are, so the notebook is the only evidence.
• Zero-shot prompts. The paper's example is "Summarize the following text: [text]. Summary:" §3.2. The code's summarization prompt is 'summarize the following text: ' with no "Summary:" suffix zeroshot_summarize_cnn.py. Sentiment uses 'what is the binary sentiment of the following sentence? ' zeroshot_sentiment_imdb.py. QA uses the question, then a context and '? Find the answer to the question in the following text: "…". Answer:' zeroshot_qa_squad.py. The prompts are constant per task across models, as the paper says §3.2.
• Output length. The paper does not state generation limits for the zero-shot models. The code sets them per architecture and task (Hyperparameters table). The paper's explanation for the decoder-only versus seq2seq gap relies on output length §4.2.2, so these limits matter to that comparison.
• Model names. The paper's BLOOMz-1B and BLOOMz-7B are bigscience/bloomz-1b7 and bigscience/bloomz-7b1 in the code zeroshot_sentiment_imdb.py.
• Model selection. The text-classification list is pulled from the Hub at run time: of the 20 most-downloaded text-classification models, the first 8 whose IDs do not contain fin text_classif_imdb.py. The IDs can change with Hub ordering; use the appendix list.
• Dataset splits. The paper gives no split. The code uses the test split for IMDB text_classif_imdb.py, the validation split for SQuAD qa_squad.py, and the train split for SAMSum summarize_samsum.py. Each script takes the first 1,000 rows qa_squad.py.
• Repetitions. The paper says each set was run 10 times §3.2. The scripts run one pass per model. The 6,513 per-run files in data/emissions/base/ are consistent with repeats, but this review did not check the repeat count per model.
• Notebook paths. The notebook reads ../emissions_sept20/... [jupyter/CO2-inference-paper.ipynb#L292]. That folder is not in this checkout; the data are under data/emissions/. Rerunning the notebook needs the paths changed.
• Evaluation fork. The zero-shot quality check uses the authors' fork of the LM Evaluation Harness, pinned at 8fbf02529d6124156ce9bcf873e14f35576aac10 requirements.txt. The paper names the harness but not the fork §4.2.

Pitfalls

• Energy scope. Use only the query model task when comparing to the paper. The whole-run CSVs cover the full tracker session, which starts before model loading text_classif_imdb.py, so their energy is higher [jupyter/CO2-inference-paper.ipynb#L294-L308] [data/emissions/complete/text_classif_imdb.csv#L1]. The paper does not state this, so the card relies on the notebook.
• Energy components. CodeCarbon's energy_consumed is the sum of CPU, GPU and RAM energy. In the first row of text_classif_imdb.csv, the parts add up to the total [data/emissions/complete/text_classif_imdb.csv#L2]. The paper does not say which parts count.
• Grid intensity. The carbon figures correspond to about 139 g CO2eq/kWh, not 297.6 §3.2 tab. 3. To estimate carbon for another grid, scale the energy column yourself. Do not multiply the paper's carbon by a factor.
• Output caps. Decoder-only classification generates one token and seq2seq classification has no cap, so the two families are not length-matched. Decoder-only QA on SQuAD allows 65 tokens and removes the end-of-sequence stop, but SciQ sets no cap at all zeroshot_qa_sciq.py. Keep the code's settings to reproduce.
• Hardware. The paper measures on A100-SXM4-80GB GPUs in one AWS region §3.2. The paper says energy should stay similar for A100s in other regions, while carbon varies with the grid §3.2. Energy on other hardware is not covered.
• Inference excludes serving. The BLOOM study's 0.004 kWh per query included API deployment; the BLOOMz figures here do not §4.3. The two are not directly comparable.
• Parity arithmetic. The parity counts use training plus fine-tuning energy, not training alone. The caption describes training only, but the numbers match a total row that is commented out in the arXiv LaTeX source (main.tex, line 433). That row gives 59,257 kWh for BLOOMz-7B, and 59,257 ÷ 0.0001 ≈ 592,570,000 tab. 5.
• Parity precision. Per-inference energy is given to two significant figures (for example 5.4 × 10⁻⁵). Each parity count is therefore uncertain by roughly 1–2%.
• Source of training energy. BLOOMz training energy comes from the BLOOM study's logs, not from the BLOOMz run §4.3 tab. 5. Fine-tuning energy is an estimate from GPU TDP that assumes 100% utilization §4.3 (footnote). No carbon is estimated for training.
• Internal inconsistencies. The text gives text generation as 0.042 kWh §4.1, while Table 2 gives 0.047 tab. 2. The text says image classification has a median of 0.0068 kWh §4.1, but Table 2 lists means tab. 2. The text calls the gap between task-specific and multi-purpose models "several orders of magnitude" §4.2.1, but its own example, 0.3 g against 10 g, is about 30× §4.2.1. The model sbcBI/sentiment_analysis_mode appears in app. A as printed; app. B calls it sbcBI/sentiment_analysis_model.
• Table 3 averaging. Table 3 says "average" but not over which tasks. Its values (for example 3.67 g for Flan-T5-base) cannot be matched to the QA values in §5 (2.36 g for Flan-T5-base). Use the per-task figures in fig. 5 or the data files, not Table 3, for per-task claims.
• Quality coverage. Only 32 of the 88 models have accuracy numbers app. B. The other 56 (image, token, masked language modelling and others) have none. The quality check is rough: the authors say the two model families cannot be compared on identical metrics §4.2.
• Figure numbering. Figure numbers here assume the teaser is Figure 1, following the LaTeX order. Check against the typeset PDF before citing a number.
• Version. This card follows arXiv v3, updated 2024-10-15 [raw/arxiv/2311.16863.json]. Earlier versions used a different smartphone charge figure (0.012 kWh rather than 0.022 kWh) for comparisons §4.1 (footnote).
• Authors' framing. The paper says it is not representative of all deployment contexts §5. Read the results as one hardware setup, not a general law.

Lineage

• Builds on: Luccioni et al. 2022, the BLOOM carbon study (arXiv:2211.02001 in the paper's bibliography). Its logs supply the BLOOM training energy the paper reuses §2 §4.3 tab. 5. Not carded.
• Related measurement work: Strubell et al. 2019 and Patterson et al. 2021 on training energy §2. Tools CodeCarbon, MLCO2 and LLMCarbon, compared in Bannour et al. 2021 §2. Not carded.
• Future work named by the authors: distillation, parameter count, hardware and numerical precision §5.
• Superseded by: not stated in the content.