Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling

In plain words

People quote very different numbers for how much electricity one chatbot answer uses. This paper builds an estimate from the bottom up: how fast a server full of AI chips can produce words, how much power that server draws, and how much extra power the data centre adds for cooling and overhead. Run over many random but realistic setups, it gives a typical answer around a third of a watt-hour for large models. Long, reasoning-heavy answers can cost more than ten times as much, so they dominate the total when they become common.

The paper then asks which improvements could cut the number: better models, smarter serving software, and newer hardware. Its estimate is that these together could reduce the energy per answer several-fold to around twenty times, and that sending easy questions to smaller models is one of the main levers. The results are the authors' own estimates, not measurements.

TL;DR

• Per-query energy is modelled as node power times time per query, divided by the efficiency gains, with PUE and a fixed constant for the unit conversion eq. 1.
• Median for models above 200B parameters on H100 nodes: 0.31 Wh per query (IQR 0.16-0.60) in the traditional regime fig. 2.
• Test-time scaling (median 5,000 output tokens): 3.91 Wh (IQR 2.15-7.05) fig. 2.
• Author-reported: line-of-sight improvements give 8-20x, and 10% long queries roughly double the daily energy of 1 billion queries fig. 3.

Problem

Public "energy per query" figures differ by more than an order of magnitude. The authors attribute this to differences in what is measured (GPU only or whole node), and to benchmarks that do not reflect batched, high-concurrency serving §1. Single "average query" figures also hide the effect of long outputs from reasoning models and agentic workflows §1. The paper's scope is steady-state, autoregressive text generation; diffusion image and video generation are excluded §4.

Method

Query definition. A query is one prompt-completion event with a single input and one generated output, served under steady state. Multi-step agentic workflows are out of scope and would be modelled as several queries, or as extra overhead that the model does not capture §2 §4.

Estimator. For a fixed GPU node, energy per query is computed from steady-state node power, the effective token length, the throughput at which the node emits tokens for the given input and output lengths, PUE, and an efficiency multiplier that is 1 in the baseline eq. 1 §2. The paper runs a Monte Carlo over 10,000 random draws of output length, node power and PUE for each model §2.

Throughput. Tokens per second for a saturated node come from published NVIDIA TensorRT-LLM benchmarks on H100 with FP8 and tensor parallelism 8 (10 for DeepSeek-R1) and continuous batching §2 [supp. S1]. A log-linear regression in input and output length is fitted to these points. Predictions are capped at the largest observed throughput, so the model is flat beyond the benchmarked range [supp. S2].

Regimes. Traditional queries use a median of 300 output tokens (IQR 129-618). Test-time-scaling queries use a median of 5,000 (IQR 2,040-9,717). Input length is fixed at 500 tokens in the main analysis §2 fig. 1.

Baseline and line-of-sight scenarios. The baseline blends models above 200B parameters, which the authors sample at alpha = 1 §2 fig. 2. Efficiency interventions are grouped into three categories. Each category has its own multiplier alpha, drawn from a log-normal distribution with the stated P5-P95 range. Categories are treated as independent and non-additive within the Monte Carlo, but a stacked, illustrative estimate multiplies them §3 fig. 2.

Datacenter scale. Energy for serving a number of queries is scaled by beta = 1.31, the product of a 1.10 networking factor, a 1.08 utilisation factor from a sinusoidal load between 50% and 100%, and a 1.10 node redundancy factor [supp. S5] §2.

Pseudocode for the per-query estimate:

# Energy per query on one fixed compute node  [eq. 1] [§2]
# Input:  p_node (kW), l_eff (tokens), tps (tokens/s), pue, alpha
# Output: Wh per query

def energy_per_query(p_node, l_eff, tps, pue, alpha):
    # Node power times the time to emit l_eff tokens; 3.6 converts kW*s
    # to Wh  [eq. 1]
    return (pue / 3.6) * (p_node * l_eff / tps) / alpha

Pseudocode for the throughput model:

# Throughput of a saturated node  [supp. S2]
# Input:  l_in, l_out (tokens); b0, b1, b2 (fitted, not stated); tps_max
# Output: tokens per second

def tps_model(l_in, l_out, tps_max):
    # Log-linear fit of log(TPS) on log(input) and log(output)  [supp. S2]
    log_tps = b0 + b1 * log(l_in) + b2 * log(l_out)
    # Cap at the largest benchmarked TPS; the decline in very long
    # generations is not modelled  [supp. S2]
    return min(exp(log_tps), tps_max)

Hyperparameters

Results

Author-reported. Setup: Monte Carlo over 10,000 draws; H100 nodes; FP8; traditional and test-time-scaling output lengths as above §2 §3.

Per model, no efficiency gains (median Wh per query, IQR in brackets) fig. 1:

Baseline blend and line-of-sight scenarios (Wh per query) fig. 2:

Illustrative stacks (text, not separate Monte Carlo runs) §3: a conservative stack gives roughly 8-9x (about 1.5x hardware, 2x serving, 2.5-3x model); an optimistic stack gives about 20x (2x hardware, 2.5x serving, 4x model). The authors note these overstate gains when layers overlap.

Power-proportional variant [supp. S4] [supp. fig. S3]: letting node power rise linearly with TPS (idle 2.7 kW to 0.9 * P_max) changes the baseline median to 0.32 Wh and the test-time median to 4.71 Wh (IQR 2.44-8.72).

Datacentre totals (1 billion queries per day, beta = 1.31) §3 fig. 3:

The authors put the 0.8 GWh/day figure at about 0.4% of the energy used to power US household television, and relate the baseline to a 30 MW H100 data centre running at 80% utilisation for under 0.6 GWh per day §3.

Comparison with prior estimates tab. 1 p. 5-6. Values are the study's reported figures, normalised to H100 FP8 where the study allows. The "within IQR" column is the study's check against its own range:

The paper concludes that most published figures for production-scale measurements fall within its IQR, and that the non-production estimates overstate energy by 4-20x tab. 1 §4.

Paper vs code

No official code. The paper says analysis scripts "will be open sourced at the time of publication" p. 13, and it links no code repository for this analysis. Searches done: the paper's text for links; the GitHub repository search API with the title words "inference energy per query test-time scaling" and "energy per query LLM inference line-of-sight", both returning zero results. The only analysis inputs are the public TensorRT-LLM tables in [supp. S1], which the paper cites from NVIDIA's own documentation.

Pitfalls

• Table 1 and Figure 1 disagree on DeepSeek-R1. Table 1 gives an IQR of 0.19-0.68 for the 0.63 Wh median tab. 1; Figure 1a gives 0.26-1.29 fig. 1. Use the figure values only after checking the source.
• Two different values for the same datacentre scenario. Figure 3a gives 0.33 GWh per day for the line-of-sight traditional scenario, while the matching bar in Figure 3b gives 0.29 GWh fig. 3. The text uses alpha in [1.5, 3] for both. The gap is not explained.
• Maximum node power. Section 2 gives P_max of 10.2 kW for 8 GPUs and 12.5 kW for 10 GPUs §2. The datacentre calculation uses 11.3 kW peak and 2.7 kW idle [supp. S5]. Pick one and document which.
• Effective length. The main analysis sets L_eff equal to L_out with fixed L_in = 500 §2. Supplemental Note S3 defines L_eff as L_in + L_out [supp. S3]. The two give different energy for long prompts: the authors report within 6.5% for traditional queries at L_in = 500, and at L_in = 5,000 the traditional distribution becomes bimodal, because long prefill dominates short outputs [supp. S3].
• Regression coefficients are not in the paper. The form of the log-linear TPS model is given, but the fitted beta values are not stated [supp. S2]. Refit from the rows of [supp. tab. S1] before reproducing the medians. The paper's fit covers outputs up to 4,000 tokens, and its cap means the throughput stays flat beyond that [supp. S2].
• Throughput is a saturated-node figure. The TPS values come from high-concurrency benchmarks, so the per-query energy assumes the node is always full. Lower utilisation raises energy per query, and the paper's power-proportional variant only partly covers this [supp. S4].
• Distributions are not fully specified. The exponential rate for L_out, the discrete values of L_in, and the log-normal sigma for P_node, PUE and alpha are not stated. The card gives only the P5-P95 ranges and centres §2 §3.
• Alpha is applied as a divisor or multiplier of energy, not throughput. Equation 1 divides by alpha, and the text says alpha can scale throughput or energy per query equally §2 eq. 1. Implement it once, in one place.
• Categories are sampled, then stacked by hand. The Monte Carlo treats categories as independent, but the combined 8-20x figures are derived by multiplying category endpoints, which the authors say overstates gains §3.
• Long outputs and agents are under-modelled. Prefill cost for very long inputs is not modelled, and agentic workloads with tool calls and several parallel calls may be dominated by overhead the paper excludes §4.
• Baseline model set. The "above 200B" label on the baseline is shorthand. The sampled models are DeepSeek-R1 671B, Llama 3.1 405B and Llama 3.1 Nemotron Ultra 253B §3 fig. 1. The 200B threshold itself is motivated by other models, such as Qwen 3 and GPT-4o, which are not sampled §3.
• Electricity scope. Most public measurements report GPU energy only, while these estimates include PUE and node overhead §3. Compare absolute values with that in mind.
• Grid and history comparisons are not model outputs. The comparisons with US grid share and with past data-centre forecasts in §3 are the authors' reasoning, drawing on cited external sources [62]. The energy model does not compute them.

Lineage

• Compared against: the production and non-production measurements and estimates in tab. 1, including ML.ENERGY (Chung et al., NeurIPS 2025) and the AI Energy Score (Salesforce, Hugging Face).
• Serving techniques used to derive the serving-platform category §3: KV cache management, disaggregated prefill and decode, speculative decoding, and FP8 quantisation. These are cited to the paper's references rather than reproduced here.
• Related wikillm pages: test-time-scaling, llm-serving, model-routing.