From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

In plain words

Answering a question with a large language model uses real electricity, and this paper measures how much. The authors ran Meta's LLaMA model in three sizes (7, 13 and 65 billion parameters) on NVIDIA graphics cards from two generations, on a university supercomputer. The largest model is too big for one card, so it was spread over up to 32 GPUs. They fed the model two sets of prompts, one of instructions and one of grade-school maths problems, and recorded how fast it produced text and how much energy the GPUs used.

The main lessons are simple. The big model needs many GPUs before it can run at all, and more GPUs mean more energy per second. Lowering the power limit on the newer GPUs saved about a quarter of the energy for the big model, while making it slower by a few percent. The authors call the work preliminary, and it covers only some settings.

TL;DR

A measurement study, not a new method: LLaMA 7B, 13B and 65B are run for inference on V100 and A100 GPUs, including 8 to 32-GPU sharded runs, with energy read from the GPUs. The headline result is that capping the power of the A100 GPUs at 175 W reduced energy by about 23% for the 65B model on one dataset, with about 7% more time §4.5 tab. III. Relevant if you need energy or cost numbers for serving a model of this size.

Problem

Most energy studies of language models cover training. The authors state that inference energy gets less attention, even though inference runs far more often §1. They point to two gaps: no public numbers for large-model inference at multi-node scale, and no view of how GPU resources are used during it §1 §5. The study targets LLaMA 65B because it is the largest model available to them and needs sharding to run §2.2 §3.1.

The paper's own limits: output quality is not measured, and the study does not test quantisation, continuous batching, query arrival rates or other optimisations §3.3.

Method

This is a benchmark protocol, not an algorithm. The steps below describe what was measured and how, so the results can be reproduced in spirit.

1. Hardware. Experiments ran on the MIT SuperCloud cluster §3. Each V100 node has two GPUs, and four nodes with A100 GPUs were also used §3 tab. I. All runs used the 25 Gb/s Ethernet fabric, not the 100 Gb/s Omnipath one §3.

   Node CPU (memory GB, TDP W)	GPU (memory GB, TDP W)
   Intel Xeon Gold 6248 (384, 150)	NVIDIA V100 (32, 250)
   Intel Xeon Platinum 8358 (503, 240)	NVIDIA A100 (80, 300)

2. Model. Pretrained LLaMA weights obtained from Meta, with Meta's evaluation scripts §3.1. Model sharding uses FairScale across GPUs and nodes §3.1. Decoding uses temperature 0.8 and top-p 0.95 §3.1. The paper does not state the numeric precision it ran in (see Pitfalls).

3. Workload. From each dataset, 4,096 inputs are sampled §3.2. The Alpaca set has 52,000 instruction tasks and the GSM8K set 8,500 maths problems §3.2. One experiment (job) is one pass over all 4,096 prompts at one batch size §3.3.

4. Generation. For each prompt the model writes until it reaches the maximum length or hits a stop word §3.3. The baseline uses batch size 64 and maximum length 256 §4.1.1. Energy sweeps add maximum lengths 512 and 1024 §4.2 §4.3 §4.4. Length 2048 appears only in the A100 utilisation study §4.6.

5. Placement. Minimum resources for LLaMA 65B are 8 V100 GPUs or 4 A100 GPUs §5 tab. II. The 65B model is sharded evenly across the GPUs, which leaves 20 to 25% of GPU memory in use §5. Shard counts of 8, 16 and 32 are run on V100s, and smaller experiments on A100s §3.3.

6. Energy measurement. nvidia-smi samples GPU usage every 100 ms. NVIDIA DCGM reports aggregate GPU energy for the rank-0 node §3.3. The default per-GPU power cap is 250 W §3.3.

7. Metrics. Words per second count space-separated words in the output. Tokens per second use LLaMA's own tokenizer. Responses per second divide responses by total run time §3.3.

Energy metrics, from the definitions in §3.3:

# Energy metrics for one job  [§3.3]
# Input:  rank0_energy_joules (DCGM, rank-0 node); node_count;
#         run_seconds; decoded_tokens; responses
# Output: energy_per_second (W), energy_per_token (J), energy_per_response (J)

def energy_metrics(rank0_energy_joules, node_count, run_seconds,
                   decoded_tokens, responses):
    # Scale rank-0 energy by the number of nodes  [§3.3]
    total_energy = rank0_energy_joules * node_count
    # Total GPU energy divided by job run time  [§3.3]
    energy_per_second = total_energy / run_seconds
    # Total GPU energy divided by decoded output tokens  [§3.3]
    energy_per_token = total_energy / decoded_tokens
    # Total GPU energy divided by number of responses  [§3.3]
    energy_per_response = total_energy / responses
    return energy_per_second, energy_per_token, energy_per_response

Hyperparameters

Results

Author-reported, all for LLaMA on MIT SuperCloud GPUs. Tables are cited as tab. I to tab. V in order of appearance; the PDF prints them in roman numerals. Most energy results are in figures, and the paper gives only approximate values for them.

Minimum hardware and batch limits tab. II, maximum response length 256:

Baselines §4.1: fig. 2 shows the A100 is faster than the V100 for all sizes, with the largest gain for 7B, and a smaller one for 65B §4.1.1. The paper attributes the smaller 65B gain to sharding across nodes, which adds latency to each forward pass §4.1.1. The 7B gain is larger on Alpaca than on GSM8K, which the paper attributes to input complexity §4.1.1. On energy, the A100 uses considerably more energy per second than the V100 at every size, most of all for 7B §4.1.2 fig. 3. The paper says the extra throughput for 65B may not justify the extra energy §4.1.2.

Energy per second, 65B §4.2 fig. 4 fig. 5: energy per second rises with the number of shards, even at equal batch size. It rises with batch size only slightly. Moving the maximum length from 512 to 1024 raises it within each shard setting, but the effect is not consistent overall §4.2.

Energy per token, 65B §4.3 fig. 6 fig. 7: energy per output token also rises with shards. Lengths 512 and 1024 give similar values. On GSM8K at maximum length 512, a batch-size increase at 16 shards lowers energy per token, a pattern that disappears at length 1024 §4.3. The authors call this a possible sweet spot that needs more experiments §4.3.

Energy per response, 65B §4.4 fig. 8 fig. 9: the pattern matches energy per token. Shards raise energy per response, and length has no clear effect. The GSM8K sweet spot at 16 shards also appears here §4.4.

GPU power capping, 65B §4.5 tab. III. Reported as the paper's table (one run per cell, no error bars given):

The paper's own summary: a 30% power cut (250 W to 175 W) gives about 6.7% more time and about 23.21% less energy; 150 W costs much more time, and the paper says a single static cap may not suit all workloads §4.5.

GPU utilisation §4.6:

The authors read the low memory use as room to co-locate several models on one GPU, using NVIDIA MPS or MIG §4.6 §5. They did not test this §4.6.

Paper vs code

No official code. The paper and arXiv record name no code repository of the authors. The repositories it cites (Meta's llama, FairScale, Stanford Alpaca, NVIDIA Megatron-LM) belong to other groups §3.1. No project page was linked. A GitHub search for "words to watts", "llama energy inference" and "llm inference energy benchmark" found no repository owned by an author, and a search restricted to a MIT Lincoln Laboratory GitHub organisation found none. A search for the author names returned only unrelated accounts.

Pitfalls

• Figures carry most energy results. Energy per second, per token and per response for 65B are given only as plots, plus rough text values (300 W to 1 kW; 3 to 4 J per token). Do not treat the text values as exact.
• Node energy is extrapolated. Total energy is the rank-0 node's reading times the node count §3.3. This assumes every node draws the same energy as rank 0. The paper does not state that assumption or measure it.
• Power-cap average mismatch. The text gives +19.49% time at 150 W §4.5. The table values (15.33, 21.70, 21.65) average to 19.56%. The LaTeX source also contains an uncompiled copy of the table with 15.14 for length 256, which averages to 19.50%. Use the compiled values and note the difference.
• Power-cap evidence is thin. One model, one dataset (GSM8K), one batch size, one GPU type, and no error bars §4.5. The paper itself says results will vary by task §4.5.
• Latency vs throughput wording. §4.1.1 says "latency" but the reported numbers are rates (words, tokens and responses per second). Read the 2x and 1.25x figures as throughput gains.
• Table IV disagrees with the text. The text says SM utilisation sits at 94 to 95% in all cases and reaches 98% only at length 2048 §4.6. Table IV shows 98.81% at length 512 and 98.85% at 1024 on the A100s.
• Caption errors. fig. 4's caption repeats batch size 256 (probably 512). fig. 9's caption says maximum length 512, but its file name and the section text say 1024.
• Typo in §4.3. It says length 512 gives 3 to 4 J "which is approximately the same amount for length 512". The 3 to 4 J figure is for length 512, and no value is given for 1024 in the text.
• Inferred node count. Table II's 8 V100 and 4 A100 GPUs are read as 4 and 2 nodes, because each node has two GPUs §3. The text gives the same minimum node counts, four for V100 and two for A100 §4.1.1.
• Unstated runtime details. Numeric precision, batch scheduling (static or continuous), context length, stop-word lists and how the 4,096 prompts were sampled are all not stated (see Hyperparameters). Results depend on them.
• Licence of the measured weights. The paper is CC BY 4.0, but the paper notes LLaMA weights were released under a non-commercial licence §2.2. This card does not assess whether reproducing the runs with those weights is permitted for commercial use.
• Hidden costs. Sharding a 65B model across nodes over Ethernet adds communication, which the paper links to the smaller A100 gain §4.1.1. More shards raise energy per second and per token §4.2 §4.3. Memory use is only 23 to 27% on the A100s, so the hardware is over-provisioned for these runs §4.6.
• Scope of the conclusions. No quality measurement, no quantisation, no continuous batching, and no query arrival model §3.3. The conclusions on power capping are stated as preliminary §5.

Lineage

• Builds on: LLaMA: the models measured. The paper makes no change to them §2.2.
• Related prior work by the same group: McDonald et al. 2022 on power capping during training of language models §4.5. The source gives no arXiv id, so it is not linked.
• Compared against: the paper compares its own model sizes and GPU types only. It benchmarks no other inference system §3.3.
• Superseded by: not stated.