--- kind: concept slug: gpu-power-capping title: GPU power capping summary: >- Limiting how much power each GPU may draw to cut the energy used per LLM output, at the cost of run time. One measurement study so far: capping A100s at 175 W running LLaMA 65B saved about 23% energy for about 7% more time on one dataset. A 150 W cap cost much more time, and the evidence covers one model, dataset and batch size. tags: [inference, energy, gpu, benchmarking] papers: ["2310.03003"] related: [llm-serving] status: draft updated_at: 2026-11-01 --- ## Overview A power cap limits how much power each GPU may draw. The only source compiled here that measures a cap during LLM inference is a benchmark of LLaMA models on NVIDIA V100 and A100 GPUs [arxiv:2310.03003 §3]. It runs at a default per-GPU cap of 250 W and tests lower caps of 175 W and 150 W on the A100s [arxiv:2310.03003 §3.3, §4.5]. The question it answers is how a cap changes the energy used per output and the time taken, compared with the default on the same workload [arxiv:2310.03003 §4.5, tab. III]. Energy is taken from the GPU readings of the rank-0 node, scaled by the node count, which the paper uses as the energy of the whole job [arxiv:2310.03003 §3.3]. Measurements are per job: one pass over the same set of prompts at one batch size [arxiv:2310.03003 §3.3]. ## Variants ### Static per-GPU cap on A100 (the case studied) The cap is fixed for the whole run and set per GPU. The study applied it to LLaMA 65B on four A100 80 GB GPUs, with GSM8K prompts at batch size 64, and compared each cap with the 250 W default [arxiv:2310.03003 §4.5, tab. III]. At 175 W, across maximum output lengths of 256, 512 and 1024, run time rose by 6.2% to 7.4%, GPU energy fell by 21.8% to 24.0% (a mean of 23.21%), and the token rate fell by 5.9% to 6.9% [arxiv:2310.03003 §4.5, tab. III]. At 150 W, run time rose by 15.3% to 21.7%, energy fell by 32.8% to 34.7%, and the token rate fell by 13.2% to 17.8% [arxiv:2310.03003 tab. III]. The paper calls the 150 W time cost much larger and says a single static cap may not suit every workload [arxiv:2310.03003 §4.5]. Card: [From Words to Watts](/papers/2310.03003). ### Caps on other GPUs and sharded layouts Not measured. The power-cap results cover one layout only: LLaMA 65B on four A100 80 GB GPUs [arxiv:2310.03003 §4.5, tab. III]. The V100 runs, including the 8, 16 and 32-GPU sharded runs, have no cap results on this page. ## When to use which | Situation | Use | Why | |---|---|---| | Batch serving where energy per output matters more than run time, on A100s, for a model around LLaMA 65B's size | Try 175 W, measured on your own workload | In the one study it cut energy by about 23% for about 7% more time [arxiv:2310.03003 §4.5, tab. III] | | Interactive or latency-bound serving | Keep the default, or cap only after measuring latency against your target | The 175 W cost (6% to 7% more time) may still exceed a tight latency budget; the study reports throughput-style rates, not latency percentiles [arxiv:2310.03003 §3.3, §4.5] | | A deeper cut to 150 W | Only if a 15% to 22% longer run time is acceptable | Energy fell about 33% to 35%, but time rose and token rate fell much more than at 175 W [arxiv:2310.03003 tab. III] | | V100 or other GPU types | No evidence on this page | The power-cap study covers A100s only [arxiv:2310.03003 §4.5] | | Layouts sharded across many nodes | Measure before relying on energy totals | Node energy is extrapolated from the rank-0 node and assumed equal across nodes [arxiv:2310.03003 §3.3] | | Mixed workloads (several models, prompt sets or batch sizes) | Sweep caps over your own mix | The authors say one static cap may not suit all workloads [arxiv:2310.03003 §4.5]. No dynamic cap was tested | ## Implementation sketch A sweep that repeats one fixed workload at each cap and reports changes against the default. The steps follow the study's measurements; the function that sets the cap is a placeholder, because the card does not record how the study applied it. ```pseudocode # Sweep per-GPU power caps over one fixed workload [§3.3] [§4.5] # Input: caps in W, e.g. [250, 175, 150]; baseline is 250 W # Output: run time, energy and token-rate change against the baseline def cap_sweep(caps, run_workload, set_power_cap, baseline=250): runs = {} for cap in caps: # Apply the same per-GPU cap to every GPU in the job [§4.5] set_power_cap(cap) # One pass over the same prompts, batch size and max length [§3.3] # Energy = rank-0 GPU energy x node count [§3.3] runs[cap] = run_workload() # -> seconds, energy_j, tokens base = runs[baseline] report = {} for cap, run in runs.items(): # Each change is relative to the default cap [§4.5] report[cap] = { "time_change": run.seconds / base.seconds - 1, "energy_change": run.energy_j / base.energy_j - 1, "token_rate_change": (run.tokens / run.seconds) / (base.tokens / base.seconds) - 1, } return report ``` Run each cap at every output length you serve, because the study reports per-length values [arxiv:2310.03003 tab. III]. ## Evidence limits - One model (LLaMA 65B), one dataset (GSM8K), one batch size (64) and one GPU type (A100 80 GB) for the cap results [arxiv:2310.03003 §4.5]. - One run per cell and no error bars [arxiv:2310.03003 §4.5]. - Output quality was not measured, and quantisation, continuous batching and request arrival rates were not studied [arxiv:2310.03003 §3.3]. - The paper calls its power-capping conclusions preliminary [arxiv:2310.03003 §5]. - The measured model's weights carry a non-commercial licence, which the paper notes [arxiv:2310.03003 §2.2]. This page does not assess whether reproducing the runs is allowed for commercial use. ## Contradictions The 150 W run-time increase is given as +19.49% in the text, while the three per-length values in the table average to 19.56% [arxiv:2310.03003 §4.5, tab. III]. Both figures are reported as they appear; the difference is small. ## Papers - [From Words to Watts](/papers/2310.03003): the only source here; measures 175 W and 150 W caps on A100s for LLaMA 65B, against a 250 W default.