GPU power capping

Overview

A power cap limits how much power each GPU may draw. The only source compiled here that measures a cap during LLM inference is a benchmark of LLaMA models on NVIDIA V100 and A100 GPUs 2310.03003 §3. It runs at a default per-GPU cap of 250 W and tests lower caps of 175 W and 150 W on the A100s [arxiv:2310.03003 §3.3, §4.5]. The question it answers is how a cap changes the energy used per output and the time taken, compared with the default on the same workload 2310.03003 §4.5, tab. III.

Energy is taken from the GPU readings of the rank-0 node, scaled by the node count, which the paper uses as the energy of the whole job [arxiv:2310.03003 §3.3]. Measurements are per job: one pass over the same set of prompts at one batch size 2310.03003 §3.3.

Variants

Static per-GPU cap on A100 (the case studied)

The cap is fixed for the whole run and set per GPU. The study applied it to LLaMA 65B on four A100 80 GB GPUs, with GSM8K prompts at batch size 64, and compared each cap with the 250 W default 2310.03003 §4.5, tab. III.

At 175 W, across maximum output lengths of 256, 512 and 1024, run time rose by 6.2% to 7.4%, GPU energy fell by 21.8% to 24.0% (a mean of 23.21%), and the token rate fell by 5.9% to 6.9% 2310.03003 §4.5, tab. III. At 150 W, run time rose by 15.3% to 21.7%, energy fell by 32.8% to 34.7%, and the token rate fell by 13.2% to 17.8% 2310.03003 tab. III. The paper calls the 150 W time cost much larger and says a single static cap may not suit every workload 2310.03003 §4.5. Card: From Words to Watts.

Caps on other GPUs and sharded layouts

Not measured. The power-cap results cover one layout only: LLaMA 65B on four A100 80 GB GPUs 2310.03003 §4.5, tab. III. The V100 runs, including the 8, 16 and 32-GPU sharded runs, have no cap results on this page.

When to use which

Implementation sketch

A sweep that repeats one fixed workload at each cap and reports changes against the default. The steps follow the study's measurements; the function that sets the cap is a placeholder, because the card does not record how the study applied it.

# Sweep per-GPU power caps over one fixed workload  [§3.3] [§4.5]
# Input:  caps in W, e.g. [250, 175, 150]; baseline is 250 W
# Output: run time, energy and token-rate change against the baseline

def cap_sweep(caps, run_workload, set_power_cap, baseline=250):
    runs = {}
    for cap in caps:
        # Apply the same per-GPU cap to every GPU in the job  [§4.5]
        set_power_cap(cap)
        # One pass over the same prompts, batch size and max length  [§3.3]
        # Energy = rank-0 GPU energy x node count  [§3.3]
        runs[cap] = run_workload()  # -> seconds, energy_j, tokens
    base = runs[baseline]
    report = {}
    for cap, run in runs.items():
        # Each change is relative to the default cap  [§4.5]
        report[cap] = {
            "time_change": run.seconds / base.seconds - 1,
            "energy_change": run.energy_j / base.energy_j - 1,
            "token_rate_change": (run.tokens / run.seconds)
            / (base.tokens / base.seconds) - 1,
        }
    return report

Run each cap at every output length you serve, because the study reports per-length values 2310.03003 tab. III.

Evidence limits

• One model (LLaMA 65B), one dataset (GSM8K), one batch size (64) and one GPU type (A100 80 GB) for the cap results 2310.03003 §4.5.
• One run per cell and no error bars 2310.03003 §4.5.
• Output quality was not measured, and quantisation, continuous batching and request arrival rates were not studied 2310.03003 §3.3.
• The paper calls its power-capping conclusions preliminary 2310.03003 §5.
• The measured model's weights carry a non-commercial licence, which the paper notes 2310.03003 §2.2. This page does not assess whether reproducing the runs is allowed for commercial use.

Contradictions

The 150 W run-time increase is given as +19.49% in the text, while the three per-length values in the table average to 19.56% [arxiv:2310.03003 §4.5, tab. III]. Both figures are reported as they appear; the difference is small.

Papers

• From Words to Watts: the only source here; measures 175 W and 150 W caps on A100s for LLaMA 65B, against a 250 W default.