Measuring the Carbon Intensity of AI in Cloud Instances

In plain words

Training and running AI models uses electricity, and the electricity has a carbon footprint that depends on where and when it is generated. The same job can emit much less CO2 in one data centre or at a different hour, because the local power grid is greener then. The authors propose a way to estimate the emissions of one cloud GPU job: measure the energy the GPU uses, then multiply it by the grid's current carbon intensity for that time and place.

They apply this to 11 workloads, from BERT language models to vision transformers, on Microsoft Azure. They then ask two practical questions. Does moving a job to another region help? Does starting it at another hour help, or pausing it while the grid is dirty? Region turns out to matter most. Shifting work in time helps too.

TL;DR

• The paper defines an operational carbon estimate for one cloud GPU job as GPU energy times the marginal grid carbon intensity at that time and place §3.1 eq. 2.
• It measures or derives energy for 11 workloads (BERT, a 6.1 B parameter Transformer, DenseNets, ViTs) §4 tab. 2.
• Across 16 Azure regions, it simulates two schedulers: Flexible Start and Pause and Resume §6 tab. 4 tab. 5.
• Region choice has the largest operational reduction; time-of-day and pause-and-resume effects are smaller and depend on job length §6.1 tab. 4 tab. 5.

Problem

Cloud users can see the price of compute but not its carbon cost. Earlier estimates often used coarse regional averages or post-hoc data that is not public §1. The paper states that a job's emissions need a per-GPU energy measurement and a grid signal, and that the signal should be the marginal rather than the average one, because the marginal signal reflects what a change in demand actually does §2 §3.1.

Assumptions: only GPU energy is measured, because GPUs are typically not shared across instances, while CPU, RAM, storage and cooling are shared or hard to attribute §3.2. The estimates are therefore lower bounds §3.2.

Method

Carbon model. The Software Carbon Intensity (SCI) rate is (E × I + M) per functional unit R eq. 1. Here E is energy in kWh, I the location-based marginal intensity in gCO2eq/kWh, M the embodied carbon allocated to the job, and R one training job §3.1. The paper defines the operational part as O = E × I eq. 2 and measures only O; embodied carbon M is left to future work §3.1 §8.

Energy measurement. The GPU is measured with nvidia-smi; CPU and DRAM power come from Intel RAPL on a server with one TITAN X GPU §3.2. The paper samples power over the run and averages per component §3.2. The energy of the 11 experiments is reported per run §4 tab. 2; the paper does not describe the measurement setup for them beyond the GPU-only rule.

Intensity data. Marginal emissions come from WattTime at 5-minute granularity for 2020 §6. The paper states these are historical values and that the optimisations therefore assume a perfect forecast §6.

Per-job emissions over a run is the sum over intervals of energy times intensity. The paper describes energy per 5-minute interval and marginal intensity per 5-minute interval, and states that time-series data gives the cumulative emission of an experiment at the end §5.

# Operational emissions of one job  [§3.1] [eq. 2] [§5]
# Input:  energy (T,) kWh per 5-min step; intensity (T,) gCO2eq/kWh
# Output: total gCO2eq

def job_emissions(energy, intensity):
    # Emission of each step: energy used times the grid intensity then  [§5]
    per_step = energy * intensity                # (T,) gCO2eq
    # Cumulative emission is the sum over all steps  [§5]
    return sum(per_step)

Flexible Start. Start the job at the time within the next N hours that minimises its total emissions. Once started, it runs to completion. The paper tries every start in 5-minute steps in the window §6. The job's energy profile is re-used from its start time.

# Flexible Start  [§6] [fig. 3]
# Input:  intensity (W,) over the window; job_energy (K,) per 5-min step
# Output: start index with the lowest emissions

def flexible_start(intensity, job_energy):
    # Try every start step in the window, in 5-min increments  [§6]
    best_start = None
    best_total = inf
    for start in range(len(intensity) - len(job_energy) + 1):
        # Energy of step k is applied to the intensity at start + k  [§6]
        total = 0.0
        for k in range(len(job_energy)):
            total += job_energy[k] * intensity[start + k]
        if total < best_total:
            best_total = total
            best_start = start
    return best_start

Pause and Resume. Over the window of N hours plus the job's own duration, select the 5-minute intervals with the lowest marginal emissions until their total length equals the job's duration, then run only in those intervals §6. The paper assumes that pausing and restarting is immediate and uses no extra energy §6.

# Pause and Resume  [§6] [fig. 4]
# Input:  intensity (W,) over N + duration; duration_steps (int)
# Output: sorted step indices in which the job runs

def pause_and_resume(intensity, duration_steps):
    # Rank every 5-min step in the window by marginal intensity  [§6]
    ranked = argsort(intensity)
    # Keep the cleanest steps until the job's run time is covered  [§6]
    run_steps = ranked[:duration_steps]
    # Run only in these steps; pausing costs no extra energy  [§6]
    return sorted(run_steps)

Extension sets. Pause and Resume is evaluated with an absolute extension of 6, 12, 18 or 24 hours, or a relative extension of 25%, 50%, 75% or 100% of the original duration §6. The threshold that corresponds to a given extension is not given as a formula; the paper says the threshold is set so that run time grows by the chosen amount §6.

Time of day. A single BERT fine-tuning run, started at different hours on three days in Central US, shows emissions that vary with start hour §5.2 tab. 3.

Hyperparameters

Results

Author-reported. Setup: 11 workloads on Azure GPUs §4 tab. 2; 16 regions, averages over the year and 5 start times per month §6.1. The gain columns are percentages that the paper reports without a formula tab. 4.

Energy per experiment tab. 2:

The 6B model was stopped at about 13% of a full run §4 tab. 2.

Power split for BERT-base on one GPU tab. 1: GPU 187.1 W (74%), CPU0 22.9 W, CPU1 9.3 W, DRAM0 23.0 W, DRAM1 9.3 W, total 251.6 W §3.2.

Region effect (BERT LM, 8 V100s, 36 h): emissions range about 7k to 26k grams across regions, from the most to the least efficient fig. 1 §5.1. Averaged over the year, per GPU-hour the most efficient region is 34 g and the least efficient 128 g; per kWh the range is 200 to 755 g §5.1.

Time of day (BERT finetune, Central US, 3 days, values as printed, units not stated) tab. 3:

The text says starting at midnight rather than 06:00 can raise emissions by up to 8% on some days §5.2.

Flexible Start and Pause and Resume, 24 h extension (average gain in %, over the year and 16 regions) tab. 4:

Flexible Start and Pause and Resume, 100% extension tab. 5:

Other extensions (6, 12, 18, 25, 50, 75 h or %) are in tab. 6 to tab. 13. Figures fig. 3 and fig. 4 show Flexible Start and Pause and Resume for DenseNet 201 and the 6B model; fig. 5 to fig. 15 show every experiment in the appendix.

Comparison with prior work §5.1. The paper compares kWh per GPU-hour (0.07 to 0.28 here; 0.22 to 0.47 and 0.36 in the two Patterson et al. papers), gCO2eq per GPU-hour (34 to 128 here; 63 to 202 and 32 there) and gCO2eq per kWh (200 to 755 here; 427 to 545, except 200 for GShard 600B, and 88 there). The paper expects prior figures to be higher because they include PUE, and says most of the prior estimates lie within its own range §5.1.

Paper vs code

No official code.

The paper links the GSF SCI specification, WattTime, and Google/Azure region pages, but no code repository or project page §3.1 §5 §6. The only repository links are the BERT code by Devlin et al. used for the BERT run §3.2, which is not the authors' code. GitHub searches for "carbon intensity cloud instances" and for "marginal emissions WattTime cloud" within the microsoft organisation returned no repositories.

Pitfalls

• BERT size is inconsistent. The energy measurement run is BERT-base §3.2 tab. 1, while the pretraining and finetuning runs are described as BERT-small §4. The paper does not say whether the 37.3 kWh pretraining energy comes from the small or base model.
• Finetune energy differs between places. The text gives 3.2 kWh §4, while tab. 2 and the appendix caption fig. 6 give 3.1 kWh.
• Full 6B run is an extrapolation. The text estimates the full run at (60/8) × 13.8 MWh ≈ 103.5 MWh §4, i.e. 103,500 kWh. The tab. 2 caption gives 103,593 kWh. The estimate assumes linear scaling from the 13% run; the paper does not validate it.
• Figure 4 caption versus Table 5. The caption says doubling the duration of the 6B run can save "up to about 25%" fig. 4. The averaged 100% result in tab. 5 is 11.4%. The caption does not say whether the 25% is a single region or month.
• Pause and Resume is an oracle. The optimisations use historical intensity and perfect foresight §6. The paper says WattTime's forecasts cover 24 h, so longer gains are upper bounds §6.
• Threshold is not given. The paper says the threshold is set so that the run grows by a chosen amount §6, but gives no procedure for choosing it in practice.
• Window for relative extensions is not stated. Flexible Start is compared with Pause and Resume at the same extension §6.1.2 tab. 5, but how the start window is set for a 100% extension is not given.
• Energy per paused step is underspecified. The paper says it "simulates running the job only during those intervals" §6 but does not say how the measured per-step energy is mapped onto the non-contiguous steps. The pseudocode above assumes the energy profile is replayed in order; this is an inference, not the paper's wording.
• GPU-only scope underestimates. CPU, DRAM, PUE, cooling and embodied carbon are excluded §3.2 §7 §8. Authors note this makes their numbers underestimates §3.2. The GHG Protocol Scope 2 rule is said to be incompatible with marginal accounting §8.
• Time-of-day evidence is thin. It comes from one BERT finetuning run on three days in one region tab. 3. The paper says the effect varies by region and season §5.2.
• Regional coverage. 16 regions: 9 in North America, 6 in Europe, 1 in Australia. The paper says other continents are missing because carbon intensity data for them was unavailable §6.
• Table units. tab. 3 has no units in its caption or cells.
• Figure and table numbers in this card are counted from the LaTeX source order. Check them against the rendered PDF before quoting.

Lineage

• Builds on: Patterson et al. (2021) and Strubell et al. (2019), cited for prior emissions estimates; the paper cites their arXiv ids nowhere, so no wiki link is given §2.
• Builds on: Green Software Foundation SCI specification, the carbon intensity definition used in eq. 1 §3.1.
• Compared against: Patterson et al. (2021, 2022) for kWh and gCO2eq per GPU hour and per kWh §5.1.
• Related tools named by the paper: Code Carbon, Carbon Tracker, ML CO2 Impact Tracker, and EnergyVis §2 §8.
• Superseded by: not stated in this wiki.