A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model

In plain words

Training a large language model uses a lot of electricity, but the training run is only part of what a project costs in energy. This paper adds up everything the Noor project used: collecting and storing text, the research experiments, the final training runs, the staff's laptops and flights, and the energy that the finished models will use when people query them. Each item is turned into electricity and then into emissions, using the carbon intensity of the place where that work ran.

The main finding is that the total is not just the training run. Other costs add up to about a third of the electricity, and the choice of data centre and region changes the emissions a great deal. The authors also argue that answering questions with a large model can, over time, cost more energy than training it. The numbers are estimates built on stated assumptions, not independently measured.

TL;DR

The paper accounts for the whole Noor project, not only pretraining: data, R&D, four model trainings, flights and a prospective inference estimate §5 §5.2. The project used 59.14 MWh of electricity fig. 2, and the authors estimate 36.5 t CO2e fig. 3 §7. Training is 65% of the emissions, flights 18%, storage 12%, R&D 4% and other items 1% fig. 3. Running the same compute in France would have given 14.9 t CO2e §7. The inference figures are a prospective estimate, not a measurement §5.1.5. Several intermediate numbers do not reproduce from the values the paper states; see Pitfalls.

Problem

Carbon reporting for machine learning usually covers only the GPU hours of training §1. The authors argue that extreme-scale models need an end-to-end assessment that includes data, research and development, staff travel, and serving, because these costs can be large relative to training §1 §6.3. They also note that open models can be deployed by anyone on hardware the developer does not control, so inference emissions are hard to forecast §5.1.5 §7.

The paper gives no operating regime beyond the Noor project itself. Hardware lifecycle (embodied carbon of the chips and servers) is explicitly excluded §4.

Method

The paper adds up electricity per phase, then converts it to CO2e with the carbon intensity of the local grid. The phases are data, R&D, pretraining, other costs and inference. A general model-size argument comes first: the number of floating-point operations to train a Transformer with N parameters on D tokens is approximated as C = 6ND, and the authors use it to project training budgets §4. The paper does not number its equations; the approximation appears inline in §4 and §5.1.4.

1. Scope. Total footprint = data storage and transfer, data processing, R&D, main training, other costs (laptops, flights) and projected inference. Each phase is located at a specific provider and region §5 §5.2 tab. 2.
2. Data storage and transfer. The average peak power of cloud storage is taken as 11.3 W per TB, which the authors convert to 99 kWh per TB per year §5.1.1. The curated data (about 2 TB) and the processed data (about 25 TB) are kept for six months; the 210 TB of raw Common Crawl dumps are downloaded once and discarded after processing §5.1.1. Model checkpoints (5.7 TB, 0.3 MWh) and transfers (1.8 MWh) are added §5.1.1. Storage and transfer together come to 3.5 MWh §5.1.1.
3. Data processing. The pipeline runs on 768 CPU cores in 16 nodes at 350 W per node, so 5.6 kW for the cluster. It ran for 381 wall-clock hours at a PUE of 1.1, giving 2.35 MWh §5.1.2.
4. R&D. The experiments used 16,800 A100-hours on a supercomputer (MeluXina), 4 A100 SXM 40 GB at 400 W TDP and 2 AMD EPYC 7763 CPUs at 280 W TDP per node, with PUE 1.35. The authors report 10.7 MWh §5.1.3.
5. Main training. The four models (1.3B, 2.7B, 6.7B, 13B) were trained on 150B tokens each §3 §5.1.4 tab. 1. Table 1 lists the budget in PF-days (footnote 1: 1 PFLOP/s sustained for a day, equal to 10 A100s), A100-hours at an effective 100 TFLOP/s per GPU, and the measured energy. The 1.3B model ran on MeluXina; the other three on the Noor-HPC cluster §5.1.4 tab. 1 [fn. 2].
6. Inference (prospective). The authors estimate energy per generated token on one A100 at batch size 1. Per hour the GPU draws 400 W and the CPU 70 W, multiplied by PUE 1.1, which gives 517 Wh per hour. At 72,000 tokens per hour this is 26 J per generated token §5.1.5. The cost is assumed constant for up to about 512 processed prompt tokens §5.1.5.
7. Inference carbon. Two grid scenarios convert tokens to CO2e: the world-average mix (475 g CO2e/kWh) gives about 300,000 tokens per kg CO2e, and the French mix (56 g CO2e/kWh) gives about 2,500,000 tokens per kg CO2e §5.1.5 §4. Scaled to GPT-3's reported 4.5 billion words per day, the authors get 30 t CO2e per day and 3.5 t per day §5.1.5.
8. Other costs. Six full-time scientists (half in France, half in the UAE) use a 70 W laptop and a 30 W external screen for 8 hours per day over six months, which the authors round to 1 MWh §5.1.6 §5.2. Three round-trip flights for four scientists between Paris and Abu Dhabi are counted at 527 kg CO2e per person per round trip §5.2.
9. Carbon per phase. Electricity is multiplied by the intensity of the region where the work ran: Bahrain (storage), Netherlands (GCP CPU cluster for preprocessing), Luxembourg (MeluXina), UAE (Noor-HPC), France (other) §5.2 tab. 2.

Per-phase accounting, our rewrite of the steps above (not a paper algorithm; the paper states no pseudocode):

# Carbon of one project phase  [§5.1] [§5.2]
# devices: count; power_watts: TDP or measured per device
# grid_g_per_kwh: local carbon intensity  [tab. 2]
def phase_carbon(devices, power_watts, hours, pue, grid_g_per_kwh):
    # Grid electricity, including data-centre overhead  [§5.1.3] [§5.1.4]
    energy_kwh = devices * power_watts * hours * pue / 1000
    # Emissions from the location's carbon intensity  [§5.2] [tab. 2]
    co2e_kg = energy_kwh * grid_g_per_kwh / 1000
    return energy_kwh, co2e_kg

Inference energy per generated token, our rewrite of §5.1.5:

# Joules per generated token on one GPU at batch size 1  [§5.1.5]
# gpu_watts and cpu_watts: draw per hour of generation
# tokens_per_hour: throughput of the model on that GPU
def joules_per_token(gpu_watts, cpu_watts, pue, tokens_per_hour):
    # Watt-hours drawn in one hour, including PUE  [§5.1.5]
    watt_hours_per_hour = (gpu_watts + cpu_watts) * pue
    # One watt-hour is 3600 joules  [§5.1.5]
    return watt_hours_per_hour * 3600 / tokens_per_hour

Hyperparameters

Values the accounting depends on. Each one is as stated in the paper unless marked derived.

Results

Author-reported. Setup: one project (Noor), four models of 1.3B to 13B parameters trained on 150B tokens §3 §5. The paper does not state the calendar period of the accounting. Table 2 and Figure 3 are the authors' summary.

The Table 2 rows sum to 30.1 t; the 36.5 t total adds the 6.4 t of flights fig. 3 §5.2.

Breakdown of electricity fig. 2: training 70%, R&D (including preprocessing) 22%, storage 6%, other 2%. Table 1 gives the training energy by model: 2.1, 4.8, 11.8 and 22.9 MWh for 1.3B, 2.7B, 6.7B and 13B, totalling 41.6 MWh tab. 1 §5.1.4. The 13B model is 55% of the training energy §5.1.4.

Breakdown of emissions fig. 3: training 65%, flights 18%, storage 12%, R&D 4%, other 1%.

Other headline results:

• Projected inference: at 26 J per token, the authors state that about 3 billion generated tokens would be needed to equal the training energy §5.1.5. See Pitfalls, which shows this does not match the stated numbers.
• Projected inference emissions for GPT-3-scale usage: 30 t CO2e per day under the world-average mix and 3.5 t per day under the French mix §5.1.5.
• Counterfactual: running all compute in France would give 14.9 t CO2e, 42% of it from flights §7.
• The paper states that the 36.5 t total is about "a little over two years" of the average US individual footprint §7.

Paper vs code

No official code. The paper links no repository. The only GitHub link is a third-party reference (GPT-J) [references]. Searches run: the GitHub repository search for noor arabic (65 results, none from TII or the authors in the 18 visible), noor org:tiiuae (0 results) and carbon footprint noor (0 results). No accounting scripts or configs are published, so the figures cannot be checked against code.

Pitfalls

• Inference catch-up arithmetic does not match. The paper says 3 billion tokens at 26 J each would equal the training costs §5.1.5. That is 3e9 × 26 J = 78 GJ, or about 21.7 MWh (our arithmetic). Matching the 41.6 MWh training energy needs about 5.8 billion tokens; matching the 59.14 MWh total needs about 8.2 billion tab. 1 fig. 2. Use the per-token figure, not the 3-billion figure.
• Storage PUE and redundancy are not visibly applied. §5.1.1 says the 99 kWh per TB per year includes PUE 1.6 and redundancy factor 2. But 11.3 W × 8,760 h = 99 kWh, so the 99 kWh figure appears to be raw power only. With both factors the figure would be about 317 kWh per TB per year (our arithmetic). The 3.5 MWh storage total depends on which reading is correct §5.1.1.
• R&D energy does not reproduce from the stated values. 16,800 GPU-hours at 4 GPUs per node is 4,200 node-hours. With 2.16 kW per node (TDP sum) and PUE 1.35 this is about 12.2 MWh; GPUs alone give about 9.1 MWh. The reported 10.7 MWh lies between and the averaging method is not stated §5.1.3 (our arithmetic).
• Training energy in Table 1 is about 16–18% above a simple TDP × hours × PUE estimate. For the 13B model: 33,000 A100-hours × 0.4 kW × 1.5 = 19.8 MWh, against 22.9 MWh reported. The other rows show a similar gap tab. 1 §5.1.4 (our arithmetic). The source of the extra energy (CPUs, memory, networking, idle time) is not stated.
• Words to tokens is not stated. The 30 t per day and 3.5 t per day figures imply about 2 tokens per GPT-3 word applied to the stated tokens-per-kg rates §5.1.5 (our inference). The paper does not state this conversion.
• Flight total. Three trips for four scientists read as 12 round trips at 527 kg each gives 6.3 t, not the 6.4 t stated §5.2. The count of trips is ambiguous.
• Laptop energy. 0.604 MWh at 100 W for six people implies about 126 eight-hour days, not six months of daily use. The working-day count is not stated; the value is then rounded to 1 MWh §5.1.6.
• Transfer energy constant. 23.9 kJ per GB equals 6.64 kWh per TB in decimal units, not 6.38 kWh per TB as stated §5.1.1. The difference is small but the conversion basis is not given.
• Processed-data storage rounding. 25 TB held for six months at 99 kWh per TB per year is about 1.24 MWh, reported as 1.3 MWh §5.1.1 (our arithmetic).
• Data-centre efficiency wording. "20% more efficient than the world average" is used for a PUE of 1.5 against a world average of 1.8 §5.1.4 §4. That is 17% less overhead, or 20% better in ratio terms. Use the PUE values, not the percentage.
• Electricity mix sources. The grid intensities in Table 2 are given without per-value sources, except France §5.1.5. Carbon intensities change by year and by provider; the paper's values should be treated as a snapshot.
• Scope gaps. Hardware embodied carbon is excluded §4. Public-release costs (downloads, traffic) are neglected §5.1.1. Non-electric sources other than flights are deferred to a later phase and not quantified §5.1.
• Inference is a prospective estimate. The assumptions are batch size 1, a constant per-token cost up to about 512 prompt tokens, and a single A100 §5.1.5. Batched serving, longer contexts, and quantized models would change the figure, and the paper does not estimate them.
• Counterfactual method. The 14.9 t CO2e "all in France" figure §7 does not state the storage intensity or the flight treatment used to compute it.
• Equations are not numbered. The only formula is C ≈ 6ND in §4 and §5.1.4; cite by section.
• Per-phase rounding. Table 2 values are rounded; for example 10.7 MWh at 60 g CO2e/kWh is 0.64 t, shown as 0.65 tab. 2. Recompute from energy and intensity when you need exact values.

Lineage

• Builds on: Kaplan et al. 2020 (compute approximation C = 6ND, cited in §4); Posani et al. 2019 (storage power per TB); Baliga et al. 2011 and Aslan et al. 2017 (transfer energy); Patterson et al. 2021 (energy of large models, cited in §2). None of these has a card on this site.
• Compared against: prior carbon studies of training runs (Patterson et al. 2021, Strubell et al. 2019, Lacoste et al. 2019) §2; the paper says training-only reporting is the usual practice and is limited §1.
• Superseded by: not applicable.