BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

In plain words

BLOOM is a very large text-prediction model, the kind that writes or answers questions when you give it a few example sentences or an instruction. Most models this large are built inside companies and kept closed. This paper describes a project in which hundreds of researchers built one and released it openly. It was trained on a deliberately multilingual collection of text covering dozens of human languages and some programming languages.

The model works like most modern chatbots' base models: it reads text from left to right and predicts the next word. The team made a few small, deliberate changes to the standard design, mainly in how the model keeps track of word order, and used a number format that is more forgiving of very large numbers. The paper reports that the model does well on many tasks, and that a second version trained further on instructions does better still.

TL;DR

BLOOM is a 176B-parameter causal decoder-only Transformer trained on ROOTS, a multilingual corpus with 46 natural and 13 programming languages §1 §3.1. It uses ALiBi positional biases and an extra LayerNorm after the embeddings §3.2. Training ran in bfloat16 mixed precision on 384 A100 80GB GPUs for about 3.5 months §3.4. The paper reports competitive zero- and few-shot results across benchmarks, and better results after multitask finetuning (BLOOMZ) §4. All numbers below are author-reported.

Problem

Large language models were mostly built by a few well-resourced organisations and not released §1. The authors aim for an openly available model at this scale that is also multilingual, since the existing large models were concentrated on English and a few other languages §1 §3.1. The design goal is zero-shot and few-shot prompt performance, because finetuning a model of this size is described as unwieldy §4.1.

Method

BLOOM is a standard causal decoder-only Transformer §3.2 with four departures from the original design, argued from the authors' own ablations §3.2:

1. Causal (left-to-right) language modelling objective, chosen over encoder-decoder and prefix objectives §3.2, "Architecture and Pretraining Objective".
2. ALiBi attention biases instead of learned or rotary position embeddings §3.2, "ALiBi Positional Embeddings". The bias is added to the attention scores and grows with key-query distance; the per-head slopes are given in the architecture figure caption as 2^(-8i/n) for head index i in 1..n, where n is the number of heads fig. caption in §3.2.
3. An extra LayerNorm directly after the token embedding table, which the authors kept to avoid training instability although it hurt zero-shot results in their ablation §3.2, "Embedding LayerNorm".
4. Tied input and output embeddings §3.5 table: "Tied emb. True".

The tokenizer is a multilingual subword tokenizer with a vocabulary of 250,680 entries §3.5 table; the paper's tokenizer section describes how it was trained with the Hugging Face Tokenizers library §3.3. The authors check it with fertility (subwords per word) against monolingual tokenizers on Universal Dependencies and OSCAR, with the target of not degrading any language by more than 10 percentage points §3.3.

The forward pass in outline. Only the ALiBi bias, the embedding LayerNorm and the tied output are stated in the paper; the internal order of each block is not given in prose, so it is left out here.

# BLOOM forward pass  [§3.2] [§3.5 table]
# Input:  token_ids (B, T) with T <= 2048
# Output: next-token logits (B, T, 250680)

def bloom_forward(token_ids):
    # Look up token embeddings; vocabulary 250,680  [§3.3] [§3.5 table]
    hidden = embedding[token_ids]                 # (B, T, 14336)
    # Extra LayerNorm after the embedding table  [§3.2 "Embedding LayerNorm"]
    hidden = embedding_layer_norm(hidden)         # (B, T, 14336)
    # ALiBi biases are added to attention logits, not to the embeddings
    # [§3.2 "ALiBi Positional Embeddings"]
    alibi = alibi_bias(112, T)                    # (112, T, T)
    # 70 transformer layers  [§3.5 table]
    for block in transformer_blocks:
        hidden = block(hidden, alibi)
    hidden = final_layer_norm(hidden)
    # Output projection reuses the input embedding  [§3.5 table]
    return hidden @ embedding.T                   # (B, T, 250680)

Slope definition for ALiBi used in the paper's figure caption, per head i in 1..n with n heads fig. caption in §3.2:

# ALiBi slopes as stated in the architecture figure caption  [§3.2]
# Input:  num_heads n
# Output: one slope per head, shape (n,)

def alibi_slopes_from_caption(num_heads):
    # Head i (1-based) gets slope 2^(-8 i / n)  [§3.2 figure caption]
    return [2 ** (-8 * i / num_heads) for i in range(1, num_heads + 1)]

The reference implementation does not use this formula for head counts that are not powers of two; see Paper vs code.

Training objective and schedule:

# Pretraining loop  [§3.5]
# Next-token cross-entropy on ROOTS; Adam with decoupled-style weight decay,
# warmup then cosine learning-rate decay

def train(model, batches):
    # Warmup then cosine decay of the learning rate  [§3.5]
    for step, batch in enumerate(batches):
        # Mean next-token cross-entropy over the batch  [§3.5]
        loss = cross_entropy(model(batch.inputs), batch.targets)
        loss.backward()
        # Clip the global gradient norm to 1.0  [§3.5 table]
        clip_grad_norm(model.parameters(), max_norm=1.0)
        # Adam with betas (0.9, 0.95), weight decay 0.1  [§3.5 table]
        optimizer.step(lr=lr_schedule(step))
        optimizer.zero_grad()

Multitask finetuning (BLOOMZ) keeps the architecture and changes only the training data and optimiser settings §3.5 "Multitask Finetuning" §4.7. Its data is the xP3 corpus described in the paper §4.7. Contrastive finetuning of the 1.3B and 7.1B models with the SGPT bi-encoder recipe gives text-embedding variants §3.5 "Contrastive Finetuning".

Hyperparameters

Values for the 176B model unless noted. Source: §3.5 hyperparameter table (columns BLOOM-560M to BLOOM-7.1B are the ablation-scale family).

Results

Author-reported. Setup: zero- and few-shot prompting; benchmarks and baselines as stated in each section. The paper's evaluation focuses on prompt-based zero- and one-shot settings rather than finetuning §4. The setup of the HumanEval runs (prompting, number of samples, decoding) is not restated in the table; check §4.5 before comparing numbers.

Code generation (HumanEval, pass@k) §4.5. The paper states that non-BLOOM numbers come from prior work.

Source: §4.5 table. The paper says BLOOM is similar to GPT models of comparable size trained on code.

SuperGLUE (zero- and one-shot) §4.2. Figure-only results; the paper reports that on entailment tasks (BoolQ, CB) BLOOM is well above chance, that average performance over prompts is near chance on most other tasks, and that BLOOM improves more than comparable models from zero- to one-shot. Exact numbers are in the figure and are not transcribed here. fig. superglue

Multitask finetuning (BLOOMZ) §4.7. The paper reports that multilingual multitask finetuning on xP3 significantly increases zero-shot performance, and compares against pretrained BLOOM and T0 on held-out tasks fig. taskgen. Plateau after 1 to 6 billion finetuning tokens §3.5.

Machine translation, summarization, embeddings, probing, bias [§4.3, §4.4, §4.8, §4.9, §4.10]. Reported in the paper's tables and figures. Not transcribed here; check the source section before relying on any number.

Paper vs code

Code: Megatron-DeepSpeed at commit 8387ae17c4704f6579f88a84500b535d19d7fbbf. The owner is the BigScience Workshop, which is the author collective named first in the author list. Code and launch scripts for the 176B run live in a second repository, bigscience-workshop/bigscience (train/tr11-176B-ml/tr11-176B-ml.slurm). It is not in the local clone; its values below were read from the GitHub file, so they are cited as external and not as local lines. Its license was not checked.

• Adam β2. Paper: 0.95 §3.5 table. Megatron default is 0.999 arguments.py. The launch script sets --adam-beta2 0.95 (external). The paper value is what the run used; the default would be wrong for BLOOM.
• Dropout. Paper: no dropout §3.5. Megatron defaults are 0.1 for both attention and hidden dropout arguments.py. The launch script does not pass either flag (external). The code default would apply, which contradicts the paper. No discrepancy can be confirmed without the training commit used.
• Embedding LayerNorm. Paper: used §3.2. Code: applied only when --embed-layernorm is set arguments.py, inside the vocabulary-parallel embedding forward layers.py and layers.py. The launch script sets the flag (external).
• ALiBi slopes. Paper caption: 2^(-8i/n) §3.2 figure caption. Code computes the power-of-two case as that same formula, but for non-powers of two it takes the slopes of the nearest lower power of two plus every other slope of the next power, truncated transformer.py. BLOOM has 112 heads, so the code's interpolation applies, and the paper's formula is not what the reference implementation computes. Which one the trained model used is a question for the authors.
• ZeRO stage. Paper: ZeRO stage 1 §3.4. Launch script: ZERO_STAGE=0 (external). This is a direct discrepancy.
• Synchronous kernel launches. Paper: asynchronous CUDA launches disabled §3.4. Launch script: CUDA_LAUNCH_BLOCKING=1 (external). Consistent.
• Schedule units. Paper: warmup 375M tokens, decay 410B tokens §3.5 table. Launch script: --lr-warmup-samples 183105 and --lr-decay-samples 200000000 (external). At sequence length 2048, these are about 375M and 410B tokens, so the paper's figures match the script's sample counts. Arithmetic is ours.
• Total budget. Paper: 366B total tokens §3.5 table. Launch script: --train-samples 220000000 (external). At 2048 tokens per sample that is about 450B tokens. The paper does not say where the run stopped.
• Initialisation std. Megatron default 0.02 arguments.py. Launch script: 0.0048 (external). The paper does not state the value ("not stated" in the paper).
• Parallel degrees. Paper describes 3D parallelism without degrees §3.4. Launch script: tensor parallel 4, pipeline parallel 12 (external). The paper does not state this.
• Micro-batch size. Paper: not stated. Launch script: 2 (external).

No other discrepancies were checked; the rest of the codebase was not compared line by line against the paper.

Pitfalls

• Non-power-of-two heads and ALiBi. The paper's caption formula and the reference code differ for 112 heads (see Paper vs code). Implement the code's rule if you want to reproduce the released weights; document it if you follow the caption.
• Dropout is ambiguous. The paper says no dropout; Megatron's default is 0.1 and the launch script does not override it. Set dropout to zero explicitly, since the paper is the stated design.
• ZeRO stage and bf16. The paper says stage 1 and the launch script says stage 0. Stage affects memory and communication, not the math of the optimiser, but the two disagree.
• Float16 vs bfloat16. Earlier 104B experiments with float16 showed divergence on V100 GPUs. The final run used bfloat16 mixed precision with float32 for gradient accumulation and softmax §3.4. The embedding LayerNorm was introduced for the float16 experiments, and the paper says bfloat16 may make it unnecessary but it was kept §3.2.
• Schedule never reaches its end. The paper states the cosine decay was never completed because warmup plus decay tokens exceed the total §3.5. Plan a schedule that stops at the actual step count; the minimum learning rate is not reached.
• Hardware failures. On average 1 to 2 GPU failures per week, handled with spare nodes and checkpoints every three hours. A data-loader deadlock and disk issues caused 5 to 10 hour downtimes §3.4. One loss spike was recovered from §3.4.
• Throughput. The fastest configuration reached 156 TFLOPs per A100 GPU, about half the theoretical peak §3.4. Use this as a reference when estimating your own run.
• Ablations at small scale. Architecture decisions were tested at 1.3B and pretraining objectives at 6.7B. The paper itself notes that outlier features appear above 6.7B, so small-scale conclusions may not transfer §3.2.
• Carbon numbers are estimates. The 25 t CO2eq is the authors' estimate of training emissions, and some values in their comparison table for other models are inferred rather than reported §3.4.
• Licenses differ. This paper is CC BY 4.0. The released model is under the BigScience Responsible AI License (RAIL), which restricts some uses; the code is Apache-2.0 per the paper §3.6. Check the model's license before reusing its weights. Our code license field records the repository's mixed notices.
• Prompt sensitivity. The paper reports that per-prompt SuperGLUE results vary widely and that the average over prompts sits near chance on most tasks §4.2. Report results per prompt, not only the best one.

Lineage

• Builds on: Megatron-LM (tensor parallelism, fused kernels) §3.4; DeepSpeed and ZeRO (optimizer sharding, pipeline) §3.4; ALiBi positional biases §3.2; the architecture ablations of the same project §3.2; GPT-3-style causal LM training §3.2.
• Compared against: GPT-3, OPT, GPT-J, Codex, T0 and GPT-NeoX §3.4 §4.
• Superseded by: not stated in this wiki.
• Related concept pages: language-model-pretraining, in-context-learning, dataset-curation, positional-encoding.