OPT: Open Pre-trained Transformer Language Models

In plain words

OPT is Meta's attempt to make a GPT-3-class language model that outside researchers can study. Full access to models of this size was limited to a few well-resourced labs, which made it hard for others to look at their weights and find their weaknesses.

The models are ordinary "predict the next word" Transformers. Their size ranges from very small to very large, so you can study how behavior changes with scale. The team trained them on a large mix of text from the web, books, Wikipedia, news and Reddit conversations, after removing duplicate documents. The largest one was trained with a learning-rate schedule and optimizer settings borrowed from GPT-3. Training a model this big was hard: the hardware failed often, and the loss sometimes blew up, so the team restarted from older saved states and lowered the learning rate in places. They published a logbook of those problems along with the code.

The authors report that the largest model performs comparably to GPT-3 on standard zero- and few-shot tests on average, and that it has about one seventh of GPT-3's carbon footprint by their estimate. On several bias and toxicity tests the models do worse than GPT-3, and the authors did not add safety mitigations for this first release.

TL;DR

OPT is a suite of decoder-only Transformers with 125M to 175B parameters, trained with a GPT-3-style recipe: AdamW, a linear learning-rate decay to 10% of peak over 300B tokens, weight decay 0.1, and a batch of 0.5M to 4M tokens §2.1 §2.2 tab. 1. The corpus is about 180B tokens from RoBERTa, the Pile, and Pushshift Reddit, deduplicated with MinHash §2.3. The 175B model ran on 992 A100 80GB GPUs with FSDP and Megatron tensor parallelism §2.4. Author-reported: OPT-175B is comparable to GPT-3 on the average of standard zero- and few-shot NLP tasks §3.1 fig. 3 fig. 4, is competitive with the supervised BlenderBot 1 on the dialogue sets, most closely on ConvAI2 tab. 2, and does worse than GPT-3 on toxicity and on CrowS-Pairs §4.2 §4.4. The reported carbon footprint is 75 tCO2eq, against an estimated 500 tons for GPT-3 §6. The paper's stated values differ from the released code for the 175B learning rate and warmup length; see Paper vs code.

Problem

Large language models show zero- and few-shot abilities, but full model access was limited to a few well-resourced labs §1. This hampered outside research on robustness, bias and toxicity §1. The paper aims to give researchers weights and a comparable-size baseline, with a reproducible recipe and a transparent account of the training run §1 §6. The authors also want a model whose carbon cost is much lower than GPT-3's §1 §6.

Method

Section map of the compiled paper: §1 Introduction; §2 Method (§2.1 Models, §2.2 Training Setup, §2.3 Pre-training Corpus, §2.4 Training Efficiency, §2.5 Training Processes); §3 Evaluations (§3.1 Prompting & Few-Shot, §3.2 Dialogue); §4 Bias & Toxicity Evaluations (§4.1 to §4.5); §5 Limitations; §6 Considerations for Release; §7 Related Work; §8 Conclusion. Appendices and the datasheet and model card follow. The paper has no numbered equations or algorithm blocks. The learning-rate schedule and initialization are described in prose §2.2, and the pseudocode below is this card's reading of that prose. Figures and tables are cited by the number they get in the compiled paper.

The model is a decoder-only Transformer. The paper states the following choices §2.1 §2.2:

• ReLU activation and sequence length 2048 §2.2.
• Weights drawn from a normal distribution with mean 0 and standard deviation 0.006, with biases initialized to zero. The output-side weights of each block are scaled by 1 / sqrt(2L), where L is the number of layers §2.2.

Architecture grid tab. 1. The LaTeX source also has a commented-out 760M row, which is not part of the table:

tab. 1. The text says "eight" models, but the table has nine rows §2.1 tab. 1; see Pitfalls.

The training loop, as described in words §2.2:

1. Keep one global batch size for the whole run, from 0.5M to 4M tokens depending on model size tab. 1.
2. Use AdamW with betas (0.9, 0.95) and weight decay 0.1 §2.2.
3. Warm the learning rate linearly from 0 to its peak over the first 2000 steps for OPT-175B, or over 375M tokens for the smaller baselines §2.2.
4. Decay the learning rate linearly to 10% of peak over 300B tokens §2.2.
5. Apply dropout 0.1 everywhere except the embeddings, and clip the global gradient norm at 1.0 §2.2.
6. Apply the mid-run changes described in §2.5: gradient clipping at 0.3 in parts of the run, learning-rate reductions, and loss-scale resets.

The learning-rate schedule as pseudocode, following the description in §2.2:

# Learning rate at a given update  [§2.2] [tab. 1]
# Linear warmup, then linear decay to 10% of peak  [§2.2]
def learning_rate(update, peak_lr, warmup_updates, total_updates):
    # Warmup: 0 to peak over the first warmup_updates  [§2.2]
    if update < warmup_updates:
        return peak_lr * update / warmup_updates
    # Decay: linear from peak to 0.1 * peak at total_updates  [§2.2]
    progress = (update - warmup_updates) / (total_updates - warmup_updates)
    return peak_lr - 0.9 * peak_lr * progress

Initialization as pseudocode §2.2:

# Weight initialization  [§2.2]
def init_weights(model, num_layers):
    # Embeddings and attention/MLP input weights: N(0, 0.006^2)  [§2.2]
    for each input weight matrix W in model:
        W = normal(mean=0.0, std=0.006)
    # Output-side weights of attention and MLP: std 0.006 / sqrt(2L)  [§2.2]
    for each output weight matrix W in model.output_projections():
        W = normal(mean=0.0, std=0.006 / sqrt(2 * num_layers))
    # Biases start at zero  [§2.2]
    for each bias b in model:
        b = 0.0

Data §2.3: the corpus concatenates RoBERTa's corpus (BookCorpus, Stories and an updated CCNews v2 crawled through 28 September 2021), a subset of the Pile (CommonCrawl, DM Mathematics, Project Gutenberg, HackerNews, OpenSubtitles, OpenWebText2, USPTO and Wikipedia), and a subset of Pushshift.io Reddit. For Reddit, the authors kept only the longest comment chain in each thread, which cut that corpus by about 66%. Documents are deduplicated with MinHash LSH at Jaccard similarity 0.95 or higher. Other Pile subsets were dropped because they caused gradient-norm spikes at 1.3B scale. The text is tokenized with the GPT-2 byte-level BPE. The total is about 180B tokens §2.3. The paper adds that the Pile contains many duplicates and recommends extra deduplication for anyone using it.

Systems §2.4: training used 992 80GB A100 GPUs with Fully Sharded Data Parallel and Megatron tensor parallelism, reaching up to 147 TFLOP/s per GPU. Adam states are kept in FP32 and sharded across hosts. Weights are FP16, and dynamic loss scaling prevents underflow §2.4 fig. 1 fig. 2.

Training events §2.5: the paper reports at least 35 manual restarts and more than 100 hosts cycled out over two months due to hardware failures, and estimates 70 or more automatic restarts. For loss divergence, the team lowered the learning rate and restarted from an earlier checkpoint whose dynamic loss scale was still at least 1.0 and whose final-layer activation norm was trending down §2.5. Divergence was correlated with the loss scale collapsing to zero and with the final-layer activation norm spiking §2.5. Other attempts were a switch to vanilla SGD (abandoned), a reset of the loss scaler (helped partly), and a newer Megatron version (reduced activation-norm pressure and improved throughput) §2.5. Learning-rate and validation-perplexity curves are in fig. 1 and fig. 2.

Hyperparameters

Paper-stated values are cited to the paper. Rows marked "code only" come from the released launcher and are not stated in the paper.

Results

All numbers are author-reported. The per-task zero-shot and few-shot results appear only as figures in the source, so they are described here and not tabulated.

Zero-shot and few-shot NLP §3.1 fig. 3 fig. 4 fig. 6 fig. 7. Setup: 16 standard NLP tasks, with prompts and evaluation following GPT-3 §3.1. The averages use 14 tasks, because MultiRC and WIC were removed §3.1. The paper says OPT's zero-shot average follows GPT-3's trend and that OPT roughly matched GPT-3 on 10 tasks and underperformed on ARC Challenge and MultiRC §3.1. One- and few-shot averages are similar to GPT-3 across models, but per-task behavior differs §3.1 fig. 4. The paper also reports unexplained scale effects on CB, BoolQ and WSC, where both OPT and GPT-3 sit near majority-class accuracy §3.1.

Dialogue tab. 2. Setup: greedy decoding up to 32 tokens with no prompt beyond "Person 1:" and "Person 2:" lines §3.2. Perplexity is normalized to the GPT-2 tokenizer §3.2. The OPT row is unsupervised; the other rows use BlenderBot 1's generation parameters §3.2.

tab. 2. The paper reports OPT-175B on the ConvAI2 hidden test set at 10.7 perplexity and .185 UF1, and on a MultiSessionChat subset at 9.7 perplexity and .177 UF1 §3.2. It searched the pre-training corpus for the first ConvAI2 conversation and found no overlap §3.2.

Hate speech detection (ETHOS) §4.1 tab. 3. Setup: yes/no or yes/no/neither prompts, zero- to few-shot, scored by F1. "Davinci" is GPT-3 via the API.

tab. 3

CrowS-Pairs (lower is better) §4.2 tab. 4.

tab. 4

StereoSet §4.3 tab. 5. LMS and ICAT are higher-is-better, SS is lower-is-better, and both models are normalized by token count §4.3.

tab. 5

RealToxicityPrompts §4.4 fig. 5. Setup: 25 continuations of 20 tokens per prompt, nucleus sampling with p = 0.9, over 10,000 sampled prompts §4.4. The paper reports that OPT has a higher toxicity rate overall than either PaLM or Davinci, and that toxicity rises with prompt toxicity for all three models §4.4 fig. 5. The figure has no numeric table in the source.

Dialogue safety §4.5 tab. 6. SaferDialogues reports PPL and F1. Unit tests report the fraction of unsafe responses across four sensitivity levels (lower is better).

tab. 6

Paper vs code

Code: facebookresearch/metaseq at commit f7ffa5fd61cf90f498a36d365c13dd7f1a912ff7, MIT license. The paper links this repository for the code §6. OPT-specific configuration is in metaseq/launcher/ and projects/OPT/.

Discrepancies:

• OPT-175B peak learning rate. The paper gives 1.2e-4 tab. 1. The released launcher sets 3e-5 for the 175b size opt_job_constants.py. The launcher's values for the other sizes in the paper's table match it, including 66B at 8e-5 opt_job_constants.py. The chronicles say the learning rate was lowered several times during the 175B run, and the run is described as changing mid-flight 56_percent_update.md 27_percent_update.md. The released value may therefore not match any single phase of the run. This card does not resolve which value was used at the start of the run.
• Warmup length. The paper says 2000 steps for OPT-175B §2.2. The launcher computes warmup as 375M tokens divided by batch size for every size opt_baselines.py. For 175B that gives 178 updates, and for 125M about 715 updates. The two values differ by about an order of magnitude for the 175B model.
• Total updates. The launcher sets total updates to 300B tokens divided by batch size opt_baselines.py, which is consistent with §2.2.
• Position encoding. The paper does not describe the position scheme. The launcher uses learned absolute positional embeddings and disables embedding scaling opt_baselines.py.
• Normalization placement. The paper does not state it. The decoder layer applies LayerNorm before attention and before the MLP, adding the residual after each sub-block transformer_decoder_layer.py.
• Adam epsilon. The paper gives no epsilon. The launcher sets 1e-8 opt_baselines.py. The optimizer is decoupled AdamW, which matches §2.2 adam.py.
• Output-side initialization scaling applies only when the launcher's --full-megatron-init flag is set opt_baselines.py. The launcher sets it, so the paper's 1/sqrt(2L) scaling is active in the released recipe utils.py multihead_attention.py.
• Gradient predivide. The paper describes splitting the world-size division into two steps §2.2. The launcher does not set a predivide factor opt_baselines.py.
• Data pipeline. The paper describes MinHash deduplication §2.3. No MinHash code was found in the repository's preprocessing folder. The launcher reads a pre-built corpus path named corpus_dedup_10_10_1_0.05_exp29 opt_baselines.py.
• Model count. The text says "eight" models §2.1. Table 1 lists nine sizes tab. 1. The launcher also defines a 760M size that the paper's table leaves out opt_job_constants.py.

Matches, for reference: batch sizes and layer, head and width counts for all sizes; peak learning rates for all sizes except 175B; ReLU activation; AdamW with betas (0.9, 0.95) and weight decay 0.1; gradient clipping at 1.0; dropout 0.1 with no embedding dropout; linear LR decay to 10% over 300B tokens; initialization std 0.006.

Pitfalls

• The 175B learning rate has two values. Use the paper's 1.2e-4 or the released launcher's 3e-5, but say which one you chose. The paper's table and the code disagree, and the chronicles describe in-run learning-rate reductions §2.5. The logbook PDF in the repository (projects/OPT/chronicles/OPT175B_Logbook.pdf) was not read for this card and may settle it.
• The 175B warmup is either 2000 steps or 178 updates. The paper's text and the launcher disagree by an order of magnitude §2.2 opt_baselines.py.
• Mid-run changes are not in the launcher. The paper mentions reduced gradient clipping (1.0 to 0.3), learning-rate reductions, loss-scaler resets and a Megatron version change §2.2 §2.5. These are not encoded in the released launcher. The exact timing and values are in the logbook §2.5.
• Loss-spike recovery is manual. Restart from a checkpoint with loss scale at least 1.0 and falling activation norm §2.5. The paper gives no automatic rule.
• Gradient norm and loss scale are coupled. Divergence was associated with the loss scale collapsing to zero and with spikes in the final-layer activation norm §2.5. Monitoring these two quantities is the paper's only suggested early-warning signal.
• Pile subsets were dropped for stability. Subsets that caused gradient-norm spikes at 1.3B scale were removed §2.3. The exact subset choice may matter for stability.
• Attention dropout is not named in the paper. The paper says dropout 0.1 throughout, except on embeddings §2.2. The launcher applies 0.1 to attention as well opt_baselines.py. The base architecture default is 0.0 transformer_lm.py; the Megatron architecture the launcher uses sets 0.1 transformer_lm.py.
• Zero-shot averages depend on task selection. MultiRC and WIC were excluded from the averages, because the paper says these datasets seem to favor GPT-3 or OPT disproportionately §3.1. The text says "underperformed in 3 tasks" but names only ARC Challenge and MultiRC §3.1.
• GPT-3 reference numbers may not be comparable. The paper could not reproduce GPT-3's MultiRC result through the Davinci API, and it calls GPT-3's reported WIC numbers questionable §3.1. Treat cross-paper comparisons on these tasks with care.
• Small validation sets. CB has 56 examples, BoolQ 277 and WSC 104, and results on them are erratic across scale §3.1.
• The safety results are worse than GPT-3 on several measures. OPT-175B scores worse on most CrowS-Pairs categories tab. 4, higher on RealToxicityPrompts fig. 5, and worst on the Unsafe unit-test level among the dialogue models tab. 6. The paper applied no mitigations in this release §5.
• Repetition and instruction following. The paper reports that OPT often loops on greedy or sampled generation, and that declarative instructions and direct questions often produce a simulated dialogue rather than an answer §5.
• Leakage on dialogue sets is checked only partially. The paper checked one ConvAI2 conversation against the corpus and found no overlap, while noting that OPT's ConvAI2 results were surprisingly strong §3.2.
• Carbon numbers are estimates. The paper's 75 tCO2eq is its estimate for developing OPT; the authors' own estimate of total cost, including ablations, baselines and downtime, is roughly twice that §6. GPT-3's 500 tons and Gopher's 380 tons are figures the paper cites from other work §6.
• Weights are not under this card's license. The paper says access to OPT is limited to the research community under a non-commercial license §6. The repository's projects/OPT/MODEL_LICENSE.md is a separate Meta agreement, and the README states that use of the weights is subject to it README.md. This card covers the paper's text and code only.
• Some implementation details are in the code only. Learned position embeddings, pre-LayerNorm placement, and Adam epsilon are not in the paper opt_baselines.py transformer_decoder_layer.py. An implementer working from the paper alone would get a different model.

Lineage

• Builds on: GPT-3 (arXiv:2005.14165), whose hyperparameters, prompts and few-shot evaluation setup OPT follows §2.2 §3.1. Megatron-LM model-parallel training and initialization §2.2 §2.4. The RoBERTa and Pile corpora and Pushshift Reddit for pre-training data §2.3. AdamW §2.2. Dynamic loss scaling from mixed-precision training §2.4. Fully Sharded Data Parallel §2.4.
• Compared against: GPT-3 175B (primary comparison throughout), GPT-3 via the Davinci API §3.1 §4.1. Dialogue models BlenderBot 1, R2C2 BlenderBot and Reddit 2.7B §3.2. PaLM for toxicity and dialogue §4.4 §4.5. Gopher and Chinchilla in the evaluation tables §3.1.
• Superseded by: not stated in this paper.