LLaMA: Open and Efficient Foundation Language Models

In plain words

Large language models are usually built by making the network as big as possible. This paper asks a different question: for a model that will be run many times after training, is it better to train a smaller model for longer? The authors train a family of models from 7 to 65 billion parameters on about a trillion tokens of text. They use only publicly available data.

The model itself is a standard text-predicting transformer with a few well-known tweaks: each block normalises its inputs rather than its outputs, the activation function is a gated one called SwiGLU, and word positions are encoded by rotating the query and key vectors instead of adding a position vector. The result is that the 13-billion-parameter model is better than GPT-3, which is more than ten times larger, on most common-sense tests, and the largest model comes close to the best closed models of its time.

The paper is mainly a recipe. The authors also list what the training cost in energy and carbon, and what biased or toxic output they measured.

TL;DR

• Decoder-only transformers with 6.7B to 65B parameters in the paper's labels (called 7B to 65B in the text), trained on 1.0T tokens (the two smaller models) or 1.4T tokens (the two larger ones) tab. 2 fig. 1.
• Pre-norm RMSNorm, SwiGLU feed-forward layers and rotary position embeddings; no absolute position embeddings §2.2.
• Training data is public web text, C4, GitHub under permissive licences, Wikipedia (20 languages), books, arXiv and Stack Exchange §2.1 tab. 1.
• Author-reported: LLaMA-13B beats GPT-3 (175B) on most common-sense benchmarks; LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B §3.1 tab. 3.
• Released code covers inference only. The official repository now implements Llama 2, not this paper's exact release (see Paper vs code).

Problem

Scaling-law work fixes a training compute budget and asks which model size and data size give the lowest loss for it. That objective ignores the cost of serving the model afterwards. If a model will serve many requests, a smaller model trained longer can be cheaper to run at the same quality §1. The authors take that view and train past the point where the scaling-law recommendation stops, reporting that a 7B model keeps improving beyond 1T tokens §1.

The paper's second constraint is data: it uses only publicly available sources and avoids proprietary or undocumented corpora §1 §2.1.

Method

The method is a recipe, not a new algorithm. It has five parts: a data mix §2.1, a transformer architecture with three named modifications §2.2, an AdamW training schedule §2.3, systems optimisations for training §2.4, and an evaluation protocol that uses log-likelihood scoring for multiple-choice tasks and exact match for generation §3.

Data §2.1 tab. 1. Text is tokenised with byte-pair encoding through SentencePiece. All digits are split into single tokens, and unknown UTF-8 characters fall back to bytes §2.1, Tokenizer. Sources are sampled at fixed proportions: English CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, Books 4.5%, arXiv 2.5% and Stack Exchange 2%. English CommonCrawl is five dumps (2017 to 2020) processed by the CCNet pipeline, with a linear classifier that keeps pages resembling Wikipedia references §2.1. GitHub is restricted to Apache, BSD and MIT projects §2.1. arXiv LaTeX has everything before the first section and the bibliography removed, comments stripped, and user macros expanded §2.1. Stack Exchange uses the 28 largest sites, with answers sorted by score §2.1. About 1.4T tokens result, and most sources are seen once, while Wikipedia and Books are seen about twice §2.1.

Architecture §2.2 tab. 2. The network is a stack of identical blocks. Each block has two residual sub-layers. Both normalise their input (not their output) with RMSNorm §2.2. The attention sub-layer is causal multi-head self-attention in which rotary embeddings are applied to queries and keys at every layer §2.2. The feed-forward sub-layer is SwiGLU, whose hidden width is 2/3 of 4 times the model width, not 4 times as in PaLM §2.2. A final RMSNorm precedes the output projection to the vocabulary (the vocabulary size is not stated in the paper) model.py model.py. The paper does not state dropout, bias terms or weight tying §2.2.

# LLaMA decoder block, pre-norm with residuals  [§2.2]
# Input and output: hidden (B, T, d)

def decoder_block(hidden, rope_angles, causal_mask):
    # Normalise the input of the attention sub-layer  [§2.2]
    normed = rms_norm(hidden, attention_gain)           # (B, T, d)
    # Causal self-attention, residual add
    # [§2.2] [code: llama/model.py#L406-L408]
    hidden = hidden + attention(normed, rope_angles, causal_mask)
    # Normalise the input of the feed-forward sub-layer  [§2.2]
    normed = rms_norm(hidden, ffn_gain)                 # (B, T, d)
    # SwiGLU feed-forward, residual add  [§2.2] [code: llama/model.py#L409]
    hidden = hidden + swiglu_ffn(normed)
    return hidden

# RMSNorm: scale by the root mean square, no mean subtraction  [§2.2]
# Epsilon is not stated in the paper; code uses 1e-5
# [code: llama/model.py#L28]

def rms_norm(x, gain):
    # Compute the inverse root mean square over the last dimension  [§2.2]
    inv_rms = rsqrt(mean(x * x, over last dim) + 1e-5)  # (B, T, 1)
    # Multiply by the learned per-channel gain  [code: llama/model.py#L63-L77]
    return x * inv_rms * gain

# Causal multi-head attention with rotary embeddings  [§2.2]
# Shapes: x (B, T, d); heads H, head size d_head = d / H  [tab. 2]

def attention(x, rope_angles, causal_mask):
    # Project to queries, keys and values  [code: llama/model.py#L273-L278]
    queries = linear(x, w_q).reshape(B, T, H, d_head)
    keys = linear(x, w_k).reshape(B, T, H, d_head)
    values = linear(x, w_v).reshape(B, T, H, d_head)
    # Rotate query and key channel pairs by position-dependent angles  [§2.2]
    queries, keys = apply_rope(queries, keys, rope_angles)
    # Scaled dot products, with future positions masked out  [§2.2]
    scores = queries @ transpose(keys) / sqrt(d_head) + causal_mask
    # Softmax is computed in float32  [code: llama/model.py#L298-L301]
    weights = softmax(scores, in float32)
    return linear(concat_heads(weights @ values), w_o)

# Rotary position embedding  [§2.2]
# theta = 10000 comes from the code; the paper does not state it.

def apply_rope(queries, keys, rope_angles):
    # Rotate channel pairs (2i, 2i+1), not the two halves of the head
    # [code: llama/model.py#L156-L161]
    queries = rotate_adjacent_pairs(queries, rope_angles)
    keys = rotate_adjacent_pairs(keys, rope_angles)
    return queries, keys

# SwiGLU feed-forward  [§2.2] [code: llama/model.py#L331-L348]
# Hidden width: 2/3 * 4d, rounded up to a multiple of 256 in the code.

def swiglu_ffn(x):
    # Gate and value branches, both projected from the input  [§2.2]
    gate = silu(linear(x, w1))                          # (B, T, hidden)
    value = linear(x, w3)                               # (B, T, hidden)
    # Elementwise product, then project back to model width  [§2.2]
    return linear(gate * value, w2)                     # (B, T, d)

Optimisation §2.3. AdamW with beta1 = 0.9 and beta2 = 0.95. The learning rate follows a cosine schedule that decays to 10% of its peak, with 2,000 warmup steps. Weight decay is 0.1 and gradient clipping is 1.0. Peak learning rate and batch size vary with model size tab. 2. The training code is not in the public repository, so these values cannot be checked against code.

Efficiency §2.4. The authors use a causal attention kernel from the xformers library, which does not store the masked attention weights and skips masked score computations, and they hand-write the backward pass of the transformer layers to save activation memory. Reducing memory further needs model and sequence parallelism, and the authors overlap computation with the network communication §2.4. Reported throughput for 65B is about 380 tokens per second per GPU on 2,048 A100-80GB GPUs, roughly 21 days for 1.4T tokens §2.4.

Evaluation protocol §3. Zero-shot and few-shot evaluation on 20 benchmarks. For multiple-choice tasks the model scores each option by its likelihood and picks the best. The likelihood is normalised by the number of characters for most datasets. For OpenBookQA and BoolQ it is divided by the likelihood of the option after an "Answer:" prompt, following GPT-3 §3. Question answering uses exact match after normalising the answer app. A.

Hyperparameters

Model rows from tab. 2; the paper labels these models 6.7B, 13.0B, 32.5B and 65.2B in the table and 7B, 13B, 33B and 65B in the text. Use the table values for implementation.

Data mixture and epoch counts tab. 1. Epochs refer to the 1.4T-token runs; the 1.0T runs use the same proportions tab. 1.

Architecture and optimisation table tab. 2, full:

Results

Author-reported. Setup: zero-shot and few-shot evaluation on 20 benchmarks §3. Baseline numbers are mostly copied from the original papers of the other models, not re-run by the authors, and the paper says so for the common-sense and instruction tables §3.1 §4. Model names follow the text (7B to 65B).

Headline claims: LLaMA-13B outperforms GPT-3 on most common-sense benchmarks despite being about 10 times smaller §3.1. LLaMA-65B beats Chinchilla-70B on all reported common-sense benchmarks except BoolQ, and beats PaLM-540B everywhere except BoolQ and WinoGrande §3.1. On TriviaQA, LLaMA-65B is reported as state-of-the-art in zero-shot and few-shot settings §3.2 tab. 5.

Common-sense reasoning, zero-shot tab. 3. Bold marks the best value in the paper's table.

Natural Questions, exact match tab. 4. Test split used for open-domain QA with 3,610 questions app. A.

TriviaQA, exact match on the filtered dev set tab. 5. Not comparable to GPT-3 and PaLM, which report on the unfiltered test set app. A.

Reading comprehension, zero-shot accuracy tab. 6.

Mathematical reasoning tab. 7. maj1@k is majority voting over k samples: k = 256 for MATH and k = 100 for GSM8k. Minerva is PaLM fine-tuned on mathematical text; LLaMA is not fine-tuned on mathematical data §3.4.

Code generation tab. 8. HumanEval is zero-shot. MBPP uses 3-shot prompts. pass@1 is sampled at temperature 0.1; pass@100 and pass@80 at 0.8 §3.5. Values marked with an asterisk are read from a figure in the PaLM paper.

MMLU, 5-shot accuracy, by domain tab. 9. The per-subject table for the 57 tasks is tab. 16 in the appendix; its rows are not reproduced here.

Instruction tuning, MMLU 5-shot tab. 10. One short fine-tuning run on instruction data following Chung et al.; the authors say this is not the focus of the paper §4.

Bias, toxicity and truthfulness §5. Table 11 is RealToxicityPrompts with greedy decoding over 100k prompts, scored by a third-party API; higher is more toxic. Table 12 is CrowS-Pairs in zero-shot; higher means the model prefers the stereotype more often. Table 13 is WinoGender co-reference accuracy. Table 14 is TruthfulQA tab. 11 tab. 12 tab. 13 tab. 14.

Carbon footprint §6 tab. 15. Estimated, not measured. The formula is energy = GPU-hours × 400 W × PUE 1.1, and emissions = MWh × 0.385 kg CO2e per kWh (US average), the same assumptions applied to OPT and BLOOM. The total for developing the models is estimated at about 2,638 MWh and 1,015 tCO2eq §6. Equations in the paper are unnumbered, so they are cited by section.

Figures are described in words, as the wiki does not carry images yet. fig. 1 plots training loss over tokens for the four models; the 33B and 65B runs use 1.4T tokens and the smaller two use 1.0T, all at 4M-token batches fig. 1. fig. 2 tracks question answering and common-sense scores during training: most rise steadily and track training perplexity, the exceptions are SIQA, which varies a lot between checkpoints, and WinoGrande, where 33B and 65B perform alike during training §3.7. fig. 3 shows the 1-shot prompt format used for NaturalQuestions and TriviaQA, with a fixed "Answer these questions:" prefix app. A.

Paper vs code

The official repository is https://github.com/facebookresearch/llama at 689c7f261b9c5514636ecc3c5fefefcbb3e6eed7. It is linked from the paper's abstract. Its README marks the repository as deprecated and says it targets Llama 2 README.md. The commit therefore does not match the release described here. Only inference code is present: llama/model.py, llama/generation.py and llama/tokenizer.py. Training code, data pipeline and the paper's LLaMA v1 weights are not in the repository. Comparisons below cover the architecture only.

• Training hyperparameters (AdamW betas, warmup, clipping, schedule) cannot be checked; no training code is in the repository §2.3.
• Efficient attention: the paper names an xformers causal-attention kernel §2.4. The repository computes attention with a plain matrix product, a softmax and an explicit mask model.py. The kernel is not in the repository.
• RoPE pairing: the code rotates adjacent channel pairs model.py. The paper says only "rotary embeddings" §2.2.
• RoPE base: the code uses theta = 10000 model.py. The paper does not state it.
• FFN width: the paper gives 2/3 of 4d §2.2. The code rounds up to a multiple of 256 model.py. For the 6.7B model this gives 11008 (derived from code).
• Grouped key/value heads: the code accepts a separate number of key/value heads and defaults to the number of query heads model.py model.py. The paper does not describe this option, and its Table 2 shows no separate key/value count tab. 2.
• Output projection: the code keeps a separate output layer and has no bias terms in any linear layer model.py [code: llama/model.py#L446-L448]. The paper does not say whether the output and embedding weights are tied, or whether biases exist.
• Context length: the code default is 2048 tokens model.py, and the RoPE table covers twice that, with a comment about a 4096-token limit for Llama 2 model.py. The paper does not state a context length.
• Sampling defaults: the code uses temperature 0.6 and top-p 0.9 [code: llama/generation.py#L134-L135]. The paper evaluates QA with greedy decoding app. A and code pass@1 at temperature 0.1 §3.5. The code defaults are not the paper's evaluation settings.
• Tokenizer: the code loads a SentencePiece model file and reads the vocabulary size from it tokenizer.py. The model file is not in the repository, so the vocabulary size stays unknown here.

License of the code. The repository's LICENSE is the Llama 2 Community License Agreement, which is custom and not MIT, Apache-2.0 or BSD. It includes additional commercial terms above 700 million monthly active users [LICENSE §2]. Under our rules no verbatim code is quoted from this repository; the pseudocode above is written from the code with citations only.

The causal mask, the pre-norm placement and the residual layout match the paper's description §2.2 model.py [code: llama/model.py#L457-L495].

Pitfalls

• Model names differ between the table and the text: 6.7B, 13.0B, 32.5B and 65.2B in tab. 2, and 7B, 13B, 33B and 65B in the text. The table values (width, layers, heads, learning rate, tokens) are the ones to implement.
• The 33B and 65B learning rate is half the 6.7B and 13B rate, and the training budget is 1.4T rather than 1.0T tokens tab. 2. Using one value for all sizes is a common mistake.
• Epoch counts in tab. 1 refer to the 1.4T-token runs. Wikipedia and Books are seen about twice, and the others about once §2.1.
• RMSNorm epsilon, RoPE base, numeric precision, dropout and vocabulary size are not in the paper. The code gives values for epsilon, theta and precision model.py model.py [code: llama/model.py#L76]. Vocabulary size is not given in the repository.
• The SwiGLU hidden width is 2/3 of 4d before rounding §2.2. The code then rounds up to a multiple of 256; for d = 4096 that moves 10922 to 11008 (derived from code). Match the code if reproducing the released shapes model.py.
• Rotary embeddings in the code rotate adjacent pairs, not split halves model.py. Other codebases use the split-half convention, which changes the attention scores relative to the code.
• The KV cache is allocated up front for the maximum batch size and sequence length, on the GPU, for every layer model.py. Memory is reserved even for short prompts. The code calls .cuda() directly, so it does not run on a CPU-only machine as written.
• TriviaQA is evaluated on the filtered dev set, not the unfiltered test set that GPT-3 and PaLM use, so the numbers are not directly comparable app. A.
• Multiple-choice scoring uses different normalisation for OpenBookQA and BoolQ than for the rest §3. Implement both or the scores will not match the table.
• Most baseline numbers are copied from other papers, under their own protocols §3.1 §4. Some PaLM code numbers are read from a figure tab. 8. Do not treat the table as a single controlled comparison.
• LLaMA-I comes from a single instruction-tuning experiment, which the authors describe as not the focus of the paper §4. Treat its MMLU number as a single data point.
• Toxicity scores rely on a third-party API the authors do not control, so they are hard to compare with other papers §5.1.
• The authors say their bias and toxicity evaluations are not sufficient to show the risks of the model §5. They single out religion as the category where the model is most biased against OPT-175B, followed by age and gender §5.2 tab. 12.
• Carbon numbers are estimates from an assumed power draw and grid intensity, not measured energy §6.
• Training throughput (380 tokens per second per GPU on 2,048 A100s) implies about 21 days for 1.4T tokens §2.4. Plan compute from this, not from the parameter count alone.

Lineage

• Builds on: GPT-3 style pre-training §2.1 §2.2; pre-normalisation, with the GPT-3 and GPTNeo references the paper cites §2.2; RMSNorm §2.2; SwiGLU §2.2; rotary embeddings §2.2; the causal attention approach the paper credits to rabe2021self, and the backward pass from dao2022flashattention §2.4.
• Scaling-law basis: Chinchilla's training-compute-optimal recipe, which the paper deliberately departs from to optimise for inference cost §1 §7.
• Compared against: GPT-3, Gopher, Chinchilla, PaLM, OPT, GPT-J, GPT-Neo, GPT-NeoX, GLM, OPT-IML, Flan-PaLM, Minerva, LaMDA §3 tab. 3 tab. 9.
• Superseded by: none recorded on this wiki yet.