LIMA: Less Is More for Alignment

In plain words

A large language model learns most of what it knows from a first, very large training stage on raw text. A second stage then teaches it to behave like an assistant that answers people's requests. This paper asks how much of that second stage is really needed.

The authors take a 65-billion-parameter LLaMa model and fine-tune it on only 1,000 hand-picked prompt-and-answer pairs. They do not use reinforcement learning or any human preference model. Judges who compared its answers with those of GPT-4 and other chatbots found them equal or better in a good share of cases.

The authors conclude that the model's knowledge mostly comes from pretraining, and that a few good examples mainly teach it the style of a helpful answer. In their tests, the variety and quality of the examples mattered more than how many there were.

TL;DR

LIMA is LLaMa 65B fine-tuned with plain supervised loss on 1,000 examples: 750 from community Q&A, 250 written by the authors. No reinforcement learning from human feedback is used. In human pairwise tests, LIMA's answer was equal to or better than GPT-4's in 43% of cases and than DaVinci003's in 65%, both author-reported. Use it as evidence that small, curated instruction data can be enough for style alignment, with caveats about evaluation.

Problem

Alignment of pretrained language models is usually done with instruction tuning over very large datasets, and then with reinforcement learning from human feedback §1. The paper says these methods need large amounts of compute and specialised data to reach ChatGPT-level behaviour §1. The authors test the Superficial Alignment Hypothesis: a model's knowledge and capabilities are learned almost entirely during pretraining, while alignment only teaches which output format to use when talking to users §2.

If the hypothesis holds, a small set of examples should suffice to tune a strong pretrained model §2. The paper treats this as a question to test, not a result it assumes.

Method

Data (1,000 training examples)

The training set has 1,000 prompt-response pairs. Inputs are diverse. Outputs share the style of a helpful AI assistant §2. The data mix is summarised in tab. 1. The table caption puts the training text at about three-quarters of a million tokens, spread across 1,000 sequences tab. 1.

Sources, from §2.1 and §2.2:

• Stack Exchange (200 STEM and 200 other): within each community, the authors keep top-scored questions whose content sits entirely in the title, with no body text; the top answer must score at least 10 §2.1.
• wikiHow (200): sample a category first (out of 19), then an article. Use the title as the prompt and the article body as the response §2.1.
• r/WritingPrompts (150): fictional premises with high-quality responses §2.1.
• Manually written, Group A (200): prompts and answers written by the authors, with a uniform assistant tone §2.2.
• Super-Natural Instructions (50): one random example from each of 50 generation tasks, lightly edited to match the manual style §2.2.

Held-out sets: a 50-example development set from Group A, and a 300-prompt test set (70 from r/AskReddit, 230 from Group B) §2 tab. 1.

Safety-related examples: 13 training prompts with toxic or malicious intent. The responses partly or fully refuse and explain why §2.2. The test set has 30 similar prompts §2.2 §4.3.

Stack Exchange filters §2.1: 179 exchanges are split into 75 STEM and 99 other. Five niche exchanges are dropped. Sampling uses temperature τ = 3 to get more uniform domain coverage. Answers are removed if shorter than 1,200 or longer than 4,096 characters, written in first person, or refer to other answers. Links, images and HTML are stripped, but code blocks and lists are kept. For each example, the prompt is randomly the title or the description.

Training

LIMA is LLaMa 65B fine-tuned on the 1,000 examples with the standard supervised loss [abstract] §3. An end-of-turn token (EOT) marks the end of each utterance, so it is not confused with the pretrained EOS token §3. The hyperparameters are in the table below.

Checkpoints are chosen by hand, between epochs 5 and 10, using the 50-example development set. The authors found that validation perplexity does not track generation quality §3 app. B.

# LIMA supervised fine-tuning  [§3]
# Input:  pretrained LLaMa 65B; 1,000 (prompt, response) pairs
# Output: one fine-tuned checkpoint, chosen from epochs 5 to 10

def finetune(model, pairs, dev_pairs):
    # Mark the end of every utterance with an EOT token  [§3]
    sequences = [add_eot_tokens(turns) for turns in pairs]

    # Trim sequences longer than 2,048 tokens  [§3]
    sequences = [trim(seq, max_len=2048) for seq in sequences]

    checkpoints = []
    for epoch in range(15):  # 15 epochs in total  [§3]
        # Batches of 32 examples, 64 for smaller models  [§3]
        for batch in make_batches(sequences, size=32):
            # Standard supervised loss  [abstract] [§3]
            loss = supervised_loss(model, batch)
            # AdamW, beta1 = 0.9, beta2 = 0.95, weight decay 0.1  [§3]
            # Learning rate 1e-5 decayed linearly to 1e-6, no warmup  [§3]
            # Residual dropout rises from 0.0 (bottom) to 0.3 (top)  [§3]
            update_weights(model, loss)
        checkpoints.append(save_copy(model))

    # Select a checkpoint by reading dev-set outputs  [§3] [app. B]
    return pick_by_manual_review(checkpoints[4:10], dev_pairs)

Generation and evaluation protocol

• Baseline generation (applied to each baseline model): nucleus sampling with p = 0.9, temperature 0.7, repetition penalty 1.2 on previously generated tokens, maximum 2,048 tokens §4.1. The text does not say explicitly that LIMA was decoded with the same settings.
• Pairwise human preference: one response per test prompt from LIMA and from each of five baselines (Alpaca 65B, DaVinci003, Bard, Claude, GPT-4). Crowd workers choose the better response or a tie. The same comparisons are repeated with GPT-4 as annotator, using the same instructions §4.1.
• Absolute analysis: the authors label 50 random test responses as Fail, Pass or Excellent §4.3.
• Ablations: a 7B model is trained on each data variant. Five responses per test prompt are graded 1 to 6 for helpfulness by ChatGPT (GPT-3.5 Turbo), with a 95% two-sided confidence interval §5 app. D.
• Multi-turn: a new model is trained on the 1,000 examples plus 30 dialogue chains (1,030 total). Ten live conversations are compared with the zero-shot model §6.

Each setting is listed below with its source.

Hyperparameters

Results

Author-reported throughout. Setups: 300 test prompts for human and GPT-4 preference §4.1; 50 test prompts for absolute analysis §4.3; 20 prompts for out-of-distribution analysis §4.3; 30 safety prompts §4.3; 10 live conversations §6; 7B models for ablations §5.

Other reported points:

• The human preference figure and the GPT-4 figure show the same trends §4.2. The text does not give the Alpaca or DaVinci003 values beyond the headline numbers above.
• Structured-output examples: without six structure examples, the model fails some prompts with required formats. With them, responses improve; no numbers are given app. E.

Paper vs code

No official code.

Search performed for step 3 of the compile procedure:

• Paper text (LaTeX source, all sections and bibliography): no repository or project link. The only GitHub link in the bibliography is the Alpaca repository, which is a cited baseline, not the authors' code.
• arXiv abstract page: no comment or code link; the "Code, Data, Media" section lists only third-party services.
• GitHub repository search via the API for "LIMA Less Is More for Alignment" and "LIMA alignment Zhou Meta": 0 results.

The source also says that data is publicly released, but that sentence is commented out (01_intro.tex line 13), so no data release is claimed.

Pitfalls

• Checkpoint selection is manual. Perplexity on held-out data correlates negatively with generation quality, as the authors report app. B. Reproducing the model means reproducing the human choice among epochs 5 to 10, and the selection criteria are not stated beyond "manually select" §3.
• Evaluation depends on human and GPT-4 judges. Agreement is 82% crowd-crowd, 81% crowd-author, 78% author-author, 78% crowd-GPT and 79% author-GPT §4.1. The authors call the task subjective §4.1. The absolute "Excellent" labels are the authors' own §4.3.
• Baselines are not controlled. Responses were sampled throughout April 2023 §4.1. Products like Claude and GPT-4 can change over time. The paper also says some baselines may have seen large amounts of user data §4.3.
• Small evaluation sets. Human comparison uses 300 prompts. The absolute analysis uses 50, the out-of-distribution analysis 20, the safety analysis 30, and multi-turn 10 live conversations §4 §6. These sizes give wide uncertainty.
• Test and training distributions are close. Groups A and B shared priors. The authors say there was significant contact between them before annotation §2.2 footnote. Test prompts were partly filtered as "problematic" §2.2.
• Unclear data steps. The paper does not say how many Stack Exchange answers were rejected overall, how communities were sampled with temperature τ = 3, or how title and description prompts were randomised beyond "randomly" §2.1. Three sources are mined from community sites (Stack Exchange, wikiHow, Reddit), and the wikiHow preprocessing heuristics for pruning links, images and sections are only summarised §2.1.
• Unclear training details. Learning-rate decay is "linear" but per step or per epoch is not stated §3. Trimming of over-long sequences is not described §3. Whether the loss covers prompt tokens is not stated. The token string of the EOT marker is not stated §3. The threshold for "smaller models" is not stated §3.
• Ablation models are 7B, not 65B. The ablations use 7B models trained with "the same hyperparameters" §5. The footnote says 1,000 examples work for 7B, but 2,000 gave more stable training §5 fn. 1. This means the ablation setup is not the same as the headline model.
• Ablation figures give no table. The diversity and quantity results are in figures; only the 0.5-point quality gap is given in the text §5. The figures are not numbered in this card, since LaTeX float placement can change numbering. Cite them by section.
• Multi-turn is not a pure 1,000-example result. The conversational model is trained on 1,030 examples, including 30 multi-turn chains, and is compared only in 10 conversations §6.
• Safety is thin. Only 13 training examples cover toxic or malicious requests. The safety analysis uses 30 test prompts. The paper says that when malicious intent is implicit, the model is more likely to give unsafe answers §4.3.
• Limits stated by the authors. Writing good examples is labour-intensive and hard to scale. LIMA is less robust than product-grade models; an unlucky sample or adversarial prompt can give a weak answer §7.
• Mixed numbers. The intro gives "43%, 46%, 58%" for GPT-4, Claude and Bard §1. Section 4 gives Bard as 42% LIMA-better, 58% at-least-as-good §4.2. These agree; the 65% DaVinci003 number appears only in the abstract [abstract].

Lineage

• Builds on: LLaMa 65B as the pretrained base §3 (bibliography key touvron2023llama; arXiv id not stated in the source). The residual-dropout scheme follows [ouyang2022training] (bibliography key) §3.
• Compared against: Alpaca 65B (fine-tuned LLaMa on 52,000 examples), OpenAI DaVinci003 (RLHF-trained), Google Bard, Anthropic Claude, OpenAI GPT-4 §4.1.
• Superseded by: not stated in this card.