Self-Instruct: Aligning Language Models with Self-Generated Instructions

In plain words

Teaching a language model to follow instructions, such as "summarize this" or "label this review", usually takes many examples that people wrote by hand. This paper asks whether the model can write most of those examples itself.

It starts from a small set of hand-written tasks. It shows a base GPT-3 a handful of them and asks it to continue the list with new tasks. For each new task, the model either writes an input and then an answer, or lists the possible labels and then writes an input for each label. Duplicate and broken items are thrown away. What is left is a dataset of new tasks and examples, and the same base model is then fine-tuned on it.

The tuned model follows instructions far better than the untuned one. It comes close to a version tuned on private user data and human labels, and it beats models tuned on public instruction datasets on a set of new, hand-written instructions. The authors release the generated data and that instruction set.

TL;DR

• The model generates its own instruction data from 175 hand-written seed tasks: about 52K instructions and 82K instances from GPT-3 davinci (175B) §3 tab. 1.
• Fine-tuning GPT-3 on that data raises zero-shot SuperNI ROUGE-L from 6.8 to 39.9 (author-reported). InstructGPT-001 scores 40.8 tab. 3.
• On 252 new user-oriented instructions, the tuned model is about 5% behind InstructGPT-001 if "acceptable with minor errors" counts as valid §4.4.
• Only 54% of a 200-instruction sample had all fields judged valid, by one author tab. 2 §3.3. Treat the data as noisy.
• The code is official and Apache-2.0. Reproducing it needs a GPT-3 davinci base model and the legacy OpenAI completions and fine-tuning endpoints, which the repo pins to openai==0.24.0 requirements.txt.

Problem

Human-written instruction data is costly, often limited in quantity and diversity, and mostly covers popular NLP tasks §1. The strongest instruction-tuned models are trained on data that is not public, so their construction is opaque §5 §7. The paper asks whether a pretrained LM can generate diverse instruction data at low cost, with almost no human labeling §1.

Method

The pipeline has four generation stages and one training stage §2 fig. 2:

1. Generate new task instructions by few-shot prompting from a pool of seed and generated tasks §2.2.
2. Decide, for each instruction, whether it is a classification task §2.2.
3. Generate instances for each instruction. Non-classification tasks use the input-first approach. Classification tasks use the output-first approach §2.2.
4. Filter invalid, duplicate and near-duplicate items §2.2.

The fine-tuning stage trains the original model on the generated instructions and instances, with several prompt templates §2.3.

Definition. A task t has an instruction I_t and n_t \ge 1 instances (X_{t,i}, Y_{t,i}). The target behaviour is M(I_t, X_{t,i}) = Y_{t,i}. The input X may be empty, for example when the instruction is complete on its own §2.1. The paper gives no numbered equations; this is stated in prose and inline notation.

Steps.

1. Seed pool. Start with 175 tasks, each with one instruction and one instance. The authors and their lab wrote them without reference to existing datasets or to the test sets §2.2 app. A.1. The split by task type is discussed in Paper vs code.
2. Instruction prompt. Each request samples 8 instructions as in-context examples. Six come from the seed set and two from the generated pool, which promotes diversity §2.2. The prompt is a fixed header followed by a numbered list, and the model writes the next item tab. 5 bootstrap_instructions.py. Sampling settings are in tab. 4.
3. Post-process completions. Split the completion on newline-plus-number. Keep an item only if its ROUGE-L score against every existing instruction is below 0.7. Drop items that are too short or too long, that contain certain keywords (the paper's examples are "image", "picture" and "graph"), or that start with punctuation §2.2 bootstrap_instructions.py bootstrap_instructions.py.
4. Classification check. Ask a few-shot prompt "Is it classification?" for each instruction. The examples are 12 classification and 19 non-classification seed instructions. Tasks with a small, limited output label space count as classification §2.2, footnote tab. 6.
5. Input-first (non-classification). Prompt with examples from other tasks. The model writes the input fields first and the output second. Items with no input are allowed §2.2 tab. 7.
6. Output-first (classification). The model writes the class label first, then an input conditioned on that label. The paper states that input-first generation produced inputs biased toward one label, for example grammatical inputs for grammar-error detection §2.2 tab. 8.
7. Filter instances. Drop an item if its input equals its output, if its output is empty, or if its input or output ends with a colon. Drop all items for a generation in which two items share a non-empty input but differ in output. Remove exact duplicates §2.2 prepare_for_finetuning.py.
8. Fine-tune. Concatenate instruction and input with one of several templates. Templates vary "Task:" prefixes, "Input:" and "Output:" labels, and line breaks. Train on the output. The paper sets the API's prompt loss weight to 0 and trains 2 epochs §2.3 §4.1 app. A.3.

# Bootstrap new instructions  [§2.2] [fig. 2]
# Input:  seed_tasks (175 instruction strings)
# Output: generated_tasks, a list of instruction strings

def bootstrap_instructions(seed_tasks, target_count):
    generated_tasks = []
    while len(generated_tasks) < target_count:
        # 2 generated + 6 seed examples, 8 in total, shuffled  [§2.2]
        examples = sample(generated_tasks, min(2, len(generated_tasks)))
        examples += sample(seed_tasks, 8 - len(examples))
        shuffle(examples)
        prompt = numbered_list(examples)
        # decoding settings: tab. 4, row 1
        completion = language_model(
            prompt,
            temperature=0.7,
            top_p=0.5,
            presence_penalty=2,
            max_tokens=1024,
        )
        for candidate in split_numbered_items(completion):
            # drop too short or too long, keyword and style-filtered items
            if not passes_text_filters(candidate):  # [code: ...#L41-L70]
                continue
            # keep only if ROUGE-L is below 0.7 for every existing one  [§2.2]
            scores = rouge_l(candidate, seed_tasks + generated_tasks)
            if max(scores) >= 0.7:
                continue
            generated_tasks.append(candidate)
    return generated_tasks

# Classification check  [§2.2] [tab. 6] [tab. 4]
# Output: True when the model answers "Yes"

def is_classification(instruction):
    # few-shot examples end with "Task:"; the new instruction follows
    prompt = few_shot_examples + " " + instruction
    prompt += "\nIs it classification?"
    # greedy decoding, at most 3 new tokens  [tab. 4, row 2]
    answer = language_model(prompt, temperature=0, max_tokens=3)
    # exact match on the answer  [code: self_instruct/generate_instances.py#L99]
    return answer.strip() in ["Yes", "yes", "YES"]

# Instance generation  [§2.2] [tab. 4, row 3] [tab. 7] [tab. 8]
# Output: raw text, parsed by clean_instances()

def generate_instances(instruction, is_classification):
    if is_classification:
        # output-first: label first, then an input for that label
        examples = output_first_examples
    else:
        # input-first: input fields first, then the output
        examples = input_first_examples
    prompt = examples + " " + instruction + "\n"
    # greedy decoding, presence penalty 1.5, at most 300 new tokens
    return language_model(
        prompt,
        temperature=0,
        top_p=0,
        presence_penalty=1.5,
        max_tokens=300,
        stop=["Task:"],
    )

# Parse and filter one instruction's raw generation  [§2.2]
# Items are (instruction, input, output) triples
# [code: self_instruct/prepare_for_finetuning.py#L142-L191]

def clean_instances(instruction, items, hit_length_limit):
    # if max_tokens cut the text off, drop only the last item
    if hit_length_limit:
        items = items[:-1]
    # drop empty output, input == output, and text ending in ":"
    items = [item for item in items if is_valid(item)]
    # two items with the same non-empty input but different outputs:
    # the whole generation is discarded  [code: ...#L110-L121]
    if has_conflicting_pair(items):
        return []
    # remove exact duplicates, then keep at most 5 at random  [code: ...#L225]
    items = unique(items)
    return sample(items, min(5, len(items)))

# Encode one training example  [§2.3] [app. A.3]
# [code: self_instruct/prepare_for_finetuning.py#L54-L92] [#L274-L284]

def encode(instruction, input_text, output_text):
    # for about half of the items, clean the input  [code: ...#L274-L284]
    if random() < 0.5:
        # if exactly one "word:" label occurs, keep only the text after it
        input_text = drop_single_field_label(input_text)
        input_text = input_text.replace("\n\n", "\n")
    # 7 templates when there is an input, 6 when there is none  [§2.3]
    if input_text != "":
        template = choice(templates_with_input)
    else:
        template = choice(templates_without_input)
    prompt = template.format(instruction, input_text)
    completion = output_text + END_OF_TEXT
    # loss is on the completion; the API's prompt loss weight is 0
    return prompt, completion

Hyperparameters

Query settings for the OpenAI API tab. 4. The code uses the same values in its scripts bootstrap_instructions.py generate_instances.py identify_clf_or_not.py. Where the code differs, see Paper vs code.

Other settings:

Results

All numbers are author-reported. Setup: GPT-3 175B models, zero-shot evaluation on the 119 unseen SuperNI tasks with 100 instances each, given only the task definition, scored with ROUGE-L §4.3 tab. 3.

Table 3 reproduced from the paper:

Notes:

• The "+33.1%" in the text is 39.9 minus 6.8, in absolute ROUGE-L points tab. 3 §4.3.
• Public instruction-tuned models come from §4.2. The paper describes InstructGPT-001 as trained with private user data and human-annotated labels §4.3. The paper compares only with text-davinci-001 on SuperNI, because newer engines may have seen the SuperNI test set §4.2.
• The paper says the vanilla GPT-3 "basically cannot follow human instructions", tends to repeat text, and does not stop §4.3.

Human evaluation. The paper uses 252 instructions written by the authors, with one instance each §4.4. Experts rated outputs on four levels, from A (valid and satisfying) to D (irrelevant or invalid) §4.4. Two authors rated every instance independently, and the paper reports one evaluator's results app. B.1. Inter-rater agreement is Cohen's kappa 0.57 in §4.4 and 0.58 in app. B.2 (see Pitfalls). For A/B versus C/D the kappa is 0.75, and Spearman's rho on the ordinal scale is 0.81 app. B.2. The per-model human-evaluation values are shown only in a bar chart §4.4; they are not stated in the text. Counting Rating-B as valid, GPT-3 + Self-Instruct is "only 5% behind" InstructGPT-001 §4.4.

Data size and quality §4.5. Human-evaluation performance rises with the amount of generated data, and the improvement levels off at about 16K instructions. On SuperNI the gain levels off at a few hundred instructions. Regenerating outputs with InstructGPT-003 and training on them gives a stated "10%" gain over the original generated outputs. The paper does not say whether this is absolute or relative.

Generated data §3 tab. 1 tab. 2.

Table 1 reproduced from the paper:

Table 2 reproduced from the paper. One author judged 200 randomly sampled instructions, each with one random instance §3.3:

Diversity §3.2: 26,559 of 52,445 instructions contain a parseable verb-noun structure. The top 20 root verbs and their top 4 direct objects cover 14% of all instructions. The paper plots the ROUGE-L overlap of generated instructions with their nearest seed; it states a "decent number" of low-overlap instructions but gives no summary value §3.2. Seed tasks have average ROUGE-L 0.21 against SuperNI test instructions and 0.34 against the 252 user-oriented instructions app. A.1.

Cost app. A.2 app. A.3: generation cost about $600 and fine-tuning about $338, at December 2022 OpenAI prices.

Paper vs code

Checked against the official repository at commit 0b26ccaa415992100fa32df62d41b994cf928e23. The paper's abstract footnote links this repository, and its GitHub owner name matches the first author. Each item lists the paper's statement first, then the code.

• Seed split. Paper: 25 classification and 150 non-classification seed tasks app. A.1. Data file: 26 classification and 149 non-classification seed_tasks.jsonl.
• Instance max tokens. Paper: 300 tab. 4. Code: 300 if the batch contains a classification task, otherwise 350 generate_instances.py.
• Instance stop sequences. Paper: Task: tab. 4. Code: also Example 6 (max instances plus one), as well as Task: generate_instances.py.
• Classification stop sequences. Paper: \n and Task: tab. 4. Code: \n and Task, with no colon identify_clf_or_not.py.
• Similarity cutoff. Paper: an instruction is kept only if ROUGE-L is below 0.7 §2.2. Code: rejects only when the maximum is above 0.7, so a score of exactly 0.7 is kept bootstrap_instructions.py.
• Instances per task. Paper: not stated. Code: keeps at most 5 random instances per instruction prepare_for_finetuning.py.
• Text filters. Paper: "too long or too short", and keywords such as image, picture and graph §2.2. Code adds a 3-word minimum and a 150-word maximum bootstrap_instructions.py. It also filters file, files, map, maps, draw, plot and "go to" bootstrap_instructions.py, drops "Write a program" starts bootstrap_instructions.py, and drops items whose first character is punctuation or non-ASCII bootstrap_instructions.py bootstrap_instructions.py.
• Capitalization. Paper: not stated. Code: applies str.capitalize() to each generated instruction, which lowercases everything after the first letter bootstrap_instructions.py.
• Conflicting pairs. Paper: "filter out instances ... with the same input but different outputs" §2.2. Code: discards every instance from that generation, not just the conflicting pair prepare_for_finetuning.py.
• Prompt wording. The printed prompts in tab. 6 to tab. 8 differ from the shipped code. For example, the output-first prompt tells the model to generate "the correct class label" in tab. 8 but "possible class labels" in code, and the example tasks differ instance_gen_template.py. The paper does not say which version produced the released data.
• Template count. Paper: "multiple templates" §2.3. Code: 7 with input and 6 without prepare_for_finetuning.py.
• Input cleaning. Paper: not described. Code: for about half of the items, drops a single "word:" label and collapses blank lines prepare_for_finetuning.py.
• Fine-tuning. Paper: 2 epochs, prompt loss weight 0, davinci §4.1. Code matches: --n_epochs 2 and --prompt_loss_weight 0 finetune_gpt3.sh.
• Dataset and eval set sizes. The released generation file has 82,439 lines, matching tab. 1. The human-evaluation file has 252 lines, matching §4.4 all_instances_82K.jsonl user_oriented_instructions.jsonl.
• SuperNI evaluation. Paper: decoding settings in tab. 4 §4.3. The repository has no SuperNI evaluation script, only the generation pipeline, the fine-tuning data and human-evaluation predictions.
• API. Paper: GPT-3 davinci through the OpenAI API §3 app. A.2. Code: legacy openai.Completion.create gpt3_api.py with openai==0.24.0 requirements.txt.

Pitfalls

• Noisy data. Only 54% of the sampled items were fully valid, and outputs were correct in only 58% of cases tab. 2. The paper's own sample was labeled by one author with no measured agreement §3.3. The repository README gives the same caution README.md. Filter again before any high-stakes use.
• Labels depend on one model. The pipeline uses GPT-3 davinci through the legacy OpenAI API, and the paper tests only that model §3 §8. Whether the recipe works with other base models is not tested. The paper also says the approach may work best for larger models §8.
• Classification check is unmeasured. The paper gives no accuracy for the classification prompt §2.2. Misclassified tasks are generated with the wrong prompt, and the output-first and input-first prompts use different formats.
• Heuristic filters bias the data. Keyword filters remove tasks that mention images, files, maps, graphs, plots or drawing, and the ASCII-first-character rule removes non-English-starting items bootstrap_instructions.py bootstrap_instructions.py. Multimodal and file-handling tasks are excluded by design, and the paper does not discuss this. The paper also notes that gains skew toward frequent tasks §8.
• Capitalization damages names. str.capitalize() lowercases proper nouns and acronyms inside generated instructions bootstrap_instructions.py. Keep the original text if you need correct names.
• Run-to-run variation. The code deduplicates with list(set(...)) prepare_for_finetuning.py. Set order for strings can change between Python processes, so output order is not stable even with the fixed seeds.
• Length cutoff. A generation that hits max_tokens loses only its last item prepare_for_finetuning.py. Other items from the same cut-off response are kept.
• Human-evaluation conflict and reuse limit. The human-evaluation instructions were written by two of the evaluators app. B.1. The repository README asks users not to use the 252-instruction set for training README.md. Do not train on it.
• Inconsistent kappa. The paper reports 0.57 §4.4 and 0.58 app. B.2 for the same agreement statistic.
• Ambiguous percentages. "+33.1%" is an absolute ROUGE-L difference tab. 3. The "10%" distillation gain is not stated as absolute or relative §4.5.
• Evaluation depends on length settings. Evaluation is greedy with no stop sequences and a 1024-token cap tab. 4. Models that do not stop, like the vanilla GPT-3 in the paper, are scored on whatever they produce before the cap §4.3. Changing the cap changes the score.
• Cost figures are dated. The $600 and $338 figures use December 2022 prices app. A.2 app. A.3.
• Licensing of the data is not stated. The paper does not say what license covers the released generated data. The repository LICENSE is Apache-2.0, and this card does not verify whether it covers the data files. The data came from a commercial API, and the paper does not discuss that provider's terms. Check both before commercial use.
• Rouge settings are unstated. The paper does not name a ROUGE tokenizer or stemming choice. The code uses rouge_score with no stemming bootstrap_instructions.py. Use the same setting or the 0.7 cutoff will move.

Lineage

• Builds on: GPT-3 (Brown et al., 2020) as the base model, and T5 as the vanilla baseline §3 §4.2.
• Uses: SuperNI (Wang et al., 2022) as the evaluation benchmark §4.3. The 175 seed tasks are written for this paper, not taken from SuperNI app. A.1.
• Compared against: T0 (Sanh et al., 2022), Tk-Instruct (Wang et al., 2022), and InstructGPT (Ouyang et al., 2022) at the 001, 002 and 003 engines §4.2.
• Related work the paper cites for scaling instruction tuning: Scaling Instruction-Finetuned Language Models §1 §5.
• Parallel work: Unnatural Instructions (Honovich et al., 2022), which uses SuperNI tasks as seeds and InstructGPT-002 for generation §5.
• Follow-ups the paper names: Alpaca, Baize and a principle-driven variant §7. No cards exist for them yet.
• Superseded by: not assessed for this card.