The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

In plain words

Instruction tuning is the step where a language model is trained on many tasks written as plain-language instructions, so that it learns to follow new instructions it has never seen. Several groups had published collections of such tasks, but they made different choices about which tasks to include, how to phrase them, and what kind of examples to show.

This paper takes one public collection, called Flan 2022, and tests each of its design choices in isolation. The authors train the same small-to-medium model with one ingredient removed at a time and see what drops. Their main finding is that mixing examples with no examples and with a few worked examples in the same training set helps the model in both situations, not just one. They also find that how much weight each source of tasks gets matters about as much as how many tasks there are, and that a few extra tricks, such as asking the model to produce the question from an answer, help on unseen benchmarks.

A practical side result is that the instruction-tuned model is a better starting point for training on one specific task: it reaches good accuracy with fewer training steps than the plain pretrained model. The data and templates are released, so others can rebuild the mixture.

TL;DR

• Flan 2022 combines earlier collections with extra chain-of-thought, dialog and program-synthesis data and many new templates §3 §2.
• Training on a mix of zero-shot and few-shot templates improves both settings; the paper reports gains of 2%+ in all settings in its abstract [Abstract] §3.2.
• Scaling to about 1.8k tasks, adding input-inverted tasks and balancing task sources each contribute, according to the ablations on T5-XL 3B tab. 1 tab. 2.
• Flan-T5 XL beats other T5-XL collections on most held-in and held-out metrics; one zero-shot cell is slightly lower tab. 1.
• Instruction-tuned T5 converges faster on single-task finetuning §4 fig. 6.

Problem

Instruction tuning is an open-source research area, and the public collections differ in tasks, templates, prompt formats and mixture weights, so it was unclear which choices matter §1 §2. Earlier collections were tuned for one prompt setting only: zero-shot or few-shot §3.2. Most public collections also did not report the mixture weights or ablate their methods, which makes comparisons hard §3.6.

The paper studies instruction tuning without human-feedback data, because such data is less public §2.

Method

The setting is T5 language models adapted to a prefix-LM objective, fine-tuned on the Flan 2022 collection with T5X. Ablations use the 3B (XL) size §3 app. A.1.

1. Collection. Flan 2022 combines Flan 2021, the P3 tasks not already in Flan 2021, Super-Natural Instructions, and extra chain-of-thought, dialog and program-synthesis data. Templatization details are deferred to the original Flan 2022 paper §3.
2. Mixed prompt settings. Each training example is shown either zero-shot or with few-shot exemplars. Few-shot examples come with 2, 3 or 5 exemplars §3.2 app. A.1. Adding a small share of few-shot templates to a zero-shot mixture already improves zero-shot results, and adding zero-shot data improves few-shot results; the best share lies in a broad range §3.2 fig. 3.
3. Task scaling. T5-LM models (Small, Base, Large, XL, XXL) are trained on random subsets of 8, 25, 50, 100, 200, 400, 800 and all 1873 tasks, always including the held-in tasks §3.3.
4. Input inversion. For a labelled example (input x, answer y), the inverted task gives y and asks the model to produce x. For chain-of-thought data, every ordering of query, answer and explanation with at least one of them as output is used §3.4 app. B fig. 7.
5. Mixture balancing. Each source is left out in turn and its contribution is measured on MMLU §3.5 tab. 2. The final weights come from these results plus practitioner judgement; the paper does not publish them as numbers in the text §3.5 app. A.1.
6. Single-task finetuning. Flan-T5 is used as a starting checkpoint for a single target task, and compared with plain T5 finetuned on that task and with Flan-T5 used without further finetuning §4 fig. 5.

# Draw one training example from the collection  [§3.5] [app. B]
# Input:  submixtures (name -> rate), ratio_zero_shot in [0, 1]
# Output: one templatized (inputs, targets) pair

def draw_example(submixtures, rates, ratio_zero_shot):
    # Pick a source in proportion to its rate  [§3.5]
    # [code: flan/v2/mixtures.py#L38-L46]
    source = weighted_choice(submixtures, rates)
    # Inverted tasks are their own sources at rate 0.3  [app. B]
    # [code: flan/v2/mixtures.py#L44-L45]
    if random_uniform() < ratio_zero_shot:
        # Zero-shot template: no exemplars  [§3.2]
        shots = 0
    else:
        # Few-shot template: exemplar count is 2, 3 or 5  [app. A.1]
        # selection rule not stated
        shots = pick_shot_count([2, 3, 5])
    return format_template(fetch_example(source), shots)

Inversion example: a dialog turn is paired with a request for the preceding conversation; program-synthesis code is paired with a request for the coding question it solves app. B. The paper mixes inverted tasks in at a rate of 30%, i.e. 3 inverted examples per 10 regular ones app. B.

Hyperparameters

Results

Author-reported. All numbers are for T5-XL (3B) with zero-shot / few-shot evaluation, as shown in tab. 1, unless stated. Held-in is the mean over 8 validation sets; CoT over 5 datasets; MMLU over 57 tasks; BBH over 23 tasks app. A.3.

Full recipe versus other T5-XL collections tab. 1:

† Values reported by the OPT-IML and GLM papers, not re-run here tab. 1. The "gap" row compares only the T5-XL models in the table, per column tab. 1.

Method ablations, Flan-T5 XL with one component removed tab. 1:

Mixture leave-one-out, equal-weight mixture, measured on MMLU and two other scores tab. 2. The table does not state the prompt setting, and its numbers differ from the full model in tab. 1:

Other findings, reported in the text and figures:

• Scaling: held-in performance peaks around 200 tasks, while held-out MMLU rises roughly log-linearly with the task count; only T5-Small peaks on held-out before the full set §3.3 fig. 4.
• Single-task finetuning: the paper calls finetuning Flan-T5 a Pareto improvement over finetuning T5 on both held-in and held-out sets; Flan-T5 without further finetuning sometimes beats T5 with task finetuning, where task data is limited §4 fig. 5. Flan-T5 converges faster and to higher accuracy on the five held-out tasks §4 fig. 6.
• Comparison with OPT-IML: Flan-T5 XL is reported above OPT-IML-Max 30B (10x larger) and 175B (58x larger) on the listed metrics §3.1 tab. 1.

Task sources and metrics used tab. 3:

Paper vs code

The official code is the flan/v2 directory of google-research/FLAN at commit e9e4ec6e2701182c7a91af176f705310da541277 mixtures.py. It holds the data-mixture and template definitions only. It has no training configuration.

• Mixture weights. The paper gives no numeric weights app. A.1. The code sets relative rates FLAN 4, T0 3, CoT 1, NIv2 1, Dialog 1, CoT-II 0.3, Dialog-II 0.3 mixtures.py. The paper gives a 46% weight for its Muffin mixture, which is Flan 2021 plus new dialog and program-synthesis tasks §3.6. The rates alone give about 38% for the FLAN source (4 / 10.6), but the example caps mixtures.py also change the effective share. I did not reconcile the two.
• Input inversion sources. The paper applies inversion to dialog, program synthesis and chain-of-thought app. B. The code defines inverted sub-mixtures only for CoT and Dialog mixtures.py. The code's inverted task names include cot_input_inversion_*, stream_input_inversion_* and *_input_inversion dialog tasks constants.py. I did not map any of these to "program synthesis".
• Inversion rate. Paper: 30% app. B. Code: 0.3 for CoT-II and Dialog-II mixtures.py. Consistent.
• Few-shot exemplar counts. Paper: 2, 3 or 5 app. A.1. Code: enum ZERO, ONE, TWO, THREE, FIVE, MULTI few_shot.py. Consistent; the code that picks a count for each example was not traced.
• Zero/few-shot and answer-option ratios. The code builds four sub-mixtures (zero-shot or few-shot, with or without answer options) from two ratios mixtures_utils.py. The paper does not give the ratio used in its final model.
• Filter naming. For Dialog, the filter excludes wiki_dialog, task_master and qrecc mixtures.py. For Dialog-II it excludes the three *_input_inversion names mixtures.py. Read literally, the two filters look swapped relative to their names. I did not trace how the filters are applied, so check before reuse.
• Training hyperparameters. The paper defers to Chung et al. 2022 app. A.1. The repository contains no learning rate, step count or batch size for instruction tuning. The only training-like values in the repository are in test files utils_test.py.
• Data version. The experiments use a slightly earlier version of the collection than the released one app. A.1. Results may differ from what the released code produces.

Pitfalls

• Task count is inconsistent. The scaling experiment lists "all 1873" tasks §3.3, the same section refers to 1836 tasks, and other sections say "1800+" §3 §6. The code count was not verified.
• The "2%+ in all settings" claim is not met in every cell. The abstract says mixed prompt training gives 2%+ gains "in all settings". Removing the few-shot templates raises zero-shot CoT (38.9 vs 35.8) and zero-shot BBH (27.6 vs 26.2) tab. 1 [Abstract].
• One zero-shot cell is below the best competitor. Flan-T5 XL is 1.9 points below the best T5-XL competitor on zero-shot BBH, although the abstract talks about gains "across evaluation settings" tab. 1 [Abstract].
• Headline numbers mix settings. The introduction's 4.2% (MMLU) gain is a zero-shot number and its 8.5% (BBH) gain is a few-shot number §1 tab. 1.
• Mixture ranking has no variance. Differences in the leave-one-out table are at most about 0.5 MMLU points for CoT, Program Synthesis and Super-Natural Instructions tab. 2. No seeds or error bars are reported. The "Dialog and Program Synthesis last" ordering rests on these small differences.
• Table 2 does not match Table 1. The weighted full mixture in Table 2 (Held-In 66.4, MMLU 48.1) differs from the full model in Table 1 (Held-In 73.8 / 74.8, MMLU 50.3 / 52.4). The paper does not explain the gap tab. 1 tab. 2.
• Held-in metrics are not all accuracy. The text defines held-in as mean accuracy app. A.3, but Table 3 lists AUC-ROC for BoolQ and RTE and F1 for ANLI and AI2 Science tab. 3. Use the per-dataset metric from the table.
• MMLU must be removed from Super-Natural Instructions. Otherwise the held-out score is contaminated app. A.3.
• Instruction-tuning hyperparameters are missing. Learning rate, schedule, step count and batch size for the Flan-T5 runs are not in this paper or its code. Obtain them from the Flan 2022 paper, which this card does not cover.
• Single-task seeds. Three seeds are used only for tasks with fewer than 1K examples app. A.2. Ablation tables are single runs.
• Compute is not quantified. The paper says instruction tuning costs more than a single-task finetune but is a one-time cost, and gives no FLOPs or hardware hours §4.
• Baselines differ. OPT-IML and GLM figures are copied from their papers and were not re-run; the templates and data differ tab. 1 §3.6.
• Overlap estimate uses OPT-IML's own task definition. The paper's 94% (2067 of 2207) figure does not deduplicate across collections §3.6.
• Single-task deltas exist only in the figure. The per-task gains in fig. 5 are printed on the plot and are not in the text; read them from the figure, not from the captions.

Lineage

• Builds on: Flan 2022 collection and Flan-T5, from Chung et al. 2022 (arXiv: 2210.11416). The paper defers templatization and the collection to that work §3.
• Compared against: Flan 2021 (Wei et al. 2021, "Finetuned language models are zero-shot learners"; no arXiv id in the source bibliography); P3 / T0 Sanh et al. 2021, arXiv: 2110.08207; Super-Natural Instructions Wang et al. 2022, arXiv: 2204.07705; OPT-IML Iyer et al. 2022, arXiv: 2212.12017; GLM-130B (reported values only) tab. 1.
• Input inversion idea: P3's "prompts not intended for the original task" and MetaICL's "noisy channel" §3.4.
• Superseded by: not stated in this paper.