CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

In plain words

Language models can learn a new text task from a few dozen labelled examples, but each new task usually starts from scratch. The authors ask whether a model that first practises on many other language tasks gets better at picking up unfamiliar ones. To test this they built a benchmark of 160 small tasks, all rewritten as "read text, write text" problems, and split them into tasks used for practice and tasks held back for testing. They compared several ways of practising. Practising on other tasks usually helped the held-back ones, but not for every task, and the choice of practice tasks mattered. The benchmark exists so that this kind of transfer can be measured.

TL;DR

• The CrossFit setup has an upstream stage on seen tasks, then a few-shot fine-tuning stage on unseen tasks §3.2.
• The NLP Few-shot Gym holds 160 English tasks in one text-to-text format, sampled with 16 examples per class (classification and regression) or 32 examples (other tasks) §4.2§4.3.
• Author-reported: every upstream method has a positive ARG on the random partition, and multi-task learning is strongest there (35.06%) tab. 1.
• Upstream learning helped on average, but 12.60% of test-task results were worse by more than 5% §6.
• Relevant if you are deciding whether to add a multi-task or meta-learning stage before per-task few-shot fine-tuning. The paper does not give the upstream recipe, so expect to tune it yourself.

Problem

Few-shot NLP work mostly improves generalisation within one task, usually with task-specific templates. Performance on each task is the usual success measure §3.1. The authors ask whether a model can also gain a general ability to learn new tasks from a set of earlier tasks, and how to measure that. They standardise the seen/unseen partitions, the data each stage may access, and the evaluation protocol so that such methods can be compared §1§3.2.

Method

Setup and metric

A task is a triple of train, dev and test sets of text-to-text examples §3.1. The repository of tasks is split into three disjoint sets: training tasks, development tasks and test tasks §3.2.

• Upstream stage. The method sees the train and dev data of the training tasks. It may use the development tasks only for validation. Test-task data is not available here §3.2.
• Few-shot stage. Each test task is fine-tuned on its own few-shot train set. Its dev set tunes fine-tuning hyperparameters, and the test set gives the score §3.2§5.
• ARG (Average Relative Gain). For each test task, take the relative change in score from direct fine-tuning to the method's score, then average over the test tasks. This is comparable across the seven metrics used §3.2. Those metrics are classification F1, accuracy, QA F1, exact match, ROUGE-L, Matthews correlation and Pearson correlation §3.2.

# Average Relative Gain over the test tasks  [§3.2]
# before: score of direct fine-tuning; after: score after upstream learning
def average_relative_gain(before, after, test_tasks):
    # Relative change per test task, whatever its metric  [§3.2]
    gains = []
    for task in test_tasks:
        change = after[task] - before[task]
        gains.append(change / before[task])
    # Plain mean over test tasks  [§3.2]
    return sum(gains) / len(gains)

Benchmark construction

• Candidates come from the Hugging Face Datasets library (626 datasets at the time). The authors keep English monolingual datasets, and drop ones that need retrieval, sequence labelling such as NER or parsing, or very long documents §4.1.
• Each dataset is converted to text-to-text form, following the T5 convention. Natural language inference becomes a premise-and-hypothesis input with a one-word label. Reading comprehension becomes a question-and-context input with the answer span as output. QA formats also follow UnifiedQA §4.2.
• Few-shot sampling follows an earlier prompt-based paper. Classification and regression tasks get 16 training examples per class, other tasks get 32. The dev set has the same size as the train set §4.3.
• Five random samples are drawn per dataset from its original train split, so the effective size is 800 few-shot tasks (160 × 5). The paper reports 160 throughout §4.3.
• The test set is the original dev split, or 20% held out when no dev split exists. It is sampled once, before the train and dev samples §4.3.

Task partitions

The 160 tasks are classified into a two-level ontology: classification, question answering, conditional generation and others §4.4fig. 2. Eight partitions are defined tab. 1fig. 3:

• Random (Partition 1): 120 training, 20 dev, 20 test tasks, drawn at random §4.4.
• Partitions 2.1 to 2.3: 10 classification tasks for dev and 10 for test. Training tasks are all classification (2.1), half classification and half not (2.2), or all non-classification (2.3) §4.4fig. 5.
• Partitions 3.1 and 3.2: held out by sub-category. 3.1 holds out NLI (57 non-NLI classification training tasks, 8 NLI test tasks). 3.2 holds out paraphrase identification (61 training, 4 test) §4.4tab. 1.
• Partitions 4.1 and 4.2: held out by QA sub-category. 4.1 holds out machine reading comprehension (42 training, 9 test). 4.2 holds out multiple-choice QA (29 training, 22 test) §4.4tab. 1.

Upstream methods

• Direct fine-tuning. The baseline. It skips the upstream stage, and every other method's ARG is measured against it §5.
• Multi-task learning (MTL). Pool the train and dev data of all training tasks, train one model on it, then fine-tune that checkpoint on each test task. The dev data is used as training data in the upstream stage §5.
• MAML. Each training task gives a support batch from its train split and a query batch from its dev split. The paper uses one inner gradient step and one outer step §5.
• First-order MAML. It drops the second-order term and takes the gradient at the adapted weights §5.
• Reptile. It makes several inner updates on a task, then moves the initial weights a fraction of the way toward the adapted weights §5. The paper does not state the inner rule, the number of steps or the data used.

# MAML upstream update for one training task  [§5]
# Hyperparameters: alpha (inner rate), beta (outer rate)  [§5]

def maml_update(theta, task):
    # Support batch from the task's train split  [§5]
    support_batch = sample_batch(task.train)
    # Query batch from the same task's dev split  [§5]
    query_batch = sample_batch(task.dev)
    # Inner loop: one gradient step on the support batch  [§5]
    adapted = theta - alpha * grad(loss(theta, support_batch))
    # Outer loop: query loss at the adapted weights  [§5]
    # Plain MAML differentiates through the inner step (second order)  [§5]
    outer_grad = grad_wrt_theta(loss(adapted, query_batch))
    theta = theta - beta * outer_grad
    return theta

Hyperparameters

Values in the paper are cited to the paper. Values marked code are taken from the official repository and are not in the paper.

Results

Author-reported. Setup: NLP Few-shot Gym, BART-Base, five few-shot samples per dataset, direct fine-tuning as the baseline §4§5app. B.

ARG per partition and upstream method tab. 1:

Across all methods and partitions, 51.47% of test tasks improved by more than 5% relative to direct fine-tuning, 35.93% stayed within ±5%, and 12.60% dropped by more than 5% §6 Q1. Per-task results are plotted in fig. 4.

Controlled comparison, same test tasks (ten classification tasks) with different training-task mixes fig. 5: all-classification and all-non-classification upstream sets gave comparable ARG §6 Q2.

Per-task gains for test tasks that appear in two partitions tab. 2:

Other findings the authors report §6app. A:

• Upstream data size: making the upstream data 2×, 4× and 8× larger left the overall ARG close, with about 4% more at 8× §6 Q3. Held-out-Paraphrase with multi-task learning was used for this test fig. 6.
• Larger gaps for medium-resource targets: with up to 1024 shots the upstream gain was still clear in §6, and the gap closed with millions of examples. Appendix A reports the gain up to 2048 shots, and for CommonsenseQA also at full size §6 Q4app. Afig. 7.
• Model size: with T5-v1.1-Base and BART-Large on the Held-out-Paraphrase partition, few-shot scores generally grew with size. Upstream learning helped in almost all cases, except FoMAML on T5-v1.1-Base, which the authors saw as unstable app. Afig. 8.
• Pattern-exploiting training (PET) applied to this encoder-decoder setup gave lower scores on average than direct fine-tuning, with or without upstream learning §6 Q5app. Afig. 9.

Paper vs code

The official repository is INK-USC/CrossFit, linked from the abstract's footnote. The footnote link is enough to tie it to the authors, so no other search was needed. It has no LICENSE file, so code_license is not stated and no code is quoted here.

• Upstream recipes are in the code, not the paper. The paper gives no learning rate, step count or batch size for upstream learning §5app. B. The README multi-task command uses 16980 steps, 1018 warmup steps, batch 32 and 10 epochs README.md. The multi-task example script uses 17450 steps and 1047 warmup steps instead upstream_multitask.sh. The README MAML command uses learning rate 1e-5, 6000 steps, 360 warmup steps, batch 1, 4-step gradient accumulation and 40 epochs README.md.
• FoMAML and Reptile are missing from the code. The paper reports both §5tab. 1. The README says first-order methods are still in progress README.md. Only one MAML variant is in run_maml.py.
• MAML is second-order in the code. The inner step is kept inside a differentiable inner loop, and the query loss is back-propagated through it run_maml.py. This matches the paper's description of plain MAML §5.
• Dev data is used differently. The paper says the dev tasks may tune the upstream method §3.2. The MAML script never evaluates on them: dev_data is set to None run_maml.py, and only last-model.pt is saved run_maml.py. The multi-task script evaluates on dev tasks only every eval_period steps, and that default is 10,000,000 cli_multitask.py. The README command does not set it, so no best checkpoint is saved in practice run_multitask.py.
• Multi-task data matches the paper. The multi-task loader pools train and dev examples of the training tasks, as the paper's footnote says run_multitask.py§5.
• Support and query batches. Each MAML item is one task. Support and query sets are sampled with replacement when a task has fewer examples than the batch size utils.py. The paper does not state this.
• Loss. Training uses label-smoothed cross-entropy with ε = 0.1 bart.py. The paper does not mention label smoothing.
• Few-shot fine-tuning keeps the best dev-scoring weights seen during the run, and evaluates test only on those run_singletask.pyrun_singletask.py. The paper says dev is checked every 100 steps, but not that the best checkpoint is kept.
• Classification F1 is macro-averaged metrics.py.
• ARG is not computed in the code. The released collector script reports per-task mean and standard deviation only collect_results.py. Any ARG must be computed from its output.
• Task partitions. The eight split files in the repository are named by partition . The random split has 120, 20 and 20 tasks, matching tab. 1.

Pitfalls

• Upstream settings are not in the paper. A reimplementation has to take learning rates, step counts, batch sizes and optimiser settings from the code, which differs between the README and the example script (see above).
• The MAML README command is probably under-trained. The random split has 120 training tasks tab. 1. Under the code's loop, 40 epochs of 120 tasks at batch size 1 with 4-step accumulation gives about 1200 optimiser steps (derived from code, not run). The schedule decays to total_steps = 6000, so the learning rate stays high at the end. The README's claim that the schedule reaches zero does not hold for this command.
• Known gaps between the two multi-task commands. The step counts differ (16980 in the README, 17450 in the script), and both assume about 6% warmup README.md.
• MAML is sensitive. The authors attribute MAML's weaker results partly to a single inner step and untuned hyperparameters and seeds §6. Plan for a search over the inner and outer rates. Neither rate is in the paper.
• Tuning effort. The authors say they did not tune the meta-learning methods exhaustively, and that results are sensitive to hyperparameters and seeds §6. Fine-tuning picks the learning rate and batch size on each test task's dev set, then reports test app. B. State which rule you used if you compare.
• Seeds and variance. Table 1 gives one ARG value per cell, with no spread. Five samples per dataset reduce few-shot variance, but seed variance for upstream training is not reported §4.3tab. 1.
• Internal inconsistencies. The paper gives T5-v1.1-Base as 248M parameters in §6 and Appendix A, and as 246M in the reproducibility appendix app. B. The text on how far upstream gains last differs between §6 (1024 shots, vanish at millions) and Appendix A (2048 shots, and for CommonsenseQA at full size).
• Δ meta column. tab. 2 labels the meta-learning column Δ_meta without saying which meta-learner it is.
• Ontology is a judgement call. The authors say their format and goal ontology is flawed as a measure of task similarity, and that the Q2 result suggests format is not what drives transfer §4.4§6 Q2.
• Data licences. The benchmark is built from third-party datasets, each with its own licence. The repository disclaims rights and says users must check each dataset's licence README.md. Check per task before reuse.
• Old dependencies. The code pins torch 1.1.0, higher 0.2.1 and an old transformers commit, and the README warns that newer versions need changes README.md.

Lineage

• Builds on: MAML and Reptile for meta-learning upstream §5; the T5 text-to-text format §4.2; BART-Base as the main model §5.
• Few-shot sampling and hyperparameters follow an earlier prompt-based few-shot paper §4.3app. B.
• Compared against: direct fine-tuning (baseline), multi-task learning, MAML, first-order MAML and Reptile §5. Pattern-exploiting training was tested as an add-on app. A.
• Related: transfer between one intermediate task and a target task §1.