Paper · 2021-04-10 · cs.CL, cs.AI
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Meta-tuning fine-tunes a pre-trained T5 on 43 classification datasets, each rewritten as Yes/No questions with 441 hand-written label descriptions, then tests zero-shot on tasks that are not similar to the training ones. Meta-tuned T5-Large beats same-size UnifiedQA by 3.3% average AUC-ROC; going from 220M to 770M parameters adds 6.3%.
Zero-shot learningInstruction tuningText classificationzero-shot-classificationmeta-tuningquestion-answering-format
arXivPDFCodeMarkdown10 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Language models learn by guessing the next word, which is only loosely related to the job of answering a prompt. This paper closes that gap by training the model directly on prompts. The authors gathered many classification tasks, such as sentiment, topic and spam detection, and rewrote each label as a yes-or-no question, for example "Does the user like this movie?". They then trained one model to answer these questions across all the tasks. To check whether it learned something general, they held out whole families of similar tasks and tested on those. The trained model beats a model of the same size that was only built for question answering, and bigger models do better still. The authors suggest that the zero-shot ability of large language models may have been underestimated.
TL;DR
Meta-tuning fine-tunes a pre-trained T5 on binary Yes/No classification questions built from 43 datasets, with 441 hand-written label questions §1 §2. It is evaluated zero-shot on dataset groups that were left out of training, scored by per-question AUC-ROC §3 §4. The author-reported gain over same-size UnifiedQA is +3.3% average AUC-ROC, and +6.3% from T5-Base to T5-Large tab. 1. Use it as a reference for building a zero-shot classifier from a fixed training mix, but note that the training data is distributed separately from the code and the paper's results depend on how that mix was built.
Problem
Zero-shot classification labels text using only label descriptions and no labelled examples §1. Large language models trained to predict the next word can be prompted this way, for instance by asking whether the next word after a review and a question is "Yes" or "No" §1. The paper argues that this training objective is correlated with, but not aligned to, answering prompts §1. Earlier work often treated a task as unseen only when its label or its dataset was absent from training. This paper also excludes training datasets that are similar to the test dataset, judged by property tags, which makes the test harsher §1 §2. The authors expect that larger models will do better still, and that zero-shot ability measured out of the box may underestimate what these models can do §1 §6.
Method
Data §2. Each classification dataset is converted to a Yes/No question answering format, with the label written into the question. Each label gets one to three questions. Null labels are skipped. Datasets with thousands of labels are turned into templated questions and excluded from evaluation §2. The authors collected 43 English sources from Kaggle, HuggingFace, SemEval and other papers §1 §2. Three authors wrote 441 questions for 204 unique labels, and each question was proofread by at least one other author §2. Noisy or overly complex datasets, such as Multi-NLI, are used only for training §2.
Similarity groups §2 app. C. Each dataset gets a set of property tags, for example social media, emotion, topic classification, review or questions. Datasets with the same tag set form a group. Training and testing never cross a group. Datasets with no tags are treated as different tasks and can always be used for training app. C. A table of the most relevant training dataset for each evaluation dataset is given as a robustness check app. F.
Input format §4. The input follows UnifiedQA: the question and the context are concatenated with a separator token. For T5, the score is the normalised probability of "Yes" versus "No" at the first decoded token §4. For BERT and RoBERTa, the paper adds a classification layer on the [CLS] or pooled output §4 app. E.3.
Meta-training distribution §4. Each training example is produced in three steps: pick a dataset uniformly at random, pick one of its label questions uniformly at random, and with probability 0.5 pick a text whose answer is "Yes". The same question and text pair is never used twice.
Training §4. Batch size 32, 5000 steps, no hyperparameter tuning. Full experiments took about 250 GPU hours on Quadro 8000 cards §4.
Evaluation §3 §5. For each label question, AUC-ROC is computed with the "Yes" answer as the positive class. F-score and accuracy are avoided because they depend on the decision threshold §3. For each held-out group, the model is trained on all other groups and then tested on the held-out datasets §4. Two models are compared with four descriptive statistics over the per-question change in AUC-ROC: its mean, the fraction of questions that gain more than t, the fraction that lose more than t, and its standard deviation §3. A model is called better only when the mean change is positive and gains exceed losses for t = 1%, 5% and 10%, under three weightings (per question, per label, per dataset) §3 app. E.
# Build the model input [§4] [code: model.py#L15-L16]
def make_input(question, context):
# Question first, then the text, joined by a separator [§4]
text = question + SEPARATOR + context
# Code lowercases, drops outer apostrophes [code: model.py#L11-L12]
return normalise(text)# Score one question and text [§4] [code: model.py#L43-L47]
def yes_no_log_probs(model, question, context):
# Encode; code cuts at 128 tokens (no length in paper) [code: model.py#L21]
tokens = tokenize(make_input(question, context), max_length=128)
# Run the encoder and take one decoder step from the start token [§4]
logits = model.first_decoder_logits(tokens)
# Keep only the "no" and "yes" logits and normalise them over the two [§4]
return log_softmax(logits[[NO_ID, YES_ID]])# Meta-tune a pre-trained T5 for 5000 steps [§4] [code: default_train.py#L48]
# Code runs 5001 updates, not 5000 [code: default_train.py#L32]
def meta_tune(model, pool, steps=5000, batch_size=32):
# Shuffle the whole pool once [code: default_train.py#L20]
pool = shuffle(pool)
# AdamW, learning rate 1e-5, weight decay 0.01 [code: default_train.py#L28]
optimizer = AdamW(model.parameters(), lr=1e-5, weight_decay=0.01)
# Linear warm-up over 100 steps, then decay [code: default_train.py#L29]
scheduler = warmup_then_decay(optimizer, warmup=100, total=steps)
for step in range(steps):
batch = next_batch(pool, step, batch_size)
# Negative log-likelihood of gold answer [code: default_train.py#L30]
loss = nll(yes_no_log_probs(model, batch), gold_answers(batch))
loss.backward()
optimizer.step()
scheduler.step()
optimizer.zero_grad()
return model# Draw the next batch [§4] [code: default_train.py#L33]
def next_batch(pool, step, batch_size):
# Paper: uniform dataset, uniform question, 50% Yes, no repeated pairs [§4]
# Code: one shuffled pool, read in order [code: default_train.py#L33]
first = step * batch_size
return [pool[j % len(pool)] for j in range(first, first + batch_size)]Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Initialisation | T5-Large, 770M parameters | §4 | Also T5-Base 220M §5.1 and T5-small 60M, BERT 41M to 340M app. E |
| Initialisation (QA) | UnifiedQA, T5-Large size | §4 §5.1 | Released training script hard-codes T5 (see Paper vs code) default_train.py |
| Meta-tuning steps | 5000 | §4 | Code runs 5001 updates default_train.py |
| Batch size | 32 | §4 default_train.py | |
| Optimiser | AdamW | not stated in paper | default_train.py |
| Learning rate | 1e-5 | not stated in paper | default_train.py |
| Weight decay | 0.01 (0 for bias and LayerNorm weights) | not stated in paper | default_train.py |
| LR schedule | 100 warm-up steps, then linear decay to 0 | not stated in paper | default_train.py |
| Max input length | 128 tokens, truncated | not stated in paper | model.py model.py |
| Loss | NLL of the log-softmax over "no" and "yes" | not stated in paper | model.py default_train.py |
| Random seed | not set in code | not stated in paper | README says results vary with random seeds README.md |
| Dropout | not stated | not stated in paper | No train() call in code default_train.py |
| Question annotation | 1 to 3 per label, 441 total | §2 | |
| Similarity groups | by tag set | §2 app. C | |
| Compute | about 250 GPU hours, Quadro 8000 | §4 | |
| Long-run ablation | 100K steps (20 times default) | §5.1 | Not in released code |
Results
Author-reported. Setup: T5 models, UnifiedQA baseline of the same size, 20 or so held-out datasets, label-question weighting §5.1 app. E.
| Comparison | E[Δ] | P[Δ>1%] | P[Δ<-1%] | Std(Δ) | Source |
|---|---|---|---|---|---|
| Meta-tuned T5-Large vs UnifiedQA (T5-Large) | 3.3% | 59.5% | 28.1% | 9.5% | tab. 1 |
| T5-Large vs T5-Base (both meta-tuned) | 6.3% | 75.1% | 15.1% | 8.1% | tab. 1 |
| Pre-trained vs random initialisation | 23.8% | 95.7% | 3.2% | 14.0% | tab. 1 |
| Training on similar datasets | 0.7% | 43.8% | 20.5% | 3.2% | tab. 1 |
| Ensembling label questions | 0.7% | 28.9% | 16.8% | 3.1% | tab. 1 |
| Initialising from UnifiedQA | 1.1% | 54.1% | 24.3% | 6.9% | tab. 1 |
Benchmark of Yin et al. (2019), best prior system vs a meta-tuned RoBERTa. Metrics are label-weighted F1 for emotion and situation, accuracy for topic §5.3:
| Model | Emotion | Situation | Topic | Source |
|---|---|---|---|---|
| Prior best (NLI ensemble, RoBERTa-Large based) | 25.2 | 38.0 | 52.1 | tab. 2 |
| Meta-tuned RoBERTa | 28.2 | 48.4 | 54.3 | tab. 2 |
Other results reported by the paper:
- Larger is better across T5-small, T5-Base and T5-Large and across BERT sizes app. E.2 app. E.3.
- The mean gain for meta-tuning over UnifiedQA is 1.2% when datasets are weighted equally, against 3.3% with label-question weighting app. E tab. 1.
- Ensembling label questions is the one row that fails the paper's own "better" rule under dataset weighting: 18.6% of datasets gain more than 1%, against 20.2% that lose more than 1% app. E.
- Performance falls after 100K steps by about 3% AUC-ROC on the three benchmark test sets. The authors call this not fatal §5.1 fig. 7.
- Accuracy is reported only for completeness and is marked uninformative by the authors app. H.
Paper vs code
Official repository: ruiqi-zhong/Meta-tuning, linked from the paper's introduction.
Its owner is the first author, so it qualifies. It has no LICENSE file, so no code
is quoted verbatim. Commit 91d2492. Searched the paper's links only; no other
repository was used. Code citations use the repository's file paths.
- Sampling. The paper draws a dataset, then a question, then Yes or No at 50%, and never repeats a pair §4. The code shuffles one flat pool of all datapoints once, and then reads it in order with wrap-around default_train.py default_train.py. There is no per-dataset draw, no 50% control and no duplicate check in the released code. The data files are a separate Google Drive download README.md; their balance cannot be checked from this repository.
- Step count. The paper says 5000 steps §4. The loop runs 5001 updates and saves at index 5000 default_train.py default_train.py default_train.py.
- Initialisation from UnifiedQA. The paper reports it §5.1. The released
training script hard-codes
t5-{size}default_train.py. The UnifiedQA path appears only in the baseline script baseline.py. - Encoder-model training. The paper describes meta-tuning BERT and RoBERTa §4 app. E.3. The repository contains no training script for them. Its demo loads a RoBERTa-base checkpoint through the library's default classification head demo.py demo.py. The README mentions a RoBERTa-large checkpoint README.md. The paper's head is described as an MLP in the main text §4 and as a linear layer in the appendix app. E.3.
- Input separator. The paper says a [SEP] token §4. The code joins the question and text with the two literal characters backslash and n model.py. The code also lowercases the text model.py. Neither is mentioned in the paper.
- Training mode. The training loop never calls
train(). Evaluation callseval()model.py. The model is loaded withfrom_pretrained, which returns it in eval mode. Inferred from reading the code, not run: meta-tuning probably runs without dropout. - Seeds and ensembling. No seed is set in the training script default_train.py. The paper's ensembling of label questions and its 100K-step run have no code in the repository.
- Metric. AUC-ROC on the Yes logit alone matches the paper evaluate_and_plot.py §3.
- Groups. Leave-one-group-out matches the paper default_train.py §4.
- Benchmark label resolution. Not in the released code; the paper's rule (highest Yes probability when several labels say Yes) §5.3 is not implemented here.
Pitfalls
- Normalisation mangles text. The regex deletes the first and last apostrophe whenever a text has at least two model.py. Reading the regex, not running it: "I don't like it's" would become "i dont like its". Reimplementations must copy this step, or scores will not match.
- Silent truncation. Inputs longer than 128 tokens are cut from the right, which removes the end of the text. The paper gives no length limit and does not say how many inputs were cut model.py model.py.
- Balance is not reproducible from code. Because batches come from a flat pool default_train.py, the dataset and label balance depends on the Drive data. Check it before comparing numbers.
- Average gains hide losses. Per-dataset AUC for the 770M meta-tuned model is below UnifiedQA on IMDB (90.5% vs 92.9%), SemEval 2019 Task 8 question type (59.3% vs 75.1%), definition detection (63.9% vs 72.8%), SUBJ (58.7% vs 62.8%), and spam (35.0% vs 57.2%). Spam is below the 50% chance line app. G.
- Weighting matters. Gains over UnifiedQA shrink from 3.3% to 1.2% when datasets are weighted equally app. E tab. 1.
- Prompt wording. Two near-synonymous questions for the same label can differ a lot in AUC. The authors suspect abstract concepts such as "atheism" are poorly grounded §6. The questions were written by three computer science students and may not represent real users §6.
- Contamination. T5 and UnifiedQA pretraining may have seen the datasets §5.2. The BERT size comparison is offered as partial evidence against memorisation app. E.3.
- Released checkpoints are not zero-shot. The released models were trained on every dataset in the paper, and the README asks readers not to compare them with zero-shot systems README.md.
- Benchmark comparison is not like-for-like. The prior system uses an NLI ensemble and auxiliary objectives §5.3. Several test sets were subsampled for speed, so numbers are not directly comparable with other papers app. H README.md.
- Step-count analysis uses test sets. The 100K-step run is evaluated on the three benchmark test sets §5.1. The 5000-step default is not tuned §4.
- Scope. Only binary, single-label classification was tested. Multi-label, few-shot and generation tasks are left as future work §6.
Lineage
- Builds on: UnifiedQA, the question-answering model used both as baseline and as an initialisation, and T5 as the base encoder-decoder §4.
- Compared against: UnifiedQA of the same size §5.1; the Yin et al. (2019) NLI-based zero-shot system §5.3.
- Related but different: multi-task and meta-learning. The model never sees examples of the target task, which is what separates it from those settings §6.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections” by Ruiqi Zhong, Kristy Lee, Zheng Zhang, Dan Klein (arXiv:2104.04670), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ruiqi-zhong/Meta-tuning at commit 91d2492be710 (none (no LICENSE file in repository)). Any quoted code is shown under that license.