Meta-learning

Overview

Meta-learning trains a model over many tasks, so that on a new task it can learn quickly from a few labelled examples. The shared setup is a distribution of tasks, each split into a training part and a test part. A base learner fits task parameters on the training part given an embedding, and the embedding is chosen to minimise the expected loss on the test part across tasks 2003.11539 §3.1 eq. 1 eq. 2. Evaluation averages the same loss over held-out tasks eq. 3.

Twelve cards list this slug, and they use the idea in different ways. The 2018 survey sorts the field by the meta-data a method reuses: past model evaluations, task meta-features, or prior models 1810.03548 §1. The position paper treats two learning speeds, a slow shared one and a fast one for the parts that change, as a meta-learning-like requirement, and runs no experiments 2011.15091 §3.7. The few-shot card asks whether the gains come from the meta-learning algorithm or from the representation it learns 2003.11539 §1. The few-shot survey treats meta-learning as one of three branches of single-modal few-shot learning, next to data augmentation and transfer learning 2205.06743 §2.6. The domain generalization survey splits training sources into meta-train and meta-test parts to simulate a shift between domains 2103.03097 §IV.C.2. The Fish paper uses the same inner-and-outer shape with training domains in place of tasks 2104.09937 alg. 1. The model editing paper trains a small editor network over many edit examples, and the card files it under this slug 2110.11309 §3.2. The anomaly detection paper adds one MAML update to a transformer detector, for use when labelled data is limited 2201.07284 §3.4. The personalized federated learning survey treats each client as a task and lists a MAML-style step among its personalization methods 2103.00710 §III-B. The AutoML paper reuses the meta-data of past datasets, not of past tasks in an episode, to warm-start a pipeline search and to pick its policy 2007.04074 §3.1 §4.1.1. The few-shot classifier that scores classes by feature-map reconstruction trains its backbone on episodes, and adds only three learned scalars on top of it 2012.01506 §3.2. The scene model that starts car object networks from a federated-learned prior is the last card that uses the term 2205.04334 §3.4. The variants below keep these uses apart.

Variants

Episodic meta-learning with a learned embedding

The embedding and often the adaptation step are trained over sampled episodes. The paper names MAML, Prototypical Networks, MetaOptNet, LEO and TADAM as the methods it compares against 2003.11539 tab. 1 tab. 2. Their published numbers come from different backbones, so the table compares published setups, not matched ones 2003.11539 tab. 1.

• MetaOptNet, ResNet-12, miniImageNet 5-way: 62.64 ± 0.61 (1-shot) and 78.63 ± 0.46 (5-shot) 2003.11539 tab. 1.
• MAML with a 32-32-32-32 backbone: 48.70 ± 1.84 (1-shot) and 63.11 ± 0.92 (5-shot) on miniImageNet 2003.11539 tab. 1.

The survey sorts this family into three kinds: learning an initialisation or a learning rule (MAML, Reptile and related methods), learning a metric (Siamese, prototype, matching and relational networks), and passing information between support and query nodes with a graph network 2205.06743 §5.1 §5.2 §5.3. It runs no experiments of its own. Its computer-vision tables copy author-reported accuracy from other papers, under mixed protocols, so the survey does not rank the methods 2205.06743 §7.1 tab. 9. Card: Song et al., 2022.

Episodic training with closed-form feature reconstruction (FRN)

The backbone is trained on episodes, but the class score is not a learned head. For each class, the support feature maps are pooled into one matrix. The query's spatial feature map is then rebuilt from that pool by ridge regression, and the class whose rebuild has the lowest squared error wins. The solve has a closed form, so no iterative solver runs at test time 2012.01506 §3.1 eq. 2 eq. 3 §3.7. The regularisation strength and a recalibration factor are learned, and so is a softmax temperature §3.2 eq. 6. The paper keeps spatial positions, which global average pooling drops, and avoids the location dependence that flattening introduces §1.

Cost is one d-by-d inversion per class per episode, with d the channel count §3.4. For mini-ImageNet and tiered-ImageNet, pre-training uses a non-episodic head with one learnable map per base class. The maps are discarded afterwards, and episodic fine-tuning follows §3.6 §4.2.

Author-reported, 5-way, ResNet-12:

• CUB, cropped: 83.16 (1-shot) and 92.59 (5-shot), against 80.80 and 91.19 for the authors' re-implementation of Deep Subspace Networks [arxiv:2012.01506 tab. 3].
• mini-ImageNet: 66.45 and 82.83, which is 0.3 points below FEAT at 1-shot tab. 7.
• Mini-ImageNet ablation: episodic training from scratch gives 63.03 and 78.01. Pre-training with FRN's head, then fine-tuning, gives 66.45 and 82.83. Plain average-pool pre-training then fine-tuning gives 62.13 and 76.28 tab. 9.
• Latency, 5-way mini-ImageNet, ResNet-12, meta-test: 63 and 79 ms per episode at 1-shot and 5-shot in the paper's own form, against 178 and 18,292 ms for DeepEMD with OpenCV. Hardware is not stated tab. 1.

Card: Wertheimer et al., 2020.

Plain pretraining with a per-episode linear probe

No episodic training. Pool all base-class data into one ordinary classification problem, train with cross-entropy, drop the classification layer, and fit a logistic regression on each support set 2003.11539 §3.2 eq. 4 eq. 5 eq. 6. The embedding is frozen at meta-test time §1 §3.2. This is the paper's "simple" baseline, and it is the same probe described in linear probing. Card: Tian et al., 2020.

• ResNet-12, miniImageNet 5-way: 62.02 ± 0.63 (1-shot) and 79.64 ± 0.44 (5-shot) 2003.11539 tab. 1.
• Against MetaOptNet in the same table, this is 0.6 points lower at 1-shot and about 1 point higher at 5-shot. It is also higher at 5-shot on tieredImageNet, CIFAR-FS and FC100 2003.11539 tab. 1 tab. 2. On CIFAR-FS 1-shot it is 1.1 points lower.

Plain pretraining plus self-distillation

Train a fresh network to imitate the previous one's soft predictions, for one or two generations, then freeze it and probe as above. Distillation is a separate step and the probe is unchanged 2003.11539 §3.3. See knowledge distillation.

• miniImageNet ResNet-12: 64.82 ± 0.60 (1-shot) and 82.14 ± 0.43 (5-shot) 2003.11539 tab. 1.
• Ablation: on miniImageNet 1-shot, adding distillation to the normalised probe raises accuracy from 62.02 to 64.82 2003.11539 tab. 4.

Meta-train and meta-test splits for domain generalization

In each step, the training sources are split into a meta-train set and a meta-test set, so the update simulates a shift between domains. The survey names MLDG, MetaReg and MetaVIB as examples and gives a bi-level formulation 2103.03097 §IV.C.2. Its benchmark has no meta-learning rows: the paper lists MLDG among nine methods, but its table has only seven, without MLDG or DDC 2103.03097 §VII.C. The cards therefore give no reported numbers for this variant. The survey's own notation is inconsistent: the update uses α as the outer step, while the text calls η and β the outer and inner rates 2103.03097 §IV.C.2. Use the MAML and MLDG papers for the exact rule. Card: Wang et al., 2021.

Fish: a first-order inner loop across training domains

Fish is a domain generalization method with a meta-learning-style outer step. Each meta-step clones the current weights, visits every training domain once in random order, and takes one plain gradient step per domain on the clone. The real weights then move a fraction ε of the way toward the clone 2104.09937 alg. 1. The authors derive the update as a first-order approximation of an objective that rewards gradients from different domains for agreeing. Exact gradient matching needs second derivatives. The alignment holds only in the limit of a small inner rate, and the paper gives no bound at the rates it uses 2104.09937 §3.4 app. A.

The inner loop differs from Reptile in which batches it uses. Reptile draws from one task, and Fish draws one batch from each domain [arxiv:2104.09937 §3.4] app. A.1.

Author-reported results:

• DomainBed, 7 datasets, average test accuracy: 67.1, against 67.0 for ERM and 67.2 for Coral. The paper ranks it second of ten by average [arxiv:2104.09937 §4.3] tab. 4.
• Wilds, main-text table: Fish is best on Poverty (0.80 Pearson r), Camelyon17 (74.7), FMoW (34.6 worst accuracy) and CivilComments (72.8 worst-group). On Amazon it is within 0.5 points of ERM (53.3 against 53.8). On iWildCam it is well behind: macro F1 22.0, against 31.0 for ERM and 32.8 for Coral 2104.09937 §4.2 tab. 3.
• The appendix gives different numbers for the same test sets. For example, Camelyon17 test accuracy is 79.5 there, against 74.7 in the main table 2104.09937 app. C.2 tab. 3. See Contradictions.
• With many domains the gain shrinks: 324 locations on iWildCam and 7,676 reviewers on Amazon, where ERM is level or ahead on the primary metric 2104.09937 §4.2.

The released code runs a different step from the one in the algorithm. It divides the outer rate by the number of domains sampled per inner loop, and it keeps the optimizer state from the ERM setup across inner loops 2104.09937 Paper vs code, items 1-2. The card's Pitfalls section lists these, so check them before reproducing the reported numbers. Card: Shi et al., 2021.

Client-as-task meta-learning in personalized federated learning

Each federated client plays the role of a task. The survey's Per-FedAvg takes one local gradient step from the shared parameters, w - α ∇f_c(w), and the outer step minimises the global loss of those adapted models 2103.00710 eq. 5 §III-B. The survey notes that the outer update needs second-order gradients through the inner step. It names a first-order variant (FO-MAML) and a Hessian-free variant (HF-MAML), but gives no approximation error for either 2103.00710 §III-B. pFedMe adds an L2 regulariser and ARUBA uses online learning for the meta step, with no formulas in the survey 2103.00710 §III-B. The survey runs no experiments, so no accuracy is reported for this variant. Card: Tan et al., 2021.

Meta-learned car prior for object networks in a driving scene

Not a few-shot classifier. Each tracked object in a driving scene gets its own small network. Car networks start from a shared initialisation, learned by federated averaging over renders of ShapeNet cars, with one shape per client. The authors say this equals Reptile 2205.04334 §3.4 alg. 1. Other object categories use a fixed density-bias starting point instead §3.4. The paper reports faster convergence after two epochs with the prior, and better results from sparse views. Both results come from ShapeNet tests app. B fig. 14 fig. 15. The driving-scene numbers, 74.28 mIoU and 21.91 PSNR on KITTI-360 tab. 2, are for the full model and do not isolate the prior. The card has no ablation of it on those scenes. Card: Kundu et al., 2022.

Learned editor networks for model editing

A small network is trained ahead of time, with one network per weight shape. At edit time it takes a layer's fine-tuning gradient, kept as two short vectors, and outputs a low-rank change to that layer's weights. One wrong input and its desired output then edit the model in one step 2110.11309 §3.1 alg. 2. Training uses an edit-success loss and a locality KL term, weighted 0.1 and 1 2110.11309 §3.2 alg. 1.

Author-reported large-model results, edit success (ES) and drawdown (DD) 2110.11309 §4.1:

• GPT-J 6B, Wikitext: MEND 0.88 ES / 0.031 DD; fine-tuning (FT) 0.80 / 0.125; KnowledgeEditor (KE) 0.01 / 0.068.
• T5-XXL 11B, zsRE: MEND 0.89 ES / under 0.001 DD; FT 0.87 / under 0.001.

Editable Neural Networks, a learned editor that is itself meta-learned, needs memory the paper calls too high at 10B+ parameters 2110.11309 app. B. The card flags a sign difference between the paper's update and the code. Check the card's Paper vs code section before reimplementing. Card: Mitchell et al., 2021.

One MAML update inside a reconstruction-based detector

Not a few-shot setting. A transformer detector for multivariate time series reconstructs short windows of sensor readings. After each pass over the training windows, it takes one MAML step: an inner update on a random batch, then an outer update that uses the loss at the inner-updated parameters 2201.07284 §3.4. The authors present the step as a help when data is limited 2201.07284 §3.4. Their ablation without MAML changes average F1 by about 1% and F1* by about 12% 2201.07284 §5.1. The paper's released code has no inner or outer MAML update, so the step cannot be checked against code. Read the card's Paper vs code section before reusing it Tuli et al., 2022.

Taxonomy of reused meta-data (survey, no method)

The survey gives one shared formulation. A meta-learner L is trained on prior evaluations P and on the new task's evaluations P_new to recommend configurations for a new task 1810.03548 §2. It then groups the field by what is reused 1810.03548 §1:

• Past evaluations: task-independent rankings, configuration-space design, configuration transfer and learning curves 1810.03548 §2.1 §2.2 §2.3 §2.4.
• Task meta-features, a hand-designed or learned vector per task, used for warm starts and meta-models 1810.03548 §3.1 §3.2 §3.3 §3.4.
• Prior models: transfer learning, learned update rules, and few-shot initialisations such as MAML and Reptile 1810.03548 §4.1 §4.2 §4.3.

The author states that meta-learning gives no benefit when a new task is unrelated to earlier ones, so task similarity is the central assumption 1810.03548 §1. The survey reports no new experiments. Its study sizes are copied from cited work and are not reproduced here 1810.03548 §2.2 §3.3. Card: Vanschoren, 2018.

Portfolio warm start and a learned policy selector (AutoML)

Here the tasks are datasets and the learner is an AutoML system that must find a good pipeline inside a time budget. Offline, the authors build a greedy portfolio of pipelines from 208 meta-training datasets, scoring each candidate on every one 2007.04074 §3.1 alg. 1 §3.4.1. On a new dataset, the portfolio members are tried first, and Bayesian optimization continues from there §3.1 §3.2.1. Budgets are spent by successive halving, which drops weak pipelines early §3.2.1.

A second, learned step picks one of eight search policies for each dataset. A random forest is trained for each pair of policies, and its input is only two meta-features, the number of datapoints and the number of features 2007.04074 §4.1.1. If no meta-dataset is at least as large on both, the paper falls back to a fixed policy, holdout with successive halving §4.1.1. The card's Paper vs code section records that the released scripts return weights over all eight policies rather than one choice, so check it before reimplementing.

Author-reported on 39 AutoML benchmark datasets: mean normalized balanced error of 3.58 at 10 minutes and 2.47 at 60 minutes, against 16.21 and 7.17 for the previous Auto-sklearn version 2007.04074 tab. 5. The authors also report a single fixed policy with the portfolio at 3.69 and 2.44, and without the portfolio the selector at 5.63 and 3.42 and the single policy at 5.37 and 3.61 tab. 8. The selector is therefore not ahead in every setting. Card: Feurer et al., 2020.

Two learning speeds (position, no experiments)

The authors argue that changes in the world should be handled by a slow, shared learner and a fast, adaptable part, so the fast part is the only one that needs adapting. They say this calls for at least two learning speeds, as in the outer and inner loops of meta-learning 2011.15091 §3.7 §2. The paper calls this and its other biases hypotheses, and it gives no training procedure or numbers 2011.15091 §1. The one simulation it discusses comes from cited work 2011.15091 §4.3. Card: Goyal and Bengio, 2020.

When to use which

Latency, memory and implementation effort are not compared in the few-shot paper, so this page makes no claim about them 2003.11539 §4.1.

Implementation sketch

Pseudocode in our own words, restating the framework in the paper's notation 2003.11539 §3.1. The paper does not implement this loop; it is the setup its baseline is compared against.

# Meta-training over episodes sampled from the base classes [eq. 2]
phi = init_embedding()
for step in range(num_steps):
    # One N-way task: a training split and a test split [§3.1]
    task = sample_task(meta_train_classes)
    # Base learner fits task parameters on the training split, phi fixed [eq. 1]
    theta = base_learner(task.train, phi)
    # Test-split loss over those parameters updates the embedding [eq. 2]
    loss = query_loss(task.test, theta, phi)
    phi = phi - lr * grad(loss, phi)

# Meta-test: same base learner on unseen classes, phi frozen [eq. 3]
for task in held_out_tasks:
    theta = base_learner(task.train, phi)
    accuracy = mean_accuracy(task.test, theta, phi)

Notes:

• The base learner is left open. The paper's baseline uses a logistic regression as the base learner and no episodic loop 2003.11539 §3.2.
• Whether the outer loop must be episodic is the question the paper tests.
• The domain-generalization split uses the same outer shape, with source domains in place of classes: meta-train domains as the training part, a held-out source as the test part 2103.03097 §IV.C.2.
• The federated survey uses it with clients in place of classes: one local gradient step on a client's data, then an outer update on the adapted models' global loss 2103.00710 eq. 5 §III-B.
• The anomaly detector's MAML step has the same inner and outer shape, applied to a reconstruction loss after each pass over the windows 2201.07284 §3.4.
• Fish keeps the shape and changes the unit: the inner loop visits each training domain once on a clone, then the outer step moves the weights a fraction ε toward the clone 2104.09937 alg. 1.
• The reconstruction classifier keeps the episodic outer loop, but replaces the base learner with a closed-form ridge fit per class. It has no inner gradient steps at test time 2012.01506 eq. 9.
• The car prior is a federated average of client updates, one per shape. The paper says this is equivalent to Reptile's update 2205.04334 §3.4 alg. 1.

Contradictions

Between papers:

• The few-shot survey says meta-learning beats transfer learning in standard few-shot learning, and that this reverses in cross-domain few-shot learning 2205.06743 §4.3. The text gives no table or citation for this. Tian et al. report that plain pretraining with a linear probe reaches or beats the meta-learning baselines in their table on miniImageNet 2003.11539 tab. 1. The survey's "transfer learning" and the probe are not the same method, so the cards do not settle which claim holds.
• Pooled pre-training: Tian et al. report that a linear probe on a plain pooled embedding matches or beats the meta-learning baselines 2003.11539 tab. 1. FRN's ablation reports that average-pool pre-training followed by episodic fine-tuning does worse than episodic training from scratch, 62.13 against 63.03 at 1-shot [arxiv:2012.01506 tab. 9]. The protocols differ: one uses a frozen probe and the other a fine-tuned network. The cards do not say which result carries over to your setting.
• The other cards study different problems, and their numbers are not comparable.

Within a paper:

• TranAD's MAML step is part of its method 2201.07284 §3.4 and is ablated 2201.07284 §5.1. The card's Paper vs code section finds no such step in the released code. This is a gap between paper and code, not a disagreement between papers.

• MEND's text reports 96% edit success and under 1% accuracy loss at 25 simultaneous edits. Its batched-edit table reports 0.89 and 0.011 2110.11309 §4.3. The card treats the table as the reported result.

• MEND claims to be the only approach that edits models above 10B parameters 2110.11309 §1. On T5-XXL the table gives FT 0.87 edit success against MEND's 0.89 2110.11309 §4.1.

• The domain-generalization survey's meta-learning update rule uses a different symbol for the outer step than its text does 2103.03097 §IV.C.2.

• Fish's Wilds numbers disagree between the main text and the appendix for the same test sets: Camelyon17 74.7 in the main table against 79.5 in the appendix, and Amazon 53.3 against 56.0 2104.09937 tab. 3 app. C.2 app. C.6. The paper does not explain the gap. The card's Pitfalls section records the others. Pick one source and say which you used.

• The AutoML paper's selector is not the best in every cell. Single best beats it with the portfolio at 60 minutes (2.44 against 2.47) and without the portfolio at 10 minutes (5.37 against 5.63) tab. 8. Compare the selector with the single-policy row for your own horizon before choosing it.

Papers

• Shi et al., 2021: Fish, a first-order approximation of inter-domain gradient matching that runs one plain gradient step per training domain on a clone of the model, then moves the weights toward it; reports DomainBed and Wilds results.
• Tian et al., 2020: compares a plain cross-entropy embedding with a per-episode linear probe against meta-learning baselines on miniImageNet, tieredImageNet, CIFAR-FS, FC100 and Meta-Dataset, and adds born-again self-distillation.
• Tan et al., 2021: personalized federated learning survey that places a Per-FedAvg-style meta-learning step under global-model personalization, with no experiments of its own.
• Wang et al., 2021: domain generalization survey that lists meta-train/meta-test splits as one learning strategy, and benchmarks seven methods on PACS and Office-Home without a meta-learning entry.
• Mitchell et al., 2021: MEND, a learned editor network that maps a layer's fine-tuning gradient to a low-rank weight edit for one-step model editing.
• Song et al., 2022: few-shot learning survey that files meta-learning under a challenge-based taxonomy, covering MAML-style initialisation, metric learning and graph-based methods, with author-reported tables copied from other papers.
• Vanschoren, 2018: survey of meta-learning organised by the meta-data reused (past evaluations, meta-features, prior models), with no experiments of its own.
• Goyal and Bengio, 2020: position paper on inductive biases for higher-level cognition, including a two-speed learning requirement that resembles meta-learning's inner and outer loops; no method or results.
• Feurer et al., 2020: Auto-sklearn 2.0, which warm-starts a pipeline search from a greedy portfolio built on past datasets, runs successive halving, and picks a search policy with a random-forest selector on two dataset meta-features.
• Tuli et al., 2022: TranAD, a transformer anomaly detector for multivariate time series that adds one MAML update per pass over the training windows.
• Wertheimer et al., 2020: FRN, an episodic few-shot classifier that scores classes by how well a ridge fit from each class's pooled support maps rebuilds the query's spatial feature map; reports fine-grained and mini-ImageNet results.
• Kundu et al., 2022: Panoptic Neural Fields, a driving-scene model with one small network per tracked object and a federated-learned car initialisation; reports KITTI-360 and KITTI results.