Meta-learning approaches for few-shot learning: A survey of recent advances

In plain words

Teaching a model to recognise something from only a handful of examples is hard for ordinary training, which needs many. Meta-learning tries to solve this by training on many small learning problems instead of one big one. Each training step pretends to be a tiny test: the model sees a few labelled examples of a few classes, must predict new examples from those classes, and is scored. Over many such rounds it gets better at picking up new classes quickly.

This survey sorts the existing methods into three families. Some compare a new example with stored examples or class averages (the "metric" family). Some keep an external memory to look things up from earlier tasks. Others learn how to set up the starting weights, how to produce a model's weights, or how to take each learning step. It explains the common method ideas, puts the main methods side by side in one accuracy table, and lists open problems.

TL;DR

A review of few-shot meta-learning from 2015 to about 2022 §1. It groups methods into metric-based, memory-based and learning-based families §3 and reports, for each method, the accuracy its original paper gave on Omniglot, miniImageNet and CUB-200-2011 tab. 1. It has no code and runs no experiments of its own, so treat its table as a pointer to the original papers.

Problem

Deep networks need many labelled samples and tend to fail when the test classes or distribution differ from training §1. Fine-tuning a pretrained network on few samples often deteriorates §1. The survey frames meta-learning as learning across a family of tasks, so that the learner's performance on new tasks improves with experience of earlier tasks §1.

Method

This is a survey, not a method. Its "method" is the taxonomy and the summary of each family. Counting the survey's own numbering: §2 covers the setup, §3 the three method families, §4 the challenges §1.

Setup: episodes, support and query §2.1

Data is split without overlap into meta-train and meta-test sets. Each of these is split again into a support set and a query set. A support set is N-way k-shot: N classes with k examples each. The query set holds q examples per class §2.1. Training runs in episodes: in each episode the learner fits the support set and is scored on the query set, and its parameters are updated from that error §2.1 fig. 2. At meta-test time the learner builds a classifier from the support set of an unseen task and is scored on that task's query set §2.1.

# Episodic meta-training  [§2.1] [fig. 2]
# Input:  meta-train set, a collection of labelled samples
# Output: trained meta-learner
def meta_train(meta_train_set, num_episodes):
    for episode in range(num_episodes):
        # Draw N classes, k support and q query samples per class [§2.1]
        support, query = sample_episode(meta_train_set)
        # Learn from the support set and predict the query set [§2.1]
        predictions = meta_learner(support, query)
        # Update the meta-learner from the query prediction error [§2.1]
        loss = prediction_loss(predictions, query)
        meta_learner = update(meta_learner, loss)
    return meta_learner

Metric-based methods §3.1

These compare query samples with support samples and predict the nearest class. Their success depends on the embedding function. The survey writes a generic form with a support embedding f, a query embedding g and a similarity s; some methods share one embedding for both §3.1 fig. 3.

# Metric-based prediction  [§3.1]
# Input:  support samples with labels, one query sample x
# Output: predicted class
def predict_metric(support, x, f, g, similarity):
    # Embed support samples with f and the query with g [§3.1]
    support_vectors = [f(s.input) for s in support]
    query_vector = g(x)
    # Pick the class of the most similar support embedding [§3.1]
    scores = [similarity(query_vector, v) for v in support_vectors]
    return support[argmax(scores)].label

Siamese networks §3.1.1 fig. 4. Two networks with shared parameters embed a pair; an energy function (the survey mentions cosine similarity or Euclidean distance) scores it. The contrastive loss is written in the survey as eq. 1:

\mathfrak{L}(E,Y) = \sum_{i=1}^{P} (1-Y).L_G(E(Z_1, Z_2)_i) + Y.L_I(E(Z_1, Z_2)_i) eq. 1

with Y = 0 for a similar pair and Y = 1 for a dissimilar pair §3.1.1. The margin form used for face verification is eq. 2:

\mathfrak{L}(E,Y) = (1-Y).E^2 + Y.\max(0, \text{margin} - E)^2 eq. 2

Prototypical networks §3.1.2 fig. 5. Each class prototype is the mean embedding of its k support samples. Queries are scored by Euclidean distance to the prototypes, and a softmax over those distances gives class probabilities. Training minimises the average negative log-probability of the correct class eq. 3:

-\frac{1}{T}\sum_{i}\log p(y_i^*\mid x_i^*, \{p_c\}) eq. 3

Variants described in §3.1.2:

• Semi-supervised prototypical networks: prototypes seed a soft k-means over labelled and unlabelled data; a "distractor" class centred at the origin and a threshold, set by a small MLP over prototype statistics, catch points that belong to no known class §3.1.2.
• Gaussian prototypical networks: each embedding gets a covariance matrix; the prototype is a covariance-weighted mean eq. 5, the class covariance is the sum of the per-sample matrices eq. 6, and queries are scored with the quadratic form eq. 4 fig. 6.

d_c^2(i) = (\vec{x_i}-\vec{p_c})^T S_c (\vec{x_i}-\vec{p_c}) eq. 4

\vec{p_c} = \frac{\sum_{i}\vec{s_i}^c \cdot \vec{x_i}^c}{\sum_{i}\vec{s_i}^c} eq. 5

\vec{S_C} = \sum_{i}\vec{s_i}^c eq. 6

The survey also lists transferable prototypical networks (RKHS discrepancy between source and target prototypes), domain-alignment prototypical networks (MMD), improved prototypical networks (attention-like sample weights and learned distance scaling), label-enhanced prototypical networks for multi-label few-shot (§3.1.2), multi-prototype networks (k-means per class), dummy prototypical networks (a dummy prototype for unseen classes), local-descriptor multi-prototype networks (LMPNet) and multi-modal prototypical networks (with text-conditioned generated features) §3.1.2 fig. 7.

Matching networks §3.1.3 fig. 8. The predicted label distribution for a query is an attention-weighted sum of one-hot support labels eq. 7. The attention is a softmax over cosine similarities between the query embedding f and the support embedding g eq. 8:

p(\hat{y}\mid\hat{x}, S) = \sum_{i=1}^{k} a(\hat{x}, x_i)\, y_i eq. 7

a(\hat{x}, x_i) = \frac{e^{\cos(f(\hat{x}), g(x_i))}}{\sum_{j=1}^{k} e^{\cos(f(\hat{x}), g(x_j))}} eq. 8

Variants in §3.1.3 include a cascaded feature matching network (CFMN), attentive matching networks (AMN, with complementary cosine loss) and schedule sampling of difficult label sets.

Relation networks §3.1.4 fig. 9. An embedding f and a learned relation module g score each query against each class; k-shot support embeddings of a class are summed element-wise. Relation scores lie in [0, 1] and the model is trained with mean squared error §3.1.4. Variants include memory-augmented relation networks, prototype-relation networks, multi-scale relation networks (features subtracted rather than concatenated), multi-local feature relation networks and multi-scale Kronecker-product relation networks §3.1.4.

Other metric methods §3.1.5. Graph neural networks treat a task as a fully connected graph whose edge weights are learned §3.1.5. Global class representation adds a registration loss that links episodic and global class representations §3.1.5. Attentive recurrent comparators look at image regions over several steps. The glimpse location is computed from the previous hidden state eq. 9 eq. 10, and the RNN update is eq. 11 fig. 10:

G_t \leftarrow (I_t, \Omega_t) eq. 9

\Omega_t = w_t h_{t-1} eq. 10

h_t \leftarrow \mathrm{RNN}(G_t, h_{t-1}) eq. 11

The region comparison network (RCN) matches region vectors and uses task-specific region weights §3.1.5 fig. 11. MACO replaces a fixed metric with a learned one: a class vector is the average of pairwise relation outputs over the class eq. 12 fig. 12:

R_\rho(s_k) = \frac{1}{\binom{n}{2}} \sum_{(x_i, x_j)\in s_k} g_\rho(x_i, x_j), \quad i \neq j eq. 12

Memory-based methods §3.2

These keep an internal or external memory that stores information from earlier inputs and reads it back.

• MANN §3.2 fig. 13: a Neural Turing Machine controller with an external memory bank. Reads are content-based; writes use least-recently-used access. The survey notes that differentiable blurry reads and writes make the bank trainable by backpropagation.
• SNAIL §3.2: temporal convolutions give wide access to past inputs and soft attention picks specific ones.
• Conditional neural processes §3.2 fig. 14: an encoder h produces representations r_i from support inputs and labels, an aggregator a combines them into one memory value, and a decoder g predicts query labels from it.
• Memory-augmented matching network §3.2: class prototypes are weighted, and each sample's weight is the inverse of its summed distance to the other samples of its class.

Learning-based methods §3.3

A meta-learner trains a base learner, and the survey splits these methods into learning the initialisation, learning the parameters and learning the optimiser.

Learning the initialisation §3.3.1 fig. 15 fig. 16. MAML learns an initialisation θ from which one or a few gradient steps adapt to a new task §3.3.1. The inner step for task T_i is eq. 13, and the outer update over a batch of tasks is eq. 14 fig. 16:

\theta_i' = \theta - \alpha \nabla_\theta \mathfrak{L}_{T_i}(f_\theta) eq. 13

\theta = \theta - \beta \nabla \sum_{T_i \sim p(T)} \mathfrak{L}_{T_i}(f_\theta) eq. 14

The survey describes α and β as hyperparameters and gives no values. Other methods in this family listed in §3.3.1: probabilistic MAML (PLATIPUS), Bayesian MAML (BMAML, Stein variational gradient descent), latent embedding optimisation (LEO) fig. 17, context adaptation (CAML), adversarial MAML (ADML), task-agnostic meta-learner (TAML), body-only inner updates (BOIL), learn-to-forget (L2F), Laplace-approximation MAML (LLAMA) and implicit MAML (iMAML) fig. 18.

Learning the parameters §3.3.2 fig. 19. A meta-learner generates the weights of the base learner or its feature layers rather than initialising them. Methods listed: neural statistician, MetaNet, learnet (pupil network), LGM-Net, dynamic learning without forgetting (attention-based weight generator), a denoising graph autoencoder for those weights (DAE), weight imprinting from normalised embeddings, incremental learning with attention attractor networks, task-aware feature embeddings (TAFE-Net) and the meta-transfer learner (MTL). The survey gives no equations for this family.

Learning the optimiser §3.3.3 fig. 20. The update rule is replaced by a learned one.

• LSTM meta-learner: the gradient step eq. 15 is replaced by an LSTM g with parameters φ, giving eq. 16 §3.3.3.
• Meta-SGD: the learning rate α becomes a vector with the same shape as θ, so the step is eq. 17; both θ and α are updated in the outer loop eq. 18 eq. 19 §3.3.3.
• Reptile: after n gradient steps on a sampled task the parameters θ_i' are moved toward θ eq. 20 eq. 21 fig. 21 §3.3.3.

\theta_t = \theta_{t-1} - \alpha_t \nabla_{\theta_{t-1}} \mathfrak{L}_{T_j}(\theta_{t-1}) eq. 15

\theta_{t+1} = \theta_t = g_\varphi\big(\nabla_{\theta_t} \mathfrak{L}_{T_j}(\theta_t)\big) eq. 16

\theta_t' = \theta_{t-1} - \alpha \odot \nabla_{\theta_{t-1}} \mathfrak{L}_{T_j}(\theta_{t-1}) eq. 17

\theta_t = \theta_{t-1} - \beta \nabla_{\theta_{t-1}} \sum_{T_i \sim p(T)} \mathfrak{L}_{T_j}(f(\theta_{i,t-1}')) eq. 18

\alpha_t = \alpha_{t-1} - \beta \nabla_{\alpha_{t-1}} \sum_{T_i \sim p(T)} \mathfrak{L}_{T_j}(f(\theta_{i,t-1}')) eq. 19

\min\; E\!\left[\tfrac{1}{2} D(\theta, \theta_i')\right] eq. 20

\theta = \theta - \epsilon\,(\theta_i' - \theta) eq. 21

• R2-D2 and LR-D2 §3.3.3: the final layer of an embedding is a base learner solved in closed form. Ridge regression gives eq. 22, rewritten with the Woodbury identity eq. 23 so that the matrix inverted is n × n in the n-shot case. For classification the output is an affine map of the query embeddings eq. 24. Logistic regression (LR-D2) is solved with Newton steps eq. 25 eq. 26. The meta-learner learns α, β and λ together with the CNN weights §3.3.3.

\Lambda(z) = \arg\min \|XW - Y\|^2 + \lambda\|W\|^2 = (X^TX + \lambda I)^{-1} X^TY eq. 22

W = \Lambda(z) = X^T(XX^T + \lambda I)^{-1} Y eq. 23

\hat{Y} = \alpha X' W + \beta eq. 24

w_i = (X^T \mathrm{diag}(s_i) X + \lambda I)^{-1} X^T \mathrm{diag}(s_i) z_i eq. 25

w_i = X^T (XX^T + \lambda\,\mathrm{diag}(s_i)^{-1})^{-1} z_i eq. 26

• MetaOptNet §3.3.3: an SVM with a differentiable quadratic-programming solver learns the embedding function's parameters §3.3.3.

# MAML meta-update  [§3.3.1] [eq. 13] [eq. 14]
# Input:  shared initialisation theta, tasks T_i, step sizes alpha, beta
# Output: updated theta
def inner_adapt(theta, task, alpha):
    # One gradient step on the task's loss  [eq. 13]
    return theta - alpha * grad(task_loss(task, theta))

def maml_step(theta, tasks, alpha, beta):
    # Outer gradient summed over the batch of tasks  [eq. 14]
    # The survey prints eq. 14 with the loss at theta, not at the
    # adapted parameters; it does not say how the inner step enters
    # the outer gradient [§3.3.1]
    total = sum(grad(task_loss(task, theta)) for task in tasks)
    return theta - beta * total

Hyperparameters

The survey runs no experiments, so no training hyperparameters are given for its own work. The symbols it introduces, and what it states about each:

Results

Author-reported, copied by the survey from the original papers tab. 1. The survey does not re-run any method. Setup per row (backbone, hardware, split protocol) is not stated in the table. Columns are 5-way and 20-way accuracy with 1 and 5 shots on Omniglot, and 5-way 1- and 5-shot on miniImageNet and CUB-200-2011 tab. 1. "±" is the 95% confidence interval as printed. "-" means the original paper did not report it tab. 1. Category: I metric-based, II memory-based, III learning the initialisation, IV learning the parameters, V learning the optimiser tab. 1.

Headline statements in the survey's text §4: on Omniglot 5-way 5-shot, Meta-SGD, prototype-relation networks, MAML and GCR are the top four, and on miniImageNet 5-way 5-shot the two best metric-based methods are MLFRNet and LMPNet §4. These figures do not match the table; see Pitfalls.

Paper vs code

No official code. The survey has no method of its own to implement, and it links no repository. Searched: the paper body, abstract and arXiv metadata for links (only author ORCID links); GitHub repository search for gharoun (returned a profile-configuration repository and an unrelated repository) and for meta-learning few-shot survey (returned a third-party resource collection). No discrepancies with an implementation can be checked.

Pitfalls

• Text and table disagree. For miniImageNet 5-way 5-shot, §4 gives MLFRNet 80.23% and LMPNet 80.00% §4; tab. 1 gives LMPNet 80.23% and MLFRNet 83.16%, and no 80.00% entry for LMPNet. For Omniglot 5-way 5-shot, §4 gives Meta-SGD 99.91%, prototype-relation 99.90%, MAML 99.90%, GCR 99.86% §4; tab. 1 gives 99.93%, 99.91%, 99.9%, 99.90%. Check the original papers before quoting any number from this survey.
• Printed inversion. tab. 1 lists "Dynamic learning without forgetting" at 74.92% for 1-shot and 58.55% for 5-shot on miniImageNet, i.e. 5-shot lower than 1-shot. Verify against the original paper.
• Unreported entries. Many rows are "-": TPN, LPN, multi-prototype PN, dummy PN, PLATIPUS, alpha MAML, pupil network, weight imprinting and TAFE-Net have no number at all tab. 1. The table cannot rank them.
• Mixed protocols. Each number comes from its own paper. Backbones, splits and training budgets are not stated in the table, so rows are not directly comparable. The survey itself says learning-based methods are expensive, but gives no cost figures §4.
• Citation key mismatch. The GNN method is cited as garcia2018few in the text §3.1.5 and garcia2017few in tab. 1.
• Gaussian prototypical network notation. eq. 4 calls S_c an inverse covariance, but eq. 6 sums per-sample covariance matrices into S_C, and eq. 5 weights prototypes by s_i. How s_i is computed, and whether S_c is the inverse of S_C, is not stated §3.1.2.
• MAML outer update. eq. 14 is printed with the loss at θ, without the prime on θ_i' from eq. 13. The survey does not say how the inner-loop result enters the outer gradient §3.3.1.
• Index inconsistency in ARCs. eq. 10 maps h_{t-1}, but the text says w_t maps h_t §3.1.5.
• Overloaded symbols. β is the outer step in eq. 14 (MAML) and in eq. 18 and eq. 19 (Meta-SGD). In eq. 15 and eq. 16 θ is used both as a parameter and as the learning rate named in the text §3.3.3.
• Reptile distance. The survey writes the target as minimising an expected distance D without defining D in eq. 20; the text says Euclidean §3.3.3.
• Prototypical distance. The survey says Euclidean distance, not squared. Whether the distance is squared is not stated §3.1.2.
• Editorial leftover. §3.2 contains a bracketed editor note in the MANN paragraph. Do not copy that sentence.
• Coverage gaps the survey flags itself. Most methods are trained on one domain; episodic training can cause forgetting of base classes; metric methods do worse on miniImageNet than on Omniglot; MAML is not robust to adversarial samples §4. Benchmarks beyond the three listed are not covered §2.2.

Lineage

• Surveys it names, not linked: Huisman et al. 2021 and Li et al. 2021 §1.
• Methods it treats as the anchors of each family: Siamese networks (1993 origin, 2015 few-shot use) §3.1.1; MAML as the pioneer of learning the initialisation §3.3.1; prototypical networks as the starting point for many metric methods §4.
• Compared against: the methods in tab. 1. Superseded: not stated.