Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

In plain words

Few-shot classification asks a model to recognise new categories from a handful of labelled pictures, sometimes just one per category. A large body of work, "meta-learning", tries to solve this by training on thousands of small practice tasks so the model learns how to learn quickly.

This paper asks whether all that machinery is needed. The authors take every training category, pool them into one ordinary classification problem, and train a standard network on it. They then throw away the final classification layer and use the rest as a fixed feature extractor. For each new few-shot task they fit a simple linear classifier (logistic regression) on the features of the few labelled examples. Nothing in the network is adapted at test time.

This plain baseline matches or beats the elaborate meta-learning methods it is compared against on four standard benchmarks. A second trick, training a fresh copy of the network to imitate the first one's soft predictions ("self-distillation"), adds a couple more points. The takeaway for practitioners: on these benchmarks, the quality of the learned features matters more than the cleverness of the adaptation algorithm.

TL;DR

Pre-train an embedding network with cross-entropy on the union of all meta-training classes, freeze it, L2-normalise its features and train a per-episode multinomial logistic regression on the support set. Optionally apply one or two rounds of born-again self-distillation before freezing. With a ResNet-12 this reaches 62.02/79.64% (simple) and 64.82/82.14% (distilled) on miniImageNet 5-way 1/5-shot, and beats the best reported Meta-Dataset baselines on 9 of 10 test domains, all author-reported. Relevant if you need a strong, cheap few-shot baseline or want to test whether a meta-learning method actually beats a linear probe.

Problem

Meta-learning methods for few-shot classification train an embedding (and often an adaptation procedure) over episodes sampled from a task distribution §3.1 eq. 1 eq. 2. It was unclear whether their gains come from the meta-learning algorithm or simply from the learned representation §1. Prior baselines either fine-tuned the network on the test support set, sometimes transductively using test queries, or performed slightly worse than meta-learning §1. The paper's setting is inductive: no query information is used and the embedding stays frozen at meta-test time §1 §3.2.

Method

The meta-training set is a collection of tasks \mathcal{T} = \{(\mathcal{D}^{train}_i, \mathcal{D}^{test}_i)\}_{i=1}^I §3.1. A base learner \mathcal{A} fits parameters \theta on a task's training split given a fixed embedding f_\phi:

\theta = \mathcal{A}(\mathcal{D}^{train}; \phi) = \arg\min_\theta \mathcal{L}^{base}(\mathcal{D}^{train}; \theta, \phi) + \mathcal{R}(\theta) eq. 1

Meta-learning chooses \phi to minimise the expected query loss \phi = \arg\min_\phi \mathbb{E}_{\mathcal{T}}[\mathcal{L}^{meta}(\mathcal{D}^{test}; \theta, \phi)] eq. 2, and evaluation averages the same loss over held-out tasks \mathcal{S} eq. 3.

Step 1: embedding by plain classification. Merge all meta-training tasks into one dataset and train with cross-entropy over all classes §3.2:

\mathcal{D}^{new} = \{(\mathbf{x}_i, y_i)\}_{k=1}^K = \cup\{\mathcal{D}^{train}_1, \ldots, \mathcal{D}^{train}_i, \ldots, \mathcal{D}^{train}_I\} eq. 4

\phi = \arg\min_\phi \mathcal{L}^{ce}(\mathcal{D}^{new}; \phi) eq. 5

The network up to the penultimate layer is kept as the feature extractor §1 fig. 1; the classification layer is dropped at meta-test time fig. 5. The authors stress the classes across tasks are mutually exclusive, so the merge produces one large many-way problem; a multi-task alternative with one head per task scored 58.53 ± 0.8 vs 62.02 ± 0.63 on miniImageNet 5-way 1-shot §4.9 §6.

Step 2: per-episode linear base learner. For each meta-test task, fit multinomial logistic regression on the frozen support features §3.2 fig. 2:

\theta = \arg\min_{\{W, b\}} \sum_{t=1}^T \mathcal{L}^{ce}_t(W f_\phi(\mathbf{x}_t) + b, y_t) + \mathcal{R}(W, b) eq. 6

Features are L2-normalised onto the unit sphere and each support image is augmented into 5 samples before fitting §4.5 tab. 4. The classifier is re-initialised per task; the embedding is never fine-tuned §3.2. The implementation uses scikit-learn §4.1.

Step 3 (optional): sequential self-distillation. Train a new network of identical architecture on the same merged data with a weighted sum of cross-entropy and KL divergence to the previous network's predictions §3.3:

\phi' = \arg\min_{\phi'} \left(\alpha\, \mathcal{L}^{ce}(\mathcal{D}^{new}; \phi') + \beta\, KL(f(\mathcal{D}^{new}; \phi'), f(\mathcal{D}^{new}; \phi))\right) eq. 7

with "usually \beta = 1-\alpha" §3.3. Repeat born-again style, generation k distilled from generation k-1 fig. 3:

\phi_k = \arg\min_\phi \left(\alpha\, \mathcal{L}^{ce}(\mathcal{D}^{new}; \phi) + \beta\, KL(f(\mathcal{D}^{new}; \phi), f(\mathcal{D}^{new}; \phi_{k-1}))\right) eq. 8

After K rounds, \phi_K is the meta-test embedding §3.3. The reported tables use generation 2 on miniImageNet, CIFAR-FS and FC100 and generation 1 on tieredImageNet, chosen on the validation set §4.6.

Algorithm: RFS (simple + distill)   [§3.2] [§3.3] [§4.1]
Input: merged meta-training set D_new (all base classes), generations K
1. phi_0 <- train network (backbone + linear head over all base classes)
   on D_new with cross-entropy                                  [eq. 5]
2. for k = 1..K:
     init a fresh network with the same architecture
       [code: train_distillation.py#L263-L264]
     train it on D_new with alpha*CE(labels) + beta*KL(student, teacher phi_{k-1}),
       teacher frozen in eval mode                     [eq. 8] [code: train_distillation.py#L366-L435]
     phi_k <- student
3. Drop the classification head; embedding = global-avg-pooled penultimate features [§4.1] [fig. 5]
Meta-test, per episode (N-way, n-shot):
4. Augment each support image into 5 randomly transformed copies   [§4.5]
5. z = f_phiK(x); z <- z / ||z||_2 for support and query            [§4.5]
6. Fit N-way multinomial logistic regression on support (z, y)    [eq. 6]
7. Predict query labels; accuracy averaged over sampled episodes

Other base learners compared: nearest neighbour with Euclidean distance (NN) and NN on normalised features, which is a cosine classifier §4.5 §4.6; linear SVM on Meta-Dataset §5.

Self-supervised variant. A ResNet-50 trained without labels on the miniImageNet meta-train set with MoCo or CMC, then linear-probed the same way §4.4 app. C.

Hyperparameters

Results

Author-reported, ResNet-12 unless noted, 5-way, accuracy (%) with 95% confidence intervals tab. 1 tab. 2.

Selected rows of Table 1 (miniImageNet and tieredImageNet, 84×84 input):

† trained on train+val. Source: tab. 1. With the WRN-28-10 backbone of Dhillon et al., the simple frozen-embedding baseline gets 61.1 ± 0.86 on miniImageNet 1-shot vs their fine-tuned 57.73 ± 0.62 §4.2.

Table 2 (CIFAR-FS and FC100):

Ablation, ResNet-12 tab. 4:

Self-supervised vs supervised ResNet-50 embeddings, miniImageNet 5-way tab. 3:

Backbones and train+val data, miniImageNet / tieredImageNet tab. 5:

Backbones and train+val data, CIFAR-FS / FC100 tab. 6:

Meta-Dataset, embedding trained on the ILSVRC train split only, ResNet-18, variable ways and shots, 1000 tasks per evaluation §5 tab. 7. "Best" is the per-dataset maximum over 7 methods and 3 architectures from the Meta-Dataset paper:

Across distillation generations, accuracy rises for the first two or three generations and then falls for LR and NN; normalised features largely prevent the drop §4.6 fig. 4. With normalised features LR and NN are similar at 1-shot, but LR stays clearly better at 5-shot §4.7 fig. 4.

Paper vs code

Checked against WangYueFt/rfs at f8c837b. The repository contains no LICENSE file, so no code is quoted here.

• Max-pooling. Paper: 2×2 max-pool after each of the first 3 blocks §4.1. Code: every block, including the fourth, ends in MaxPool2d(2) (84 → 42 → 21 → 10 → 5 before avg-pool) resnet.py resnet.py resnet.py.
• DropBlock details are only in code: drop rate 0.1, DropBlock only in blocks 3 and 4, block size 5 for ImageNet derivatives and 2 for CIFAR, with the drop rate ramped linearly over the first 40,000 forward passes of each block resnet.py util.py util.py. Blocks 1 and 2 get ordinary dropout with p=0.1 instead, since their drop_block flag is off resnet.py resnet.py.
• KD temperature. Not stated in the paper; code uses T=4 and multiplies the KL by T² train_distillation.py criterion.py.
• Loss-weight naming. Paper weights CE by α and KL by β, α=β=0.5 §4.1 eq. 7. Code calls the CE weight gamma (-r) and the KL weight alpha (-a); defaults are 1 and 0, i.e. no distillation unless both flags are passed train_distillation.py train_distillation.py. The example script uses -r 0.5 -a 0.5 and notes -a 1.0 gives similar results run.sh.
• Which features feed the classifier. Paper: penultimate-layer features §1 §3.2, except ConvNet-4 on miniImageNet where logits were used app. B. Code: meta_test defaults to using the logits of the classification layer; eval_fewshot.py prints both logit-based and feature-based accuracy on val and test without saying which is reported meta_eval.py meta_eval.py eval_fewshot.py.
• Episode count. Paper: 1000 (ImageNet) or 3000 (CIFAR) episodes per run, median of 3 runs §4.2 §4.3. Code default: 600 episodes, one run, mean with t-distribution 95% interval eval_fewshot.py meta_eval.py.
• LR schedule. Only the miniImageNet default (decay at epochs 60 and 80 of 100) is in code; decay epochs for the other datasets are given in neither paper nor code train_supervised.py train_supervised.py.
• Base-learner settings come only from code: L2 penalty, C=1.0, lbfgs, 1000 iterations, multinomial; the SVM is a StandardScaler plus linear SVC with C=1 meta_eval.py.
• Test-time augmentation uses the training transform (random crop with padding, colour jitter, flip) on 5 tiled copies of each support image; no un-augmented copy is kept, and queries are not augmented mini_imagenet.py.
• Not released: Meta-Dataset (ResNet-18, 128×128) and the MoCo/CMC self-supervised training; the README points to the separate CMC repository for the latter README.md.

Pitfalls

• Logits vs features. Using classifier logits instead of pooled features is the code default and changes the embedding dimension from 640 to the number of base classes. Decide explicitly; the paper's method says penultimate features §3.2, but the evaluation script reports both eval_fewshot.py.
• L2 normalisation matters. It adds about 2-4 points of 1-shot accuracy (e.g. 58.74 → 61.56 on miniImageNet) tab. 4 and prevents the late-generation accuracy drop in distillation fig. 4. Normalise both support and query.
• Distillation needs a fresh student. The student is re-initialised, not warm-started from the teacher, and each generation is a full training run with the same schedule §4.1 train_distillation.py. Two generations triple pre-training cost.
• Generation count was tuned on validation per dataset (2, or 1 for tieredImageNet) §4.6; more generations eventually hurt for un-normalised LR and NN fig. 4.
• β is not always 1−α. Benchmarks use α=β=0.5 §4.1; Meta-Dataset uses α=0.5, β=1.0 §5.
• Architecture details are code-only (LeakyReLU 0.1, max-pool after all 4 blocks, DropBlock schedule, 1×1 stride-1 shortcut convs) resnet.py resnet.py. Results will not reproduce from the paper's text description alone.
• Code's resnet18 is not the standard ResNet-18. It stacks the same 3-conv blocks as [1,1,2,2] resnet.py. For variants with more than one block per stage, _make_layer passes use_se into the drop_block argument slot resnet.py; ResNet-12 is unaffected.
• Relies on disjoint, mergeable classes. The authors attribute the baseline's strength to merging mutually exclusive tasks into one many-way problem §4.9 §6; benchmarks without that structure may behave differently.
• Test-time augmentation is a small effect (≤0.5 points, sometimes negative: FC100 1-shot 42.77 → 42.59) tab. 4 but multiplies support-set feature extraction by 5.
• Data: the code expects MetaOptNet's pickled splits, renamed by the authors README.md; class counts are hard-coded per dataset (e.g. 64/80 miniImageNet, 351/448 tieredImageNet) train_supervised.py.
• Weight decay hits all parameters, including BatchNorm, because the optimiser receives all model parameters train_supervised.py.

Lineage

• Builds on: born-again networks (Furlanello et al.) for sequential self-distillation §3.3; knowledge distillation (Hinton et al.) §3.3; the ResNet-12 of MetaOptNet (Lee et al.) §4.1; the cross-entropy pre-training stage of TADAM and LEO §3.2.
• Compared against: MAML, Matching Networks, Prototypical Networks, Relation Networks, R2D2, TADAM, MetaOptNet, LEO, transductive fine-tuning (Dhillon et al.), Meta-Dataset baselines tab. 1 tab. 2 tab. 7.
• Self-supervised embeddings: MoCo and CMC §4.4.