Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference

In plain words

Few-shot learning is about teaching a model a new category from only a handful of labelled pictures. Most research in this area designs clever training schemes for small, old-fashioned networks. This paper asks a more practical question: what happens if you simply start from a network that someone already trained on a huge pile of unlabelled images, and then add a simple few-shot method on top?

The recipe has three stages. First, the network learns general picture features from unlabelled data. Second, it is trained on many small simulated few-shot tasks, where each class is represented by the average feature of its example pictures and a new picture is labelled by finding the closest class average. Third, when a new task comes from a very different kind of picture, the network is briefly adjusted on the few labelled examples, using random flips, colour changes and shifts to create varied copies of them.

The authors find that the choice of pre-training data and of a modern transformer backbone matters more than the few-shot training method itself. Adjusting the network at test time helps most when the new task looks unlike the training data. The results are strong, but they come from a setting that uses much more external data than most earlier papers, so they are not a like-for-like comparison.

TL;DR

P>M>F stands for pre-training, then ProtoNet meta-training, then fine-tuning. The pre-training uses external unlabelled data such as ImageNet-1k, and the backbone can be a ResNet or a ViT. The meta-trained model classifies by nearest class prototype, and it can optionally be fine-tuned per domain with augmentation at test time. The authors report results above many prior methods on miniImageNet, CIFAR-FS, Meta-Dataset and cross-domain benchmarks. They warn that these numbers are not directly comparable to prior work, which often uses no external data. §1 §4.2

Problem

Standard few-shot benchmarks restrict the source data to the meta-training split and use small networks such as CNN-4-64 or ResNet-12. The authors argue that this setup hides the effect of two practical choices: the external data used for pre-training and the architecture used, including modern vision transformers. A third choice is whether to fine-tune the backbone on the labelled support set at test time, which is often avoided. §1

The paper assumes few-shot tasks that are sampled from a set of domains (for example Meta-Dataset) and that a small labelled support set is the only labelled data available for a new task. It assumes that unlabelled external data exists for the modality. §1 §5

Method

The pipeline has three phases §3:

1. Pre-training of the backbone on unlabelled external data with an unsupervised loss. The paper considers DINO, BEiT and CLIP style checkpoints, with ResNet or ViT backbones, and does not train them itself §3.1. The code loads released checkpoints and contains no pre-training script [code: models/init.py#L70-L80].
2. Meta-training with ProtoNet on simulated few-shot episodes from labelled base classes. Each class prototype is the mean embedding of its support images. The query is classified by a softmax over negative cosine distances to the prototypes §3.2 eq. 1. The variant is "various-way-various-shot": the number of classes and shots changes from episode to episode §3.2 §4.
3. Meta-testing, with optional fine-tuning. By default the meta-trained ProtoNet is applied to new tasks without any change §3.3. If the new task is from an unseen domain, the whole backbone is fine-tuned for a few gradient steps on the support set. Each step builds a pseudo query set by augmenting the support images, and the prototypes are recomputed from the un-augmented support images with the current weights §3.3 alg. 1.

The learning rate for fine-tuning is chosen automatically per domain. The authors sample N = 5 extra tasks from each domain, test a fixed grid of learning rates on them, and reuse the best rate for every task in that domain §3.3. The paper does not say which split these tasks come from.

The core ProtoNet classifier in pseudocode form (the paper's eq. 1 with the code's scaling):

# ProtoNet classification of one episode  [§3.2] [eq. 1]
# Input:  supp_x (B, nSupp, C, H, W)
#         supp_y (B, nSupp)
#         qry_x  (B, nQry, C, H, W)
# Output: logits (B, nQry, nClasses)

def protonet_logits(supp_x, supp_y, qry_x):
    # 1. Embed support and query images with the same backbone  [§3.2]
    supp_f = backbone(supp_x)                   # (B, nSupp, d)
    qry_f = backbone(qry_x)                     # (B, nQry, d)
    # 2. Prototype of class k = mean of its support embeddings  [§3.2]
    protos = class_mean(supp_f, supp_y)         # (B, nClasses, d)
    # 3. Cosine similarity of L2-normalised vectors
    #    [eq. 1] [code: models/protonet.py#L22-L23]
    cosine = normalize(qry_f) @ normalize(protos).T
    # 4. Code multiplies by a learned scale and adds a learned bias
    #    [code: models/protonet.py#L26]
    logits = scale_cls * (cosine + bias)
    return logits

Test-time fine-tuning, following alg. 1 with the choices made in the code:

# Meta-test fine-tuning of the whole backbone  [alg. 1] [§3.3]
# Input:  supp_x, supp_y, qry_x, meta-trained weights, learning rate lr
# Output: logits for qry_x

def finetune_and_classify(supp_x, supp_y, qry_x, lr):
    # 0. Start every task from the meta-trained weights
    #    [code: models/deploy.py#L198]
    load_meta_trained_weights(backbone)
    # 1. Adam over backbone parameters only, betas (0.9, 0.999), no weight decay
    #    [code: models/deploy.py#L214-L217]
    opt = Adam(backbone.parameters(), lr=lr)
    for step in range(num_steps):             # 50 in [supp. §3]; see Pitfalls
        # 2. With probability 0.9, augment the support batch (flip, colour,
        #    translation)  [§3.3] [supp. §3] [code: models/utils.py#L9-L20]
        aug_x = maybe_augment(supp_x, prob=0.9)
        # 3. Prototypes from the un-augmented support set, with the
        #    current weights  [alg. 1]
        protos = class_mean(backbone(supp_x), supp_y)
        # 4. Classify the augmented support images against those
        #    prototypes  [alg. 1]
        logits = cosine_logits(backbone(aug_x), protos)
        loss = cross_entropy(logits, supp_y)
        opt.step(loss)
    # 5. Recompute prototypes with the adapted backbone, then classify
    #    the queries  [alg. 1]
    protos = class_mean(backbone(supp_x), supp_y)
    return cosine_logits(backbone(qry_x), protos)

Meta-training schedule §4: the backbone is trained for 100 epochs, each of 2000 episodes. The learning rate warms up linearly from 1e-6 to 5e-5 over 5 epochs and then decays with a cosine schedule to 1e-6. Validation episodes decide which checkpoint is kept. The paper uses one common schedule for every architecture and dataset, which it says may be sub-optimal for some of them. Strong regularisation and data augmentation are said to be turned off §4. The optimiser is not named in the paper.

Key equation eq. 1: the probability of class k for query x is a softmax over the negative cosine distances to all class prototypes, where each prototype c_k is the mean of the support embeddings of class k. Eq. 1 has no temperature or bias term; the code adds both (see Paper vs code).

Hyperparameters

Results

Author-reported. Setup: ProtoNet meta-test with a nearest-centroid classifier unless stated otherwise tab. 1. Benchmarks: miniImageNet and CIFAR-FS (5-way, 1 or 5 shot) §4; Meta-Dataset with 8 source domains, or ImageNet-only §4; a cross-domain benchmark meta-trained on miniImageNet with four target domains §4 tab. 6.

Table 1 summarises pre-training and architecture on Meta-Dataset (average over target datasets), miniImageNet and CIFAR-FS (5-way-5-shot) tab. 1. "MetaTr" is the meta-training method; "-" means no meta-training.

Table 2 shows other few-shot learners on top of the same features: MetaQDA and MetaOptNet with ResNet and with DINO ViT-small. Each row lists 5-way-1-shot and 5-way-5-shot for miniImageNet and CIFAR-FS tab. 2.

Table 3 shows the effect of test-time fine-tuning on Meta-Dataset (average over all domains, and the out-of-distribution subset) tab. 3 fig. 3. Meta-training used either ImageNet only (IN) or all of Meta-Dataset (MD).

Table 4 compares with state-of-the-art few-shot methods on miniImageNet and CIFAR-FS §4.2 tab. 4. The P>M>F rows carry the same numbers as the ProtoNet rows without fine-tuning in Table 1. Columns are CIFAR-FS 5w1s and 5w5s, miniImageNet 5w1s and 5w5s, and the external data and labels flags.

Table 5 gives Meta-Dataset results for the 8 in-domain training sources and for ImageNet-only meta-training tab. 5. The ViT-small average of 83.13 equals the tuned fine-tuning row in Table 3 tab. 3, so the P>M>F rows here appear to include fine-tuning. The text does not state this for every row.

Meta-Dataset, 8 in-domain sources:

Meta-Dataset, ImageNet-only meta-training (columns: in-domain ImageNet, then out-of-domain datasets, then average):

Table 6 is the cross-domain benchmark. Each entry is 5-way 5-, 20- or 50-shot accuracy, on ChestX, ISIC, EuroSAT and CropDisease tab. 6:

Supplementary numbers [supp. §1], averages over all Meta-Dataset target domains, without test-time fine-tuning: the first column meta-trains on the 8 in-domain sources, the second on ImageNet only. A dash means the supplement does not report that row.

Supplementary miniImageNet and CIFAR-FS table [supp. §2], 5-way accuracy:

Figure 1 (teaser) places the ProtoNet + ViT + CLIP result against results aggregated from prior few-shot studies, and includes ProtoNet with a randomly initialised ViT for comparison fig. 1 §1. Table 1 shows the randomly initialised ViT performing poorly tab. 1 §3.1. Figure 3 describes fine-tuning on Meta-Dataset: held-out datasets such as traffic signs and COCO benefit from it, and so do datasets far from ImageNet such as Omniglot and QuickDraw fig. 3 §4.1.2. The supplementary ablation (fine-tuning hyperparameters on validation domains) finds that the learning rate dominates, that a higher augmentation probability helps, and that 50 steps work well at the right learning rate [supp. §3].

Paper vs code

Code: github.com/hushell/pmf_cvpr22, Apache-2.0, commit 25bf65f013405b6596ed89c3e3bb4f9dda917694. The paper links this repository in §4.2. The abstract links a project page, which the README also links.

• Distance and scale. Eq. 1 uses cosine distance with no temperature or bias §3.2 eq. 1. Because exp(-(1 - cos)) is proportional to exp(cos), eq. 1 as written has an implicit scale of 1. The code multiplies the cosine by a learned scale initialised at 10 and adds a learned bias initialised at 0, and both are trained during meta-training protonet.py protonet.py.
• Peak learning rate under fp16. The paper states 5e-5 as the peak §4. The code multiplies the peak by world size / 8 whenever fp16 is on, and fp16 is on by default. A single-GPU run therefore peaks at 6.25e-6 for --lr 5e-5. The 8-GPU case is the one where the code's scaling is neutral. The paper does not give a GPU count main.py args.py.
• Optimiser. The paper does not name one. The code uses SGD with momentum 0.9 and no weight decay, and the AdamW default in the arguments is not used main.py.
• Model mode. The code calls the model with training mode off during meta-training, so any batch-norm or dropout layers run in evaluation mode engine.py main.py.
• Early stopping. The paper says validation decides when to stop §4. The code keeps the checkpoint with the best validation accuracy but runs all epochs main.py.
• Epoch length. The paper says each epoch is 2000 episodes §4. The Meta-Dataset loader's length is nEpisode multiplied by the number of base sources, which is 16,000 with the eight default sources meta_h5_dataset.py meta_h5_dataset.py.
• Image size. The paper says 84 × 84 for miniImageNet and 32 × 32 for CIFAR-FS §4. The code default is 224 for both, unless --img-size is passed args.py mini_imagenet.py cifar_fs.py.
• Evaluation split for miniImageNet and CIFAR-FS. The paper reports 600 test episodes §4. With --eval the code evaluates the fixed 1000-episode validation file (val1000Episode JSON) [code: datasets/init.py#L72-L73] [code: datasets/init.py#L52-L53] mini_imagenet.py. The Meta-Dataset path uses the test split with 600 episodes, as in the paper meta_h5_dataset.py.
• Augmentation during meta-training. The paper says data augmentation is turned off §4. The miniImageNet training transform still applies random resized crops (scale 0.05 to 1), colour jitter of 0.4 and horizontal flips mini_imagenet.py.
• Learning-rate search for fine-tuning. The paper uses N = 5 tasks per domain §3.3. The Meta-Dataset script scores each candidate learning rate on the first six episodes of the test-split loader, using query labels, then reports accuracy on the same test loader test_meta_dataset.py engine.py meta_h5_dataset.py. The reported accuracy is therefore chosen on the same split it reports. The paper does not say which split the N tasks come from.
• Learning-rate grid for CDFSL. The paper uses {0.01, 0.001, 0.0001, 0} §3.3. The Meta-Dataset script uses {0, 1e-4, 1e-3, 1e-2} test_meta_dataset.py, and the CDFSL script uses {0, 1e-4, 5e-4, 1e-3} test_bscdfsl.py.
• Fine-tuning steps and default. The ablation fixes 50 steps and probability 0.9 [supp. §3]. The argument default is 40 args.py, the class default is 50 deploy.py, and the README examples use 100 steps [README: Fine-tuning on meta-test tasks].
• Augmentation decision. The code decides once per step whether to augment the whole support batch, rather than per image utils.py.
• Per-episode learning-rate search. The code also has a finetune_autolr option that holds out part of each support set to choose the learning rate per episode deploy.py. The paper describes a per-domain search, and the README and test scripts use the per-domain version. The LaTeX source contains a commented-out supplementary paragraph saying the main results used per-episode fine-tuning. It is not in the compiled paper, so this card follows the main text.
• Algorithm 1 notation. The pseudocode writes the logits as the norm of the feature times the norm of the prototype. The code normalises each vector separately and applies the learned scale and bias alg. 1 protonet.py deploy.py.
• README Meta-Dataset learning rate. The README gives --lr 5e-4 for Meta-Dataset, ten times the paper's common 5e-5 [README: On Meta-Dataset with 8 source domains] §4.
• Pre-training code. The repository has no pre-training script. It loads released checkpoints such as DINO ViT-small from the DINO release URL [code: models/init.py#L70-L80].

Pitfalls

• Learning-rate selection leaks into reported numbers in the code. The Meta-Dataset and CDFSL scripts pick the fine-tuning learning rate with query labels from the test split, then report accuracy on that split test_meta_dataset.py test_bscdfsl.py. Treat the code's fine-tuning numbers as optimistic until a held-out split is used. The paper's intent is a few labelled tasks per domain, but it does not say where they come from §3.3.
• Fine-tuning is expensive. Each test task backpropagates through the full backbone for many steps. The authors say this may not run on embedded platforms without backpropagation §5. The cost scales with the number of steps (50 in the ablation, 40 in the code default, 100 in the README examples) [supp. §3] args.py.
• Learning rate sensitivity. A single fixed rate such as 0.01 helps some Meta-Dataset domains and hurts others. The paper attributes the inconsistent fine-tuning gains to this §4.1.2 tab. 3.
• Peak learning rate depends on the number of GPUs. Under the code's fp16 scaling, the effective peak is proportional to the number of GPUs. Check the logged learning rate before comparing runs main.py.
• Unstated details. The paper does not state the optimiser, weight decay, batch of episodes per update, number of GPUs, Meta-Dataset image size, validation episode count, pre-training hyperparameters, or whether the model is in training mode during meta-training §4. The code fills most of these, but the values are code defaults and may not be what the paper's numbers used.
• Class overlap. Pre-training on ImageNet-1k shares classes with several meta-test datasets, and the paper says CUB, aircraft and COCO have class overlap with ImageNet §4.1.1 (footnote). The paper argues that overlap is hard to avoid in practical benchmarks and that the self-supervised community does not treat it as a problem. It notes the overlap is less common in FSL pipelines §4.1.1.
• Inconsistent averages inside the paper. Table 1 gives 78.4 and 79.2 for the Meta-Dataset average of DINO ViT-small and ViT-base with ProtoNet, while the supplement gives 78.528 and 79.507 for the same rows tab. 1 [supp. §1]. Use the table you are reproducing and cite it.
• Not directly comparable. The authors say their numbers are not directly comparable to much prior work, because the architecture and external data differ §4.2. Do not read the rows in Tables 4 to 6 as a fair ranking.
• Domain is undefined outside Meta-Dataset. The learning-rate search is per domain §3.3, but the paper does not say how a new task's domain is identified beyond the Meta-Dataset datasets.
• Table 4 and Tables 5 to 6 may use different protocols. Table 4 rows match the no-fine-tuning ProtoNet numbers in Table 1. Table 5's ViT-small row matches the tuned fine-tuning row in Table 3. The text does not state the fine-tuning setting for Table 6 tab. 1 tab. 3 tab. 4 tab. 5 tab. 6.

Lineage

• Builds on: ProtoNet (Snell et al., prototypical networks; cited in the paper as [snell2017prototypical]) for the meta-training classifier. The change is cosine distance, a ViT or ResNet backbone, and external pre-training §3.2.
• Builds on: test-time fine-tuning of the support set, as in the works cited as [guo2020broader] and [li2021improvingFSL]. The change is that augmented support images form the pseudo query set, rather than the support set itself, and the whole backbone is updated rather than part of it §3.3.
• Builds on: self-supervised foundation models such as DINO, BEiT and CLIP, used as released checkpoints §3.1.
• Compared against: MetaOptNet, MetaQDA, SIB, PT-MAP, CNAPs, SUR, URT, URL, ITA, FLUTE and others in Tables 2 and 4 to 6 tab. 2 tab. 4 tab. 5 tab. 6.
• Superseded by: none recorded in this wiki.