Deep Metric Learning for Few-Shot Image Classification: A Review of Recent Developments

In plain words

Teaching a computer to recognise a new kind of object from only a handful of example photos is hard, because a model trained on a few examples tends to memorise them rather than learn what makes the class special. A popular way around this is to learn a way of measuring how alike two images are, so that a new photo can be labelled by finding which known examples it resembles most.

This paper is a review of recent work that takes this route. It sorts the methods into three questions a designer has to answer: how to turn images into useful numbers (features), how to summarise a class from its few examples (a representative point, a distribution or a subspace), and how to compare a new image with a class (a fixed formula or a small learned network). It also explains how these systems are tested, and closes with open problems. It is a map of the field rather than a new method, and it contains no experiments of its own.

TL;DR

• A review of 2018 to 2022 deep metric learning for few-shot image classification, as of v2 (June 2022) § Introduction.
• Methods are grouped by the stage they improve: feature embeddings §2.1, class representations §2.2 and distance or similarity measures §2.3.
• It defines the C-way K-shot task and its episodic evaluation §1.1 §1.2, and lists the benchmark splits §1.3.
• It reports no new experiments and no results tables; numbers must be taken from the original papers.
• Open problems: generalised feature learning, episodic training, robustness, and cross-domain transfer §3.

Problem

Deep networks need large labelled sets, while few-shot classification must learn a rule from only one to five labelled examples per class § Introduction. The main risk is overfitting to those few samples § Introduction. Metric learning avoids learning new parameters for each new class once the metric is learned, which is why the review focuses on it § Introduction.

Few-shot classification uses two disjoint label sets. A base dataset with labels from one set is used to learn the model. A novel dataset with labels from a different set is split into a support set and a query set §1.1. Cross-domain few-shot classification is the case where base and novel data differ in feature space or marginal distribution §1.1. Generalised few-shot classification must classify queries from both base and novel classes §1.1.

Method

The review's own method is the evaluation procedure in alg. 1 and the taxonomy in fig. 2. Each family below is summarised in prose, with the section of the review that discusses it.

Task definition §1.1. In C-way K-shot classification, the support set holds K labelled examples for each of C classes. The task is one-shot when K = 1. The paper says support sizes are typically 1 to 5 §1.1.

Evaluation procedure §1.2 alg. 1. Each episode samples C classes from the novel label set, K support examples per class, and M query examples from the remaining images of those classes. A classifier is built from the base data and the support set, it predicts the query labels, and the episode accuracy is recorded. The score is the mean over E episodes §1.2 alg. 1.

# Evaluation of C-way K-shot classification  [alg. 1]
# Input:  base set, novel set, number of episodes E
# Output: mean query accuracy over the E episodes

def evaluate(base_set, novel_set, num_episodes):
    # Repeat for e = 1..E  [alg. 1, steps 2-8]
    accuracies = []
    for episode in range(num_episodes):
        # Randomly pick C classes from the novel labels  [alg. 1, step 4]
        classes = sample_classes(novel_set, C)
        # K labelled examples per class form the support set  [alg. 1, step 5]
        support = sample_per_class(novel_set, classes, K)
        # M other examples of those classes form the query set  [alg. 1, step 6]
        query = sample_remaining(novel_set, classes, M)
        # Predict query labels with f built from base data and support  [alg. 1, step 7]
        predicted = f(query.inputs, base_set, support)
        # Fraction of query labels predicted correctly  [alg. 1, step 8]
        accuracies.append(mean(predicted == query.labels))
    # Average over all episodes  [alg. 1, return]
    return mean(accuracies)

Stage 1, learning feature embeddings §2.1. The premise is that the network extracts features discriminative enough to generalise to new classes. The review covers four ways of doing this.

• Task-agnostic features §2.1.1. Siamese networks with a weighted L1 distance §2.1.1; Matching Networks, which embed support and query images in the context of the full support set, introduce episodic training and classify by a softmax over cosine similarities §2.1.1. Later work removes the order dependence of the set embedding by pooling or self-attention §2.1.1. Transductive label propagation (TPN, PTN) uses query images without their labels §2.1.1. Feature-wise transformation layers simulate shifted feature distributions for cross-domain robustness §2.1.1. PARN adds position encoding and deformable convolutions §2.1.1.
• Fine-grained features §2.1.2. DN4 uses local descriptors and an image-to-class nearest-neighbour rule; ATL-Net learns thresholds and episodic attention over these descriptors; COMET learns one embedding per fixed concept mask; MSML learns one embedding per image scale; LRPABN and its follow-up add spatial alignment and low-rank second-order pooling §2.1.2.
• Task-specific features §2.1.3. TADAM applies task-conditioned affine transforms to each convolution, with an auxiliary standard classification loss on the base set §2.1.3. LGM-Net generates the weights of a Matching Network target from the support set §2.1.3. A Category Traversal Module produces a feature mask that can be added to most metric methods §2.1.3. FEAT applies a Transformer set-to-set mapping to the support features §2.1.3. XtarNet adapts features and prototypes for generalised few-shot classification §2.1.3.
• Augmentation and multi-task learning §2.1.4 §2.1.5. Hallucination networks generate extra support features §2.1.4; semantic feature synthesis perturbs word vectors and projects them into visual space §2.1.4. Auxiliary losses include a global classifier over base classes (TADAM), rotation prediction (Gidaris et al.), attribute prediction and neural architecture search §2.1.5.

Stage 2, learning class representations §2.2. Prototypes are reference vectors per class, built as averages or learned end-to-end. The review covers three kinds.

• Feature-embedding prototypes §2.2.1. Prototypical Networks take the mean support embedding as the prototype and classify by Euclidean distance, trained with cross-entropy over episodes §2.2.1. Variants use covariance matrices, semi-supervised soft k-means with unlabelled data, infinite mixtures of Gaussians, query-dependent attention weights, a robust attention version against outliers and label noise, and a Voronoi-diagram interpretation §2.2.1.
• Point-based learnable prototypes §2.2.2. Prototypes are a learned function of the class's embeddings (shot-free) §2.2.2. TapNet learns prototype parameters and projects them into a task space with a fixed linear nulling operator §2.2.2. KGTN and its extension propagate prototypes over a graph of classes §2.2.2. SOC learns several prototypes per novel class §2.2.2. PHR refines prototypes iteratively using all query images §2.2.2. One method treats prototypes as latent variables with amortised variational inference §2.2.2.
• Distribution or subspace representations §2.2.3. A variational Bayesian framework models each class distribution with a KL-based distance §2.2.3. Deep Subspace Networks build a per-class subspace by SVD and classify by the nearest subspace §2.2.3.

Stage 3, learning distance or similarity measures §2.3. Earlier methods use a fixed measure, such as Euclidean distance or cosine similarity §2.3. The review covers two families.

• Analytical measures §2.3.1. TADAM analyses how the metric scale affects the loss, and many later works tune this scale by cross-validation §2.3.1. Two scale parameters can separate the true class from the others, and they are tuned with self-paced scheduling §2.3.1. Simple CNAPS uses a Mahalanobis distance with regularised task and class covariances §2.3.1. TEAM learns a task-specific Mahalanobis metric §2.3.1. DeepEMD uses an Earth Mover's Distance over dense features §2.3.1. BDC uses a characteristic-function-based distance §2.3.1.
• Learned similarity networks §2.3.2. The Relation Network learns the similarity of support and query embeddings with a small convolutional and fully connected module §2.3.2. BSNet adds a cosine branch alongside it §2.3.2. SAML uses an MLP over a local-feature relation matrix §2.3.2. Graph-based methods (GNN-FSL, EGNN, TRPN, DPGN) treat samples or support-query pairs as graph nodes and learn edges or relations §2.3.2.

Challenges §3. The review names four open problems: learning generalised features from few samples, rethinking episodic training (including episode quality and underfitting of the base set), robustness to support and query perturbations and distribution shift, and cross-domain metric learning §3.

Hyperparameters

The review has no training configuration of its own. The evaluation and dataset settings it describes are below. Splits are the standard ones as written by the review §1.3.

• Meta-Dataset: multi-domain setting trains on the official training splits of the first eight of its 13 datasets and tests in-domain on those eight and out-of-domain on the other five. Single-domain setting trains on ImageNet only and tests on ImageNet and 12 other datasets §1.3.
• Episode parameters: C-way K-shot with K typically 1 to 5 §1.1. The number of episodes E, and the value of M, are not stated.

Results

No results of its own: the review runs no experiments and gives no results table § Conclusions. The abstract's claim that these methods reach state-of-the-art accuracy is the authors' summary and cites no numbers § Introduction. For figures, use the original papers, and check the setting (dataset, backbone, shot count, transductive or not) before comparing.

The review does quote two numbers from other sources, and they are not results of this paper: the Omniglot 5-way 1-shot accuracy above 99% §1.3 and the finding that simple transductive fine-tuning beats meta-trained methods under large domain shift §3. Treat both as the review's citations.

Paper vs code

No official code. The review is a survey with no method implementation, and the LaTeX source contains no code link [raw/src/2105.08149/main.tex]. No GitHub search was run for this card, because a survey has no method code to find.

Pitfalls

• No numbers to reuse. The review gives no comparison tables. Anyone who needs accuracies must take them from each cited paper, under that paper's own setting.
• Algorithm 1 typo. Step 6 of alg. 1 labels the query set with the support symbol D_S^(e). The text makes clear the query set is meant §1.2 alg. 1.
• Accuracy notation. The accuracy in alg. 1 is a mean of an element-wise indicator over vectors of labels. For the usual single-query case, it is simply the fraction of correct predictions. The review does not spell out how ties or multiple queries are handled.
• Overlapping categories. Some methods appear under two stages, for example those that learn features and also a distance. The taxonomy figure says so fig. 2.
• Coverage ends in 2022. The review covers 2018 to 2022 § Introduction. Later methods are not included.
• Dataset counts. Omniglot's count of 6,492 includes rotated copies of the 1,623 characters §1.3. Read the split before quoting class totals.
• Meta-Dataset protocols. The review describes two training procedures and two evaluation protocols, and calls multi-domain training the more common one. It gives no performance numbers for either §1.3.

Lineage

• Surveys it compares itself with: Shu et al. (2018), Wang et al. (2020), Lu et al. (2020) and Li et al. (2021) § Introduction. The review says earlier surveys cover metric learning either thinly or only as meta-learning.
• Method families it builds on: Siamese Networks §2.1.1, Matching Networks §2.1.1, Prototypical Networks §2.2.1 and Relation Networks §2.3.2.
• Later methods it reviews: TADAM §2.1.3, FEAT §2.1.3, TPN §2.1.1, DN4 §2.1.2, DeepEMD §2.3.1 and the GNN family §2.3.2.