Free Lunch for Few-shot Learning: Distribution Calibration

In plain words

When a classifier sees only one or five examples of a class, it tends to learn those few examples rather than the class as a whole. The spread of the examples is unreliable because there are so few of them.

The paper's idea is to borrow that spread from classes that have plenty of data. Each class is described as a cloud of feature vectors, with an average and a spread. For a new class with few examples, the average and spread are pulled toward those of the most similar well-studied classes. From that corrected cloud, the method draws many made-up feature vectors and trains an ordinary logistic regression on the real and invented vectors together. Nothing new is learned, and no generator network is needed.

This matters because the approach sits on top of any pretrained feature extractor and any simple classifier, so it is cheap to try. The catch is that it depends on the features roughly following a bell curve per dimension, which the paper handles with a power transform.

TL;DR

A training-free feature augmentation for few-shot classification. Each support feature gets a Gaussian whose mean and covariance are averaged from its k = 2 nearest base classes §3.2.2 eq. 4–6, after a Tukey power transform with λ = 0.5 §3.2.2 eq. 3. The method samples 750 features per class from these Gaussians §3.2.3 eq. 7 and trains a logistic regression on support plus sampled features eq. 8. The backbone is frozen and no parameters are learned §3.2. Author-reported 5-way results with logistic regression are 68.57 and 82.88% (miniImageNet 1/5-shot), 79.56 and 90.67% (CUB) and 78.19 and 89.90% (tieredImageNet) tab. 2 tab. 3. Read it when you have a fixed pretrained embedding and few labels per class. Do not treat the numbers as controlled comparisons: the CUB α and several details differ from the released code (see Paper vs code).

Problem

Few-shot classification trains on base classes with plenty of data and tests on novel classes with only N × K labelled support examples §3.1. With few examples the estimated per-class distribution is biased, and the classifier overfits to those samples §1. Estimating a class's distribution from only a few samples is ill-posed §3.2. Prior generators (GANs, VAEs, hallucination networks) fix this by learning a model with extra loss terms and parameters, which the authors view as costly §2.

The method assumes that each feature dimension is roughly Gaussian and that similar classes share similar means and variances §1 §3.2. The paper motivates this with class-similarity percentages for one example class tab. 1. It assumes the features are non-negative, which holds for post-ReLU pooled features §4.1.3.

Method

The pipeline has three parts. Base-class statistics are computed once from base features. Each novel support feature is then calibrated into a Gaussian, and a logistic regression is trained on support plus sampled features. Query features are transformed the same way at test time §3.2.2.

Base-class statistics §3.2.1. For each base class i with n_i feature vectors, the mean is the average feature eq. 1. The covariance divides by n_i − 1, the unbiased estimator eq. 2. These are computed once and reused for every task.

Tukey power transform §3.2.2. The support and query features of a task are transformed by a power λ. For λ ≠ 0 the transform is the λ-th power. For λ = 0 it is the logarithm eq. 3. λ = 1 leaves features unchanged. Lowering λ reduces positive skew §3.2.2. The paper uses λ = 0.5 §4.1.3 and reports it as the best value in its sweep §4.5, fig. 3.

Neighbour selection §3.2.2. For each transformed support feature x, compute the squared Euclidean distance to every base mean μ_i eq. 4. The paper selects the k base classes with the smallest distance, written as a top-k over negated distances eq. 5. The paper uses k = 2 §4.1.3.

Calibrated Gaussian §3.2.2. The calibrated mean averages the k neighbour means and the support feature itself, so the sum divides by k + 1 eq. 6. The calibrated covariance averages the k neighbour covariances and then adds a constant α eq. 6. Section 4.6 clarifies that α is added to every element of the matrix §4.6. Each support feature of a class yields its own Gaussian, so a K-shot class gets K Gaussians §3.2.2.

Sampling §3.2.3. The total number of generated features per class is a hyperparameter, set to 750 §4.1.3. It is split equally across the K Gaussians of that class §3.2.3 eq. 7.

Classifier §3.2.3. The classifier is trained with cross-entropy over the transformed support features and the generated features eq. 8. The paper uses scikit-learn's logistic regression with default settings §4.1.3. The paper also reports maximum-likelihood and SVM variants trained on the same data tab. 2 tab. 3.

The pseudocode below follows alg. 1 and the equations above.

# Base-class statistics, computed once from base features  [§3.2.1]
for each base class c:
    # Mean of the n_c feature vectors of class c  [eq. 1]
    base_means[c] = mean(base_features[c], axis=0)
    # Unbiased covariance, dividing by n_c - 1  [eq. 2]
    base_covs[c] = covariance(base_features[c])

# One N-way K-shot task  [alg. 1]
# support: (N*K, d) non-negative features; query: (N*q, d)
# base_means: (C_b, d); base_covs: (C_b, d, d)
# alpha: 0.21 (miniImageNet, tieredImageNet), 0.3 (CUB)  [§4.1.3]

def calibrate_and_fit(support, labels, query, base_means, base_covs):
    # Tukey power transform with lambda = 0.5 on both sets  [§3.2.2] [eq. 3]
    support = support ** 0.5
    query = query ** 0.5
    generated_features = []
    generated_labels = []
    # Calibrate one Gaussian per support feature  [§3.2.2]
    for feature, label in zip(support, labels):
        # Indices of the k = 2 nearest base means; the paper uses squared
        # distance, which ranks the same as plain distance  [eq. 4] [eq. 5]
        nearest = k_nearest(base_means, feature, k=2)
        # Calibrated mean: average of neighbour means and the feature  [eq. 6]
        mean = (base_means[nearest].sum(axis=0) + feature) / 3
        # Calibrated covariance: neighbour average plus alpha  [eq. 6] [§4.6]
        # alpha is added to every entry, not only the diagonal  [§4.6]
        cov = base_covs[nearest].sum(axis=0) / 2 + alpha
        # Draw 750 / K features per support feature  [§4.1.3] [eq. 7]
        count = 750 // K
        draws = sample_gaussian(mean, cov, count)
        generated_features.append(draws)
        generated_labels.append(repeat(label, count))
    # Train logistic regression on support and generated features  [eq. 8]
    classifier = fit_logistic_regression(
        concat(support, generated_features),
        concat(labels, generated_labels),
    )
    # Predict the transformed query features  [§3.2.2]
    predictions = classifier.predict(query)
    return predictions

Figure descriptions only, since the card carries no images. Figure 1 shows a classifier overfitting a few examples versus one trained on calibrated samples fig. 1. Figure 2 is a t-SNE view of support, generated and query features for one class fig. 2. Figure 3 shows accuracy against λ and against the number of generated features fig. 3. Figures 4 and 5 sweep k and α fig. 4 fig. 5. Figure 6 in the appendix shows base and novel class distributions before and after the Tukey transform fig. 6.

Hyperparameters

Results

Author-reported. Setup: 5-way tasks sampled from novel classes, averaged over 10,000 tasks, top-1 accuracy with 95% confidence intervals §4.1.2 tab. 2 tab. 3. The paper names the feature extractor and the splits but does not give the GPU or hardware. The baselines are the paper's own table entries, not re-runs.

miniImageNet and CUB, 5-way, 1-shot and 5-shot accuracy (%) tab. 2

The paper bolds its methods where they overlap the best method's interval tab. 2. Against the best baseline in this table, the logistic regression variant is ahead by about 4.8 points at miniImageNet 1-shot (E3BM, 63.80) and 1.9 points at 5-shot (Negative-Cosine, 80.94). On CUB it is ahead by 6.6 points at 1-shot (Prototypical Net, 72.99) and 1.3 points at 5-shot (Negative-Cosine, 89.40). These gaps are computed from the table, not stated by the paper.

tieredImageNet, 5-way, 1-shot and 5-shot accuracy (%) tab. 3

Ablation on miniImageNet 5-way, accuracy (%) tab. 4

Backbones, miniImageNet 5-way 1-shot, without and with DC tab. 5

Baselines, miniImageNet 5-way 1-shot, without and with DC tab. 6

Appendix results, all miniImageNet 5-way 1-shot: nearest-class retrieval with 1 feature gives 62.39 ± 0.49, with 5 gives 59.73 ± 0.42, with 10 gives 58.93 ± 0.49 and with 100 gives 57.33 ± 0.48. Sampling 100 features from the calibrated distribution gives 68.53 ± 0.32 tab. 7 app. A. Using only the base mean without the query feature in the mean gives 59.38 ± 0.73, against 68.57 ± 0.55 with it tab. 8 app. B. The paper reports that DC's gain grows with the similarity of the nearest base classes, over three novel classes only tab. 9 app. D.

Paper vs code

Code: github.com/ShuoYang-1998/Few_Shot_Distribution_Calibration at commit 5aab53eb4b5f102119ce9c71a6fda8b528cba48f. The repository has no LICENSE file, so it carries no license grant. We cite it and do not quote it. The paper links it in its abstract, and the owner name matches the first author; this is the basis for treating it as the authors' code.

Differences found:

• CUB α. The paper uses α = 0.3 for CUB §4.1.3. The repository uses the function default α = 0.21 evaluate_DC.py and never overrides it when calling, so CUB runs with 0.21 evaluate_DC.py. The evaluation script runs only miniImageNet as its dataset value evaluate_DC.py. The CUB value is therefore not wired anywhere in the released code.
• Logistic regression settings. The paper says the scikit-learn defaults are used §4.1.3. The code sets max_iter=1000 evaluate_DC.py.
• Confidence intervals. The paper reports 95% intervals tab. 2 tab. 3. The script prints only the mean accuracy over tasks and computes no interval evaluate_DC.py. The paper does not state its interval method.
• Datasets. The paper evaluates tieredImageNet §4.1.1. The code's loader knows only miniImageNet and CUB FSLTask.py. tieredImageNet results cannot be reproduced from this repository.
• Classifier variants. The paper reports maximum-likelihood and SVM variants tab. 2 tab. 3. The code trains only the logistic regression evaluate_DC.py.
• Validation. The paper says hyperparameters were chosen on validation classes §4.6. The evaluation script runs only on novel tasks and has no validation loop evaluate_DC.py.
• Query count. The paper gives no number of query samples per class, only q §3.1. The code uses 15 evaluate_DC.py.
• Base statistics. The paper does not say whether base statistics use the Tukey-transformed features. The code computes them from untransformed base features evaluate_DC.py, while the support and query features are transformed evaluate_DC.py.
• Distance. The paper uses squared distances with a negated top-k eq. 4 eq. 5. The code takes the k smallest plain Euclidean distances evaluate_DC.py. The selected neighbours are the same.
• Generated count. The code uses int(750 / K) per support feature evaluate_DC.py, which totals 750 per class for K that divides 750. The paper's per-class total is consistent §3.2.3.

Agreements: λ = 0.5 evaluate_DC.py; k = 2 evaluate_DC.py; the mean and covariance rules evaluate_DC.py; the unbiased covariance evaluate_DC.py; 10,000 tasks evaluate_DC.py; and post-ReLU pooled features wrn_model.py save_plk.py.

Pitfalls

• Where α goes. Equation 6 is typeset as a plus α, which a reader may take as adding α to the diagonal. The text and code add α to every entry of the covariance matrix §4.6 evaluate_DC.py. The result is a covariance that is not diagonal, so sampled dimensions are correlated.
• Singular base covariance. Each base covariance has rank at most n_i − 1. For miniImageNet, n_i is 600 §4.1.1, so the rank is at most 599. The feature dimension is not stated. If it exceeds 599, the averaged covariance is singular and only α makes it sampleable. This is a derived consequence, not a stated result.
• Mixed transforms. The query is transformed with λ = 0.5, while base means and covariances come from untransformed base features evaluate_DC.py. The paper does not state this choice. Check that your base statistics use the same features as the training path.
• Power of negatives. The Tukey transform needs non-negative inputs §4.1.3. The code raises features to 0.5 without checking the sign evaluate_DC.py. Non-negative features from ReLU are fine. Anything else gives NaN.
• Shot counts that do not divide 750. The code floors 750 / K per support feature, so the total per class falls short of 750 when K does not divide 750 evaluate_DC.py. The paper does not discuss this.
• Ablation text. The paper says generated features help below 500 per class §4.5 but uses 750 as its default §4.1.3. Section 4.5 calls the 1-shot gain "12% relative", but Table 4 shows 12.2 points absolute (56.37 to 68.57) tab. 4. Use the table numbers.
• Per-class truncation. The data loader truncates every class to the smallest class size in the dataset FSLTask.py. The paper does not state this. It matters only if classes are unevenly sized.
• Abstract mismatch. The arXiv metadata abstract says "two datasets" and the LaTeX abstract says "three datasets". The tables cover three, so this card follows the LaTeX.
• Weak evidence in appendix. The similarity analysis app. D rests on three novel classes. Treat the trend as anecdotal.
• Unstated. The feature dimension, the logistic-regression regularisation strength beyond defaults, the query count per class and the hardware are not stated. The CI method is not stated.

Lineage

• Builds on: the WideResNet backbone trained as S2M2_R, cited for feature extraction and not compiled here §4.1.3 README.md.
• Related generation methods, named but not compiled: MetaGAN, Delta-Encoder, TriNet, Meta Variance Transfer, and the hallucination models cited in §2 tab. 2.
• Compared against: MAML, Meta-SGD, LEO, E3BM, Matching Net, Prototypical Net, Baseline++, Variational Few-shot, Negative-Cosine, DeepEMD tab. 2 tab. 3.
• Superseded by: not stated in this paper.