Few-shot learning

Overview

Few-shot learning asks a model to predict over a label set it has only seen a few labelled examples of, typically one to five per class 2003.11539 §1. Most variants split the problem into a support set (the labelled examples for the new task) and query inputs to label. Most of them leave the base network's weights unchanged at meta-test time: the embedding is never fine-tuned 2003.11539 §3.2, the calibration route keeps its backbone frozen and learns no parameters, using only base-class statistics stored once 2101.06395 §3.2.1 §3.2, the language model's weights are untouched when demonstrations are placed in the prompt 2202.12837 §3, the vision cache is set from the support set without gradient steps 2111.03930 §3.2, grammar-constrained decoding changes only which tokens may be chosen 2305.13971 §2.2, and the in-context segmenter takes the support set as input with no gradient steps 2304.06131 §3. Some routes do change weights. The adapter route freezes the base model and trains a small set of added scaling vectors 2205.05638 §3.3. SetFit fine-tunes a sentence encoder with a contrastive loss on pairs built from the support set 2209.11055 §3.1. Channel prompt tuning trains 20 prompt vectors on the support set 2108.04106 §4.2.3. Four more routes learn prompt vectors, not discrete words. PPT pre-trains the vectors on unlabelled text before tuning 2109.04332 §3.1. MaPLe trains prompt tokens in both CLIP branches 2210.03117 §3.2. P-Tuning trains continuous slots around a hand-written pattern, and in its few-shot SuperGLUE setting also tunes the model 2103.10385 §2.2 §3.3. ProGrad trains CLIP context vectors but drops the part of each cross-entropy gradient that works against a zero-shot teacher 2205.14865 §3.2. GlobalE changes no weights at all: it uses the few examples only to choose the order of the demonstrations 2104.08786 §3.

Detection is covered too. FSCE fine-tunes a detector on a few boxes per novel class and reports nAP50 rather than accuracy 2103.05950 §4.1. Its numbers are not classification results.

Scope. "Few-shot" also names few input views in neural rendering. FreeNeRF regularizes a radiance field fitted to a handful of photos 2303.07418 §3.2. It predicts no labels, so its results are not evidence for the routes below. It is listed only to mark the difference. DietNeRF is a second case of the same kind: it fits a radiance field to eight training views and uses a frozen CLIP encoder only as a loss on unseen poses 2104.00677 §4.1 alg. 1. Its reported PSNR and FID are for rendered views 2104.00677 tab. 1, so it is not evidence for the routes below either. FLEURS shares the title phrase but is a multilingual speech benchmark. Its baselines are fine-tuned on all 102 locales 2205.12446 §4.1, and the results on its card do not come from a few-shot setting, so they are not evidence for the routes below either.

Not classifiers. Two variants use the few examples to make data or drafts rather than predictions. Promptagator has an LLM write training queries, and the dual encoder and reranker trained on them are what run at query time 2209.11755 §4.1. The LLM annotation study produces draft labels that experts correct. Their results measure the data or the annotation effort, not the accuracy of a few-shot predictor.

Map of the field. A survey groups few-shot methods by the problem each one addresses: data augmentation, transfer learning, meta-learning and multimodal learning 2205.06743 §2.6. Use it to find where to look, not as evidence. It runs no experiments, and its computer-vision tables copy numbers from cited papers under mixed protocols 2205.06743 §7 tab. 9. A broader meta-learning survey lists few-shot learners as learned initialisations, non-parametric matchers, prototype models and memory-based learners 1810.03548 §4.3. It reports no experiments and has no code.

Variants

Frozen embedding plus a per-episode linear classifier

Pool all base-class training data into one ordinary classification problem, train a network with cross-entropy, discard its classification head, and fit an L2-normalised multinomial logistic regression on each test support set 2003.11539 §3.2 eq. 4 eq. 5 eq. 6. Card: Rethinking Few-Shot Image Classification.

• Reported with a ResNet-12 on miniImageNet 5-way: 62.02 ± 0.63 (1-shot) and 79.64 ± 0.44 (5-shot) for the simple version 2003.11539 tab. 1.
• On Meta-Dataset, trained on ILSVRC only with a ResNet-18, the same linear base learner beats the best prior per-dataset number on 9 of 10 test sets, but trails on Aircraft 2003.11539 §5 tab. 7.
• The authors attribute the strength of this baseline to merging mutually exclusive tasks into one many-way problem. A multi-head alternative scored lower on miniImageNet 5-way 1-shot 2003.11539 §4.9.

Born-again distillation on top of the embedding

Train a fresh network of the same architecture to match both the labels and the soft predictions of the previous generation, then repeat for a chosen number of generations 2003.11539 §3.3 eq. 7 eq. 8. The new network is re-initialised, not warm-started 2003.11539 §4.1. Same card as above.

• Reported: 64.82 ± 0.60 (1-shot) and 82.14 ± 0.43 (5-shot) on miniImageNet with ResNet-12, using two generations 2003.11539 tab. 1 §4.6.
• Generation count was picked on the validation set: two for miniImageNet, CIFAR-FS and FC100, one for tieredImageNet 2003.11539 §4.6.
• Cost: every generation is a full training run with the same schedule 2003.11539 §4.1, so two generations add two more training runs.

Feature distribution calibration with sampling

Keep the frozen embedding, but do not fit the support set alone. Each support feature is turned into a Gaussian whose mean and covariance are averaged from its two nearest base classes, after a Tukey power transform with λ = 0.5 2101.06395 §3.2.2 eq. 3 eq. 6. The method then samples 750 features per class from these Gaussians and fits a logistic regression on the support set plus the samples 2101.06395 §3.2.3 eq. 7 eq. 8. No parameters are learned 2101.06395 §3.2. Card: Free Lunch for Few-shot Learning: Distribution Calibration.

• Reported 5-way results with logistic regression on WideResNet features trained as S2M2_R 2101.06395 §4.1.3: 68.57 ± 0.55 (1-shot) and 82.88 ± 0.42 (5-shot) on miniImageNet; 79.56 ± 0.87 and 90.67 ± 0.35 on CUB 2101.06395 tab. 2; 78.19 ± 0.25 and 89.90 ± 0.41 on tieredImageNet tab. 3. Author-reported; the baselines are the paper's table entries, not re-runs §4.1.2.
• Ablation on miniImageNet 1-shot: 56.37 with neither part, 64.30 with the transform only, 63.70 with sampling only, 68.57 with both [arxiv:2101.06395 tab. 4].
• The method assumes each feature dimension is roughly Gaussian, that similar classes share similar statistics, and that features are non-negative, as post-ReLU pooled features are 2101.06395 §1 §3.2 §4.1.3. The power transform is the paper's answer to skew.

Reconstructing spatial feature maps from the support set (FRN)

Keep the backbone's spatial feature map rather than pooling it. For each class, stack the feature vectors of all support images, across every spatial location, into one pool. For a query map, solve a closed-form ridge regression that rebuilds the query from that pool, and score the class by the mean squared reconstruction error over locations 2012.01506 §3.1 eq. 1 eq. 4. A learned penalty and a learned rescaling of the reconstruction are added to the backbone, together with a softmax temperature, three learned parameters in all §3.2 eq. 6. No gradient step runs at test time §4.2. Pre-training on the base classes gives each class one learnable map as its pool. That head is discarded afterwards §3.6. Card: Few-Shot Classification with Feature Map Reconstruction Networks.

• Author-reported, ResNet-12, 5-way, cropped CUB: 83.16 (1-shot) and 92.59 (5-shot), against 80.80 and 91.19 for the best listed baseline, a re-implemented Deep Subspace Network 2012.01506 tab. 3. On uncropped CUB the figures are 83.55 and 92.92 tab. 4.
• The gains on the ImageNet-derived benchmarks are mixed. On mini-ImageNet, FRN is 0.33 points below FEAT at 1 shot (66.45 against 66.78) and 0.42 points above DeepEMD at 5 shots 2012.01506 tab. 7. On 84-pixel tiered-ImageNet it is below the best listed baseline in both columns tab. 7.
• Test-time cost, as the paper's latency table reports it: 63 ms per 5-way mini-ImageNet episode at 1 shot and 79 ms at 5 shots, against 178 ms and 18,292 ms for DeepEMD with OpenCV. The hardware is not stated [arxiv:2012.01506 tab. 1] §4.2 [SM Computational Efficiency].
• Training matters. On mini-ImageNet, episodic training from scratch gives 63.03 and 78.01, average-pool pre-training followed by fine-tuning gives 62.13 and 76.28, and FRN's own pre-training followed by fine-tuning gives 66.45 and 82.83 2012.01506 tab. 9. A model trained on 1-shot episodes is worse than one trained on 5-shot episodes, even when tested at 1 shot §5.1.
• The auxiliary loss and the learned penalty and scale change results only a little, and not consistently across backbones [arxiv:2012.01506 SM Auxiliary Loss and Regularization Parameters].
• The released code differs from the paper in the auxiliary loss, in the pre-training penalty and in two small constants. Check the card's Paper vs code section before reproducing the auxiliary term.

Contrastive fine-tuning of a sentence encoder (SetFit)

Build pairs from the labelled examples: two examples with the same label form a positive pair, two with different labels a negative pair. Tune a sentence-encoder body so that the cosine similarity of each pair matches its 0 or 1 target, then fit a logistic-regression head on the body's embeddings 2209.11055 §3.1. No prompts or verbalizers are written §1. Card: Efficient Few-Shot Learning Without Prompts.

• Author-reported, with 8 examples per class over five text classification datasets (AG News left out of the average): 62.3 for the 110M-parameter MPNet body, against 63.4 for T-Few 3B and 58.3 for ADAPET 2209.11055 tab. 2. At 64 per class, the same five-dataset average is 75.3 against 70.3 for T-Few 3B tab. 2.
• Cost is estimated from FLOP formulas, not measured: about 19 times fewer inference FLOPs than T-Few 3B at 8 per class 2209.11055 tab. 5 §7.2.
• Standard deviations are large at 8 per class. For the Amazon Counterfactual set the score is 40.3 ± 11.8 tab. 2, so compare methods on the spread as well as the mean.
• With a multilingual body at 8 per class, the average error over six languages of MARC is 86.4 to 88.3 (lower is better), against 117.2 to 121.9 for fine-tuned XLM-R base. The two use different bodies [arxiv:2209.11055 tab. 4] §6.

Training-free cache over CLIP features (Tip-Adapter)

Store the normalised CLIP image feature of each few-shot training image as a key and its one-hot label as a value. For a test image, compute an exponential affinity to every key, sum the labels weighted by affinity, scale the sum by α, and add it to CLIP's zero-shot logits 2111.03930 §3.2 eq. 5 eq. 6. Nothing is trained for this variant §3.2. Card: Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling.

• ImageNet, ResNet-50, 16 shots: 62.03 training-free, against 60.33 for zero-shot CLIP and 63.59 for CLIP-Adapter, which learns its adapter by gradient descent 2111.03930 tab. 2.
• The keys can be tuned for 20 epochs (Tip-Adapter-F). That gives 65.51 at 16 shots, against 63.59 for CLIP-Adapter after 200 epochs tab. 2. The paper reports 5 minutes on one NVIDIA GeForce RTX 3090 for Tip-Adapter-F, against 50 minutes for CLIP-Adapter tab. 3.
• The cost moves from training time to memory: the cache holds NK × C features 2111.03930 §3.2. The paper gives no separate timing for building it tab. 3.
• Per-dataset results at 16 shots appear only in a figure [arxiv:2111.03930 fig. 4]. The paper does not say how the fine-tuning epoch was chosen, so check the card's Paper vs code section before reproducing the numbers.

Grammar-constrained decoding with few-shot prompts (GCD)

Keep the model frozen and prompt it with a short instruction and a few demonstrations, as in few-shot prompting. At each step a parser returns the tokens that keep the output inside a grammar, and every other token is removed before the choice 2305.13971 §2.2 §2.3. The grammar may depend on the input: for entity disambiguation, the candidate set for each mention is part of the grammar §3.2. The constraint fixes the form of the answer and leaves the demonstrations as they are. Card: Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning.

• Author-reported, LLaMA-33B with four demonstrations. Closed information extraction F1 rises from 17.5 unconstrained to 36.0 constrained 2305.13971 tab. 1. Entity disambiguation average accuracy rises from 54.1 to 80.3 with input-dependent candidates tab. 2. Constituency parsing F1 rises from 42.9 to 54.6, and valid trees from 64.2% to 100% tab. 3.
• The method needs the next-token distribution, so models behind an API that does not expose it cannot be constrained 2305.13971 §2.2 §6.
• Latency: the 2.7M-entity extraction grammar adds about 69 ms per token on a consumer CPU, against 54 to 136 ms per token for the LLM forward pass on an A100 tab. 4 §4.4.
• With grammars that allow an empty output, the most probable complete output can be empty. For LLaMA-13B the paper reports an empty top output at length exponents 1.0 to 2.0, and none at 2.5 or higher §5 tab. 5.

Mixed zero-shot and few-shot templates in instruction tuning (Flan)

Instruction tuning on a mixture in which each training example is shown either zero-shot or with 2, 3 or 5 worked exemplars 2301.13688 §3.2 [app. A.1]. The model is then tested in both settings. This is a training recipe, not an inference method, so it applies only when you can tune the model. Card: The Flan Collection: Designing Data and Methods for Effective Instruction Tuning.

• Ablation on T5-XL 3B: removing the few-shot templates lowers few-shot held-in accuracy from 74.8 to 62.2 and few-shot MMLU from 52.4 to 38.7 2301.13688 tab. 1. Zero-shot held-in accuracy falls less, from 73.8 to 72.5 tab. 1.
• A small share of few-shot templates already helps zero-shot results, and the best share lies in a broad range. The paper does not state the share used in its final mixture §3.2 fig. 3.
• The ablations are single runs with no error bars 2301.13688 tab. 1 tab. 2.

In-context demonstrations for a frozen language model

Place k labelled input-label pairs in the prompt, then ask for the label of a new input. Each candidate label is scored by the model, and the best one is returned 2202.12837 §3 §4.1. Card: Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?.

• Setting used by the study: k = 16 demonstrations, direct and channel scoring, 26 datasets of classification and multi-choice, six decoder-only models from 774M parameters to GPT-3 2202.12837 §3 tab. 1.
• Replacing gold labels with uniform random labels costs about 0-5 points absolute on nearly all 12 model/inference pairs 2202.12837 §4.1 fig. 3. The per-dataset gap reaches nearly 14 points for one model on one dataset 2202.12837 §4.1 app. C.2.
• Ablations point to the label space, the input text distribution and the input-label pair format as the main sources of the gain, more than label correctness 2202.12837 §5.
• Scope: the findings cover classification and multi-choice with label words the model already understands. Generation is not studied, and later work cited by the authors reports that negated labels hurt substantially 2202.12837 Limitation.

Ordering demonstrations without labels (GlobalE)

Build candidate prompts by ordering the same demonstrations, up to 24 orders. For each candidate, the frozen LM writes unlabelled probe inputs, and every candidate then labels all probe inputs. A candidate scores by the entropy of the label distribution it predicts over those probes. The top four are kept 2104.08786 §3.1 §3.2 §4. The reasoning is that an order which predicts one label for almost every probe scores low 2104.08786 §2. No gold labels are used in the selection §3.2. Card: Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity.

• Author-reported: GlobalE gives an average relative gain of 13% over the unselected baseline on 11 text classification tasks. The alternative score, LocalE, gives 9.6% 2104.08786 §5 tab. 2.
• Example: GPT-2 0.8B on SST-2 moves from 74.5 (mean over orders) to 84.8 with GlobalE, against 88.9 for an oracle that picks by validation accuracy 2104.08786 tab. 2. The oracle uses labels, so it is an upper bound, not a few-shot method.
• Cost: each candidate needs its own generation, and every probe input needs a forward pass under every candidate. The GPT-3 175B runs were capped at 12 orders and two sample sets 2104.08786 §4.
• Entropy does not see which label wins. Two labels split 80/20 in either direction score the same, and the paper does not discuss ties 2104.08786 §3.2.
• The released code implements GlobalE only. LocalE is reported in the paper but has no code in the repository, so check the card's Paper vs code section before reproducing it.

Few-shot parameter-efficient fine-tuning (T-Few recipe)

Start from an instruction-tuned encoder-decoder (T0, 11B parameters), freeze its weights, and learn (IA)3 scaling vectors on each task's few-shot set. The training objective is the language-modelling loss plus an unlikelihood loss on wrong answers and a length-normalised softmax over the answer choices, summed with unit weights 2205.05638 §3.2 eq. 1 eq. 2. The vectors are first pre-trained on the T0 multitask mixture §3.4. The fine-tuning settings are the same for every task: 1,000 steps, Adafactor, learning rate 3e-3 §3.5. Card: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning.

• Author-reported, on nine held-out T0 tasks with median accuracy over five few-shot subsets: 72.4% average, against 66.9% for zero-shot T0 and 66.6% for GPT-3 175B in-context learning 2205.05638 tab. 1 §4.2. The GPT-3 numbers are copied from the GPT-3 paper, on different few-shot subsets app. F.
• Inference cost: the paper estimates about 1,000 times fewer inference FLOPs per prediction than GPT-3 175B in-context learning tab. 1. Training the vectors costs about as much as running GPT-3 175B in-context on 20 examples §4.2.
• Ablations on the held-out tasks: removing the (IA)3 pre-training gives 70.8, removing the two extra losses gives 68.3, against 72.4 for the full recipe 2205.05638 §4.4 app. G.
• On RAFT, T-Few scores 75.8% against 73.5% for the human baseline, with the same recipe and no per-task tuning tab. 2 §4.3.
• The text says (IA)3 is the only PEFT method that beats full fine-tuning. The appendix averages do not support "only": LoRA is ahead of full fine-tuning in two of three printed settings, by small margins §3.3 app. D.
• Storage does not reconcile: the paper reports 4.2 MB for the vectors, but its appendix lists 540K parameters, about 1.05M at 4 bytes each tab. 1 app. D. Measure the checkpoint before budgeting storage.
• Scope: classification and multiple-choice tasks only; generation is future work §5. Prompt tuning in the same comparison was unstable in training §3.3.

Noisy-channel scoring of few-shot prompts

Score each label by how well it explains the input: the probability of the input text given the verbalized label, under a frozen causal language model. Under a uniform label prior this replaces the usual probability of the label given the input 2108.04106 §3. Demonstrations are either concatenated into one context or run one at a time and their probabilities multiplied. The second form, the ensemble, needs O(K) memory rather than O(K²) and does not depend on example order §4.1.2 §4.1.3. The route can also tune 20 prompt vectors on the support set while the LM stays frozen §4.2.3. Card: Noisy Channel Language Model Prompting for Few-Shot Text Classification.

• Author-reported, GPT-2 Large, K = 16 training examples over 11 text classification datasets: channel prompt tuning averages 61.7 with worst case 53.0, against 57.7 (47.1) for direct head tuning 2108.04106 tab. 4. Averages are over verbalizers and seeds §5.4.
• Without tuning, channel scoring with concatenated demonstrations averages 58.9 (worst case 50.8), above the best direct demonstration model, ensemble Direct++ at 55.8 (43.6) 2108.04106 tab. 3 §6.1.
• Direct head tuning is better on TREC (54.1 against 37.1 for channel prompt tuning) and Subj (86.3 against 75.5) tab. 4 §6.2. The paper calls these tasks far from language modelling §6.2. With K = 64, head tuning beats channel prompt tuning §6.3.
• The paper states that direct models cannot predict labels unseen in training, and channel prompt tuning outperforms all finetuning on the 2-way datasets §6.4.
• Cost grows with the number of labels, because each label needs its own pass over the input, and with the number of demonstrations for the ensemble form §3 §4.1.3.
• The released code differs from the appendix in training schedule and several details, so check the card's Paper vs code section before reproducing numbers.

Segmentation with episodic support sets and cost aggregation (VAT)

A dense variant. For a query image and K support image-mask pairs, build the correlation between every query position and every support position from frozen ImageNet ResNet features, after masking the support features to the object. A 4D convolutional Swin Transformer then cleans up that volume across a three-level pyramid 2207.10866 §3.3 §3.4. An appearance-aware transformer decodes the mask §3.5. Each support pair is run once, and the per-pixel foreground votes are summed for K-shot prediction §3.6. The class is unseen in training, and no retraining is done for it §3.1. Card: Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation.

• Author-reported 1-shot results: PASCAL-5i mIoU 67.9 against 66.2 for HSNet, both ResNet-101 2207.10866 tab. 1; COCO-20i 41.3 against 39.2 with ResNet-50 tab. 2; FSS-1000 90.3 against 86.5 with ResNet-101 tab. 3.
• Memory is the limit, because the full volume is built for every query-support pair. A standard transformer over it ran out of memory, at about 84 GB in the authors' table, while VAT's aggregator used 3.8 GB. Batch size and shot count for that table are not stated 2207.10866 tab. 5.
• Swapping the aggregator for a linear transformer or Fastformer gives 87.7 and 87.8 mIoU on FSS-1000, against 90.3 for VAT tab. 5.
• The paper states no training augmentation §4.1, but the released configs enable one, and the code's K-shot vote uses a different threshold from the paper's rule §3.6. The card's Paper vs code section lists both. Settle which setting you need before comparing with reported numbers.

Frozen diffusion features with a pixel-classifier ensemble (DDPM segmentation)

A dense variant with no episodes. A pretrained diffusion model is used as a fixed feature extractor. Each labelled image is corrupted to three diffusion steps, t = 50, 150 and 250, and the activations of five UNet decoder blocks are read at each step. Each activation is upsampled to image size, and all of them are concatenated into 8448 features per pixel 2112.03126 §3.2. Ten small MLPs are trained on the pixel features and labels of the few annotated images, and each test pixel takes a majority vote across them §3.2 app. C. The annotated images form one training set for the ensemble, with no per-episode support set §3.2. Card: Label-Efficient Semantic Segmentation with Diffusion Models.

• Author-reported mean IoU, averaged over five runs with different data splits, with 20 to 50 training images per dataset: 49.4 on Bedroom-28 against 47.9 for DatasetDDPM and 45.0 for MAE, and 65.0 on Horse-21 against 63.4 for MAE 2112.03126 tab. 1 tab. 2.
• Fewer labels cost accuracy. Bedroom-28 falls from 49.4 at 40 images to 38.2 at 10, and Horse-21 from 65.0 at 30 images to 56.9 at 10 [arxiv:2112.03126 tab. 4].
• The noise draw barely matters. Sharing it between train and test, or across steps, moves Bedroom-28 mIoU by at most 0.4 points 2112.03126 tab. 5.
• Memory is the limit. The paper's sizes give 256 × 256 × 8448 float32 values per image, about 2.2 GB by calculation (derived). Each image also needs three full UNet forward passes, one per step 2112.03126 §3.2.
• The method needs a diffusion model trained on the target domain. The authors expect this to be hard for complex domains such as ImageNet or MSCOCO 2112.03126 §5.
• The paper says each MLP trains for about four epochs [arxiv:2112.03126 app. C]. The card's Paper vs code section records that the released code stops early on a loss criterion instead, so check it before reproducing numbers.

Few-shot detection with a contrastive proposal head (FSCE)

Start from a base-class Faster R-CNN with a ResNet-101 backbone and FPN. Fine- tune it on K shots of each novel class plus sampled base-class instances, with the ResNet frozen and the FPN, RPN and RoI head trained 2103.05950 §3 fig. 3. A contrastive head encodes each sampled RoI into a 128-dimensional vector. A supervised contrastive loss then pulls together the encodings of same-class RoIs in a batch and pushes the others apart, using only RoIs whose proposal IoU reaches a threshold φ §3.2 §3.3 eq. 2 eq. 3. The loss enters the total with weight λ = 0.5 eq. 5. The fine-tuning recipe also doubles the RPN proposals kept after NMS and halves the sampled RoIs per image §3.1. Card: FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding.

• Author-reported novel nAP50 on PASCAL VOC split 3 at 10 shots: 58.5, against 50.2 for TFA w/ fc, the strongest unstarred baseline in that column 2103.05950 tab. 2. On split 1 at 3 shots it is 51.4, tying MPSR tab. 2.
• Ablation on split 1 at 10 shots: the recipe changes give 61.4, the contrastive loss adds 1.3 points, and proposal consistency adds 0.7, for 63.4 in total 2103.05950 tab. 5.
• On COCO at 30 shots, novel AP is 16.4 against 13.7 for TFA w/ cos tab. 3.
• Head size and temperature moved the 10-shot result by at most 0.5 points in the ablation tab. 6.
• Cost: the loss builds an M × M similarity matrix over every sampled RoI in the batch. For 16 images at 256 RoIs each, that is about 16.8 million entries. This is computed from the config values, not measured 10shot_CL_IoU.yml.
• The released configs differ from the paper: the loss weight is 0.2 or 0.1 in some splits, φ varies from 0.5 to 0.8, and a loss-weight decay not described in the paper is enabled. Some paper cells have no released config, including COCO. The card's Paper vs code section lists them 3shot_CL_IoU.yml.

LLM-written training data for a retriever (Promptagator)

A frozen instruction-tuned LLM gets a short task description and 2 to 8 labelled query-document examples. It writes queries for documents sampled from the target corpus 2209.11755 §3.1. A round-trip check drops pairs whose source document a retriever trained on the synthetic pairs does not rank in its top K §3.2. The retriever is a 110M T5 dual encoder, and a 110M cross-encoder is trained as a reranker on the same data §3.3 §4.1. Only the dual encoder and the reranker run at query time §4.1. Card: Promptagator: Few-shot Dense Retrieval From 8 Examples.

• Author-reported nDCG@10 over 11 BEIR datasets: 47.8 for the few-shot dual encoder, above SPLADE v2 (46.6) and ColBERT v2 (46.2), both trained on MS MARCO. With the reranker the average is 52.8 2209.11755 tab. 2.
• The round-trip filter improves 8 of 11 datasets and adds about 2.5 points on average. It lowers NFCorpus and SciFact 2209.11755 §4.3 fig. 2(a).
• The generator matters: replacing the FLAN model with one trained without NQ and Quora costs about 1.5 points of average nDCG@10 tab. 3.
• On BEIR datasets with only a test split, the examples come from the evaluation set and must be counted as misses §2.2.
• The 137B generator is described as provided by the authors, and the paper gives no public release §3.1 §7. It promises prompts and examples "will be released" §1; the card found no official code.

In-context segmentation with a support set (UniverSeg)

One network is trained once on many medical segmentation tasks. At use time it takes a query image and a support set of image-label pairs and returns a mask, with no gradient steps for the new task 2304.06131 §3 §3.1. CrossBlocks exchange information between query and support features through a shared convolution and a mean over the support entries, so the support set can have any size §3.1 eq. 1 eq. 2. Card: UniverSeg: Universal Medical Image Segmentation.

• Author-reported mean Dice over six held-out datasets, on 2D mid-slices: 71.8 for UniverSeg, against 50.1 for SE-net and 41.8 for PANet, and 84.4 for supervised nnUNet with one network per task 2304.06131 tab. 1. The few-shot baselines were run with the paper's support size and augmentation, not their original settings §5.1.
• More support pairs help. Average held-out Dice rises from 53.7 at one pair to 69.9 at 16, then levels off near 71 at 64 §5.3 fig. 7. The evaluation averages five random support sets per test subject §5.1 §3.3.
• Each support entry adds a pairwise convolution, so cost grows with the support set §3.1 eq. 1.
• Each run predicts one binary label on a 2D slice, so multi-label tasks need one run per label §4 app. A.
• The paper's CC BY terms cover its text. The released weights carry a separate licence that the card records as research-only, so check it before any use beyond research. See the card's Pitfalls.

Few-shot LLM pre-labelling for expert annotation

A few-shot prompted LLM, PaLM 2 with no tuning, labels medication fields in chunks of discharge summaries. Its text output is parsed and matched back to the source text, and an expert then refines the labels 2312.02296 §5 §4. Two prompt schemas run separately and are merged by an ensemble that was chosen on the development set §5.3 §5.4. This is an annotation step: the LLM's output is a draft, not a deployed extractor. Card: LLMs Accelerate Annotation for Medical Information Extraction.

• Author-reported on 216 test documents: refinement time falls from 26.67 to 11.32 minutes per document (-58%), and phrase-level vertical F1 is 0.907 against 0.910 for human base labels with refinement 2312.02296 tab. 2 §6.1.
• The time comparison is not like-for-like. The LLM base has no rater time and LLM compute is excluded, and the authors say the two cannot be compared directly app. B §6.1.
• On the development set, the ensemble's F2 is 84.9 against 84.0 for the best IOB-Token configuration alone 2312.02296 tab. 5.
• The results cover five medication fields. The reason field was dropped §3.1 tab. 7.

Trainable continuous prompt slots (P-Tuning)

Keep a hand-written pattern around the input and add trainable vectors at chosen positions. A small encoder, an LSTM or an MLP, links the vectors to each other, and the result replaces the ordinary word embeddings at those positions 2103.10385 §2.2 §2.3. The model can stay frozen, as in LAMA knowledge probing, or be tuned, as in SuperGLUE §3. Card: GPT Understands, Too.

• Frozen BERT-base on LAMA-34k: precision@1 of 48.3 against 43.3 for the discrete AutoPrompt search, author-reported 2103.10385 tab. 3. Use the same-size comparison; the paper's headline range mixes model sizes §3.1.
• Few-shot SuperGLUE with ALBERT-xxlarge-v2, mean of four random splits: 71.81 average, against 70.74 for PET-FT and 58.56 for prompt tuning 2103.10385 tab. 5. The gain over PET is about one point and sits within the reported standard deviations. RTE is lower than PET tab. 5.
• Robustness to the pattern: on LAMA-P17 with BERT-base, the spread across six manual prompts falls from 10.1 to 0.46 points tab. 6. That is one relation, not the whole benchmark.
• The released LAMA code leaves out the manual relation prompt that the paper places around the slots. A faithful reimplementation has to add it, and the card's Paper vs code section lists the other differences.

Pre-trained soft prompts (PPT)

Before tuning on a labelled task, train the soft prompt on unlabelled text with self-supervised cloze tasks. The model stays frozen. The paper uses three pre-training pairs, one per task shape: sentence pairs, multiple choice over six candidates, and single sentences labelled by a RoBERTa-base model 2109.04332 §3.2. A unified variant turns every classification task into multiple choice, so one prompt serves all tasks §3.3. Downstream, the prompt starts from the pre-trained values instead of random ones §3.1. Card: PPT: Pre-trained Prompt Tuning for Few-shot Learning.

• Author-reported, T5-XXL (11B), 32 training and 32 dev examples per task, mean ± standard deviation over five seeds: SST-2 93.5 ± 0.3, against 91.4 ± 0.8 for full fine-tuning and 70.5 ± 15.5 for prompt tuning from random initialisation 2109.04332 tab. 4.
• The random-start variance is the problem the pre-training targets. On SST-2 the standard deviation falls from 15.5 points to 0.3 tab. 4.
• Cost: pre-training runs for up to 200,000 steps on about 10 GB of text per format 2109.04332 app. C. Wall-clock time is not stated. Pre-training is done once and its prompts are reused across tasks app. E. One SST-2 optimisation step takes 4,416 ms and 259 GB for full fine-tuning, against 794 ms and 72 GB for prompt tuning app. E tab. 10.
• The released repository names a different base checkpoint from the paper, and its batch arithmetic differs from the appendix. The card's Paper vs code section lists these before you reproduce anything.

Prompt tokens on both CLIP branches (MaPLe)

Freeze a CLIP ViT-B/16 and learn prompt tokens in its text encoder and its image encoder. Each vision prompt is a learned linear function of the text prompt in the same layer, which ties the two branches together. Prompts are inserted in the first nine blocks for base-to-novel generalization 2210.03117 §3.2 §3.2.3 §4.1. Training uses 16 labelled images per class, and the prompt parameters total about 3.55M, about 2.85% of CLIP tab. 6. Card: MaPLe: Multi-modal Prompt Learning.

• Base-to-novel, average over 11 datasets, author-reported: harmonic mean of 78.55 against 75.83 for Co-CoOp, and novel-class accuracy of 75.14 against 71.69 2210.03117 tab. 3.
• Ablation: deep prompts on the text side alone reach 77.56 harmonic mean, and independent prompts for the two sides reach 77.90. Full MaPLe is 78.55 2210.03117 tab. 1.
• Cross-dataset, trained on ImageNet with 16 shots: 66.30 average over the ten targets, against 65.74 for Co-CoOp and 63.88 for CoOp tab. 4.
• Cost: at batch 100, MaPLe runs at 1365 frames per second. At batch 1 it is slower than Co-CoOp, 60.2 against 64.6, so the speed advantage holds only for larger batches tab. 6 §4.6.
• Longer prompts keep base accuracy but lose novel accuracy. Depth 9 gave the best result in the paper's sweep §4.6 fig. 4.
• The learning rate differs between the text and the appendix. Section 4.1 and the released config use 0.0035, while appendix A gives 0.0025 2210.03117 §4.1 app. A.

Prompt tuning with a zero-shot gradient guard (ProGrad)

Start from CoOp-style context vectors in front of each class name, with CLIP's image and text encoders frozen. A zero-shot teacher gives class probabilities from the hand-written prompt 2205.14865 §3.1 eq. 1. Take the gradient of the cross-entropy with respect to the context vectors, g_ce, and the gradient of a KL-type loss toward the teacher, g_kl §3.2 eq. 3. If the two have a negative dot product, subtract λ times the part of g_ce that lies along g_kl eq. 4. Otherwise keep g_ce unchanged. λ = 1 removes the conflicting part fully, and λ = 0 gives back plain CoOp §3.2. Card: Prompt-aligned Gradient for Prompt Tuning.

• Author-reported, ResNet-50, 3 seeds averaged over 11 datasets, 16 context tokens: 62.61 at 1 shot against 59.46 for CoOp, and 74.28 at 16 shots against 73.49 2205.14865 tab. 1. The method is above CoOp at every shot count in that table.
• Base-to-new at 4 shots: harmonic mean 69.06 against 65.58 for CoOp. The New-class accuracy is 65.96 against 61.26 2205.14865 tab. 3.
• Trained on ImageNet, the same method scores 62.17 on ImageNet itself against 61.34 for CoOp, and 23.05 on ImageNet-A against 22.08 tab. 2. The robustness table covers ResNet-50, ResNet-101, ViT-B/32 and ViT-B/16.
• The λ ablation covers only DTD and OxfordPets. Its best setting differs by dataset and shot count: λ = 0.9 on DTD at 16 shots, λ = 1.0 on OxfordPets at 16 shots 2205.14865 §4. The main 16-shot setting, λ = 0.8, is not in the table.
• The projection cannot protect what the zero-shot prompt gets wrong. The paper's failure analysis finds many ProGrad errors that zero-shot CLIP also makes 2205.14865 §4 fig. 5. The zero-shot teacher is therefore a ceiling on what the method can keep.
• The paper gives no timing or memory figures for the extra backward pass 2205.14865 §3.2. The card's Paper vs code section lists the released code's differences in base-to-new context length, λ at 16 shots and ProGrad++.

When to use which

Implementation sketch

Stage 1 is a one-off pre-training step. Stage 2 runs per episode. Both are from the first card; the embedding-plus-linear-probe path is shown. The adapter route is sketched after the in-context variant. The other routes carry their own pseudocode on their cards.

# Stage 1, once: classification pre-training on all base classes merged
# [arxiv:2003.11539 eq. 4-5]
def pretrain(base_data, num_base_classes):
    # one head over every base class, cross-entropy loss  [§3.2]
    net = ResNet12(num_classes=num_base_classes)
    train(net, base_data, loss="cross_entropy")
    # keep the pooled penultimate features, drop the head  [§4.1]
    return net.backbone  # 640-d output for ResNet-12  [§4.1]

# Stage 2, per episode: fit a linear classifier on the support set
# [arxiv:2003.11539 §4.5] [eq. 6]
def classify_episode(embed, support_x, support_y, query_x):
    # augment each support image into five random copies  [§4.5] [tab. 4]
    aug_x, aug_y = augment_each(support_x, support_y, copies=5)
    # L2-normalise features onto the unit sphere, support and query  [§4.5]
    z_s = normalise(embed(aug_x))
    z_q = normalise(embed(query_x))
    # multinomial logistic regression, re-initialised for each episode  [§3.2]
    # regularisation strength and iteration limit: not stated in paper
    clf = LogisticRegression(multi_class="multinomial")
    clf.fit(z_s, aug_y)
    return clf.predict(z_q)

The calibration variant 2101.06395 alg. 1 keeps Stage 1 and replaces the augmentation in Stage 2 with per-feature Gaussian sampling. Its base-class means and covariances are computed once from base features §3.2.1.

Optional born-again step, run before Stage 2 2003.11539 §3.3 eq. 8:

# Optional born-again distillation  [arxiv:2003.11539 §3.3]
# alpha = beta = 0.5 on miniImageNet  [§4.1]; KL temperature: not stated
def distill(base_data, teacher, generations):
    for k in range(generations):
        # fresh student, same architecture, not warm-started  [§4.1]
        student = ResNet12(num_classes=num_base_classes)
        # alpha * CE(labels) + beta * KL(student || teacher)  [eq. 8]
        train(student, base_data, ce_weight=0.5, kl_weight=0.5,
              teacher=teacher)
        teacher = student
    return teacher.backbone

In-context variant, for comparison 2202.12837 §3 alg. 1:

# In-context few-shot prediction, direct scoring  [arxiv:2202.12837 §3]
def icl_predict(demos, x, label_set):
    # k (input, label) pairs, each pair on its own lines  [app. B]
    prompt = ""
    for x_i, y_i in demos:
        prompt += x_i + "\n" + y_i + "\n\n\n"
    # score each candidate label as the continuation of the input
    # paper writes argmax of P(y | prompt, x); released code length-normalises,
    # see the card's Paper vs code section
    scores = {y: log_prob(prompt + x + "\n", y) for y in label_set}
    return argmax(scores)

Adapter route, per task, for comparison 2205.05638 §3.5 §3.3:

# Few-shot PEFT: learn (IA)3 vectors on one task's few-shot set
def peft_fit(backbone, support, templates):
    # base weights stay frozen; only the scaling vectors are trained  [§3.3]
    ia3 = init_ones(backbone.scaling_shapes)  # start equal to T0  [§3.3]
    for step in range(1000):  # [§3.5]
        batch = sample(support, size=8)  # [§3.5]
        # one random P3 template per example, drawn at each step  [§3.1]
        # batch reduction over examples: not stated in paper
        loss = batch_mean(tfew_loss(backbone, ia3, ex, random(templates))
                          for ex in batch)  # [§3.2] [eq. 1] [eq. 2]
        ia3 = adafactor_step(ia3, loss, lr=3e-3)  # [§3.5]
    return ia3

Contradictions

No two cards report different numbers for the same setting. The numbers are not comparable across cards, though. The ResNet-12 embedding in 2003.11539 tab. 1 and the WideResNet features behind 2101.06395 tab. 2 differ in backbone, so the 5-shot figures (79.64 and 82.88 on miniImageNet) should not be read as one method beating the other. The in-context results 2202.12837 concern language tasks and are not measured on the image benchmarks above.

The adapter route's in-context baseline is not like-for-like either. Its GPT-3 numbers come from the GPT-3 paper, on that paper's subsets, while the adapter results use the authors' own subsets 2205.05638 §3.1 app. F. The study in 2202.12837 uses 16 demonstrations over 26 datasets §3. It measures what drives in-context gains, not the accuracy of the adapter comparison, so its results and the T-Few tables answer different questions.

The two T-Few 3B figures in this page are also different evaluations. The SetFit card's 63.4 average 2209.11055 tab. 2 is over five text datasets, with a median of five seeds for T-Few. The 72.4 average in 2205.05638 tab. 1 is over nine held-out T0 tasks. Do not place them side by side §4.3.

The Tip-Adapter abstract calls the training-free variant comparable to CLIP-Adapter. The paper's own ImageNet table puts it below CLIP-Adapter at every shot count, by 0.5 to 1.6 points 2111.03930 tab. 2.

The Flan paper's abstract reports gains of 2% or more from mixed prompt training "in all settings". Its own table has exceptions: removing the few-shot templates raises zero-shot CoT (38.9 against 35.8) and zero-shot BBH (27.6 against 26.2) 2301.13688 tab. 1.

The segmentation, retrieval and annotation numbers measure other things. VAT reports mIoU on PASCAL-5i, COCO-20i and FSS-1000 2207.10866 tab. 1; UniverSeg reports Dice on medical slices 2304.06131 tab. 1; Promptagator reports nDCG@10 on BEIR 2209.11755 tab. 2; and the annotation study reports rater minutes 2312.02296 tab. 2. The diffusion-feature method reports mIoU, as VAT does, but on six datasets of its own with 20 to 50 training images each 2112.03126 tab. 1 tab. 2, so the two mIoU tables are not comparable. FSCE reports nAP50 and nAP on PASCAL VOC and COCO [arxiv:2103.05950 tab. 2] tab. 3. None of them is comparable with the others or with the classification accuracies above.

The PPT paper says PPT beats full fine-tuning on all Chinese datasets and most English ones 2109.04332 §4.2. The Chinese part holds in its table: PPT on CPM-2 is above fine-tuned CPM-2 on all seven datasets tab. 4. On the English datasets, the task-group PPT row is above fine-tuned T5-XXL on two of seven (SST-2 and SST-5) and below on five. Unified PPT is above on three (SST-2, SST-5 and RTE) tab. 4. The table supports the gain on SST-2 and SST-5, not on most English tasks.

The FSCE paper's +8.8 nAP50 gain on split 3 is measured against MPSR. Against TFA w/ fc, the strongest unstarred baseline in that column, the gap is 8.3 2103.05950 §4.2 tab. 2.

Some cards record gaps between a paper and its released code, and the code changes the setting behind the reported numbers. VAT's paper says it uses no training augmentation 2207.10866 §4.1, while its released configs enable one. The noisy-channel code runs a 400-step schedule, where the appendix describes 100 steps 2108.04106 app. B. Check the card's Paper vs code section before matching a reported number.

The order-sensitivity claims also conflict. The GlobalE card records that a 2021 study by Liu et al. found order matters little once retrieved examples are used, and that the GlobalE paper argues the opposite for fixed few-shot sets. It also records that Gao et al. 2020 found fine-tuning less order-sensitive than in-context learning 2104.08786 §6. The setups differ, so both sides are stated as reported.

The prompt-vector numbers are not comparable with the classification accuracies above. MaPLe reports a harmonic mean over base and novel CLIP classes, and P-Tuning reports LAMA precision@1 and an ALBERT SuperGLUE average over seven tasks 2210.03117 tab. 3 2103.10385 tab. 5. ProGrad's 11-dataset averages use a CLIP backbone and 16 context tokens, but its base-to-new table uses 4 tokens in the paper and 16 in the released scripts, so its base-to-new harmonic mean 2205.14865 tab. 3 is not the same setting as the 4-shot few-shot rows. The released code also uses λ = 1.0 at 16 shots, where the paper's text gives 0.8 §4.

The survey's mini-ImageNet table reports 5-way 1-shot accuracy up to 82.92% (PT+MAP) and 91.5% at 5 shots with extra data 2205.06743 tab. 9. These are copied from other papers under different protocols and training data, so they do not compete with the ResNet-12 and WideResNet rows above. The survey itself says the tables are not on a common protocol §7.1.

Papers

• Rethinking Few-Shot Image Classification: pooled cross-entropy embedding with a per-episode logistic regression, and born-again distillation on top.
• PPT: Pre-trained Prompt Tuning for Few-shot Learning: soft prompts pre-trained on unlabelled text with cloze tasks, then tuned per task on a frozen T5-XXL.
• FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding: fine-tuned Faster R-CNN with a supervised contrastive head on proposal encodings.
• Free Lunch for Few-shot Learning: Distribution Calibration: training-free calibration of support features from base-class neighbours, with feature sampling and a logistic regression.
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?: ablations on which properties of in-context demonstrations drive the gain over zero-shot prompting.
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning: (IA)3 adapters on T0 with unlikelihood and length-normalised losses, compared with in-context learning on held-out classification tasks.
• Efficient Few-Shot Learning Without Prompts: contrastive fine-tuning of a sentence encoder on pairs from a few labelled examples, with a logistic-regression head.
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling: a training-free cache of CLIP features added to zero-shot CLIP logits, with optional key tuning.
• Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning: decoding-time grammar masks on a frozen LLM with few-shot prompts, for extraction, linking and parsing.
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning: instruction-tuning mixtures with zero-shot and few-shot templates, with ablations on T5-XL.
• Noisy Channel Language Model Prompting for Few-Shot Text Classification: scores each label by the probability of the input given the label, with a frozen GPT-2 and demonstrations or tuned prompt vectors.
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation: episodic few-shot segmentation by transformer aggregation over the query-support correlation volume.
• Label-Efficient Semantic Segmentation with Diffusion Models: ten MLPs on frozen diffusion decoder features from a few annotated images, with a majority vote.
• Promptagator: Few-shot Dense Retrieval From 8 Examples: an LLM writes queries from 2 to 8 examples, and a dual encoder and reranker are trained on them.
• UniverSeg: Universal Medical Image Segmentation: in-context medical segmentation from a support set, with no per-task training.
• LLMs Accelerate Annotation for Medical Information Extraction: a few-shot prompted LLM pre-labels medication fields for expert refinement.
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech: multilingual speech benchmark that shares the name phrase. Scope note only.
• FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization: few-input-view neural rendering, a different problem that shares the name. Scope note only.
• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis: few-view radiance field trained with a CLIP semantic-consistency loss on unseen poses. Scope note only.
• GPT Understands, Too: P-Tuning, trainable continuous prompt slots in a frozen or tuned masked language model.
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity: GlobalE, which orders in-context demonstrations by the entropy of labels predicted on LM-written probe inputs.
• MaPLe: Multi-modal Prompt Learning: prompt tokens on both CLIP branches, with the vision prompts coupled to the text prompts.
• Prompt-aligned Gradient for Prompt Tuning: CoOp-style CLIP context vectors, with the cross-entropy gradient projected away from a zero-shot KL direction.
• Few-Shot Classification with Feature Map Reconstruction Networks: closed-form ridge reconstruction of query feature maps from each class's pooled support maps, with a learned penalty and scale.
• Meta-Learning: A Survey: survey of meta-learning grouped by the meta-data used, with few-shot learners as one family. A map, not evidence for the routes above.
• A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities: survey organised by challenge, with computer-vision comparison tables copied from other papers. A map, not evidence for the routes above.