--- kind: concept slug: few-shot-learning title: Few-shot learning summary: >- Predicting new labels from a few labelled examples. Routes: a frozen embedding with a linear classifier (79.64% 5-shot on miniImageNet; author-reported), in-context demonstrations, adapter, contrastive or prompt-vector tuning, a training-free CLIP cache, noisy-channel scoring, in-context segmentation, or LLM-written training data. Main trade-off: what can you train, and what does each query cost? tags: [few-shot, meta-learning, in-context-learning, linear-probe, prompting, parameter-efficient-fine-tuning, contrastive, training-free, constrained-decoding, noisy-channel, segmentation, dense-retrieval, medical-imaging, annotation, diffusion, object-detection] papers: ["2003.11539", "2101.06395", "2202.12837", "2205.05638", "2303.07418", "2209.11055", "2111.03930", "2305.13971", "2301.13688", "2205.12446", "2108.04106", "2207.10866", "2209.11755", "2304.06131", "2312.02296", "2103.10385", "2104.08786", "2210.03117", "2112.03126", "2205.14865", "2205.06743", "2104.00677", "1810.03548", "2103.05950", "2109.04332", "2012.01506"] related: [linear-probing, data-augmentation, in-context-learning, meta-learning, metric-learning, parameter-efficient-fine-tuning, contrastive-learning, zero-shot-learning, constrained-decoding, instruction-tuning, soft-prompt-tuning, semantic-segmentation, dense-retrieval, prompt-engineering, named-entity-recognition, cost-aggregation, vision-language-pretraining, diffusion-models, transfer-learning, gradient-projection, neural-radiance-field, novel-view-synthesis, object-detection] status: draft updated_at: 2026-11-01 --- ## Overview Few-shot learning asks a model to predict over a label set it has only seen a few labelled examples of, typically one to five per class [arxiv:2003.11539 §1]. Most variants split the problem into a support set (the labelled examples for the new task) and query inputs to label. Most of them leave the base network's weights unchanged at meta-test time: the embedding is never fine-tuned [arxiv:2003.11539 §3.2], the calibration route keeps its backbone frozen and learns no parameters, using only base-class statistics stored once [arxiv:2101.06395 §3.2.1] [§3.2], the language model's weights are untouched when demonstrations are placed in the prompt [arxiv:2202.12837 §3], the vision cache is set from the support set without gradient steps [arxiv:2111.03930 §3.2], grammar-constrained decoding changes only which tokens may be chosen [arxiv:2305.13971 §2.2], and the in-context segmenter takes the support set as input with no gradient steps [arxiv:2304.06131 §3]. Some routes do change weights. The adapter route freezes the base model and trains a small set of added scaling vectors [arxiv:2205.05638 §3.3]. SetFit fine-tunes a sentence encoder with a contrastive loss on pairs built from the support set [arxiv:2209.11055 §3.1]. Channel prompt tuning trains 20 prompt vectors on the support set [arxiv:2108.04106 §4.2.3]. Four more routes learn prompt vectors, not discrete words. PPT pre-trains the vectors on unlabelled text before tuning [arxiv:2109.04332 §3.1]. MaPLe trains prompt tokens in both CLIP branches [arxiv:2210.03117 §3.2]. P-Tuning trains continuous slots around a hand-written pattern, and in its few-shot SuperGLUE setting also tunes the model [arxiv:2103.10385 §2.2] [§3.3]. ProGrad trains CLIP context vectors but drops the part of each cross-entropy gradient that works against a zero-shot teacher [arxiv:2205.14865 §3.2]. GlobalE changes no weights at all: it uses the few examples only to choose the order of the demonstrations [arxiv:2104.08786 §3]. Detection is covered too. FSCE fine-tunes a detector on a few boxes per novel class and reports nAP50 rather than accuracy [arxiv:2103.05950 §4.1]. Its numbers are not classification results. **Scope.** "Few-shot" also names few input views in neural rendering. [FreeNeRF](/papers/2303.07418) regularizes a radiance field fitted to a handful of photos [arxiv:2303.07418 §3.2]. It predicts no labels, so its results are not evidence for the routes below. It is listed only to mark the difference. [DietNeRF](/papers/2104.00677) is a second case of the same kind: it fits a radiance field to eight training views and uses a frozen CLIP encoder only as a loss on unseen poses [arxiv:2104.00677 §4.1] [alg. 1]. Its reported PSNR and FID are for rendered views [arxiv:2104.00677 tab. 1], so it is not evidence for the routes below either. [FLEURS](/papers/2205.12446) shares the title phrase but is a multilingual speech benchmark. Its baselines are fine-tuned on all 102 locales [arxiv:2205.12446 §4.1], and the results on its card do not come from a few-shot setting, so they are not evidence for the routes below either. **Not classifiers.** Two variants use the few examples to make data or drafts rather than predictions. [Promptagator](/papers/2209.11755) has an LLM write training queries, and the dual encoder and reranker trained on them are what run at query time [arxiv:2209.11755 §4.1]. The [LLM annotation study](/papers/2312.02296) produces draft labels that experts correct. Their results measure the data or the annotation effort, not the accuracy of a few-shot predictor. **Map of the field.** A survey groups few-shot methods by the problem each one addresses: data augmentation, transfer learning, meta-learning and multimodal learning [arxiv:2205.06743 §2.6]. Use it to find where to look, not as evidence. It runs no experiments, and its computer-vision tables copy numbers from cited papers under mixed protocols [arxiv:2205.06743 §7] [tab. 9]. A broader meta-learning survey lists few-shot learners as learned initialisations, non-parametric matchers, prototype models and memory-based learners [arxiv:1810.03548 §4.3]. It reports no experiments and has no code. ## Variants ### Frozen embedding plus a per-episode linear classifier Pool all base-class training data into one ordinary classification problem, train a network with cross-entropy, discard its classification head, and fit an L2-normalised multinomial logistic regression on each test support set [arxiv:2003.11539 §3.2] [eq. 4] [eq. 5] [eq. 6]. Card: [Rethinking Few-Shot Image Classification](/papers/2003.11539). - Reported with a ResNet-12 on miniImageNet 5-way: 62.02 ± 0.63 (1-shot) and 79.64 ± 0.44 (5-shot) for the simple version [arxiv:2003.11539 tab. 1]. - On Meta-Dataset, trained on ILSVRC only with a ResNet-18, the same linear base learner beats the best prior per-dataset number on 9 of 10 test sets, but trails on Aircraft [arxiv:2003.11539 §5] [tab. 7]. - The authors attribute the strength of this baseline to merging mutually exclusive tasks into one many-way problem. A multi-head alternative scored lower on miniImageNet 5-way 1-shot [arxiv:2003.11539 §4.9]. ### Born-again distillation on top of the embedding Train a fresh network of the same architecture to match both the labels and the soft predictions of the previous generation, then repeat for a chosen number of generations [arxiv:2003.11539 §3.3] [eq. 7] [eq. 8]. The new network is re-initialised, not warm-started [arxiv:2003.11539 §4.1]. Same card as above. - Reported: 64.82 ± 0.60 (1-shot) and 82.14 ± 0.43 (5-shot) on miniImageNet with ResNet-12, using two generations [arxiv:2003.11539 tab. 1] [§4.6]. - Generation count was picked on the validation set: two for miniImageNet, CIFAR-FS and FC100, one for tieredImageNet [arxiv:2003.11539 §4.6]. - Cost: every generation is a full training run with the same schedule [arxiv:2003.11539 §4.1], so two generations add two more training runs. ### Feature distribution calibration with sampling Keep the frozen embedding, but do not fit the support set alone. Each support feature is turned into a Gaussian whose mean and covariance are averaged from its two nearest base classes, after a Tukey power transform with λ = 0.5 [arxiv:2101.06395 §3.2.2] [eq. 3] [eq. 6]. The method then samples 750 features per class from these Gaussians and fits a logistic regression on the support set plus the samples [arxiv:2101.06395 §3.2.3] [eq. 7] [eq. 8]. No parameters are learned [arxiv:2101.06395 §3.2]. Card: [Free Lunch for Few-shot Learning: Distribution Calibration](/papers/2101.06395). - Reported 5-way results with logistic regression on WideResNet features trained as S2M2_R [arxiv:2101.06395 §4.1.3]: 68.57 ± 0.55 (1-shot) and 82.88 ± 0.42 (5-shot) on miniImageNet; 79.56 ± 0.87 and 90.67 ± 0.35 on CUB [arxiv:2101.06395 tab. 2]; 78.19 ± 0.25 and 89.90 ± 0.41 on tieredImageNet [tab. 3]. Author-reported; the baselines are the paper's table entries, not re-runs [§4.1.2]. - Ablation on miniImageNet 1-shot: 56.37 with neither part, 64.30 with the transform only, 63.70 with sampling only, 68.57 with both [arxiv:2101.06395 tab. 4]. - The method assumes each feature dimension is roughly Gaussian, that similar classes share similar statistics, and that features are non-negative, as post-ReLU pooled features are [arxiv:2101.06395 §1] [§3.2] [§4.1.3]. The power transform is the paper's answer to skew. ### Reconstructing spatial feature maps from the support set (FRN) Keep the backbone's spatial feature map rather than pooling it. For each class, stack the feature vectors of all support images, across every spatial location, into one pool. For a query map, solve a closed-form ridge regression that rebuilds the query from that pool, and score the class by the mean squared reconstruction error over locations [arxiv:2012.01506 §3.1] [eq. 1] [eq. 4]. A learned penalty and a learned rescaling of the reconstruction are added to the backbone, together with a softmax temperature, three learned parameters in all [§3.2] [eq. 6]. No gradient step runs at test time [§4.2]. Pre-training on the base classes gives each class one learnable map as its pool. That head is discarded afterwards [§3.6]. Card: [Few-Shot Classification with Feature Map Reconstruction Networks](/papers/2012.01506). - Author-reported, ResNet-12, 5-way, cropped CUB: 83.16 (1-shot) and 92.59 (5-shot), against 80.80 and 91.19 for the best listed baseline, a re-implemented Deep Subspace Network [arxiv:2012.01506 tab. 3]. On uncropped CUB the figures are 83.55 and 92.92 [tab. 4]. - The gains on the ImageNet-derived benchmarks are mixed. On mini-ImageNet, FRN is 0.33 points below FEAT at 1 shot (66.45 against 66.78) and 0.42 points above DeepEMD at 5 shots [arxiv:2012.01506 tab. 7]. On 84-pixel tiered-ImageNet it is below the best listed baseline in both columns [tab. 7]. - Test-time cost, as the paper's latency table reports it: 63 ms per 5-way mini-ImageNet episode at 1 shot and 79 ms at 5 shots, against 178 ms and 18,292 ms for DeepEMD with OpenCV. The hardware is not stated [arxiv:2012.01506 tab. 1] [§4.2] [SM Computational Efficiency]. - Training matters. On mini-ImageNet, episodic training from scratch gives 63.03 and 78.01, average-pool pre-training followed by fine-tuning gives 62.13 and 76.28, and FRN's own pre-training followed by fine-tuning gives 66.45 and 82.83 [arxiv:2012.01506 tab. 9]. A model trained on 1-shot episodes is worse than one trained on 5-shot episodes, even when tested at 1 shot [§5.1]. - The auxiliary loss and the learned penalty and scale change results only a little, and not consistently across backbones [arxiv:2012.01506 SM Auxiliary Loss and Regularization Parameters]. - The released code differs from the paper in the auxiliary loss, in the pre-training penalty and in two small constants. Check the card's Paper vs code section before reproducing the auxiliary term. ### Contrastive fine-tuning of a sentence encoder (SetFit) Build pairs from the labelled examples: two examples with the same label form a positive pair, two with different labels a negative pair. Tune a sentence-encoder body so that the cosine similarity of each pair matches its 0 or 1 target, then fit a logistic-regression head on the body's embeddings [arxiv:2209.11055 §3.1]. No prompts or verbalizers are written [§1]. Card: [Efficient Few-Shot Learning Without Prompts](/papers/2209.11055). - Author-reported, with 8 examples per class over five text classification datasets (AG News left out of the average): 62.3 for the 110M-parameter MPNet body, against 63.4 for T-Few 3B and 58.3 for ADAPET [arxiv:2209.11055 tab. 2]. At 64 per class, the same five-dataset average is 75.3 against 70.3 for T-Few 3B [tab. 2]. - Cost is estimated from FLOP formulas, not measured: about 19 times fewer inference FLOPs than T-Few 3B at 8 per class [arxiv:2209.11055 tab. 5] [§7.2]. - Standard deviations are large at 8 per class. For the Amazon Counterfactual set the score is 40.3 ± 11.8 [tab. 2], so compare methods on the spread as well as the mean. - With a multilingual body at 8 per class, the average error over six languages of MARC is 86.4 to 88.3 (lower is better), against 117.2 to 121.9 for fine-tuned XLM-R base. The two use different bodies [arxiv:2209.11055 tab. 4] [§6]. ### Training-free cache over CLIP features (Tip-Adapter) Store the normalised CLIP image feature of each few-shot training image as a key and its one-hot label as a value. For a test image, compute an exponential affinity to every key, sum the labels weighted by affinity, scale the sum by α, and add it to CLIP's zero-shot logits [arxiv:2111.03930 §3.2] [eq. 5] [eq. 6]. Nothing is trained for this variant [§3.2]. Card: [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](/papers/2111.03930). - ImageNet, ResNet-50, 16 shots: 62.03 training-free, against 60.33 for zero-shot CLIP and 63.59 for CLIP-Adapter, which learns its adapter by gradient descent [arxiv:2111.03930 tab. 2]. - The keys can be tuned for 20 epochs (Tip-Adapter-F). That gives 65.51 at 16 shots, against 63.59 for CLIP-Adapter after 200 epochs [tab. 2]. The paper reports 5 minutes on one NVIDIA GeForce RTX 3090 for Tip-Adapter-F, against 50 minutes for CLIP-Adapter [tab. 3]. - The cost moves from training time to memory: the cache holds NK × C features [arxiv:2111.03930 §3.2]. The paper gives no separate timing for building it [tab. 3]. - Per-dataset results at 16 shots appear only in a figure [arxiv:2111.03930 fig. 4]. The paper does not say how the fine-tuning epoch was chosen, so check the card's Paper vs code section before reproducing the numbers. ### Grammar-constrained decoding with few-shot prompts (GCD) Keep the model frozen and prompt it with a short instruction and a few demonstrations, as in few-shot prompting. At each step a parser returns the tokens that keep the output inside a grammar, and every other token is removed before the choice [arxiv:2305.13971 §2.2] [§2.3]. The grammar may depend on the input: for entity disambiguation, the candidate set for each mention is part of the grammar [§3.2]. The constraint fixes the form of the answer and leaves the demonstrations as they are. Card: [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](/papers/2305.13971). - Author-reported, LLaMA-33B with four demonstrations. Closed information extraction F1 rises from 17.5 unconstrained to 36.0 constrained [arxiv:2305.13971 tab. 1]. Entity disambiguation average accuracy rises from 54.1 to 80.3 with input-dependent candidates [tab. 2]. Constituency parsing F1 rises from 42.9 to 54.6, and valid trees from 64.2% to 100% [tab. 3]. - The method needs the next-token distribution, so models behind an API that does not expose it cannot be constrained [arxiv:2305.13971 §2.2] [§6]. - Latency: the 2.7M-entity extraction grammar adds about 69 ms per token on a consumer CPU, against 54 to 136 ms per token for the LLM forward pass on an A100 [tab. 4] [§4.4]. - With grammars that allow an empty output, the most probable complete output can be empty. For LLaMA-13B the paper reports an empty top output at length exponents 1.0 to 2.0, and none at 2.5 or higher [§5] [tab. 5]. ### Mixed zero-shot and few-shot templates in instruction tuning (Flan) Instruction tuning on a mixture in which each training example is shown either zero-shot or with 2, 3 or 5 worked exemplars [arxiv:2301.13688 §3.2] [app. A.1]. The model is then tested in both settings. This is a training recipe, not an inference method, so it applies only when you can tune the model. Card: [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](/papers/2301.13688). - Ablation on T5-XL 3B: removing the few-shot templates lowers few-shot held-in accuracy from 74.8 to 62.2 and few-shot MMLU from 52.4 to 38.7 [arxiv:2301.13688 tab. 1]. Zero-shot held-in accuracy falls less, from 73.8 to 72.5 [tab. 1]. - A small share of few-shot templates already helps zero-shot results, and the best share lies in a broad range. The paper does not state the share used in its final mixture [§3.2] [fig. 3]. - The ablations are single runs with no error bars [arxiv:2301.13688 tab. 1] [tab. 2]. ### In-context demonstrations for a frozen language model Place k labelled input-label pairs in the prompt, then ask for the label of a new input. Each candidate label is scored by the model, and the best one is returned [arxiv:2202.12837 §3] [§4.1]. Card: [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](/papers/2202.12837). - Setting used by the study: k = 16 demonstrations, direct and channel scoring, 26 datasets of classification and multi-choice, six decoder-only models from 774M parameters to GPT-3 [arxiv:2202.12837 §3] [tab. 1]. - Replacing gold labels with uniform random labels costs about 0-5 points absolute on nearly all 12 model/inference pairs [arxiv:2202.12837 §4.1] [fig. 3]. The per-dataset gap reaches nearly 14 points for one model on one dataset [arxiv:2202.12837 §4.1] [app. C.2]. - Ablations point to the label space, the input text distribution and the input-label pair format as the main sources of the gain, more than label correctness [arxiv:2202.12837 §5]. - Scope: the findings cover classification and multi-choice with label words the model already understands. Generation is not studied, and later work cited by the authors reports that negated labels hurt substantially [arxiv:2202.12837 Limitation]. ### Ordering demonstrations without labels (GlobalE) Build candidate prompts by ordering the same demonstrations, up to 24 orders. For each candidate, the frozen LM writes unlabelled probe inputs, and every candidate then labels all probe inputs. A candidate scores by the entropy of the label distribution it predicts over those probes. The top four are kept [arxiv:2104.08786 §3.1] [§3.2] [§4]. The reasoning is that an order which predicts one label for almost every probe scores low [arxiv:2104.08786 §2]. No gold labels are used in the selection [§3.2]. Card: [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](/papers/2104.08786). - Author-reported: GlobalE gives an average relative gain of 13% over the unselected baseline on 11 text classification tasks. The alternative score, LocalE, gives 9.6% [arxiv:2104.08786 §5] [tab. 2]. - Example: GPT-2 0.8B on SST-2 moves from 74.5 (mean over orders) to 84.8 with GlobalE, against 88.9 for an oracle that picks by validation accuracy [arxiv:2104.08786 tab. 2]. The oracle uses labels, so it is an upper bound, not a few-shot method. - Cost: each candidate needs its own generation, and every probe input needs a forward pass under every candidate. The GPT-3 175B runs were capped at 12 orders and two sample sets [arxiv:2104.08786 §4]. - Entropy does not see which label wins. Two labels split 80/20 in either direction score the same, and the paper does not discuss ties [arxiv:2104.08786 §3.2]. - The released code implements GlobalE only. LocalE is reported in the paper but has no code in the repository, so check the card's Paper vs code section before reproducing it. ### Few-shot parameter-efficient fine-tuning (T-Few recipe) Start from an instruction-tuned encoder-decoder (T0, 11B parameters), freeze its weights, and learn (IA)3 scaling vectors on each task's few-shot set. The training objective is the language-modelling loss plus an unlikelihood loss on wrong answers and a length-normalised softmax over the answer choices, summed with unit weights [arxiv:2205.05638 §3.2] [eq. 1] [eq. 2]. The vectors are first pre-trained on the T0 multitask mixture [§3.4]. The fine-tuning settings are the same for every task: 1,000 steps, Adafactor, learning rate 3e-3 [§3.5]. Card: [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](/papers/2205.05638). - Author-reported, on nine held-out T0 tasks with median accuracy over five few-shot subsets: 72.4% average, against 66.9% for zero-shot T0 and 66.6% for GPT-3 175B in-context learning [arxiv:2205.05638 tab. 1] [§4.2]. The GPT-3 numbers are copied from the GPT-3 paper, on different few-shot subsets [app. F]. - Inference cost: the paper estimates about 1,000 times fewer inference FLOPs per prediction than GPT-3 175B in-context learning [tab. 1]. Training the vectors costs about as much as running GPT-3 175B in-context on 20 examples [§4.2]. - Ablations on the held-out tasks: removing the (IA)3 pre-training gives 70.8, removing the two extra losses gives 68.3, against 72.4 for the full recipe [arxiv:2205.05638 §4.4] [app. G]. - On RAFT, T-Few scores 75.8% against 73.5% for the human baseline, with the same recipe and no per-task tuning [tab. 2] [§4.3]. - The text says (IA)3 is the only PEFT method that beats full fine-tuning. The appendix averages do not support "only": LoRA is ahead of full fine-tuning in two of three printed settings, by small margins [§3.3] [app. D]. - Storage does not reconcile: the paper reports 4.2 MB for the vectors, but its appendix lists 540K parameters, about 1.05M at 4 bytes each [tab. 1] [app. D]. Measure the checkpoint before budgeting storage. - Scope: classification and multiple-choice tasks only; generation is future work [§5]. Prompt tuning in the same comparison was unstable in training [§3.3]. ### Noisy-channel scoring of few-shot prompts Score each label by how well it explains the input: the probability of the input text given the verbalized label, under a frozen causal language model. Under a uniform label prior this replaces the usual probability of the label given the input [arxiv:2108.04106 §3]. Demonstrations are either concatenated into one context or run one at a time and their probabilities multiplied. The second form, the ensemble, needs O(K) memory rather than O(K²) and does not depend on example order [§4.1.2] [§4.1.3]. The route can also tune 20 prompt vectors on the support set while the LM stays frozen [§4.2.3]. Card: [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106). - Author-reported, GPT-2 Large, K = 16 training examples over 11 text classification datasets: channel prompt tuning averages 61.7 with worst case 53.0, against 57.7 (47.1) for direct head tuning [arxiv:2108.04106 tab. 4]. Averages are over verbalizers and seeds [§5.4]. - Without tuning, channel scoring with concatenated demonstrations averages 58.9 (worst case 50.8), above the best direct demonstration model, ensemble Direct++ at 55.8 (43.6) [arxiv:2108.04106 tab. 3] [§6.1]. - Direct head tuning is better on TREC (54.1 against 37.1 for channel prompt tuning) and Subj (86.3 against 75.5) [tab. 4] [§6.2]. The paper calls these tasks far from language modelling [§6.2]. With K = 64, head tuning beats channel prompt tuning [§6.3]. - The paper states that direct models cannot predict labels unseen in training, and channel prompt tuning outperforms all finetuning on the 2-way datasets [§6.4]. - Cost grows with the number of labels, because each label needs its own pass over the input, and with the number of demonstrations for the ensemble form [§3] [§4.1.3]. - The released code differs from the appendix in training schedule and several details, so check the card's Paper vs code section before reproducing numbers. ### Segmentation with episodic support sets and cost aggregation (VAT) A dense variant. For a query image and K support image-mask pairs, build the correlation between every query position and every support position from frozen ImageNet ResNet features, after masking the support features to the object. A 4D convolutional Swin Transformer then cleans up that volume across a three-level pyramid [arxiv:2207.10866 §3.3] [§3.4]. An appearance-aware transformer decodes the mask [§3.5]. Each support pair is run once, and the per-pixel foreground votes are summed for K-shot prediction [§3.6]. The class is unseen in training, and no retraining is done for it [§3.1]. Card: [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](/papers/2207.10866). - Author-reported 1-shot results: PASCAL-5i mIoU 67.9 against 66.2 for HSNet, both ResNet-101 [arxiv:2207.10866 tab. 1]; COCO-20i 41.3 against 39.2 with ResNet-50 [tab. 2]; FSS-1000 90.3 against 86.5 with ResNet-101 [tab. 3]. - Memory is the limit, because the full volume is built for every query-support pair. A standard transformer over it ran out of memory, at about 84 GB in the authors' table, while VAT's aggregator used 3.8 GB. Batch size and shot count for that table are not stated [arxiv:2207.10866 tab. 5]. - Swapping the aggregator for a linear transformer or Fastformer gives 87.7 and 87.8 mIoU on FSS-1000, against 90.3 for VAT [tab. 5]. - The paper states no training augmentation [§4.1], but the released configs enable one, and the code's K-shot vote uses a different threshold from the paper's rule [§3.6]. The card's Paper vs code section lists both. Settle which setting you need before comparing with reported numbers. ### Frozen diffusion features with a pixel-classifier ensemble (DDPM segmentation) A dense variant with no episodes. A pretrained diffusion model is used as a fixed feature extractor. Each labelled image is corrupted to three diffusion steps, t = 50, 150 and 250, and the activations of five UNet decoder blocks are read at each step. Each activation is upsampled to image size, and all of them are concatenated into 8448 features per pixel [arxiv:2112.03126 §3.2]. Ten small MLPs are trained on the pixel features and labels of the few annotated images, and each test pixel takes a majority vote across them [§3.2] [app. C]. The annotated images form one training set for the ensemble, with no per-episode support set [§3.2]. Card: [Label-Efficient Semantic Segmentation with Diffusion Models](/papers/2112.03126). - Author-reported mean IoU, averaged over five runs with different data splits, with 20 to 50 training images per dataset: 49.4 on Bedroom-28 against 47.9 for DatasetDDPM and 45.0 for MAE, and 65.0 on Horse-21 against 63.4 for MAE [arxiv:2112.03126 tab. 1] [tab. 2]. - Fewer labels cost accuracy. Bedroom-28 falls from 49.4 at 40 images to 38.2 at 10, and Horse-21 from 65.0 at 30 images to 56.9 at 10 [arxiv:2112.03126 tab. 4]. - The noise draw barely matters. Sharing it between train and test, or across steps, moves Bedroom-28 mIoU by at most 0.4 points [arxiv:2112.03126 tab. 5]. - Memory is the limit. The paper's sizes give 256 × 256 × 8448 float32 values per image, about 2.2 GB by calculation (derived). Each image also needs three full UNet forward passes, one per step [arxiv:2112.03126 §3.2]. - The method needs a diffusion model trained on the target domain. The authors expect this to be hard for complex domains such as ImageNet or MSCOCO [arxiv:2112.03126 §5]. - The paper says each MLP trains for about four epochs [arxiv:2112.03126 app. C]. The card's Paper vs code section records that the released code stops early on a loss criterion instead, so check it before reproducing numbers. ### Few-shot detection with a contrastive proposal head (FSCE) Start from a base-class Faster R-CNN with a ResNet-101 backbone and FPN. Fine- tune it on K shots of each novel class plus sampled base-class instances, with the ResNet frozen and the FPN, RPN and RoI head trained [arxiv:2103.05950 §3] [fig. 3]. A contrastive head encodes each sampled RoI into a 128-dimensional vector. A supervised contrastive loss then pulls together the encodings of same-class RoIs in a batch and pushes the others apart, using only RoIs whose proposal IoU reaches a threshold φ [§3.2] [§3.3] [eq. 2] [eq. 3]. The loss enters the total with weight λ = 0.5 [eq. 5]. The fine-tuning recipe also doubles the RPN proposals kept after NMS and halves the sampled RoIs per image [§3.1]. Card: [FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding](/papers/2103.05950). - Author-reported novel nAP50 on PASCAL VOC split 3 at 10 shots: 58.5, against 50.2 for TFA w/ fc, the strongest unstarred baseline in that column [arxiv:2103.05950 tab. 2]. On split 1 at 3 shots it is 51.4, tying MPSR [tab. 2]. - Ablation on split 1 at 10 shots: the recipe changes give 61.4, the contrastive loss adds 1.3 points, and proposal consistency adds 0.7, for 63.4 in total [arxiv:2103.05950 tab. 5]. - On COCO at 30 shots, novel AP is 16.4 against 13.7 for TFA w/ cos [tab. 3]. - Head size and temperature moved the 10-shot result by at most 0.5 points in the ablation [tab. 6]. - Cost: the loss builds an M × M similarity matrix over every sampled RoI in the batch. For 16 images at 256 RoIs each, that is about 16.8 million entries. This is computed from the config values, not measured [code: configs/PASCAL_VOC/split1/10shot_CL_IoU.yml#L23]. - The released configs differ from the paper: the loss weight is 0.2 or 0.1 in some splits, φ varies from 0.5 to 0.8, and a loss-weight decay not described in the paper is enabled. Some paper cells have no released config, including COCO. The card's Paper vs code section lists them [code: configs/PASCAL_VOC/split1/3shot_CL_IoU.yml#L29]. ### LLM-written training data for a retriever (Promptagator) A frozen instruction-tuned LLM gets a short task description and 2 to 8 labelled query-document examples. It writes queries for documents sampled from the target corpus [arxiv:2209.11755 §3.1]. A round-trip check drops pairs whose source document a retriever trained on the synthetic pairs does not rank in its top K [§3.2]. The retriever is a 110M T5 dual encoder, and a 110M cross-encoder is trained as a reranker on the same data [§3.3] [§4.1]. Only the dual encoder and the reranker run at query time [§4.1]. Card: [Promptagator: Few-shot Dense Retrieval From 8 Examples](/papers/2209.11755). - Author-reported nDCG@10 over 11 BEIR datasets: 47.8 for the few-shot dual encoder, above SPLADE v2 (46.6) and ColBERT v2 (46.2), both trained on MS MARCO. With the reranker the average is 52.8 [arxiv:2209.11755 tab. 2]. - The round-trip filter improves 8 of 11 datasets and adds about 2.5 points on average. It lowers NFCorpus and SciFact [arxiv:2209.11755 §4.3] [fig. 2(a)]. - The generator matters: replacing the FLAN model with one trained without NQ and Quora costs about 1.5 points of average nDCG@10 [tab. 3]. - On BEIR datasets with only a test split, the examples come from the evaluation set and must be counted as misses [§2.2]. - The 137B generator is described as provided by the authors, and the paper gives no public release [§3.1] [§7]. It promises prompts and examples "will be released" [§1]; the card found no official code. ### In-context segmentation with a support set (UniverSeg) One network is trained once on many medical segmentation tasks. At use time it takes a query image and a support set of image-label pairs and returns a mask, with no gradient steps for the new task [arxiv:2304.06131 §3] [§3.1]. CrossBlocks exchange information between query and support features through a shared convolution and a mean over the support entries, so the support set can have any size [§3.1] [eq. 1] [eq. 2]. Card: [UniverSeg: Universal Medical Image Segmentation](/papers/2304.06131). - Author-reported mean Dice over six held-out datasets, on 2D mid-slices: 71.8 for UniverSeg, against 50.1 for SE-net and 41.8 for PANet, and 84.4 for supervised nnUNet with one network per task [arxiv:2304.06131 tab. 1]. The few-shot baselines were run with the paper's support size and augmentation, not their original settings [§5.1]. - More support pairs help. Average held-out Dice rises from 53.7 at one pair to 69.9 at 16, then levels off near 71 at 64 [§5.3] [fig. 7]. The evaluation averages five random support sets per test subject [§5.1] [§3.3]. - Each support entry adds a pairwise convolution, so cost grows with the support set [§3.1] [eq. 1]. - Each run predicts one binary label on a 2D slice, so multi-label tasks need one run per label [§4] [app. A]. - The paper's CC BY terms cover its text. The released weights carry a separate licence that the card records as research-only, so check it before any use beyond research. See the card's Pitfalls. ### Few-shot LLM pre-labelling for expert annotation A few-shot prompted LLM, PaLM 2 with no tuning, labels medication fields in chunks of discharge summaries. Its text output is parsed and matched back to the source text, and an expert then refines the labels [arxiv:2312.02296 §5] [§4]. Two prompt schemas run separately and are merged by an ensemble that was chosen on the development set [§5.3] [§5.4]. This is an annotation step: the LLM's output is a draft, not a deployed extractor. Card: [LLMs Accelerate Annotation for Medical Information Extraction](/papers/2312.02296). - Author-reported on 216 test documents: refinement time falls from 26.67 to 11.32 minutes per document (-58%), and phrase-level vertical F1 is 0.907 against 0.910 for human base labels with refinement [arxiv:2312.02296 tab. 2] [§6.1]. - The time comparison is not like-for-like. The LLM base has no rater time and LLM compute is excluded, and the authors say the two cannot be compared directly [app. B] [§6.1]. - On the development set, the ensemble's F2 is 84.9 against 84.0 for the best IOB-Token configuration alone [arxiv:2312.02296 tab. 5]. - The results cover five medication fields. The reason field was dropped [§3.1] [tab. 7]. ### Trainable continuous prompt slots (P-Tuning) Keep a hand-written pattern around the input and add trainable vectors at chosen positions. A small encoder, an LSTM or an MLP, links the vectors to each other, and the result replaces the ordinary word embeddings at those positions [arxiv:2103.10385 §2.2] [§2.3]. The model can stay frozen, as in LAMA knowledge probing, or be tuned, as in SuperGLUE [§3]. Card: [GPT Understands, Too](/papers/2103.10385). - Frozen BERT-base on LAMA-34k: precision@1 of 48.3 against 43.3 for the discrete AutoPrompt search, author-reported [arxiv:2103.10385 tab. 3]. Use the same-size comparison; the paper's headline range mixes model sizes [§3.1]. - Few-shot SuperGLUE with ALBERT-xxlarge-v2, mean of four random splits: 71.81 average, against 70.74 for PET-FT and 58.56 for prompt tuning [arxiv:2103.10385 tab. 5]. The gain over PET is about one point and sits within the reported standard deviations. RTE is lower than PET [tab. 5]. - Robustness to the pattern: on LAMA-P17 with BERT-base, the spread across six manual prompts falls from 10.1 to 0.46 points [tab. 6]. That is one relation, not the whole benchmark. - The released LAMA code leaves out the manual relation prompt that the paper places around the slots. A faithful reimplementation has to add it, and the card's Paper vs code section lists the other differences. ### Pre-trained soft prompts (PPT) Before tuning on a labelled task, train the soft prompt on unlabelled text with self-supervised cloze tasks. The model stays frozen. The paper uses three pre-training pairs, one per task shape: sentence pairs, multiple choice over six candidates, and single sentences labelled by a RoBERTa-base model [arxiv:2109.04332 §3.2]. A unified variant turns every classification task into multiple choice, so one prompt serves all tasks [§3.3]. Downstream, the prompt starts from the pre-trained values instead of random ones [§3.1]. Card: [PPT: Pre-trained Prompt Tuning for Few-shot Learning](/papers/2109.04332). - Author-reported, T5-XXL (11B), 32 training and 32 dev examples per task, mean ± standard deviation over five seeds: SST-2 93.5 ± 0.3, against 91.4 ± 0.8 for full fine-tuning and 70.5 ± 15.5 for prompt tuning from random initialisation [arxiv:2109.04332 tab. 4]. - The random-start variance is the problem the pre-training targets. On SST-2 the standard deviation falls from 15.5 points to 0.3 [tab. 4]. - Cost: pre-training runs for up to 200,000 steps on about 10 GB of text per format [arxiv:2109.04332 app. C]. Wall-clock time is not stated. Pre-training is done once and its prompts are reused across tasks [app. E]. One SST-2 optimisation step takes 4,416 ms and 259 GB for full fine-tuning, against 794 ms and 72 GB for prompt tuning [app. E] [tab. 10]. - The released repository names a different base checkpoint from the paper, and its batch arithmetic differs from the appendix. The card's Paper vs code section lists these before you reproduce anything. ### Prompt tokens on both CLIP branches (MaPLe) Freeze a CLIP ViT-B/16 and learn prompt tokens in its text encoder and its image encoder. Each vision prompt is a learned linear function of the text prompt in the same layer, which ties the two branches together. Prompts are inserted in the first nine blocks for base-to-novel generalization [arxiv:2210.03117 §3.2] [§3.2.3] [§4.1]. Training uses 16 labelled images per class, and the prompt parameters total about 3.55M, about 2.85% of CLIP [tab. 6]. Card: [MaPLe: Multi-modal Prompt Learning](/papers/2210.03117). - Base-to-novel, average over 11 datasets, author-reported: harmonic mean of 78.55 against 75.83 for Co-CoOp, and novel-class accuracy of 75.14 against 71.69 [arxiv:2210.03117 tab. 3]. - Ablation: deep prompts on the text side alone reach 77.56 harmonic mean, and independent prompts for the two sides reach 77.90. Full MaPLe is 78.55 [arxiv:2210.03117 tab. 1]. - Cross-dataset, trained on ImageNet with 16 shots: 66.30 average over the ten targets, against 65.74 for Co-CoOp and 63.88 for CoOp [tab. 4]. - Cost: at batch 100, MaPLe runs at 1365 frames per second. At batch 1 it is slower than Co-CoOp, 60.2 against 64.6, so the speed advantage holds only for larger batches [tab. 6] [§4.6]. - Longer prompts keep base accuracy but lose novel accuracy. Depth 9 gave the best result in the paper's sweep [§4.6] [fig. 4]. - The learning rate differs between the text and the appendix. Section 4.1 and the released config use 0.0035, while appendix A gives 0.0025 [arxiv:2210.03117 §4.1] [app. A]. ### Prompt tuning with a zero-shot gradient guard (ProGrad) Start from CoOp-style context vectors in front of each class name, with CLIP's image and text encoders frozen. A zero-shot teacher gives class probabilities from the hand-written prompt [arxiv:2205.14865 §3.1] [eq. 1]. Take the gradient of the cross-entropy with respect to the context vectors, g_ce, and the gradient of a KL-type loss toward the teacher, g_kl [§3.2] [eq. 3]. If the two have a negative dot product, subtract λ times the part of g_ce that lies along g_kl [eq. 4]. Otherwise keep g_ce unchanged. λ = 1 removes the conflicting part fully, and λ = 0 gives back plain CoOp [§3.2]. Card: [Prompt-aligned Gradient for Prompt Tuning](/papers/2205.14865). - Author-reported, ResNet-50, 3 seeds averaged over 11 datasets, 16 context tokens: 62.61 at 1 shot against 59.46 for CoOp, and 74.28 at 16 shots against 73.49 [arxiv:2205.14865 tab. 1]. The method is above CoOp at every shot count in that table. - Base-to-new at 4 shots: harmonic mean 69.06 against 65.58 for CoOp. The New-class accuracy is 65.96 against 61.26 [arxiv:2205.14865 tab. 3]. - Trained on ImageNet, the same method scores 62.17 on ImageNet itself against 61.34 for CoOp, and 23.05 on ImageNet-A against 22.08 [tab. 2]. The robustness table covers ResNet-50, ResNet-101, ViT-B/32 and ViT-B/16. - The λ ablation covers only DTD and OxfordPets. Its best setting differs by dataset and shot count: λ = 0.9 on DTD at 16 shots, λ = 1.0 on OxfordPets at 16 shots [arxiv:2205.14865 §4]. The main 16-shot setting, λ = 0.8, is not in the table. - The projection cannot protect what the zero-shot prompt gets wrong. The paper's failure analysis finds many ProGrad errors that zero-shot CLIP also makes [arxiv:2205.14865 §4] [fig. 5]. The zero-shot teacher is therefore a ceiling on what the method can keep. - The paper gives no timing or memory figures for the extra backward pass [arxiv:2205.14865 §3.2]. The card's Paper vs code section lists the released code's differences in base-to-new context length, λ at 16 shots and ProGrad++. ## When to use which | Situation | Use | Why | |---|---|---| | You can train the embedding and have many labelled base classes | Frozen embedding plus linear classifier | Strong reported 1- and 5-shot results with no meta-training loop [arxiv:2003.11539 tab. 1] | | The same, and you can pay roughly three times the pre-training cost | Add one or two born-again generations | +2-3 points on miniImageNet 5-way [arxiv:2003.11539 tab. 1]; each generation is a full run [§4.1] | | The embedding is fixed and you cannot retrain it, but you can store base-class features | Feature distribution calibration | No parameters learned [arxiv:2101.06395 §3.2]; highest mean in its own miniImageNet and CUB table at both shots [tab. 2]; needs non-negative features [§4.1.3] | | You have an 11B-class encoder-decoder you can fine-tune, a few dozen labelled examples per task, and a high query volume | Few-shot PEFT (T-Few recipe) | Author-reported gain over GPT-3 175B in-context, at about 1,000 times fewer inference FLOPs [arxiv:2205.05638 tab. 1]; one training run per task [§4.2] | | Text classification, a sentence encoder of about 110M parameters, around 8 to 64 labelled examples per class, and no budget for prompt design | SetFit | Author-reported 62.3 at 8 per class against 63.4 for T-Few 3B; about 19 times fewer inference FLOPs, as an estimate [arxiv:2209.11055 tab. 2] [tab. 5] | | Image classification with a CLIP-style model, a few labelled images per class, and no training budget | Training-free cache (Tip-Adapter) | No gradient steps; 62.03 at 16 shots against 60.33 zero-shot, but below CLIP-Adapter's 63.59 [arxiv:2111.03930 tab. 2]; memory grows with NK × C [§3.2] | | The same CLIP setting, with about 5 minutes of tuning available | Tip-Adapter-F (key tuning) | 65.51 at 16 shots after 20 epochs [arxiv:2111.03930 tab. 2] [tab. 3] | | Output must follow a formal grammar (fact triples, one of a candidate list, bracketed trees), on an open-weight model you can decode with | Grammar-constrained decoding with input-dependent grammar | Reported gains over unconstrained few-shot prompting on all three tasks [arxiv:2305.13971 tab. 1] [tab. 2] [tab. 3]; validity 100% on parsing [tab. 3] | | Your only model is behind an API that does not expose token probabilities | Not grammar-constrained decoding | Decoding needs the next-token distribution [arxiv:2305.13971 §2.2] | | You will tune an instruction model and want it to handle both prompt styles | Mix zero-shot and few-shot templates | Removing the few-shot templates lowers few-shot MMLU from 52.4 to 38.7 [arxiv:2301.13688 tab. 1]; a small few-shot share already helps [§3.2] | | You have only a frozen LLM reachable through a prompt | In-context demonstrations | Needs no training; results above are for the models and datasets tested [arxiv:2202.12837 §3] | | The new task needs an input-to-label mapping the model did not learn in pre-training | Do not expect demonstrations alone to teach it | In the tested models the mapping mattered least; demonstrations mainly supplied format and label space [arxiv:2202.12837 §5] | | Unlabelled inputs only, or label-free prompts | Check against the baselines before relying on it | Inputs-only and labels-only prompts landed close to or below no demonstrations [arxiv:2202.12837 §5.3] | | Few labelled texts, a frozen causal LM whose log-probabilities you can read, and labels that are imbalanced or unseen at test time | Noisy-channel scoring with channel prompt tuning | Highest mean and worst case of the four tuning methods on the card's table, at K = 16 [arxiv:2108.04106 tab. 4]; direct heads cannot predict unseen labels [§6.4] | | The same, but with question-type or subjectivity labels, or with K = 64 or more | Direct head tuning | Head tuning wins on TREC and Subj [arxiv:2108.04106 tab. 4] and at K = 64 [§6.3] | | Fine-grained classes where spatial detail matters, a ResNet-12-class backbone you can train episodically, and no budget for test-time optimisation | Feature-map reconstruction (FRN) | Cropped CUB 5-way at 83.16 (1-shot) against 80.80 for the best listed baseline [arxiv:2012.01506 tab. 3]; on mini-ImageNet 1-shot it does not beat FEAT [tab. 7] | | Dense labels (masks) for classes unseen in training, a frozen backbone, and GPU memory for a 4D correlation volume | Cost aggregation (VAT) | 67.9 PASCAL-5i 1-shot with ResNet-101 [arxiv:2207.10866 tab. 1]; 3.8 GB against about 84 GB for a standard transformer on the volume [tab. 5] | | Dense labels for a fixed label set, a diffusion model trained on unlabelled images of the same domain, and 10 to 40 annotated images | Frozen diffusion features with a pixel-classifier ensemble | Mean IoU 49.4 on Bedroom-28 against 45.0 for MAE [arxiv:2112.03126 tab. 2]; 38.2 with 10 annotated images [tab. 4]; about 2.2 GB of features per image, derived [§3.2] | | Object detection with a few boxes per novel class, a base detector you can fine-tune (Faster R-CNN with ResNet-101 and FPN), and room for a small contrastive head | FSCE | Novel nAP50 58.5 at 10 shots on VOC split 3, against 50.2 for TFA w/ fc [arxiv:2103.05950 tab. 2]; the released configs differ from the paper, see the card | | A 2D medical scan of a structure with no task-specific network to train, and you can label a few support pairs | In-context segmentation (UniverSeg) | Mean Dice 71.8 on unseen datasets, against 84.4 for per-task nnUNet [arxiv:2304.06131 tab. 1]; more support pairs help up to about 16 [§5.3]. Check the weight licence first | | A new corpus with no labelled queries, access to a large LLM that writes queries, and a budget for a dual encoder | Few-shot query generation, then distillation (Promptagator) | 47.8 nDCG@10 average over 11 BEIR sets, above SPLADE v2 at 46.6 [arxiv:2209.11755 tab. 2]; the round-trip filter helps 8 of 11 [§4.3] | | Draft labels for expert annotation of clinical text, where expert time is the bottleneck | Few-shot LLM pre-labelling with expert refinement | Refinement time falls from 26.67 to 11.32 minutes per document, with refined F1 close to the human baseline [arxiv:2312.02296 tab. 2]; time is not like-for-like [app. B] | | A CLIP-style vision-language model, 16 labelled images per class, and room for about 3.5M trainable prompt parameters | Prompt tokens on both branches (MaPLe) | Base-to-novel harmonic mean 78.55 against 75.83 for Co-CoOp [arxiv:2210.03117 tab. 3]; batch-100 inference much faster than Co-CoOp [tab. 6] | | A CLIP-style model, a few labelled images per class, and prompt tuning that must not drift far from the zero-shot prompt, including on unseen classes | Gradient-guarded prompt tuning (ProGrad) | 62.61 against 59.46 for CoOp at 1 shot, 11-dataset average [arxiv:2205.14865 tab. 1]; New-class accuracy 65.96 against 61.26 [tab. 3]; the hand-written zero-shot prompt caps the gain [§4] | | A masked language model, a few labelled examples per task, and the ability to tune the model | Continuous prompt slots (P-Tuning) | Few-shot SuperGLUE average 71.81 against 70.74 for PET-FT, a gap inside the spread [arxiv:2103.10385 tab. 5] | | A frozen 11B-class encoder-decoder, 32 labelled examples per task, and a budget for one pre-training run on unlabelled text | Pre-trained soft prompts (PPT) | SST-2 93.5 against 91.4 for full fine-tuning, with SD 0.3 against 15.5 for random-start prompt tuning [arxiv:2109.04332 tab. 4]; on five of seven English tasks the PPT row is below full fine-tuning [tab. 4] | | A frozen LM with few-shot demonstrations, no labelled dev set to choose the order, and a budget for up to 24 candidate prompts | Order by GlobalE entropy | Author-reported 13% average relative gain [arxiv:2104.08786 §5]; GPT-2 0.8B on SST-2 from 74.5 to 84.8 [tab. 2] | ## Implementation sketch Stage 1 is a one-off pre-training step. Stage 2 runs per episode. Both are from the first card; the embedding-plus-linear-probe path is shown. The adapter route is sketched after the in-context variant. The other routes carry their own pseudocode on their cards. ```pseudocode # Stage 1, once: classification pre-training on all base classes merged # [arxiv:2003.11539 eq. 4-5] def pretrain(base_data, num_base_classes): # one head over every base class, cross-entropy loss [§3.2] net = ResNet12(num_classes=num_base_classes) train(net, base_data, loss="cross_entropy") # keep the pooled penultimate features, drop the head [§4.1] return net.backbone # 640-d output for ResNet-12 [§4.1] # Stage 2, per episode: fit a linear classifier on the support set # [arxiv:2003.11539 §4.5] [eq. 6] def classify_episode(embed, support_x, support_y, query_x): # augment each support image into five random copies [§4.5] [tab. 4] aug_x, aug_y = augment_each(support_x, support_y, copies=5) # L2-normalise features onto the unit sphere, support and query [§4.5] z_s = normalise(embed(aug_x)) z_q = normalise(embed(query_x)) # multinomial logistic regression, re-initialised for each episode [§3.2] # regularisation strength and iteration limit: not stated in paper clf = LogisticRegression(multi_class="multinomial") clf.fit(z_s, aug_y) return clf.predict(z_q) ``` The calibration variant [arxiv:2101.06395 alg. 1] keeps Stage 1 and replaces the augmentation in Stage 2 with per-feature Gaussian sampling. Its base-class means and covariances are computed once from base features [§3.2.1]. Optional born-again step, run before Stage 2 [arxiv:2003.11539 §3.3] [eq. 8]: ```pseudocode # Optional born-again distillation [arxiv:2003.11539 §3.3] # alpha = beta = 0.5 on miniImageNet [§4.1]; KL temperature: not stated def distill(base_data, teacher, generations): for k in range(generations): # fresh student, same architecture, not warm-started [§4.1] student = ResNet12(num_classes=num_base_classes) # alpha * CE(labels) + beta * KL(student || teacher) [eq. 8] train(student, base_data, ce_weight=0.5, kl_weight=0.5, teacher=teacher) teacher = student return teacher.backbone ``` In-context variant, for comparison [arxiv:2202.12837 §3] [alg. 1]: ```pseudocode # In-context few-shot prediction, direct scoring [arxiv:2202.12837 §3] def icl_predict(demos, x, label_set): # k (input, label) pairs, each pair on its own lines [app. B] prompt = "" for x_i, y_i in demos: prompt += x_i + "\n" + y_i + "\n\n\n" # score each candidate label as the continuation of the input # paper writes argmax of P(y | prompt, x); released code length-normalises, # see the card's Paper vs code section scores = {y: log_prob(prompt + x + "\n", y) for y in label_set} return argmax(scores) ``` Adapter route, per task, for comparison [arxiv:2205.05638 §3.5] [§3.3]: ```pseudocode # Few-shot PEFT: learn (IA)3 vectors on one task's few-shot set def peft_fit(backbone, support, templates): # base weights stay frozen; only the scaling vectors are trained [§3.3] ia3 = init_ones(backbone.scaling_shapes) # start equal to T0 [§3.3] for step in range(1000): # [§3.5] batch = sample(support, size=8) # [§3.5] # one random P3 template per example, drawn at each step [§3.1] # batch reduction over examples: not stated in paper loss = batch_mean(tfew_loss(backbone, ia3, ex, random(templates)) for ex in batch) # [§3.2] [eq. 1] [eq. 2] ia3 = adafactor_step(ia3, loss, lr=3e-3) # [§3.5] return ia3 ``` ## Contradictions No two cards report different numbers for the same setting. The numbers are not comparable across cards, though. The ResNet-12 embedding in [arxiv:2003.11539 tab. 1] and the WideResNet features behind [arxiv:2101.06395 tab. 2] differ in backbone, so the 5-shot figures (79.64 and 82.88 on miniImageNet) should not be read as one method beating the other. The in-context results [arxiv:2202.12837] concern language tasks and are not measured on the image benchmarks above. The adapter route's in-context baseline is not like-for-like either. Its GPT-3 numbers come from the GPT-3 paper, on that paper's subsets, while the adapter results use the authors' own subsets [arxiv:2205.05638 §3.1] [app. F]. The study in [arxiv:2202.12837] uses 16 demonstrations over 26 datasets [§3]. It measures what drives in-context gains, not the accuracy of the adapter comparison, so its results and the T-Few tables answer different questions. The two T-Few 3B figures in this page are also different evaluations. The SetFit card's 63.4 average [arxiv:2209.11055 tab. 2] is over five text datasets, with a median of five seeds for T-Few. The 72.4 average in [arxiv:2205.05638 tab. 1] is over nine held-out T0 tasks. Do not place them side by side [§4.3]. The Tip-Adapter abstract calls the training-free variant comparable to CLIP-Adapter. The paper's own ImageNet table puts it below CLIP-Adapter at every shot count, by 0.5 to 1.6 points [arxiv:2111.03930 tab. 2]. The Flan paper's abstract reports gains of 2% or more from mixed prompt training "in all settings". Its own table has exceptions: removing the few-shot templates raises zero-shot CoT (38.9 against 35.8) and zero-shot BBH (27.6 against 26.2) [arxiv:2301.13688 tab. 1]. The segmentation, retrieval and annotation numbers measure other things. VAT reports mIoU on PASCAL-5i, COCO-20i and FSS-1000 [arxiv:2207.10866 tab. 1]; UniverSeg reports Dice on medical slices [arxiv:2304.06131 tab. 1]; Promptagator reports nDCG@10 on BEIR [arxiv:2209.11755 tab. 2]; and the annotation study reports rater minutes [arxiv:2312.02296 tab. 2]. The diffusion-feature method reports mIoU, as VAT does, but on six datasets of its own with 20 to 50 training images each [arxiv:2112.03126 tab. 1] [tab. 2], so the two mIoU tables are not comparable. FSCE reports nAP50 and nAP on PASCAL VOC and COCO [arxiv:2103.05950 tab. 2] [tab. 3]. None of them is comparable with the others or with the classification accuracies above. The PPT paper says PPT beats full fine-tuning on all Chinese datasets and most English ones [arxiv:2109.04332 §4.2]. The Chinese part holds in its table: PPT on CPM-2 is above fine-tuned CPM-2 on all seven datasets [tab. 4]. On the English datasets, the task-group PPT row is above fine-tuned T5-XXL on two of seven (SST-2 and SST-5) and below on five. Unified PPT is above on three (SST-2, SST-5 and RTE) [tab. 4]. The table supports the gain on SST-2 and SST-5, not on most English tasks. The FSCE paper's +8.8 nAP50 gain on split 3 is measured against MPSR. Against TFA w/ fc, the strongest unstarred baseline in that column, the gap is 8.3 [arxiv:2103.05950 §4.2] [tab. 2]. Some cards record gaps between a paper and its released code, and the code changes the setting behind the reported numbers. VAT's paper says it uses no training augmentation [arxiv:2207.10866 §4.1], while its released configs enable one. The noisy-channel code runs a 400-step schedule, where the appendix describes 100 steps [arxiv:2108.04106 app. B]. Check the card's Paper vs code section before matching a reported number. The order-sensitivity claims also conflict. The GlobalE card records that a 2021 study by Liu et al. found order matters little once retrieved examples are used, and that the GlobalE paper argues the opposite for fixed few-shot sets. It also records that Gao et al. 2020 found fine-tuning less order-sensitive than in-context learning [arxiv:2104.08786 §6]. The setups differ, so both sides are stated as reported. The prompt-vector numbers are not comparable with the classification accuracies above. MaPLe reports a harmonic mean over base and novel CLIP classes, and P-Tuning reports LAMA precision@1 and an ALBERT SuperGLUE average over seven tasks [arxiv:2210.03117 tab. 3] [arxiv:2103.10385 tab. 5]. ProGrad's 11-dataset averages use a CLIP backbone and 16 context tokens, but its base-to-new table uses 4 tokens in the paper and 16 in the released scripts, so its base-to-new harmonic mean [arxiv:2205.14865 tab. 3] is not the same setting as the 4-shot few-shot rows. The released code also uses λ = 1.0 at 16 shots, where the paper's text gives 0.8 [§4]. The survey's mini-ImageNet table reports 5-way 1-shot accuracy up to 82.92% (PT+MAP) and 91.5% at 5 shots with extra data [arxiv:2205.06743 tab. 9]. These are copied from other papers under different protocols and training data, so they do not compete with the ResNet-12 and WideResNet rows above. The survey itself says the tables are not on a common protocol [§7.1]. ## Papers - [Rethinking Few-Shot Image Classification](/papers/2003.11539): pooled cross-entropy embedding with a per-episode logistic regression, and born-again distillation on top. - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](/papers/2109.04332): soft prompts pre-trained on unlabelled text with cloze tasks, then tuned per task on a frozen T5-XXL. - [FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding](/papers/2103.05950): fine-tuned Faster R-CNN with a supervised contrastive head on proposal encodings. - [Free Lunch for Few-shot Learning: Distribution Calibration](/papers/2101.06395): training-free calibration of support features from base-class neighbours, with feature sampling and a logistic regression. - [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](/papers/2202.12837): ablations on which properties of in-context demonstrations drive the gain over zero-shot prompting. - [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](/papers/2205.05638): (IA)3 adapters on T0 with unlikelihood and length-normalised losses, compared with in-context learning on held-out classification tasks. - [Efficient Few-Shot Learning Without Prompts](/papers/2209.11055): contrastive fine-tuning of a sentence encoder on pairs from a few labelled examples, with a logistic-regression head. - [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](/papers/2111.03930): a training-free cache of CLIP features added to zero-shot CLIP logits, with optional key tuning. - [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](/papers/2305.13971): decoding-time grammar masks on a frozen LLM with few-shot prompts, for extraction, linking and parsing. - [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](/papers/2301.13688): instruction-tuning mixtures with zero-shot and few-shot templates, with ablations on T5-XL. - [Noisy Channel Language Model Prompting for Few-Shot Text Classification](/papers/2108.04106): scores each label by the probability of the input given the label, with a frozen GPT-2 and demonstrations or tuned prompt vectors. - [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](/papers/2207.10866): episodic few-shot segmentation by transformer aggregation over the query-support correlation volume. - [Label-Efficient Semantic Segmentation with Diffusion Models](/papers/2112.03126): ten MLPs on frozen diffusion decoder features from a few annotated images, with a majority vote. - [Promptagator: Few-shot Dense Retrieval From 8 Examples](/papers/2209.11755): an LLM writes queries from 2 to 8 examples, and a dual encoder and reranker are trained on them. - [UniverSeg: Universal Medical Image Segmentation](/papers/2304.06131): in-context medical segmentation from a support set, with no per-task training. - [LLMs Accelerate Annotation for Medical Information Extraction](/papers/2312.02296): a few-shot prompted LLM pre-labels medication fields for expert refinement. - [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](/papers/2205.12446): multilingual speech benchmark that shares the name phrase. Scope note only. - [FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization](/papers/2303.07418): few-input-view neural rendering, a different problem that shares the name. Scope note only. - [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](/papers/2104.00677): few-view radiance field trained with a CLIP semantic-consistency loss on unseen poses. Scope note only. - [GPT Understands, Too](/papers/2103.10385): P-Tuning, trainable continuous prompt slots in a frozen or tuned masked language model. - [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](/papers/2104.08786): GlobalE, which orders in-context demonstrations by the entropy of labels predicted on LM-written probe inputs. - [MaPLe: Multi-modal Prompt Learning](/papers/2210.03117): prompt tokens on both CLIP branches, with the vision prompts coupled to the text prompts. - [Prompt-aligned Gradient for Prompt Tuning](/papers/2205.14865): CoOp-style CLIP context vectors, with the cross-entropy gradient projected away from a zero-shot KL direction. - [Few-Shot Classification with Feature Map Reconstruction Networks](/papers/2012.01506): closed-form ridge reconstruction of query feature maps from each class's pooled support maps, with a learned penalty and scale. - [Meta-Learning: A Survey](/papers/1810.03548): survey of meta-learning grouped by the meta-data used, with few-shot learners as one family. A map, not evidence for the routes above. - [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](/papers/2205.06743): survey organised by challenge, with computer-vision comparison tables copied from other papers. A map, not evidence for the routes above.