--- kind: concept slug: linear-probing title: Linear probing summary: >- Fitting a linear classifier on frozen features, with the network never updated. In few-shot use, the probe is refit per test episode from labelled support images. A supervised ResNet-12 with an L2-normalised logistic-regression probe reaches 62.02% (1-shot) and 79.64% (5-shot) on miniImageNet 5-way, author-reported. Trade-off: cheap to fit, but capped by how good the frozen features already are. tags: [few-shot, self-supervised, representation, evaluation, transfer, fine-tuning, distribution-shift, interpretability, vision-language, activation-steering, truthfulness] papers: ["2003.11539", "2101.06395", "1610.02055", "2202.10054", "2102.06810", "2111.06377", "2204.07141", "2603.17839", "2108.04840", "1812.11446", "2204.08790", "2104.14294", "2306.03341", "2110.09348", "2011.00362", "2202.03026"] related: [few-shot-learning, meta-learning, self-supervised-learning, knowledge-distillation, data-augmentation, transfer-learning, dataset-curation, masked-image-modeling, vision-transformer, domain-generalization, contrastive-learning, activation-patching, calibration, greedy-layerwise-training, vision-language-pretraining, zero-shot-learning, activation-steering, hallucination-mitigation, mechanistic-interpretability, nearest-neighbor-search] status: draft updated_at: 2026-10-24 --- ## Overview A linear probe trains only a linear classifier on top of an embedding that stays fixed. The shared invariant is that the backbone's weights do not change when the probe is fit. In the first card on this page, the probe is refit for every new few-shot task from the support set alone, and the embedding is never fine-tuned at meta-test time [arxiv:2003.11539 §3.2] [eq. 6]. The same probe is also used to score self-supervised embeddings [arxiv:2003.11539 §4.4]. The second card keeps the backbone frozen too, but its classifier is also trained on features sampled from per-class statistics of the base classes, so it needs base-class data at fit time [arxiv:2101.06395 §3.2.1] (see the distribution-calibration variant below). The third card uses the same mechanism for transfer rather than few-shot learning. Its CNNs are trained on scene labels, and their frozen features are scored with a linear SVM on eight target datasets [arxiv:1610.02055 §3.3] [tab. 3]. The remaining thirteen cards use the probe in six other roles. One turns it into the first stage of a fine-tuning recipe, where the probed head initialises full fine-tuning [arxiv:2202.10054 §4.2]. Seven use it as the evaluation for self-supervised pretraining, where the probe measures the features and is not part of the method [arxiv:2111.06377 tab. 1] [arxiv:2204.07141 tab. 3] [arxiv:2102.06810 §4] [arxiv:2104.14294 tab. 2] [arxiv:2110.09348 tab. 1] [arxiv:2011.00362 tab. 1] [arxiv:2202.03026 tab. 1]. Two use probes as diagnostics: a linear model reads a property out of hidden activations, to show where information sits rather than to classify a task [arxiv:2603.17839 §2.6], and a survey treats such probes as a way to test linguistic information [arxiv:2108.04840 §14]. One uses a probe to choose which attention heads to edit at inference time. The probe's accuracy ranks the heads, and the edit direction is a mean difference, not the probe's weights [arxiv:2306.03341 §3.2] [§3.3]. One fits linear classifiers at each depth of a CNN trained one block at a time, to measure how separable and how transferable its frozen features are [arxiv:1812.11446 §4.2] [app. C.3]. The last uses the probe as the protocol for a language-augmented model, and starts the probe's weights from the text encoder instead of at random [arxiv:2204.08790 §4.2]. Sixteen paper cards cite this slug. The first two of the three few-shot and transfer cards report author-reported results on miniImageNet-style 5-way benchmarks, with different backbones and protocols, so their numbers are not directly comparable (see Contradictions). The benchmarks are miniImageNet, tieredImageNet, CIFAR-FS, FC100 and a Meta-Dataset setup [arxiv:2003.11539 tab. 1] [tab. 2] [tab. 7], plus miniImageNet, CUB and tieredImageNet [arxiv:2101.06395 tab. 2] [tab. 3]. The third card's benchmarks are SUN397, MIT Indoor67, Scene15, SUN Attribute, Caltech101, Caltech256, Action40 and Event8 [arxiv:1610.02055 tab. 3]. The LP-FT card uses ten distribution-shift sets [arxiv:2202.10054 tab. 2], and the ViT pretraining cards use ImageNet-1K linear evaluation and low-label splits [arxiv:2111.06377 tab. 1] [arxiv:2204.07141 tab. 1]. The DINO card uses ImageNet-1K linear and k-NN evaluation [arxiv:2104.14294 tab. 2]. The DirectCLR card uses ImageNet-1K linear probing with a ResNet-50 [arxiv:2110.09348 tab. 1]. The survey card collects linear-probe numbers on ImageNet, Places and PASCAL VOC from other papers [arxiv:2011.00362 tab. 1] [tab. 2] [tab. 3]. The context-autoencoder card uses ImageNet-1K linear and attentive evaluation [arxiv:2202.03026 tab. 1]. The greedy layer-wise card uses Caltech-101 with 30 training images per class over ten splits [arxiv:1812.11446 app. C.3] [tab. 8], the ELEVATER card averages over 20 image-classification datasets [arxiv:2204.08790 tab. 2], and the ITI card uses TruthfulQA [arxiv:2306.03341 §3.1]. ## Variants ### Per-episode logistic regression on a supervised embedding Pool all base-class training data into one ordinary classification problem, train a network with cross-entropy, drop its classification layer, and fit a multinomial logistic regression on each test support set using L2-normalised features [arxiv:2003.11539 §3.2] [eq. 4] [eq. 5] [eq. 6]. The paper's implementation uses scikit-learn [arxiv:2003.11539 §4.1]. Card: [Tian et al., 2020](/papers/2003.11539). - ResNet-12, miniImageNet 5-way: 62.02 ± 0.63 (1-shot) and 79.64 ± 0.44 (5-shot) [arxiv:2003.11539 tab. 1]. - Normalisation matters. On miniImageNet 1-shot, plain logistic regression gives 58.74 and adding L2 normalisation gives 61.56; support augmentation brings it to 62.02 [arxiv:2003.11539 tab. 4]. - Against a WRN-28-10 backbone, the frozen probe scored 61.1 ± 0.86 on miniImageNet 1-shot, above the 57.73 ± 0.62 that Dhillon et al. reported for fine-tuning the same backbone [arxiv:2003.11539 §4.2]. - On Meta-Dataset (trained on ILSVRC only, ResNet-18), the mean accuracy over ten test sets is 68.02, against 60.94 for the best prior number per dataset. It is ahead of the best prior on 9 of 10 sets and behind on Aircraft [arxiv:2003.11539 tab. 7] [§5]. ### Probe on a distilled embedding Train the embedding with born-again self-distillation before freezing it. The probe itself does not change [arxiv:2003.11539 §3.3]. This is the same mechanism as [knowledge distillation](/concepts/knowledge-distillation) applied across generations. - miniImageNet ResNet-12: 64.82 ± 0.60 (1-shot) and 82.14 ± 0.43 (5-shot), using two generations [arxiv:2003.11539 tab. 1] [§4.6]. - Generation count was chosen on the validation set: two for miniImageNet, CIFAR-FS and FC100, one for tieredImageNet [arxiv:2003.11539 §4.6]. - Each generation is a full training run with the same schedule, and the new network is re-initialised, not warm-started [arxiv:2003.11539 §4.1]. ### Probe on a self-supervised embedding Train a ResNet-50 without labels on the miniImageNet meta-training split, using MoCo or CMC, then fit the same per-episode probe [arxiv:2003.11539 §4.4] [app. C]. Frozen features from supervised training are the reference point. | Embedding (ResNet-50) | mini 1-shot | mini 5-shot | |---|---|---| | Supervised | 57.56 ± 0.79 | 73.81 ± 0.63 | | MoCo | 54.19 ± 0.93 | 73.04 ± 0.61 | | CMC | 56.10 ± 0.89 | 73.87 ± 0.65 | Source: [arxiv:2003.11539 tab. 3]. The CMC 5-shot result sits within the confidence intervals of the supervised baseline. ### Distribution calibration before the probe Keep the backbone frozen and make the support set larger before fitting. For each base class, store the mean and the unbiased covariance of its features once [arxiv:2101.06395 §3.2.1] [eq. 1] [eq. 2]. For each support feature, after a Tukey power transform with λ = 0.5 [arxiv:2101.06395 §3.2.2] [eq. 3], average the statistics of its two nearest base classes with its own feature to get a Gaussian [§3.2.2] [eq. 4–6]. Sample 750 features per class from these Gaussians [§3.2.3] [eq. 7] and fit a logistic regression on the support features plus the samples [eq. 8]. No parameters are learned and no generator is trained [§3.2]. Card: [Yang et al., 2021](/papers/2101.06395). - miniImageNet 5-way, WRN28 backbone with a rotation loss: 68.57 ± 0.55 (1-shot) and 82.88 ± 0.42 (5-shot) with logistic regression [arxiv:2101.06395 tab. 2]. - Same backbone, ablation on miniImageNet 5-way 1-shot: plain logistic regression 56.37, Tukey transform alone 64.30, generated features alone 63.70, both 68.57 [arxiv:2101.06395 tab. 4]. The 5-shot figures are 79.03, 81.33, 82.26 and 82.88 [tab. 4]. - The gain depends on the backbone. For WRN28 without the rotation loss, the 1-shot figure rises from 54.53 to 64.38 [arxiv:2101.06395 tab. 5]. - The paper says the gain grows with the similarity of the nearest base classes, but this rests on three novel classes [arxiv:2101.06395 app. D]. ### Linear SVM on a frozen CNN trained on a source dataset Train a CNN on a large labelled source set, take a fixed layer as a feature vector, and fit a linear SVM with C = 1 on each target dataset's training split [arxiv:1610.02055 §3.3]. For AlexNet and VGG the feature is the fc7 layer (4096-d). For GoogLeNet it is the globally pooled layer (1024-d) [arxiv:1610.02055 §3.3]. Card: [Zhou et al., 2016](/papers/1610.02055). - Source choice changes the ranking. Places365-VGG scores 63.24 on SUN397 and ImageNet-VGG scores 48.29 [arxiv:1610.02055 tab. 3]. The order flips on Caltech101, where the figures are 67.63 for Places365-VGG and 88.42 for ImageNet-VGG [tab. 3]. - Averaged over the eight targets, ImageNet-VGG reaches 77.05 and Places365-VGG 73.18. A VGG trained on both label sets (Hybrid1365-VGG) reaches 81.48, the best average in the table [tab. 3] [§3.3]. - The authors say the Places365-VGG feature is the best single feature for SUN397 that they know of [§3.3]. This is a claim about the paper's own comparison, not a general result. ### Per-depth probe on a layer-wise trained CNN Greedy layer-wise training fits each block with a small auxiliary classifier and then freezes the block [arxiv:1812.11446 §3.2]. Those auxiliary classifiers are trained together with their block, so they are part of the training method and not probes. A probe appears in two analyses of the finished network. The first fits a linear classifier on the frozen features at each depth, and the paper reports that linear separability rises with depth on CIFAR-10 [arxiv:1812.11446 §4.2] [fig. 3]. The second trains a linear model on the frozen features of each ImageNet layer and tests it on Caltech-101 [arxiv:1812.11446 app. C.3] [tab. 8]. Card: [Belilovsky et al., 2018](/papers/1812.11446). - Caltech-101, 30 training images per class, 10 splits: a ConvNet trained from scratch reaches 46 ± 1.7%, layer 2 reaches 59.9 ± 0.9, layer 4 reaches 75.0 ± 1.0 and layer 8 reaches 82.6 ± 0.9 [arxiv:1812.11446 tab. 8]. Layer 1 is below the from-scratch baseline, at 45.5 ± 0.9 [tab. 8]. - The separability study averages three widths (64, 128 and 256). Its standard deviation over five seeds averages 0.58 points [arxiv:1812.11446 §4.2]. ### Linear probe as the head before fine-tuning (LP-FT) Fit a linear head with the features frozen, then fine-tune all layers starting from that head. The paper's argument is that when the head starts random, its early gradients distort the pretrained features, and that starting from a probed head avoids this for the perfect-feature case [arxiv:2202.10054 §3.2.2] [§3.4]. Card: [Kumar et al., 2022](/papers/2202.10054). - Ten shifts, averaged. In-distribution accuracy is 85.7 for LP-FT, 85.1 for full fine-tuning and 82.9 for probing alone. Out-of-distribution accuracy is 68.9, 59.3 and 66.2 respectively [arxiv:2202.10054 tab. 1] [tab. 2]. - The paper calls LP-FT a first step, not a state-of-the-art method, and says its analysis for imperfect features is not closed-form [§3.4] [app. B]. - On ImageNet, the recipe is five epochs of probing followed by five of fine-tuning, which the paper sets to equal compute with plain fine-tuning [arxiv:2202.10054 app. B]. ### Language-initialised linear head for a language-augmented model For a model with an image encoder and a text encoder, such as CLIP, the linear head can start from the text embeddings of the class names instead of random weights. Each class vector is built from the averaged prompt embeddings, and the bias starts at zero [arxiv:2204.08790 §4.2] [fig. 2 (d)]. The Separate-Head variant keeps the projected dimension. Merge-Head puts the head on the backbone output instead, which keeps the trainable parameter count equal to that of a language-free model [fig. 2 (e)] [§5.1]. The paper's argument is that the starting classifier then reproduces the zero-shot choice [app. C]. Card: [Li et al., 2022](/papers/2204.08790). - Image classification, CLIP ViT-B/32, averaged over 20 datasets, 5-shot linear probing: 58.09 ± 2.80 with a random head, 65.35 ± 1.24 with Separate-Head and 65.88 ± 0.79 with Merge-Head [arxiv:2204.08790 tab. 2]. - The zero-shot accuracy of the same checkpoint is 56.64, so the 5-shot probes with language initialisation score above it [tab. 2]. The paper states this as a headline finding [§5.1] [fig. 3]. - With full data the three inits are close: 78.38 random, 78.40 Separate-Head and 78.96 Merge-Head [arxiv:2204.08790 tab. 2]. The difference is in the few-shot columns. - Object detection with GLIP shows the same ordering of adaptation methods. The linear probe is below prompt tuning at every shot count listed, for example 22.2 against 29.7 at one shot [arxiv:2204.08790 tab. 3]. - Adding knowledge text to class names changes zero-shot accuracy of UniCL Swin-Tiny from 27.15 to 33.95 with Wiktionary and GPT-3 together [fig. 5]. On K-CLIP the 5-shot probe gains 0.48 points and wins on 15 of 20 datasets [arxiv:2204.08790 tab. 4]. ### Linear evaluation of masked-image pretraining Masked autoencoders pretrain a ViT by reconstructing hidden pixels, then a linear head is fit on frozen features. The probe is the evaluation, not the method. Card: [He et al., 2021](/papers/2111.06377). - The probe uses LARS with base learning rate 0.1, a batch of 16,384, 90 epochs, and a BatchNorm layer before the linear head [arxiv:2111.06377 app. A.1]. - Linear probing gives 68.0 (ViT-B), 75.8 (ViT-L) and 76.6 (ViT-H) top-1 on ImageNet-1K, against fine-tuned results in the mid-80s [arxiv:2111.06377 app. B] [tab. 3]. - The decoder's depth changes the probe far more than fine-tuning. With one decoder block the linear score is 65.5 and with eight it is 73.5, while fine-tuning stays at 84.8–84.9 [arxiv:2111.06377 tab. 1]. ### Linear evaluation of context-autoencoder pretraining The context autoencoder hides about half the patches, encodes only the visible ones, and regresses the hidden patches' encoder representations. The probe is the evaluation, not the method. The paper reports two frozen-encoder probes: a linear head, which this page covers, and an attentive probe with a learned query token, which is a different head [arxiv:2202.03026 §5.2]. Card: [Chen et al., 2022](/papers/2202.03026). - ImageNet-1K, ViT-B, 1600 epochs: linear 71.4 with the d-VAE tokenizer and 70.4 with the DALL-E tokenizer. The attentive probe gives 77.4 and 77.1 [arxiv:2202.03026 tab. 1]. - ViT-L, 1600 epochs: linear 77.9 with the d-VAE tokenizer and 78.1 with DALL-E, against 76.0 for MAE in the same table [arxiv:2202.03026 tab. 1]. - The linear probe uses LARS with momentum 0.9, 90 epochs, batch 16,384 and a learning rate of 6.4, with a BatchNorm layer without affine parameters before the classifier [arxiv:2202.03026 §5.2]. - Ablation, ViT-B at 300 epochs: linear 60.3 with neither the decoder nor the alignment term, and 64.1 with both [arxiv:2202.03026 tab. 5]. - Reconstruction target, ViT-B at 1600 epochs: linear 70.4 with DALL-E codes, 71.4 with d-VAE codes and 72.4 with raw RGB pixels. The authors conclude the target does not matter much [arxiv:2202.03026 tab. 8]. ### Linear evaluation of siamese ViT pretraining Masked Siamese Networks match prototype assignments with an EMA teacher, and they are evaluated both with many labels and with very few. The probe is an L2-regularised logistic regression on frozen CLS features [arxiv:2204.07141 app. A.2]. Card: [Assran et al., 2022](/papers/2204.07141). - Full ImageNet-1K labels, linear evaluation: MSN ViT-L/7 reaches 80.7 at 200 epochs. The table lists MAE ViT-H/14 at 76.6, close to the MAE card's own number [arxiv:2204.07141 tab. 3]. - Low labels, 1, 2 or 5 per class, averaged over three splits: MSN ViT-B/4 reaches 72.4 with 5 labels per class [arxiv:2204.07141 tab. 1]. - The same low-label table lists MAE ViT-B/16 at 8.2, 25.0 and 40.5. DINO ViT-B/16 at 41.8, 51.9 and 61.4 on the same protocol [arxiv:2204.07141 tab. 1]. ### Linear evaluation of self-distilled ViT features DINO trains a ViT without labels by matching a momentum teacher. Its frozen [CLS] features are scored in two ways: a linear probe, and a weighted k-NN classifier with no fitted parameters. The probe is the evaluation, not the method. The linear head uses SGD with batch 1024 for 100 epochs, no weight decay, and a learning rate that is swept [arxiv:2104.14294 app. F]. The paper uses k-NN as a hyperparameter-free check, because linear probes vary a lot with the learning rate [arxiv:2104.14294 §3.2]. Card: [Caron et al., 2021](/papers/2104.14294). - ImageNet-1K, ViT-S/16: 77.0 linear and 74.5 k-NN top-1, against 79.8 linear for a supervised ViT-S of the same size [arxiv:2104.14294 tab. 2]. - The linear head reads the last four [CLS] tokens for ViT-S. A single final layer gives 76.1 and the four-layer concatenation gives 77.0 [app. F]. - With 1% of ImageNet labels, the frozen ViT-S/16 scores 64.5 with logistic regression and 61.3 with k-NN [arxiv:2104.14294 app. A]. ### Probe as the metric for non-contrastive SSL The DirectPred paper reports linear-probe top-1 as its metric. STL-10 and CIFAR-10 use ResNet-18, and ImageNet uses ResNet-50 with top-1 and top-5 [arxiv:2102.06810 app. C] [§4]. The probe is only the readout. The paper's change is in the predictor, which is set from an eigendecomposition instead of trained by gradient descent [arxiv:2102.06810 §4]. Card: [Tian et al., 2021](/papers/2102.06810). - ImageNet, 300 epochs, symmetric setting: 72.4 top-1 for DirectPred against 69.9 for a gradient-trained linear predictor, and a 72.5 reference for the two-layer BYOL predictor [arxiv:2102.06810 tab. 9]. - STL-10, 100 epochs: 77.86 ± 0.16 for DirectPred against 74.51 ± 0.47 for the gradient-trained predictor [arxiv:2102.06810 tab. 6] [tab. 3]. ### Survey tables of linear-probe results A contrastive-learning survey lists linear-probe results from many original papers. It runs no experiments of its own [arxiv:2011.00362 §7]. Its linear evaluation uses a frozen encoder with a linear classifier, reported as top-1 [arxiv:2011.00362 tab. 1] [tab. 2] [tab. 3]. Card: [Jaiswal et al., 2020](/papers/2011.00362). - ImageNet, ResNet-50, top-1: SwAV 75.3, MoCo v2 71.1, SimCLR 69.3 and MoCo 60.6, against 76.5 for the supervised ResNet-50 [arxiv:2011.00362 tab. 1]. - Places, linear classifier on ImageNet-pretrained ResNet-50 features: SwAV 56.7, against 53.2 for the supervised model [arxiv:2011.00362 tab. 2]. - PASCAL VOC linear classification, top-1: SwAV 88.9, against 87.5 for the supervised ResNet-50 [arxiv:2011.00362 tab. 3]. The survey's text says these linear models were pretrained on VOC7, but the caption does not [§7]. ### Probe with and without a projector for contrastive SSL Contrastive pretraining usually places a projector between the encoder and the loss. The probe is fit on the encoder output, so the projector changes the probed features only through training. The paper's no-projector rows make this explicit: the probe reads the encoder output there [arxiv:2110.09348 tab. 1]. DirectCLR removes the projector and applies the contrastive loss to a fixed sub-vector of the encoder output instead [arxiv:2110.09348 §6.2]. Card: [Jing et al., 2021](/papers/2110.09348). - ImageNet, ResNet-50, 100 epochs, SimCLR: 51.5 with no projector, 61.1 with a one-layer linear projector and 66.5 with a two-layer projector [arxiv:2110.09348 tab. 1]. - DirectCLR, no projector: 62.7, above the one-layer linear projector and below the two-layer one [arxiv:2110.09348 tab. 1]. - The ablation's fixed low-rank diagonal projector, which matches the DirectCLR setting, scores 62.7. A trainable diagonal projector scores 60.2 [arxiv:2110.09348 tab. 2]. ### Diagnostic probe on model activations Train a linear model on hidden activations to predict a property of the model's output, not to perform the task. In the Gemma 3 27B study, each layer and token position gets an L2 logistic regression for whether an answer is correct (scored with AUROC) and a ridge regression for the midpoint of the stated confidence class (scored with R²). Both use 5-fold cross-validation on z-scored activations from 3,000 TriviaQA trials [arxiv:2603.17839 app. C.1.8]. Card: [Kumaran et al., 2026](/papers/2603.17839). - Activations at the post-answer newline explain verbal-confidence variance beyond six answer log-probability summaries [arxiv:2603.17839 §2.6] [app. D.3]. - For correctness, a log-probability mean baseline reaches AUROC 0.75. The paper reports that the post-answer position decodes correctness earlier than the other position it compares [arxiv:2603.17839 §2.6] [fig. 6]. - The study also uses steering, patching, noising, swaps and attention blocking, so probes are one line of evidence among several [arxiv:2603.17839 §2]. ### Probe-selected heads for activation steering Fit one linear probe per attention head, on the activation at the last token of each labelled true or false answer. Rank the heads by held-out probe accuracy, keep the top K, and add a scaled direction to each kept head's output at every decoding step [arxiv:2306.03341 §3.2] [§3.3] [eq. 2]. The direction is the mean of the true activations minus the mean of the false ones, which the paper calls mass mean shift [§3.3] [tab. 3]. Here the probe selects heads, and its weights are not the edit. Card: [Li et al., 2023](/papers/2306.03341). - The strongest single head reaches 83.3% validation accuracy on the paper's probe, and most of the signal is in early and middle layers [§3.2] [fig. 2]. - LLaMA-7B on TruthfulQA, 2-fold cross-validation: true*informative rises from 30.5 to 42.3 with mass mean shift at alpha 20 [arxiv:2306.03341 tab. 3]. - Selection matters. Editing all heads at alpha 5 gives 35.4, head-wise selection gives 42.3, and point-wise top coefficients give 39.2 with a cross-entropy of 4.01 [arxiv:2306.03341 tab. 5]. - Instruction-tuned models, 2-fold cross-validation: Alpaca goes from 32.5 to 65.1 and Vicuna from 51.5 to 74.0 [arxiv:2306.03341 tab. 2]. - Transfer without tuning on other benchmarks gains 4.7, 1.5 and 4.45 points on Natural Questions, TriviaQA and MMLU [tab. 4]. The authors call this "somewhat better" [§5.3]. ## When to use which | Situation | Use | Why | |---|---|---| | Labelled base classes that can be pooled into one classification problem, and you can train a backbone | Supervised embedding plus per-episode logistic regression | Author-reported baseline that matches or beats the meta-learning methods it was compared with [arxiv:2003.11539 tab. 1] | | The probe is not enough and you can afford extra full training runs | Add born-again distillation, usually two generations | Raises miniImageNet 5-shot from 79.64 to 82.14 [arxiv:2003.11539 tab. 1] [§4.6] | | Only unlabelled images for pretraining | Self-supervised embedding plus probe | Competitive at 5-shot, but MoCo trails supervised at 1-shot [arxiv:2003.11539 tab. 3] | | You need to know whether a meta-learning method's gains come from its algorithm or from the features | Use a linear probe on the same backbone as the baseline | The paper's motivating question [arxiv:2003.11539 §1] | | Your classes cannot be merged into one disjoint label set | Treat the paper's result with caution | The authors attribute the baseline's strength to merging mutually exclusive tasks [arxiv:2003.11539 §4.9] [§6] | | A frozen embedding, few labels per class, and base-class features you can store to compute per-class means and covariances | Distribution calibration, then logistic regression | No training and no extra parameters. Plain logistic regression on the same backbone gains 12.2 points at 1-shot [arxiv:2101.06395 tab. 4] | | Features that can be negative, for example before a ReLU | Per-episode probe with L2 normalisation | The Tukey transform assumes non-negative inputs [arxiv:2101.06395 §4.1.3]. The other card L2-normalises instead [arxiv:2003.11539 §4.5] | | Target data is scene-centric (indoor or outdoor places) and a labelled scene set exists to pretrain on | Backbone trained on scene labels, then a linear SVM on its fc7 or pooled features | Places365-VGG beats ImageNet-VGG on SUN397 (63.24 against 48.29) [arxiv:1610.02055 tab. 3]. This is one paper's comparison on its own targets | | Target data is object-centric and an ImageNet-scale object backbone is available | Object-label backbone plus linear SVM | ImageNet-VGG beats Places365-VGG on Caltech101 (88.42 against 67.63) [arxiv:1610.02055 tab. 3] | | You need one feature for several target types and cannot pick a source per target | A backbone trained on both label sets | Hybrid1365-VGG has the best average over the eight targets, 81.48 [arxiv:1610.02055 tab. 3] | | A pretrained backbone will be fine-tuned, and the target may differ from the pretraining data | Linear probe first, then fine-tune (LP-FT) | Better average OOD accuracy over ten shifts than fine-tuning alone, 68.9 against 59.3 [arxiv:2202.10054 tab. 2]. The paper's theory covers only the perfect-feature case [§3.4] | | Pretrained features are good and the target is close to the pretraining data | Plain fine-tuning is acceptable | Fine-tuning has the higher in-distribution average over the ten shifts, 85.1 against 82.9 for probing alone [arxiv:2202.10054 tab. 1]. This is an average over the paper's own shifts | | Masked-image pretraining where the features will be fine-tuned | Fine-tune and do not judge the features by their probe | The probe score is much lower than the fine-tuned score, and it depends strongly on decoder depth [arxiv:2111.06377 tab. 1] [tab. 3] | | Masked-image pretraining compared against a paper that reports both probes | Name the probe and report linear and attentive numbers separately | For CAE ViT-B at 1600 epochs the attentive probe is about 6 points above the linear one, 77.4 against 71.4 [arxiv:2202.03026 tab. 1] | | Comparing self-supervised ViTs on few labels | Report the frozen-feature low-label protocol with its classifier | The MSN table shows large gaps between methods at one to five labels [arxiv:2204.07141 tab. 1] | | You want to know where a model's confidence or correctness is decodable from its activations | Diagnostic probe on residual activations, per layer and position | The paper finds decodability depends on position and layer [arxiv:2603.17839 §2.6] [fig. 6] | | A language-aligned model (CLIP-style), few labels per class | Language-initialised head, Merge-Head or Separate-Head | At 5 shots the language-initialised probes average 65.35 to 65.88, against 58.09 for a random head [arxiv:2204.08790 tab. 2]. Author-reported over 20 datasets with one checkpoint. Merge-Head matches the trainable-parameter count of a language-free model [fig. 2 (e)] | | A language-aligned model with many labels per class | Either init gives about the same probe | With full data the three inits are within 0.6 points of each other [arxiv:2204.08790 tab. 2] | | Layer-wise trained CNN, and you need to know how separable or transferable each depth is | Linear probe at each depth | Caltech-101 probe accuracy rises from 59.9 at layer 2 to 82.6 at layer 8 [arxiv:1812.11446 tab. 8]. One paper, one dataset, 30 images per class | | Contrastive pretraining, probe is the metric, and you want fewer trainable parts | Drop the projector and contrast a fixed sub-vector of the encoder output (DirectCLR) | Beats a one-layer linear projector (62.7 against 61.1), but not a two-layer projector (66.5) [arxiv:2110.09348 tab. 1]. Single ImageNet setting, author-reported. If the probe score is the target, the two-layer projector scored highest | | Choosing among contrastive SSL methods by their ImageNet linear-probe numbers, with no time to re-run them | Read the survey's table as an index, then check the original paper's setting | The survey re-runs no method and collects numbers under differing protocols [arxiv:2011.00362 §7]. Its highest self-supervised ImageNet entry is SwAV at 75.3, just under the supervised 76.5 [tab. 1] | | Self-supervised ViT with no labels, compared against other SSL methods | Report k-NN and a linear probe with a learning-rate sweep | k-NN needs no fitted parameters, and the linear probe varies with the learning rate [arxiv:2104.14294 §3.2]. ViT-S/16 scores 74.5 k-NN and 77.0 linear [tab. 2] | | You can access a model's activations and have labelled true and false answers, and want to shift outputs at inference | Probe each head, keep the top K, and shift along the mean-difference direction | Raises TruthfulQA true*informative from 30.5 to 42.3 on LLaMA-7B, 2-fold [arxiv:2306.03341 tab. 3]. Needs white-box access and labelled pairs for every head [§4.1] | | Only API access, or no labelled true and false answers | Not this recipe | The method needs hidden activations and labels, and the paper's evaluation uses a fine-tuned judge model [arxiv:2306.03341 §4.1] | Where a method needs test queries, transductive fine-tuning is a different setting, and this paper does not use it [arxiv:2003.11539 §1] [§3.2]. ## Implementation sketch Pseudocode in our own words. The embedding `f` is trained and frozen, with its classification head removed. ```pseudocode # Support set: N classes, k labelled images each. Query set: unlabelled. def probe_episode(f, support_x, support_y, query_x): # Make five augmented copies of every support image [§4.5] aug_x, aug_y = augment_copies(support_x, support_y, copies=5) # Embed, then L2-normalise onto the unit sphere (both sides) [§4.5] z_support = normalise(f(aug_x)) z_query = normalise(f(query_x)) # Fit an N-way multinomial logistic regression on the support features [eq. 6] probe = fit_logistic_regression(z_support, aug_y) return probe.predict(z_query) ``` Notes: - The probe is re-initialised for each episode. Only the support set is used to fit it [arxiv:2003.11539 §3.2]. - The regularisation strength and solver are not given in the paper text [arxiv:2003.11539 §4.1]. - For distribution calibration, replace the augmentation step with sampling from the calibrated Gaussians of each support feature, and fit the same logistic regression on support plus samples [arxiv:2101.06395 §3.2.3]. - For LP-FT, run the same probe on the full training set, then pass the fitted head to the fine-tuning loop as its starting weights [arxiv:2202.10054 §4.2]. - For a language-initialised head, build each class column from the averaged prompt embeddings of its class name, and start the bias at zero [arxiv:2204.08790 §4.2]. The ELEVATER card records that the released code also normalises each prompt embedding before averaging, which the paper does not describe. ## Pitfalls - **Which layer to probe.** The paper's method uses the pooled penultimate features [arxiv:2003.11539 §3.2]. The released evaluation code can also use the classifier's logits, which changes the feature size to the number of base classes. The paper does not clearly say which one its reported numbers use. Check the [card](/papers/2003.11539) before reproducing. - **Projector choice changes the probe.** In the one paper that varies only the projector, the SimCLR probe ranges from 51.5 with none to 66.5 with a two-layer projector [arxiv:2110.09348 tab. 1]. Name the projector with any probe number on a contrastive model. - **Sub-vector length is not in the text.** For DirectCLR, the length d0 used for the main result is not stated in the paper text, and its sweep is only in a figure [arxiv:2110.09348 app. E]. - **Depth matters for transfer.** In the greedy layer-wise study, the Caltech-101 probe scores about 23 points higher at layer 8 than at layer 2 [arxiv:1812.11446 tab. 8]. A probe on the wrong layer can mislead about a model's features. - **Normalise support and query alike.** The paper L2-normalises both sides [arxiv:2003.11539 §4.5], and dropping normalisation costs about 2.8 points at 1-shot [arxiv:2003.11539 tab. 4]. - **Augmentation is a small effect.** Support augmentation changes results by at most about half a point, and sometimes lowers them (FC100 1-shot 42.77 to 42.59) [arxiv:2003.11539 tab. 4]. It multiplies the support-set feature extraction by five [arxiv:2003.11539 §4.5]. - **Reporting protocol.** The paper reports medians over three runs of 1000 episodes (ImageNet-derived sets) or 3000 (CIFAR-derived sets) [arxiv:2003.11539 §4.2] [§4.3]. Compare numbers only under the same episode count. - **Where the covariance's α goes.** The distribution-calibration equation is typeset with a plus α, which can be read as adding α to the diagonal only. The text says α is added to every entry [arxiv:2101.06395 §4.6], so the calibrated covariance is not diagonal. - **Mixed transforms.** The paper does not say whether base statistics use Tukey-transformed features, while support and query are transformed [arxiv:2101.06395 §3.2.2]. The released code uses untransformed base features, so check the [card](/papers/2101.06395) before reproducing. - **Non-negative inputs only.** The Tukey transform with λ = 0.5 applied to negative features gives NaN. Post-ReLU pooled features are fine [arxiv:2101.06395 §4.1.3]. - **Probes can learn the task.** A probe with enough capacity and data can learn the target label itself, so a good probe score does not show what the model encodes. Control tasks and information-theoretic measures such as minimum description length are the proposed fixes [arxiv:2108.04840 §14.2]. - **Probe recipes differ between SSL papers.** MAE's probe uses LARS and a large batch [arxiv:2111.06377 app. A.1]. MSN's probe uses SGD with Nesterov momentum at learning rate 6.4 for 100 epochs [arxiv:2204.07141 app. A.3]. DINO's probe uses SGD at batch 1024 for 100 epochs, with the learning rate swept [arxiv:2104.14294 app. F]. Compare SSL numbers only under the same recipe. The MSN card also flags a batch-size mismatch between paper and code for linear evaluation [arxiv:2204.07141 app. A.3]. - **Attentive is not linear.** The CAE card reports an attentive probe beside its linear one, and it is about 6 points higher on ViT-B [arxiv:2202.03026 tab. 1]. The attentive head has a cross-attention module, so a number from one probe does not stand for the other [arxiv:2202.03026 §5.2]. Check which head a table reports before comparing. - **Survey probe numbers mix protocols.** The survey's ImageNet table joins rows from many original papers and gives no per-method probe settings [arxiv:2011.00362 tab. 1]. Use it to find candidates, then check the cited original before quoting a number. - **Steering strength is in units of sigma.** ITI's edit is alpha times sigma times a unit vector [arxiv:2306.03341 eq. 2]. Alpha 20 means 20 standard deviations along the direction [app. B], and sigma is computed per model. Run a random direction at the same alpha as a control [arxiv:2306.03341 app. B] [tab. 3]: it leaves LLaMA-7B nearly unchanged. - **Unstated ImageNet details.** For DirectPred on ImageNet, the learning rate, weight decay, EMA value and LARS settings are not stated in the paper. It says they follow BYOL, but BYOL's values are not given there [arxiv:2102.06810 §4]. - **Probe regularisation on ImageNet.** The LP-FT paper does not sweep the probe's L2 strength on ImageNet [arxiv:2202.10054 app. B]. - **Best test epoch in the ELEVATER toolkit.** The ELEVATER card reports that the image-classification fine-tuning script keeps the best epoch measured on the test split [arxiv:2204.08790 §4.1]. The card does not confirm this for the linear-probing rows. Check before comparing against them, or report a fixed-epoch number. - **Few-shot sizes before the split.** The ELEVATER card reports that its code applies N before an 80/20 train and validation split, so the search trains on about 80% of N. Its image-classification script also raises N = 1 to 2 [arxiv:2204.08790 §3.3] [§4.1]. Name the split when you report few-shot results. - **Probe on a frozen image backbone.** The ELEVATER paper defines linear probing as training only the classifier, with the image backbone frozen [arxiv:2204.08790 §3.3]. Its object-detection linear probe also updates the projection matrices [§4.2], so the frozen-backbone reading does not apply there. ## Contradictions - **Feature normalisation.** The first card L2-normalises support and query features and reports that dropping it costs about 2.8 points at 1-shot (61.56 to 58.74) [arxiv:2003.11539 tab. 4]. The second card applies a Tukey power transform instead and states no normalisation; its code applies none beyond that transform [arxiv:2101.06395 §4.1.3]. The papers do not test each other's preprocessing, so the two recommendations are not reconciled here. - **Numbers across cards.** The first card reports a plain probe at 58.74 (1-shot, ResNet-12) and the second a plain logistic regression at 56.37 (1-shot, WRN28 with a rotation loss) [arxiv:2003.11539 tab. 4] [arxiv:2101.06395 tab. 4]. The protocols also differ: the first uses medians of three runs of 1000 episodes [§4.2], the second averages 10,000 tasks [§4.1.2]. Compare across the two cards only with that caveat. - **Source of the probe's training data.** The first card's probe sees only the support set. The second's sees generated features, which depend on base-class statistics. Results from the two are not a like-for-like test of the probe. - **Benchmarks and protocols.** The third card's eight-target average is a different test from the 5-way few-shot numbers of the first two cards. Its values are not on the same scale, and no card tests another's probe [arxiv:1610.02055 tab. 3] [arxiv:2003.11539 tab. 1]. The ELEVATER 5-shot averages over 20 image datasets with CLIP ViT-B/32 are not comparable to either of these [arxiv:2204.08790 tab. 2]. - **Linear probe of MAE.** The MAE card reports 68.0 for ViT-B on ImageNet-1K [arxiv:2111.06377 app. B]. ELEVATER's MAE linear probe averages 68.70 over 20 image datasets [arxiv:2204.08790 tab. 5]. Both are linear probes, but on different data, so the two numbers do not contradict each other. - **Is the probe a fair readout for masked pretraining?** The MAE paper calls linear probing weak for its features and argues that fine-tuning shows them better [arxiv:2111.06377 §4.3]. The MSN paper measures MAE on a frozen-feature low-label protocol, where MAE ViT-B/16 scores 8.2 at one label per class against 41.8 for DINO ViT-B/16 [arxiv:2204.07141 tab. 1]. Both are consistent with the same features being weak under a probe. The two papers measure different things, so their numbers are not interchangeable. - **MAE's linear-probe number across cards.** The MAE card reports 68.0 for ViT-B and 75.8 for ViT-L on ImageNet-1K [arxiv:2111.06377 app. B] [tab. 3]. The CAE paper's table gives MAE 67.8 and 76.0 for the same two models [arxiv:2202.03026 tab. 1]. The two tables come from different papers, so the differences are not resolved here. - **Probing versus fine-tuning out of distribution.** LP-FT reports that full fine-tuning does worse than probing on average over ten shifts [arxiv:2202.10054 tab. 2]. The MAE card reports the opposite ordering for ImageNet-1K, where fine-tuning is far stronger than probing [arxiv:2111.06377 tab. 3]. These are different tests: one is distribution shift, the other in-distribution accuracy. - **Probing versus fine-tuning with few labels.** With a random head, ELEVATER reports 29.75 for 5-shot fine-tuning against 58.09 for probing, which agrees with LP-FT's account of random heads distorting features. With a language-initialised head, fine-tuning is 63.29 against 65.35 at 5 shots, and 72.19 against 71.69 at 20 shots [arxiv:2204.08790 tab. 2]. LP-FT does not test few-shot sizes, and ELEVATER does not test distribution shift, so neither paper settles the other's question. - **Probing as a measure of ability.** The diagnostic-probe card uses probes to show where information is decodable [arxiv:2603.17839 §2.6]. The survey warns that a probe may learn the task, so decodability does not establish that the model uses the information [arxiv:2108.04840 §14.2]. The two papers do not test each other's use. The ITI card reads a gap of about 40 points between probe accuracy and generation accuracy on LLaMA-7B as evidence that the model represents the truth without expressing it [arxiv:2306.03341 §1]. That reading is the one the survey's warning targets, and the ITI card reports no control-task test. - **SimCLR's linear probe across sources.** The survey lists SimCLR at 69.3 top-1 with ResNet-50 [arxiv:2011.00362 tab. 1]. DirectCLR reports SimCLR at 66.5 with a two-layer projector and 100 epochs [arxiv:2110.09348 tab. 1]. The survey attaches no setting to its SimCLR row, so the two numbers may come from different schedules and are not a like-for-like test. The first card's own text and its released code differ on which features are probed, as noted under Pitfalls. ## Papers - [Tian et al., 2020](/papers/2003.11539) (arXiv:2003.11539): a supervised ResNet-12 embedding with a per-episode L2-normalised logistic-regression probe, compared against meta-learning baselines on miniImageNet, tieredImageNet, CIFAR-FS, FC100 and Meta-Dataset, plus self-supervised MoCo and CMC embeddings. - [Yang et al., 2021](/papers/2101.06395) (arXiv:2101.06395): training-free distribution calibration of support features from base-class statistics, followed by logistic regression on support plus 750 sampled features per class, on a frozen WRN28 backbone. - [Zhou et al., 2016](/papers/1610.02055) (arXiv:1610.02055): the Places scene dataset, with AlexNet, GoogLeNet and VGG scene CNNs whose frozen features are scored with a linear SVM on eight target datasets. - [Kumar et al., 2022](/papers/2202.10054) (arXiv:2202.10054): fine-tuning can distort pretrained features and underperform out of distribution; LP-FT, which probes first and then fine-tunes, is evaluated on ten shifts. - [Tian et al., 2021](/papers/2102.06810) (arXiv:2102.06810): analysis of why non-contrastive SSL avoids collapse, with DirectPred, which sets the linear predictor from the input correlation matrix instead of training it; linear-probe top-1 is the reported metric. - [He et al., 2021](/papers/2111.06377) (arXiv:2111.06377): masked autoencoders for ViT pretraining, with linear-probe and fine-tuning results on ImageNet-1K and transfer tasks. - [Chen et al., 2022](/papers/2202.03026) (arXiv:2202.03026): context autoencoder, which encodes only visible patches and regresses the hidden patches' representations; linear and attentive probes on ImageNet-1K are the probe results here. - [Assran et al., 2022](/papers/2204.07141) (arXiv:2204.07141): Masked Siamese Networks for label-efficient ViT pretraining, with linear evaluation at full labels and frozen-feature low-label protocols at one, two and five labels per class. - [Kumaran et al., 2026](/papers/2603.17839) (arXiv:2603.17839): how Gemma 3 27B computes verbal confidence, using per-layer, per-position linear probes on activations alongside causal interventions. - [Madsen et al., 2021](/papers/2108.04840) (arXiv:2108.04840): a survey of post-hoc interpretability for neural NLP, which covers probing, including the critique that probes can learn the task. - [Belilovsky et al., 2018](/papers/1812.11446) (arXiv:1812.11446): greedy layer-wise CNN training with shallow auxiliary classifiers, scaled to ImageNet; its linear separability and frozen-feature Caltech-101 transfer studies are the probe uses here. - [Li et al., 2022](/papers/2204.08790) (arXiv:2204.08790): the ELEVATER benchmark and toolkit for language-augmented visual models, with linear probing whose head is initialised from the text encoder. - [Caron et al., 2021](/papers/2104.14294) (arXiv:2104.14294): DINO, label-free self-distillation for ViTs, evaluated with a linear probe and a frozen weighted k-NN classifier on ImageNet. - [Li et al., 2023](/papers/2306.03341) (arXiv:2306.03341): Inference-Time Intervention, which fits a probe per attention head on TruthfulQA and shifts the top-K heads along a truthful direction during decoding. - [Jing et al., 2021](/papers/2110.09348) (arXiv:2110.09348): dimensional collapse in contrastive SSL, with DirectCLR, which removes the projector and applies the loss to a fixed sub-vector; ImageNet linear-probe top-1 is the reported metric. - [Jaiswal et al., 2020](/papers/2011.00362) (arXiv:2011.00362): a survey of contrastive self-supervised learning in vision and NLP, whose tables collect linear-probe and fine-tuning results from the original papers on ImageNet, Places, VOC and Kinetics.