Linear probing

Overview

A linear probe trains only a linear classifier on top of an embedding that stays fixed. The shared invariant is that the backbone's weights do not change when the probe is fit. In the first card on this page, the probe is refit for every new few-shot task from the support set alone, and the embedding is never fine-tuned at meta-test time 2003.11539 §3.2 eq. 6. The same probe is also used to score self-supervised embeddings 2003.11539 §4.4.

The second card keeps the backbone frozen too, but its classifier is also trained on features sampled from per-class statistics of the base classes, so it needs base-class data at fit time 2101.06395 §3.2.1 (see the distribution-calibration variant below).

The third card uses the same mechanism for transfer rather than few-shot learning. Its CNNs are trained on scene labels, and their frozen features are scored with a linear SVM on eight target datasets 1610.02055 §3.3 tab. 3.

The remaining thirteen cards use the probe in six other roles. One turns it into the first stage of a fine-tuning recipe, where the probed head initialises full fine-tuning 2202.10054 §4.2. Seven use it as the evaluation for self-supervised pretraining, where the probe measures the features and is not part of the method 2111.06377 tab. 1 2204.07141 tab. 3 2102.06810 §4 2104.14294 tab. 2 2110.09348 tab. 1 2011.00362 tab. 1 2202.03026 tab. 1. Two use probes as diagnostics: a linear model reads a property out of hidden activations, to show where information sits rather than to classify a task 2603.17839 §2.6, and a survey treats such probes as a way to test linguistic information 2108.04840 §14. One uses a probe to choose which attention heads to edit at inference time. The probe's accuracy ranks the heads, and the edit direction is a mean difference, not the probe's weights 2306.03341 §3.2 §3.3. One fits linear classifiers at each depth of a CNN trained one block at a time, to measure how separable and how transferable its frozen features are 1812.11446 §4.2 app. C.3. The last uses the probe as the protocol for a language-augmented model, and starts the probe's weights from the text encoder instead of at random 2204.08790 §4.2.

Sixteen paper cards cite this slug. The first two of the three few-shot and transfer cards report author-reported results on miniImageNet-style 5-way benchmarks, with different backbones and protocols, so their numbers are not directly comparable (see Contradictions). The benchmarks are miniImageNet, tieredImageNet, CIFAR-FS, FC100 and a Meta-Dataset setup [arxiv:2003.11539 tab. 1] tab. 2 tab. 7, plus miniImageNet, CUB and tieredImageNet 2101.06395 tab. 2 tab. 3. The third card's benchmarks are SUN397, MIT Indoor67, Scene15, SUN Attribute, Caltech101, Caltech256, Action40 and Event8 1610.02055 tab. 3. The LP-FT card uses ten distribution-shift sets 2202.10054 tab. 2, and the ViT pretraining cards use ImageNet-1K linear evaluation and low-label splits 2111.06377 tab. 1 2204.07141 tab. 1. The DINO card uses ImageNet-1K linear and k-NN evaluation 2104.14294 tab. 2. The DirectCLR card uses ImageNet-1K linear probing with a ResNet-50 2110.09348 tab. 1. The survey card collects linear-probe numbers on ImageNet, Places and PASCAL VOC from other papers 2011.00362 tab. 1 tab. 2 tab. 3. The context-autoencoder card uses ImageNet-1K linear and attentive evaluation 2202.03026 tab. 1. The greedy layer-wise card uses Caltech-101 with 30 training images per class over ten splits 1812.11446 app. C.3 tab. 8, the ELEVATER card averages over 20 image-classification datasets 2204.08790 tab. 2, and the ITI card uses TruthfulQA 2306.03341 §3.1.

Variants

Per-episode logistic regression on a supervised embedding

Pool all base-class training data into one ordinary classification problem, train a network with cross-entropy, drop its classification layer, and fit a multinomial logistic regression on each test support set using L2-normalised features 2003.11539 §3.2 eq. 4 eq. 5 eq. 6. The paper's implementation uses scikit-learn 2003.11539 §4.1. Card: Tian et al., 2020.

• ResNet-12, miniImageNet 5-way: 62.02 ± 0.63 (1-shot) and 79.64 ± 0.44 (5-shot) 2003.11539 tab. 1.
• Normalisation matters. On miniImageNet 1-shot, plain logistic regression gives 58.74 and adding L2 normalisation gives 61.56; support augmentation brings it to 62.02 2003.11539 tab. 4.
• Against a WRN-28-10 backbone, the frozen probe scored 61.1 ± 0.86 on miniImageNet 1-shot, above the 57.73 ± 0.62 that Dhillon et al. reported for fine-tuning the same backbone 2003.11539 §4.2.
• On Meta-Dataset (trained on ILSVRC only, ResNet-18), the mean accuracy over ten test sets is 68.02, against 60.94 for the best prior number per dataset. It is ahead of the best prior on 9 of 10 sets and behind on Aircraft 2003.11539 tab. 7 §5.

Probe on a distilled embedding

Train the embedding with born-again self-distillation before freezing it. The probe itself does not change 2003.11539 §3.3. This is the same mechanism as knowledge distillation applied across generations.

• miniImageNet ResNet-12: 64.82 ± 0.60 (1-shot) and 82.14 ± 0.43 (5-shot), using two generations 2003.11539 tab. 1 §4.6.
• Generation count was chosen on the validation set: two for miniImageNet, CIFAR-FS and FC100, one for tieredImageNet 2003.11539 §4.6.
• Each generation is a full training run with the same schedule, and the new network is re-initialised, not warm-started 2003.11539 §4.1.

Probe on a self-supervised embedding

Train a ResNet-50 without labels on the miniImageNet meta-training split, using MoCo or CMC, then fit the same per-episode probe 2003.11539 §4.4 app. C. Frozen features from supervised training are the reference point.

Source: 2003.11539 tab. 3. The CMC 5-shot result sits within the confidence intervals of the supervised baseline.

Distribution calibration before the probe

Keep the backbone frozen and make the support set larger before fitting. For each base class, store the mean and the unbiased covariance of its features once 2101.06395 §3.2.1 eq. 1 eq. 2. For each support feature, after a Tukey power transform with λ = 0.5 2101.06395 §3.2.2 eq. 3, average the statistics of its two nearest base classes with its own feature to get a Gaussian §3.2.2 eq. 4–6. Sample 750 features per class from these Gaussians §3.2.3 eq. 7 and fit a logistic regression on the support features plus the samples eq. 8. No parameters are learned and no generator is trained §3.2. Card: Yang et al., 2021.

• miniImageNet 5-way, WRN28 backbone with a rotation loss: 68.57 ± 0.55 (1-shot) and 82.88 ± 0.42 (5-shot) with logistic regression [arxiv:2101.06395 tab. 2].
• Same backbone, ablation on miniImageNet 5-way 1-shot: plain logistic regression 56.37, Tukey transform alone 64.30, generated features alone 63.70, both 68.57 2101.06395 tab. 4. The 5-shot figures are 79.03, 81.33, 82.26 and 82.88 tab. 4.
• The gain depends on the backbone. For WRN28 without the rotation loss, the 1-shot figure rises from 54.53 to 64.38 2101.06395 tab. 5.
• The paper says the gain grows with the similarity of the nearest base classes, but this rests on three novel classes 2101.06395 app. D.

Linear SVM on a frozen CNN trained on a source dataset

Train a CNN on a large labelled source set, take a fixed layer as a feature vector, and fit a linear SVM with C = 1 on each target dataset's training split 1610.02055 §3.3. For AlexNet and VGG the feature is the fc7 layer (4096-d). For GoogLeNet it is the globally pooled layer (1024-d) 1610.02055 §3.3. Card: Zhou et al., 2016.

• Source choice changes the ranking. Places365-VGG scores 63.24 on SUN397 and ImageNet-VGG scores 48.29 1610.02055 tab. 3. The order flips on Caltech101, where the figures are 67.63 for Places365-VGG and 88.42 for ImageNet-VGG tab. 3.
• Averaged over the eight targets, ImageNet-VGG reaches 77.05 and Places365-VGG 73.18. A VGG trained on both label sets (Hybrid1365-VGG) reaches 81.48, the best average in the table tab. 3 §3.3.
• The authors say the Places365-VGG feature is the best single feature for SUN397 that they know of §3.3. This is a claim about the paper's own comparison, not a general result.

Per-depth probe on a layer-wise trained CNN

Greedy layer-wise training fits each block with a small auxiliary classifier and then freezes the block 1812.11446 §3.2. Those auxiliary classifiers are trained together with their block, so they are part of the training method and not probes. A probe appears in two analyses of the finished network. The first fits a linear classifier on the frozen features at each depth, and the paper reports that linear separability rises with depth on CIFAR-10 1812.11446 §4.2 fig. 3. The second trains a linear model on the frozen features of each ImageNet layer and tests it on Caltech-101 1812.11446 app. C.3 tab. 8. Card: Belilovsky et al., 2018.

• Caltech-101, 30 training images per class, 10 splits: a ConvNet trained from scratch reaches 46 ± 1.7%, layer 2 reaches 59.9 ± 0.9, layer 4 reaches 75.0 ± 1.0 and layer 8 reaches 82.6 ± 0.9 1812.11446 tab. 8. Layer 1 is below the from-scratch baseline, at 45.5 ± 0.9 tab. 8.
• The separability study averages three widths (64, 128 and 256). Its standard deviation over five seeds averages 0.58 points 1812.11446 §4.2.

Linear probe as the head before fine-tuning (LP-FT)

Fit a linear head with the features frozen, then fine-tune all layers starting from that head. The paper's argument is that when the head starts random, its early gradients distort the pretrained features, and that starting from a probed head avoids this for the perfect-feature case 2202.10054 §3.2.2 §3.4. Card: Kumar et al., 2022.

• Ten shifts, averaged. In-distribution accuracy is 85.7 for LP-FT, 85.1 for full fine-tuning and 82.9 for probing alone. Out-of-distribution accuracy is 68.9, 59.3 and 66.2 respectively 2202.10054 tab. 1 tab. 2.
• The paper calls LP-FT a first step, not a state-of-the-art method, and says its analysis for imperfect features is not closed-form §3.4 app. B.
• On ImageNet, the recipe is five epochs of probing followed by five of fine-tuning, which the paper sets to equal compute with plain fine-tuning 2202.10054 app. B.

Language-initialised linear head for a language-augmented model

For a model with an image encoder and a text encoder, such as CLIP, the linear head can start from the text embeddings of the class names instead of random weights. Each class vector is built from the averaged prompt embeddings, and the bias starts at zero 2204.08790 §4.2 fig. 2 (d). The Separate-Head variant keeps the projected dimension. Merge-Head puts the head on the backbone output instead, which keeps the trainable parameter count equal to that of a language-free model fig. 2 (e) §5.1. The paper's argument is that the starting classifier then reproduces the zero-shot choice app. C. Card: Li et al., 2022.

• Image classification, CLIP ViT-B/32, averaged over 20 datasets, 5-shot linear probing: 58.09 ± 2.80 with a random head, 65.35 ± 1.24 with Separate-Head and 65.88 ± 0.79 with Merge-Head 2204.08790 tab. 2.
• The zero-shot accuracy of the same checkpoint is 56.64, so the 5-shot probes with language initialisation score above it tab. 2. The paper states this as a headline finding §5.1 fig. 3.
• With full data the three inits are close: 78.38 random, 78.40 Separate-Head and 78.96 Merge-Head 2204.08790 tab. 2. The difference is in the few-shot columns.
• Object detection with GLIP shows the same ordering of adaptation methods. The linear probe is below prompt tuning at every shot count listed, for example 22.2 against 29.7 at one shot 2204.08790 tab. 3.
• Adding knowledge text to class names changes zero-shot accuracy of UniCL Swin-Tiny from 27.15 to 33.95 with Wiktionary and GPT-3 together fig. 5. On K-CLIP the 5-shot probe gains 0.48 points and wins on 15 of 20 datasets 2204.08790 tab. 4.

Linear evaluation of masked-image pretraining

Masked autoencoders pretrain a ViT by reconstructing hidden pixels, then a linear head is fit on frozen features. The probe is the evaluation, not the method. Card: He et al., 2021.

• The probe uses LARS with base learning rate 0.1, a batch of 16,384, 90 epochs, and a BatchNorm layer before the linear head [arxiv:2111.06377 app. A.1].
• Linear probing gives 68.0 (ViT-B), 75.8 (ViT-L) and 76.6 (ViT-H) top-1 on ImageNet-1K, against fine-tuned results in the mid-80s [arxiv:2111.06377 app. B] tab. 3.
• The decoder's depth changes the probe far more than fine-tuning. With one decoder block the linear score is 65.5 and with eight it is 73.5, while fine-tuning stays at 84.8–84.9 2111.06377 tab. 1.

Linear evaluation of context-autoencoder pretraining

The context autoencoder hides about half the patches, encodes only the visible ones, and regresses the hidden patches' encoder representations. The probe is the evaluation, not the method. The paper reports two frozen-encoder probes: a linear head, which this page covers, and an attentive probe with a learned query token, which is a different head 2202.03026 §5.2. Card: Chen et al., 2022.

• ImageNet-1K, ViT-B, 1600 epochs: linear 71.4 with the d-VAE tokenizer and 70.4 with the DALL-E tokenizer. The attentive probe gives 77.4 and 77.1 2202.03026 tab. 1.
• ViT-L, 1600 epochs: linear 77.9 with the d-VAE tokenizer and 78.1 with DALL-E, against 76.0 for MAE in the same table 2202.03026 tab. 1.
• The linear probe uses LARS with momentum 0.9, 90 epochs, batch 16,384 and a learning rate of 6.4, with a BatchNorm layer without affine parameters before the classifier 2202.03026 §5.2.
• Ablation, ViT-B at 300 epochs: linear 60.3 with neither the decoder nor the alignment term, and 64.1 with both 2202.03026 tab. 5.
• Reconstruction target, ViT-B at 1600 epochs: linear 70.4 with DALL-E codes, 71.4 with d-VAE codes and 72.4 with raw RGB pixels. The authors conclude the target does not matter much 2202.03026 tab. 8.

Linear evaluation of siamese ViT pretraining

Masked Siamese Networks match prototype assignments with an EMA teacher, and they are evaluated both with many labels and with very few. The probe is an L2-regularised logistic regression on frozen CLS features [arxiv:2204.07141 app. A.2]. Card: Assran et al., 2022.

• Full ImageNet-1K labels, linear evaluation: MSN ViT-L/7 reaches 80.7 at 200 epochs. The table lists MAE ViT-H/14 at 76.6, close to the MAE card's own number 2204.07141 tab. 3.
• Low labels, 1, 2 or 5 per class, averaged over three splits: MSN ViT-B/4 reaches 72.4 with 5 labels per class 2204.07141 tab. 1.
• The same low-label table lists MAE ViT-B/16 at 8.2, 25.0 and 40.5. DINO ViT-B/16 at 41.8, 51.9 and 61.4 on the same protocol [arxiv:2204.07141 tab. 1].

Linear evaluation of self-distilled ViT features

DINO trains a ViT without labels by matching a momentum teacher. Its frozen [CLS] features are scored in two ways: a linear probe, and a weighted k-NN classifier with no fitted parameters. The probe is the evaluation, not the method. The linear head uses SGD with batch 1024 for 100 epochs, no weight decay, and a learning rate that is swept 2104.14294 app. F. The paper uses k-NN as a hyperparameter-free check, because linear probes vary a lot with the learning rate 2104.14294 §3.2. Card: Caron et al., 2021.

• ImageNet-1K, ViT-S/16: 77.0 linear and 74.5 k-NN top-1, against 79.8 linear for a supervised ViT-S of the same size 2104.14294 tab. 2.
• The linear head reads the last four [CLS] tokens for ViT-S. A single final layer gives 76.1 and the four-layer concatenation gives 77.0 app. F.
• With 1% of ImageNet labels, the frozen ViT-S/16 scores 64.5 with logistic regression and 61.3 with k-NN 2104.14294 app. A.

Probe as the metric for non-contrastive SSL

The DirectPred paper reports linear-probe top-1 as its metric. STL-10 and CIFAR-10 use ResNet-18, and ImageNet uses ResNet-50 with top-1 and top-5 2102.06810 app. C §4. The probe is only the readout. The paper's change is in the predictor, which is set from an eigendecomposition instead of trained by gradient descent 2102.06810 §4. Card: Tian et al., 2021.

• ImageNet, 300 epochs, symmetric setting: 72.4 top-1 for DirectPred against 69.9 for a gradient-trained linear predictor, and a 72.5 reference for the two-layer BYOL predictor 2102.06810 tab. 9.
• STL-10, 100 epochs: 77.86 ± 0.16 for DirectPred against 74.51 ± 0.47 for the gradient-trained predictor 2102.06810 tab. 6 tab. 3.

Survey tables of linear-probe results

A contrastive-learning survey lists linear-probe results from many original papers. It runs no experiments of its own 2011.00362 §7. Its linear evaluation uses a frozen encoder with a linear classifier, reported as top-1 2011.00362 tab. 1 tab. 2 tab. 3. Card: Jaiswal et al., 2020.

• ImageNet, ResNet-50, top-1: SwAV 75.3, MoCo v2 71.1, SimCLR 69.3 and MoCo 60.6, against 76.5 for the supervised ResNet-50 2011.00362 tab. 1.
• Places, linear classifier on ImageNet-pretrained ResNet-50 features: SwAV 56.7, against 53.2 for the supervised model 2011.00362 tab. 2.
• PASCAL VOC linear classification, top-1: SwAV 88.9, against 87.5 for the supervised ResNet-50 2011.00362 tab. 3. The survey's text says these linear models were pretrained on VOC7, but the caption does not §7.

Probe with and without a projector for contrastive SSL

Contrastive pretraining usually places a projector between the encoder and the loss. The probe is fit on the encoder output, so the projector changes the probed features only through training. The paper's no-projector rows make this explicit: the probe reads the encoder output there [arxiv:2110.09348 tab. 1]. DirectCLR removes the projector and applies the contrastive loss to a fixed sub-vector of the encoder output instead 2110.09348 §6.2. Card: Jing et al., 2021.

• ImageNet, ResNet-50, 100 epochs, SimCLR: 51.5 with no projector, 61.1 with a one-layer linear projector and 66.5 with a two-layer projector [arxiv:2110.09348 tab. 1].
• DirectCLR, no projector: 62.7, above the one-layer linear projector and below the two-layer one 2110.09348 tab. 1.
• The ablation's fixed low-rank diagonal projector, which matches the DirectCLR setting, scores 62.7. A trainable diagonal projector scores 60.2 [arxiv:2110.09348 tab. 2].

Diagnostic probe on model activations

Train a linear model on hidden activations to predict a property of the model's output, not to perform the task. In the Gemma 3 27B study, each layer and token position gets an L2 logistic regression for whether an answer is correct (scored with AUROC) and a ridge regression for the midpoint of the stated confidence class (scored with R²). Both use 5-fold cross-validation on z-scored activations from 3,000 TriviaQA trials 2603.17839 app. C.1.8. Card: Kumaran et al., 2026.

• Activations at the post-answer newline explain verbal-confidence variance beyond six answer log-probability summaries 2603.17839 §2.6 [app. D.3].
• For correctness, a log-probability mean baseline reaches AUROC 0.75. The paper reports that the post-answer position decodes correctness earlier than the other position it compares 2603.17839 §2.6 fig. 6.
• The study also uses steering, patching, noising, swaps and attention blocking, so probes are one line of evidence among several [arxiv:2603.17839 §2].

Probe-selected heads for activation steering

Fit one linear probe per attention head, on the activation at the last token of each labelled true or false answer. Rank the heads by held-out probe accuracy, keep the top K, and add a scaled direction to each kept head's output at every decoding step 2306.03341 §3.2 §3.3 eq. 2. The direction is the mean of the true activations minus the mean of the false ones, which the paper calls mass mean shift §3.3 tab. 3. Here the probe selects heads, and its weights are not the edit. Card: Li et al., 2023.

• The strongest single head reaches 83.3% validation accuracy on the paper's probe, and most of the signal is in early and middle layers §3.2 fig. 2.
• LLaMA-7B on TruthfulQA, 2-fold cross-validation: true*informative rises from 30.5 to 42.3 with mass mean shift at alpha 20 2306.03341 tab. 3.
• Selection matters. Editing all heads at alpha 5 gives 35.4, head-wise selection gives 42.3, and point-wise top coefficients give 39.2 with a cross-entropy of 4.01 2306.03341 tab. 5.
• Instruction-tuned models, 2-fold cross-validation: Alpaca goes from 32.5 to 65.1 and Vicuna from 51.5 to 74.0 2306.03341 tab. 2.
• Transfer without tuning on other benchmarks gains 4.7, 1.5 and 4.45 points on Natural Questions, TriviaQA and MMLU tab. 4. The authors call this "somewhat better" §5.3.

When to use which

Where a method needs test queries, transductive fine-tuning is a different setting, and this paper does not use it 2003.11539 §1 §3.2.

Implementation sketch

Pseudocode in our own words. The embedding f is trained and frozen, with its classification head removed.

# Support set: N classes, k labelled images each. Query set: unlabelled.
def probe_episode(f, support_x, support_y, query_x):
    # Make five augmented copies of every support image [§4.5]
    aug_x, aug_y = augment_copies(support_x, support_y, copies=5)
    # Embed, then L2-normalise onto the unit sphere (both sides) [§4.5]
    z_support = normalise(f(aug_x))
    z_query = normalise(f(query_x))
    # Fit an N-way multinomial logistic regression on the support features [eq. 6]
    probe = fit_logistic_regression(z_support, aug_y)
    return probe.predict(z_query)

Notes:

• The probe is re-initialised for each episode. Only the support set is used to fit it 2003.11539 §3.2.
• The regularisation strength and solver are not given in the paper text 2003.11539 §4.1.
• For distribution calibration, replace the augmentation step with sampling from the calibrated Gaussians of each support feature, and fit the same logistic regression on support plus samples 2101.06395 §3.2.3.
• For LP-FT, run the same probe on the full training set, then pass the fitted head to the fine-tuning loop as its starting weights 2202.10054 §4.2.
• For a language-initialised head, build each class column from the averaged prompt embeddings of its class name, and start the bias at zero 2204.08790 §4.2. The ELEVATER card records that the released code also normalises each prompt embedding before averaging, which the paper does not describe.

Pitfalls

• Which layer to probe. The paper's method uses the pooled penultimate features 2003.11539 §3.2. The released evaluation code can also use the classifier's logits, which changes the feature size to the number of base classes. The paper does not clearly say which one its reported numbers use. Check the card before reproducing.
• Projector choice changes the probe. In the one paper that varies only the projector, the SimCLR probe ranges from 51.5 with none to 66.5 with a two-layer projector 2110.09348 tab. 1. Name the projector with any probe number on a contrastive model.
• Sub-vector length is not in the text. For DirectCLR, the length d0 used for the main result is not stated in the paper text, and its sweep is only in a figure 2110.09348 app. E.
• Depth matters for transfer. In the greedy layer-wise study, the Caltech-101 probe scores about 23 points higher at layer 8 than at layer 2 [arxiv:1812.11446 tab. 8]. A probe on the wrong layer can mislead about a model's features.
• Normalise support and query alike. The paper L2-normalises both sides 2003.11539 §4.5, and dropping normalisation costs about 2.8 points at 1-shot 2003.11539 tab. 4.
• Augmentation is a small effect. Support augmentation changes results by at most about half a point, and sometimes lowers them (FC100 1-shot 42.77 to 42.59) 2003.11539 tab. 4. It multiplies the support-set feature extraction by five 2003.11539 §4.5.
• Reporting protocol. The paper reports medians over three runs of 1000 episodes (ImageNet-derived sets) or 3000 (CIFAR-derived sets) 2003.11539 §4.2 §4.3. Compare numbers only under the same episode count.
• Where the covariance's α goes. The distribution-calibration equation is typeset with a plus α, which can be read as adding α to the diagonal only. The text says α is added to every entry 2101.06395 §4.6, so the calibrated covariance is not diagonal.
• Mixed transforms. The paper does not say whether base statistics use Tukey-transformed features, while support and query are transformed 2101.06395 §3.2.2. The released code uses untransformed base features, so check the card before reproducing.
• Non-negative inputs only. The Tukey transform with λ = 0.5 applied to negative features gives NaN. Post-ReLU pooled features are fine 2101.06395 §4.1.3.
• Probes can learn the task. A probe with enough capacity and data can learn the target label itself, so a good probe score does not show what the model encodes. Control tasks and information-theoretic measures such as minimum description length are the proposed fixes 2108.04840 §14.2.
• Probe recipes differ between SSL papers. MAE's probe uses LARS and a large batch 2111.06377 app. A.1. MSN's probe uses SGD with Nesterov momentum at learning rate 6.4 for 100 epochs 2204.07141 app. A.3. DINO's probe uses SGD at batch 1024 for 100 epochs, with the learning rate swept 2104.14294 app. F. Compare SSL numbers only under the same recipe. The MSN card also flags a batch-size mismatch between paper and code for linear evaluation 2204.07141 app. A.3.
• Attentive is not linear. The CAE card reports an attentive probe beside its linear one, and it is about 6 points higher on ViT-B [arxiv:2202.03026 tab. 1]. The attentive head has a cross-attention module, so a number from one probe does not stand for the other 2202.03026 §5.2. Check which head a table reports before comparing.
• Survey probe numbers mix protocols. The survey's ImageNet table joins rows from many original papers and gives no per-method probe settings 2011.00362 tab. 1. Use it to find candidates, then check the cited original before quoting a number.
• Steering strength is in units of sigma. ITI's edit is alpha times sigma times a unit vector 2306.03341 eq. 2. Alpha 20 means 20 standard deviations along the direction app. B, and sigma is computed per model. Run a random direction at the same alpha as a control 2306.03341 app. B tab. 3: it leaves LLaMA-7B nearly unchanged.
• Unstated ImageNet details. For DirectPred on ImageNet, the learning rate, weight decay, EMA value and LARS settings are not stated in the paper. It says they follow BYOL, but BYOL's values are not given there 2102.06810 §4.
• Probe regularisation on ImageNet. The LP-FT paper does not sweep the probe's L2 strength on ImageNet 2202.10054 app. B.
• Best test epoch in the ELEVATER toolkit. The ELEVATER card reports that the image-classification fine-tuning script keeps the best epoch measured on the test split 2204.08790 §4.1. The card does not confirm this for the linear-probing rows. Check before comparing against them, or report a fixed-epoch number.
• Few-shot sizes before the split. The ELEVATER card reports that its code applies N before an 80/20 train and validation split, so the search trains on about 80% of N. Its image-classification script also raises N = 1 to 2 2204.08790 §3.3 §4.1. Name the split when you report few-shot results.
• Probe on a frozen image backbone. The ELEVATER paper defines linear probing as training only the classifier, with the image backbone frozen [arxiv:2204.08790 §3.3]. Its object-detection linear probe also updates the projection matrices §4.2, so the frozen-backbone reading does not apply there.

Contradictions

• Feature normalisation. The first card L2-normalises support and query features and reports that dropping it costs about 2.8 points at 1-shot (61.56 to 58.74) 2003.11539 tab. 4. The second card applies a Tukey power transform instead and states no normalisation; its code applies none beyond that transform 2101.06395 §4.1.3. The papers do not test each other's preprocessing, so the two recommendations are not reconciled here.

• Numbers across cards. The first card reports a plain probe at 58.74 (1-shot, ResNet-12) and the second a plain logistic regression at 56.37 (1-shot, WRN28 with a rotation loss) 2003.11539 tab. 4 2101.06395 tab. 4. The protocols also differ: the first uses medians of three runs of 1000 episodes §4.2, the second averages 10,000 tasks §4.1.2. Compare across the two cards only with that caveat.

• Source of the probe's training data. The first card's probe sees only the support set. The second's sees generated features, which depend on base-class statistics. Results from the two are not a like-for-like test of the probe.

• Benchmarks and protocols. The third card's eight-target average is a different test from the 5-way few-shot numbers of the first two cards. Its values are not on the same scale, and no card tests another's probe 1610.02055 tab. 3 2003.11539 tab. 1. The ELEVATER 5-shot averages over 20 image datasets with CLIP ViT-B/32 are not comparable to either of these 2204.08790 tab. 2.

• Linear probe of MAE. The MAE card reports 68.0 for ViT-B on ImageNet-1K 2111.06377 app. B. ELEVATER's MAE linear probe averages 68.70 over 20 image datasets 2204.08790 tab. 5. Both are linear probes, but on different data, so the two numbers do not contradict each other.

• Is the probe a fair readout for masked pretraining? The MAE paper calls linear probing weak for its features and argues that fine-tuning shows them better 2111.06377 §4.3. The MSN paper measures MAE on a frozen-feature low-label protocol, where MAE ViT-B/16 scores 8.2 at one label per class against 41.8 for DINO ViT-B/16 2204.07141 tab. 1. Both are consistent with the same features being weak under a probe. The two papers measure different things, so their numbers are not interchangeable.

• MAE's linear-probe number across cards. The MAE card reports 68.0 for ViT-B and 75.8 for ViT-L on ImageNet-1K 2111.06377 app. B tab. 3. The CAE paper's table gives MAE 67.8 and 76.0 for the same two models 2202.03026 tab. 1. The two tables come from different papers, so the differences are not resolved here.

• Probing versus fine-tuning out of distribution. LP-FT reports that full fine-tuning does worse than probing on average over ten shifts [arxiv:2202.10054 tab. 2]. The MAE card reports the opposite ordering for ImageNet-1K, where fine-tuning is far stronger than probing 2111.06377 tab. 3. These are different tests: one is distribution shift, the other in-distribution accuracy.

• Probing versus fine-tuning with few labels. With a random head, ELEVATER reports 29.75 for 5-shot fine-tuning against 58.09 for probing, which agrees with LP-FT's account of random heads distorting features. With a language-initialised head, fine-tuning is 63.29 against 65.35 at 5 shots, and 72.19 against 71.69 at 20 shots 2204.08790 tab. 2. LP-FT does not test few-shot sizes, and ELEVATER does not test distribution shift, so neither paper settles the other's question.

• Probing as a measure of ability. The diagnostic-probe card uses probes to show where information is decodable 2603.17839 §2.6. The survey warns that a probe may learn the task, so decodability does not establish that the model uses the information 2108.04840 §14.2. The two papers do not test each other's use. The ITI card reads a gap of about 40 points between probe accuracy and generation accuracy on LLaMA-7B as evidence that the model represents the truth without expressing it 2306.03341 §1. That reading is the one the survey's warning targets, and the ITI card reports no control-task test.

• SimCLR's linear probe across sources. The survey lists SimCLR at 69.3 top-1 with ResNet-50 2011.00362 tab. 1. DirectCLR reports SimCLR at 66.5 with a two-layer projector and 100 epochs 2110.09348 tab. 1. The survey attaches no setting to its SimCLR row, so the two numbers may come from different schedules and are not a like-for-like test.

The first card's own text and its released code differ on which features are probed, as noted under Pitfalls.

Papers

• Tian et al., 2020 (arXiv:2003.11539): a supervised ResNet-12 embedding with a per-episode L2-normalised logistic-regression probe, compared against meta-learning baselines on miniImageNet, tieredImageNet, CIFAR-FS, FC100 and Meta-Dataset, plus self-supervised MoCo and CMC embeddings.
• Yang et al., 2021 (arXiv:2101.06395): training-free distribution calibration of support features from base-class statistics, followed by logistic regression on support plus 750 sampled features per class, on a frozen WRN28 backbone.
• Zhou et al., 2016 (arXiv:1610.02055): the Places scene dataset, with AlexNet, GoogLeNet and VGG scene CNNs whose frozen features are scored with a linear SVM on eight target datasets.
• Kumar et al., 2022 (arXiv:2202.10054): fine-tuning can distort pretrained features and underperform out of distribution; LP-FT, which probes first and then fine-tunes, is evaluated on ten shifts.
• Tian et al., 2021 (arXiv:2102.06810): analysis of why non-contrastive SSL avoids collapse, with DirectPred, which sets the linear predictor from the input correlation matrix instead of training it; linear-probe top-1 is the reported metric.
• He et al., 2021 (arXiv:2111.06377): masked autoencoders for ViT pretraining, with linear-probe and fine-tuning results on ImageNet-1K and transfer tasks.
• Chen et al., 2022 (arXiv:2202.03026): context autoencoder, which encodes only visible patches and regresses the hidden patches' representations; linear and attentive probes on ImageNet-1K are the probe results here.
• Assran et al., 2022 (arXiv:2204.07141): Masked Siamese Networks for label-efficient ViT pretraining, with linear evaluation at full labels and frozen-feature low-label protocols at one, two and five labels per class.
• Kumaran et al., 2026 (arXiv:2603.17839): how Gemma 3 27B computes verbal confidence, using per-layer, per-position linear probes on activations alongside causal interventions.
• Madsen et al., 2021 (arXiv:2108.04840): a survey of post-hoc interpretability for neural NLP, which covers probing, including the critique that probes can learn the task.
• Belilovsky et al., 2018 (arXiv:1812.11446): greedy layer-wise CNN training with shallow auxiliary classifiers, scaled to ImageNet; its linear separability and frozen-feature Caltech-101 transfer studies are the probe uses here.
• Li et al., 2022 (arXiv:2204.08790): the ELEVATER benchmark and toolkit for language-augmented visual models, with linear probing whose head is initialised from the text encoder.
• Caron et al., 2021 (arXiv:2104.14294): DINO, label-free self-distillation for ViTs, evaluated with a linear probe and a frozen weighted k-NN classifier on ImageNet.
• Li et al., 2023 (arXiv:2306.03341): Inference-Time Intervention, which fits a probe per attention head on TruthfulQA and shifts the top-K heads along a truthful direction during decoding.
• Jing et al., 2021 (arXiv:2110.09348): dimensional collapse in contrastive SSL, with DirectCLR, which removes the projector and applies the loss to a fixed sub-vector; ImageNet linear-probe top-1 is the reported metric.
• Jaiswal et al., 2020 (arXiv:2011.00362): a survey of contrastive self-supervised learning in vision and NLP, whose tables collect linear-probe and fine-tuning results from the original papers on ImageNet, Places, VOC and Kinetics.