Transfer learning

Overview

Transfer learning starts from a network whose weights were learned on a source task, often a large dataset, and adapts it to a target task with fewer labelled examples. What every variant shares is the starting point: the target model is not trained from random weights. The variants differ in how much of the source network is kept fixed, what is added on top, and whether the source is reused through weights, through shared parameters or through a merge.

The cards on this page show these routes:

• a fixed backbone whose features feed a linear classifier or an SVM 1610.02055 §3.3 1706.06480 §4.3.2, or gradient-boosted trees over features pooled across augmented crops 1802.00752 §2.5;
• several pretrained networks, each trained on a different source task, whose features are merged with a baseline model's features and classified by an SVM 1610.08815 §5.2;
• a fixed image encoder whose frozen per-pixel features feed a linear dense head 2011.05499 §4.1;
• a backbone with its last layer removed, a new head added, and the network trained on the target set 2009.08369 §4.2;
• an encoder-decoder segmentation network whose encoder starts from ImageNet weights while its decoder starts from random values 1801.05746 §3;
• a segmentation network fine-tuned in stages, each starting from the weights of the previous one 1706.06480 §4.3.3;
• a linear probe first, then fine-tuning of all layers from that head 2202.10054 §4.2;
• a frozen backbone whose last layer is retrained with an extra "others" logit per group of classes 2105.01879 §3.1;
• a pretrained network with a new 20-way head, trained with the head alone and then with all layers 2506.17455 §4.1;
• a classifier head for an image-text model initialised from the text encoder 2204.08790 §4.2;
• a frozen image model with small trainable adapters inside each block, for video 2206.13559 §3.2, or an image ViT with added temporal modules 2211.09552 §3.1;
• a frozen pretrained transformer with only its input, output and normalisation parameters trained 2103.05247 §2.2;
• a sparse subset of the weights, chosen from gradients, is tuned 2211.15583 §4;
• a weighted average of parameters from models that share an initialisation 2111.09832 §2.4, or an interpolation between zero-shot and fine-tuned weights 2208.05592 §2;
• joint training on a labelled source language and a small target language, with a language vector 2404.09383 §3.1;
• contrastive pre-training on unlabelled ECG, where positives are segments of the same patient 2005.13249 §4.2;
• pre-training on a large unlabelled corpus, then fine-tuning on each downstream task 1912.05372 §3.2;
• a SMILES transformer pre-trained on unlabelled molecule strings, then fine-tuned per property 2209.01712 §3;
• a transformer pre-trained on unlabelled EEG with a contrastive masked-vector task, then fine-tuned on five BCI and sleep tasks 2101.12037 §2.4;
• a frozen DINOv2 encoder that supplies coarse features to a dense matcher whose fine encoder and decoder are trained at their own learning rates 2305.15404 §3.2 2305.15404 §4.2;
• one sequence model trained on many games, then fine-tuned on held-out games 2205.15241 §4.5;
• a new column per task, with lateral inputs from frozen earlier columns 1606.04671 §2;
• a convolutional trunk kept fixed while the fully connected layers are fine-tuned with an alignment penalty, across users 1907.09173 §3.4;
• a 2D image checkpoint inflated to 3D video weights 2106.13230 §3.4;
• a video backbone trained end to end on sampled fragments of high-resolution frames, initialised from Kinetics-400 2207.02595 §3.1 2207.02595 §4.1;
• one policy co-trained on pooled datasets from many robots 2310.08864 §4.3;
• a network trained on labelled source data, then fine-tuned on unlabelled target data with a penalty that pulls the two feature distributions together 1804.07265 §III-A;
• further training on in-domain text before fine-tuning on a task 2103.11943 p. 3;
• a layered library's recipe: freeze or group the layers, fit a head, then train with a separate learning rate per group 2002.04688 §4.1;
• pre-training on a cleaned ImageNet-21K with one softmax per WordNet depth, then transfer to other image tasks 2104.10972 §3.3;
• a frozen 7B language model, with only its input and output layers trained, reprogrammed to forecast time series 2310.01728 §3;
• the last layers of a pruned and quantized network reset and retrained on a microcontroller, with 8-bit tensors 2407.10734 §4.1.

Thirteen cards are surveys or reviews and report no experiments of their own. They are used for guidance and taxonomy only: 1601.06615 §2.6, 2310.10196 §3, 2108.07258 §4.3, 2104.05314 p. 8–9, 1807.11809 §4.6, 1709.00786 §4.2.3, 2003.04960 §3.2, 1802.07569 §1, 1810.03548 §1, 2205.06743 §4, 2103.11943 p. 1, 2303.13569 §IV-B and 2103.01415 §4. Three cards are benchmarks or datasets: one compares pretrained CNNs without naming a single winner 2605.11989 §5, another is a benchmark for language-augmented visual models 2204.08790 §3, and the third introduces a labelled underwater image set and fine-tunes 13 networks on it 2506.17455 §3.1 2506.17455 §3.2.

Scope. All cards are under CC BY, CC BY-SA or CC0 licences. Each result comes from one paper, usually one dataset and one split, and the numbers are author-reported. Treat them as evidence for a direction, not as a general ranking of methods. Backbone rankings do not carry over between papers, because each one uses its own datasets, metrics and training budget.

Two broader points come from the reviews. A survey of 40 agricultural deep-learning studies found that most studies using popular architectures applied transfer learning 1807.11809 §4.6, but only 8 of 40 tested on data different from their training data 1807.11809 §4.10. The review of machine learning and deep learning lists transfer learning as a practical challenge, and names backdoors in pretrained models as a risk, reported from other work 2104.05314 p. 8–9. The cards do not measure either risk.

Transfer learning also has a risk on the data side. In a transfer victim the feature extractor is frozen and only a linear head is fitted, on data that an attacker can partly influence. A clean-label poison keeps its correct label, and the convex polytope attack places such poisons so that a target image is classed with them. The authors use 5 poisons among 500 fine-tuning images, which is 1% 1905.05897 §4. They report success rates at or near 50% for substitute-trained poisons, and say feature collision never exceeds 50% in the same transfer setting 1905.05897 §4.1. Against end-to-end victims, the last-layer version of the attack almost never succeeds, and a multi-layer variant is needed 1905.05897 §4.3. The per-victim rates appear only in a figure, so the values are estimates 1905.05897 fig. 5. Card: Transferable Clean-Label Poisoning Attacks on Deep Neural Nets.

Variants

Frozen features with a linear classifier

Keep the source network fixed, take one of its internal layers as a feature vector, and fit a simple classifier on those features. The Places paper uses the fully connected layer fc7 for AlexNet and VGG, and the globally pooled layer for GoogLeNet, with a linear SVM at C = 1 1610.02055 §3.3. The steel paper uses the layer before the classifier of a pretrained VGG19 with an RBF-kernel SVM, and does not state C or gamma 1706.06480 §4.3.2. Cards: Places: An Image Database for Deep Scene Understanding, Advanced Steel Microstructural Classification by Deep Learning Methods.

The source domain matters for this variant. Features from a scene-trained network scored higher than ImageNet features on scene benchmarks, and lower on object benchmarks 1610.02055 tab. 3. A network trained on both the ImageNet classes and the Places classes had the best average over the eight benchmarks in that comparison 1610.02055 §3.3 1610.02055 tab. 3.

The survey gives the same basic advice in general terms. Activations of a trained network can serve as generic features, and for fine-tuning one should start from the trained weights, use a much lower learning rate than the original, and re-learn fewer layers when the new task is closer to the original and data are scarcer 1601.06615 §2.6.1 1601.06615 §2.6.2. The survey gives no values for the learning rate or the number of layers.

A few-shot survey describes the same baseline: pre-train a backbone on a large labelled dataset, freeze most of its parameters, and train only the classification layer on the few labelled target examples. It cites works where fine-tuning improves 5-way 1-shot accuracy by 2% to 7% over this baseline, but gives no table for that gain 2205.06743 §4.1. Card: A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities.

Rakhlin et al. keep three ImageNet networks frozen (ResNet-50, InceptionV3 and VGG-16) and classify breast histology slides with gradient-boosted trees instead of a linear model. Each slide gets 50 colour-augmented copies, and 20 crops are taken from each copy. The crop features are pooled with a power mean into one vector per copy 1802.00752 §2.3. The blend of models reaches 87.2 ± 2.6% four-class accuracy under 10-fold cross-validation, against 84.2 ± 4.2% for the best single encoder 1802.00752 tab. 1. All numbers come from that one dataset of 400 images 1802.00752 §2.1. The authors say fine-tuning did not work well for them, but give no numbers [arxiv:1802.00752 §2.2]. Card: Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis.

The same frozen-backbone idea serves out-of-distribution (OOD) detection. MOS retrains only the last fully connected layer of an ImageNet-1k BiT-S-R101x1 model. The 1,000 classes are split into 8 super-classes, and each group gets an "others" logit 2105.01879 §3.1 2105.01879 §4.2. The score is the negated smallest "others" probability across groups [arxiv:2105.01879 §3.2]. The backbone is pretrained on ImageNet-1k, which is also the in-distribution set 2105.01879 §4.1. On four curated OOD sets, the paper reports an average FPR95 of 39.97%, against 54.30% for KL matching and 76.96% for maximum softmax probability. Each figure is from a single run with no variance 2105.01879 tab. 1. Card: MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space.

Pre-trained feature extractors from several source tasks

Train small networks on other labelled tasks, use each one as a feature extractor on the target inputs, and merge their outputs with the target model's own features. The sarcasm paper uses three sources: sentiment on SemEval 2014 tweets, emotion on blog sentences, and personality on essays, with one binary network per trait. Their fully connected layers give 100, 150 and 750 features 1610.08815 §4.2 1610.08815 §4.3 1610.08815 §4.4. A baseline sarcasm CNN adds 100 more, and an SVM is fitted to the 1,100 concatenated values 1610.08815 §5.2 1610.08815 §5.3. The paper does not say whether the extractors' weights are updated during sarcasm training, so frozen and fine-tuned extraction are both open readings 1610.08815 §4.2.

Within each dataset the gains are small. On Dataset 1 the all-feature SVM reaches 97.71% F1 against 97.60% for the baseline features alone, and on Dataset 3 it reaches 93.30% against 92.20% 1610.08815 tab. 2. The pre-trained features without the baseline reach 84.43% on Dataset 3 1610.08815 tab. 2. The cross-dataset result is larger. Trained on Dataset 3 and tested on Dataset 1, the model reaches 76.78%, against 53.02% for the authors' re-implementation of Ptacek et al. 1610.08815 tab. 3. The reverse direction falls to 33.05% 1610.08815 §5.6. Card: A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks.

Replace the head and fine-tune

Load an ImageNet-pretrained network, remove its last classification layer, attach a new head and train on the target set. The face-mask paper uses InceptionV3 with a head of average pooling, a dense ReLU layer, dropout and a two-way softmax 2009.08369 §4.2. The card does not say which backbone layers were frozen or fine-tuned. Its target set is small by design, because access to real faces is restricted 2009.08369 §4.

The steel paper also fine-tunes an ImageNet-pretrained VGG16 on single objects, resized to 224 x 224 px, with SGD at momentum 0.9. Its learning rate is stated as 0.001 in one sentence and as 0.0001 in another, so the value is open 1706.06480 §4.3.1. Cards: Face Mask Detection using Transfer Learning of InceptionV3, Advanced Steel Microstructural Classification by Deep Learning Methods.

The DeepWeeds baselines take the same route on field photos of weeds. An ImageNet-pretrained ResNet-50 or Inception-v3 loses its 1000-way head, which is replaced by nine sigmoid units, one per class, so that an image can score for more than one species 1810.05726 § Deep Learning. Training uses Adam at learning rate 1e-4 with binary cross-entropy and batches of 32, and the rate is halved after 16 epochs without a validation improvement 1810.05726 § Deep Learning. The paper does not state whether layers are frozen. The authors' code trains all layers. Averaged over five folds, the weighted top-1 accuracy is 95.7% for ResNet-50 and 95.1% for Inception-v3 1810.05726 tab. 2. Card: DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning.

A benchmark of eleven ImageNet CNNs adapted to five image datasets compares the same recipe under two modes: only the classifier is retrained, or every layer is retrained. It trains with Adam, batch size 10 and a learning-rate grid of 1e-2, 1e-3 and 1e-4, and keeps the weights from the best validation epoch 2605.11989 §4. Its scores are accuracy divided by parameter count, and it reports rankings from figures only 2605.11989 §5.1. Card: A Transfer Learning Evaluation of Deep Neural Networks for Image Classification.

A rice-variety study fine-tunes ten ImageNet CNNs with Adam at learning rate 1e-4, batch size 32 and 25 epochs 2609.10524 tab. 3. The four best reach 97.5% to 99.2% validation accuracy 2609.10524 tab. 4. Their outputs are then combined by a meta-learner 2609.10524 §4.4. Card: Precision in Rice Variety Classification using Stacking-Based Ensemble Learning.

A mammography study moves the same recipe from a patch to a whole image. A patch classifier is first trained with ImageNet weights on lesion-centred patches, which beats random initialisation on the harder set (ResNet50: 0.89 against 0.63 accuracy) 1708.09427 §3.1.3 1708.09427 tab. 1. Its convolutional layers are then reused on whole images, with two new convolutional blocks and global average pooling on top 1708.09427 §2.2. The network is fine-tuned on INbreast, a digital dataset, with Adam at learning rate 1e-5 for 200 epochs and weight decay 0.01 1708.09427 §3.2.2. On the INbreast validation split, each transferred model reaches AUC 0.95 and the ensemble 0.98 1708.09427 tab. 4 1708.09427 §3.2.2. The paper reports no baseline trained on INbreast alone, so these numbers do not show what the transfer adds. Card: Deep Learning to Improve Breast Cancer Early Detection on Screening Mammography.

An ImageNet-pretrained AlexNet is fine-tuned on 650 photos of smartphone backs in 12 classes. Its last three layers are replaced, and SGD with momentum at learning rate 3e-4 reaches 98.35 ± 2.02% accuracy over seven trials, against 95.26 ± 3.04% for Adam and 93.20 ± 3.86% for RMSProp 2606.23286 tab. 1. These are validation numbers: the paper describes no separate test set, and it is not stated whether augmentation was applied before the split 2606.23286 §4, p. 5. Card: Transfer learning-based method for automated ewaste recycling in smart cities.

A fine-tuning benchmark follows the same two-step recipe. Thirteen ImageNet-pretrained networks, from SqueezeNet to ConvNeXt and ViT-B/16, are fine-tuned on 8,171 underwater images in 20 classes 2506.17455 §3.2. Each classifier is replaced by a 20-way output layer. Training runs in two stages: first only the new head, then all layers 2506.17455 §4.1. The paper reports that most models peak after the first stage, and that only EfficientNetB0 improves in the second 2506.17455 §4.1. The best model, ConvNeXt, reaches 90.69% top-1 accuracy and 88.92% macro-F1 on the test split 2506.17455 tab. 3. The paper reports no repeats or spread 2506.17455 §4. The text gives a 32×32 input size, which is not the 224×224 used in the released notebook, so the setting behind the numbers is unclear 2506.17455 §4.1. Card: AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions.

Fine-tune with settings chosen by source-target similarity

The deep transfer survey argues against one recipe for every target. It orders source types from most to least preferred: a very similar and much larger labelled source first, and supervised training on a less related target alone last 2205.09904 §4.2. Its one experiment extends Big Transfer (BiT) fine-tuning to less similar targets, with ResNet-50x1 pretrained on ImageNet-21k 2205.09904 §3.4 2205.09904 tab. 3. The recommended BiT settings are best or near best on Cifar10, Cifar100 and Oxford Pets. On Cars and Aircraft, a learning rate of 0.02 raises the mean accuracy from 81.8 to 87.6, and from 82.4 to 87.3. Changing only the decay leaves both near 81% 2205.09904 tab. 3. The survey calls the extension to less related targets a hypothesis 2205.09904 §3.4.

The same paper's similarity measures (earth mover's distance, Wasserstein and Fréchet distance) place Cifar10 and Cifar100 closest to the source, and Cars and Aircraft furthest 2205.09904 tab. 2. The table caption names ImageNet-1k as the source, while the text names ImageNet-21k for the same measures 2205.09904 §3.4. The source is therefore unclear.

The survey also discusses penalties that keep weights near their pretrained values. L2-SP pulls weights toward the pretrained values [arxiv:2205.09904 §6.2.1], and distillation-based penalties compare feature maps with the source network 2205.09904 §6.2.2. It presents the guidance on when such penalties help as prose, not as measured results 2205.09904 §7.3 2205.09904 §7.7. A meta-learning survey notes that transfer from a large pretrained image model can fail when the target task is not similar 1810.03548 §4.1. Cards: Deep transfer learning for image classification: a survey, Meta-Learning: A Survey.

Freeze the body, fit a head, then unfreeze with per-group learning rates

The fastai library packages this recipe in a few calls. Its learner splits the model into parameter groups, which can be frozen or given their own learning rates 2002.04688 §4.1. The Pets example replaces the head of an ImageNet-pretrained ResNet-34 and runs one-cycle training for four epochs, with batch size 64 and no learning rate stated 2002.04688 §2.1. The paper reports that result as close to the state of the art, without a number 2002.04688 §2.1. It has no baseline and no variance, so the example is a recipe rather than evidence of a gain. The authors single out batch-norm handling. In their tests it changed the error rate by up to 300% across configurations, and they argue for keeping batch-norm layers trainable with their statistics still updating 2002.04688 §4.1. Card: fastai: A Layered API for Deep Learning.

Frozen coarse encoder inside a trained matcher

Keep one large encoder frozen and train the rest of the network, each part with its own learning rate. RoMa takes the patch tokens of a frozen DINOv2 ViT-L/14 as coarse features, projects them to 512 dimensions, and trains a Gaussian-process encoder and a Transformer decoder on top [arxiv:2305.15404 §3.2] 2305.15404 §3.3. A VGG19 pyramid supplies the fine features and is trained. The paper gives a learning rate of 1e-4 for the decoder and 5e-6 for the encoders, at batch size 8 2305.15404 §4.2. The paper keeps DINOv2 frozen throughout training 2305.15404 §3.2, and the cards do not report a run that fine-tunes it inside RoMa.

The frozen-feature probe in the paper compares backbones with a linear layer and a kernel nearest-neighbour matcher. DINOv2 gives an endpoint error of 27.1 on MegaDepth, against 60.2 for ResNet-50 and 87.6 for VGG19 [arxiv:2305.15404 tab. 1]. That probe does not train the matcher, so it is not evidence for the full system. On WxBS, the full RoMa system scores 80.1 mAA@10px, against 58.9 for DKM 2305.15404 tab. 4. This result also includes the trained parts, so it does not isolate the effect of the frozen backbone. Card: RoMa: Robust Dense Feature Matching.

Pretrained encoder in a segmentation network

Keep the encoder of a U-Net-style network, initialise it from an ImageNet classifier, and train the decoder from scratch. TernausNet uses a VGG11 encoder. On 30 validation images of one aerial building dataset, after 100 epochs, validation IoU is 0.686 with the pretrained encoder and 0.593 with LeCun-uniform initialisation for every layer, a difference of 0.093 1801.05746 §3. The paper runs one trial per setting and reports no seeds or variance, so this is a single measurement. A network pretrained on Carvana reaches 0.687, but the paper does not state the size, epochs or loss of that pretraining 1801.05746 §3. Card: TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation.

A video saliency network follows the same pattern. Its static network takes the first five convolution blocks of an ImageNet-trained VGG-16 and trains the rest from random values. Its dynamic network also starts from VGG-16, except for its first convolution, which is Gaussian-initialised because its input has seven channels 1702.00871 §3.2 1702.00871 §5.1. The ablation table does not isolate the ImageNet start, so its gain is not measured 1702.00871 tab. 2. Card: Video Salient Object Detection via Fully Convolutional Networks.

Linear probe, then fine-tune (LP-FT)

Train only a new head on frozen features first, then fine-tune all parameters starting from that head. The paper calls this LP-FT. The reason given is that fine-tuning from a random head changes the features that the training data touches, which hurts out-of-distribution (OOD) accuracy 2202.10054 §3.2. Starting the fine-tuning from the probed head is the paper's remedy 2202.10054 §3.4 2202.10054 §4.2. In the paper's experiments, the learning rate and the early-stopping point are chosen on in-distribution validation data 2202.10054 §4.1.

The effect depends on two conditions. It appears when the pretrained features are good and the shift is large. With a weaker MoCo-v1 checkpoint, fine-tuning beat linear probing on the OOD set (71% against 66%) 2202.10054 §4.3, and on a small shift (CIFAR-10 to CIFAR-10.1) fine-tuning also beat linear probing OOD 2202.10054 §4.3. Card: Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution.

Text-initialised head for image-text models

An image-text model such as CLIP classifies by comparing an image embedding with the embeddings of prompts that name each class. The usual adaptation replaces this with a linear head that starts from random weights. ELEVATER instead starts the linear head from the text encoder's class vectors, with a zero bias, and reports this as the better choice across settings 2204.08790 §4.2 2204.08790 §5.1.

Averaged over 20 image-classification datasets, CLIP ViT-B/32 with a linear probe at 5 shots reaches 58.09% with a random head and 65.35% with the text-initialised head (Separate-Head) 2204.08790 tab. 2. Fine-tuning at 5 shots gives 29.75% with a random head and 63.29% with the text-initialised head 2204.08790 tab. 2. The authors report that with the initialised head, few-shot results are above zero-shot results, and that fine-tuning overtakes linear probing from about 20 shots 2204.08790 §5.1 2204.08790 tab. 2. A knowledge-augmented CLIP, with WordNet and Wiktionary text added to the prompts, gains 0.48 points over CLIP at 5 shots with a linear probe 2204.08790 tab. 4. The same paper reports object detection as well, but its code covers classification only, so the detection numbers are not checked against code 2204.08790 §4.1. Card: ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models.

Frozen backbone with small trainable adapters

Keep the pretrained network frozen and insert a small bottleneck module into each block. The module down-projects, applies a depth-wise 3D convolution over neighbouring frames, and up-projects back, with a residual connection 2206.13559 §3.2. Only the adapters and a linear classifier are trained 2206.13559 §4.1. The paper uses this to turn an image ViT into a video recogniser. Card: ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning.

The paper's numbers depend on the backbone. With CLIP ViT-B/16 on Kinetics-400, the adapter reached 82.0% top-1 against 81.7% for full fine-tuning, with 7.20M trained parameters against 121.57M 2206.13559 tab. 1. With ImageNet-21K pretraining on the same dataset it reached 76.6%, against 78.0% for full fine-tuning 2206.13559 tab. 1. The authors also report that with a Swin-B backbone it underperforms full fine-tuning on ImageNet-21K pretraining 2206.13559 tab. 10.

A related route keeps the image ViT's blocks and adds video modules to them. UniFormerV2 puts a temporal relation module in front of each pretrained block and adds a query-based cross-attention module on the last layers, so video training starts from image weights without a separate video pretraining stage 2211.09552 §3.1 2211.09552 §3.2. On Kinetics-400 with CLIP-ViT-B/16 and 8 x 4 x 3 testing, the full model reaches 84.4% top-1 2211.09552 tab. 9. The paper's best headline, 90.0% with ViT-L/14 at 336 px after Kinetics-710 post-pretraining, uses 64 frames and far more compute 2211.09552 tab. 1. Card: UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer.

Freeze the blocks, train input, output and normalisation

A pretrained transformer can be reused on a different kind of data while its attention and feed-forward blocks stay frozen. Only a new input layer, a new output layer, the layer-norm parameters and the position embeddings are trained 2103.05247 §2.2. The source model is a GPT-2 base trained on natural language, and the targets are bit strings, MNIST and CIFAR-10 patches, ListOps and protein sequences 2103.05247 §2.1. Card: Pretrained Transformers as Universal Computation Engines.

The layer-norm parameters carry much of the effect. On CIFAR-10, training only the layer norms gives 54% and training only the output layer gives 25% 2103.05247 tab. 15. Freezing the layer norms in the full recipe drops CIFAR-10 from 68.2% to 61.5% 2103.05247 tab. 16. The frozen model reaches 100% on bit memory and 98.0% on MNIST, against 99.1% for a fully trained transformer on MNIST 2103.05247 tab. 1. Its gains over random initialisation are large on bit memory, where a randomly initialised frozen model reaches 75.8% 2103.05247 tab. 2. The paper reports these results as author-measured, and the released code trains more parameters than the text states (see the card's Paper vs code section).

Sparse fine-tuning with a chosen subset

Several parameter-efficient methods, including random masks, rule-based masks (BitFit, magnitude pruning) and adapters, can be written as sparse fine-tuning: only a fixed subset of weights changes 2211.15583 §2.2. The open question is which subset to pick. The SAM method ranks each weight by the square of its gradient, summed over a few burn-in batches at the pre-trained weights, keeps the top fraction, and freezes the rest 2211.15583 §4 2211.15583 §5.1. Card: On the Effectiveness of Parameter-Efficient Fine-Tuning.

On seven GLUE and SuperGLUE tasks with RoBERTa-base at a tunable fraction of 0.5%, SAM averages 77.12 ± 1.51, against 74.78 ± 2.60 for full fine-tuning and 76.03 ± 2.35 for an adapter 2211.15583 tab. 1. The paper links stability and performance with a rank correlation of 0.4356 2211.15583 §5.2. In its sparsity sweep, the mean and standard deviation on most tasks fall as the tunable fraction falls, until a threshold below which performance drops sharply and training error rises 2211.15583 §5.2 2211.15583 fig. 5. Masking does not reduce compute here: SAM averages 2.02 hours of training, against 1.67 for full fine-tuning 2211.15583 tab. 3.

Merge or interpolate weights

Average the parameters of models that start from the same initialisation, but weight each parameter by its diagonal Fisher information instead of taking a plain mean 2111.09832 §2.4 eq. 4. The paper uses the merge for ensembles of fine-tuned checkpoints, for WiSE-FT robust fine-tuning, and for moving skills from a donor task to a target task 2111.09832 §3.1 2111.09832 §3.2 2111.09832 §3.3. The Fisher is estimated from training examples, and the authors say its cost is about that of training on the same number of examples 2111.09832 §2.3. Card: Merging Models with Fisher-Weighted Averaging.

For a donor merge into a GLUE target, the paper compares Fisher merging with gradient-based intermediate-task training, and the authors estimate the merge at about six times less compute than fine-tuning 2111.09832 §3.3. The exception matters: with an MNLI donor on RTE, the gradient-trained model scores 77.4 ± 1.6 against 73.2 ± 5.1 for the merge 2111.09832 tab. A1 2111.09832 tab. A3.

PAINT is a different interpolation: it fine-tunes a zero-shot CLIP encoder on one patch task and then mixes the two sets of weights with one coefficient alpha in [0, 1] 2208.05592 §2. The text head is frozen, so the interpolation touches only the visual encoder 2208.05592 §3. With ViT-L/14, the authors report gains of 15 to 60 points on nine patch tasks, and ImageNet accuracy changes by less than one point 2208.05592 §1 2208.05592 §4. Joint patching of all nine tasks reaches a combined accuracy gain of 15.8 points 2208.05592 §5. Card: Patching open-vocabulary models by interpolating weights.

PAINT's interpolation also helps sequential patching. With ViT-B/32, sequential patching without interpolation lost 4.6 points of combined accuracy, and with interpolation it gained 11 points 2208.05592 §5. Forgetting is reduced but not removed 2208.05592 §5.

Joint training with a related high-resource language

Train one model on the labelled target data and the labelled data of a related source language at the same time. The two languages share the character encoder, the sentence BiLSTM, the tag layers and the transition weights. Each language keeps its own word embeddings and a learned language vector 2404.09383 §3.1. The objective is the target log-likelihood plus a weighted source log-likelihood, and the weight is a trade-off parameter that the paper does not give a value for 2404.09383 §3.1 eq. 7. Card: Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields.

With 100 labelled target sentences, the transfer setting raised F1 over the monolingual log-linear CRF in every listed pair, up to 9.80 points for Galician with French as source 2404.09383 tab. 2. Without a source, the neural model was below the log-linear baseline on Galician, 49.19 against 57.64 F1 2404.09383 tab. 2. With 10,000 target sentences the transfer gains were small, and positive in every row 2404.09383 tab. 2. The table reports no seeds or variance 2404.09383 tab. 2.

Contrastive and self-supervised pre-training, then reuse

Pre-train an encoder on unlabelled data with a contrastive or self-supervised loss, then use it frozen for linear evaluation or as the starting point for fine-tuning.

For ECG, the CLOCS family defines positives as segments or leads that share a patient, not only the same instance 2005.13249 §4.2 [arxiv:2005.13249 §4.1]. On Chapman, with 4 leads, an embedding size of 128 and half the labels, the best variant (CMSC) reached a linear AUC of 0.896, against 0.738 for SimCLR 2005.13249 tab. 1. For fine-tuning from PhysioNet 2020 to Cardiology, the best value was 0.714 for CMSC, against 0.658 for SimCLR and 0.678 for random initialisation 2005.13249 tab. 2. These gains are reported in AUC points, not percent of the baseline 2005.13249 tab. 1. Card: CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients.

For dense prediction, VADeR learns one embedding per pixel. Two augmented views are matched pixel to pixel using the known view geometry, and a contrastive loss is applied per pixel 2011.05499 §3.1 2011.05499 §3.2. The encoder starts from MoCo v2 and the decoder from random weights 2011.05499 §3.2. With frozen features and a linear 1 x 1 convolution on top, it reaches 56.7% mIoU on VOC against 43.0% for MoCo and 54.4% for supervised ImageNet 2011.05499 tab. 1. On NYU depth it reaches an RMSE of 0.964, against 1.136 for MoCo and 0.994 for supervised ImageNet (lower is better) 2011.05499 tab. 1. With 2% of the VOC labels in fine-tuning it reaches 47.7% against 44.7% for MoCo 2011.05499 app. A.3. The gains are not uniform: at 100% of the NYU labels, MoCo's RMSE is 0.640 and VADeR's 0.641 2011.05499 app. A.3. Card: Unsupervised Learning of Dense Visual Representations.

For EEG, BENDR pre-trains a convolutional encoder and an 8-layer transformer on 60-second windows of unlabelled recordings. The loss asks for the true masked vector among 20 distractors 2101.12037 §2.4.1 [arxiv:2101.12037 eq. 2]. The pre-training corpus is about 1.5 TB of EEG before preprocessing 2101.12037 §2.1. For each downstream task the paper compares six configurations: a new softmax on the start token, average pooling of the sequence with a new classifier, and variants that keep the first stage fixed. Two of the six, configurations 3 and 5, use random initialisation as the baseline 2101.12037 §2.4.2. The pooled classifier was the best configuration on four of five datasets, and the no-pre-training full model was generally ineffective, except on sleep staging 2101.12037 §3.2. The paper's own numbers are in 2101.12037 tab. 2. The official code's linear-head setting also freezes the input projection and position convolution, which the paper does not say, so an implementation should not assume the paper's freezing scheme. See the card's Paper vs code section. Card: BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data.

Pre-training on a large corpus, then task fine-tuning

FlauBERT is a BERT-style encoder pre-trained with masked language modelling on a 71 GB French corpus, in a base and a large size 1912.05372 §3.1 1912.05372 §3.2. Each downstream task is then fine-tuned separately. For text classification, the paper grid-searches four learning rates over 30 epochs with batch size 16 1912.05372 §5.1. Some tasks need no fine-tuning at all: verb sense disambiguation scores each sense by the cosine similarity between a test embedding and the mean embedding of that sense's examples 1912.05372 §5.6. Card: FlauBERT: Unsupervised Language Model Pre-training for French.

The same recipe appears for speech. The FLEURS benchmark fine-tunes two 600M pretrained speech models on 102 languages, and reports a character error rate of 14.1% for the speech-only model and 14.6% for the speech-and-text model 2205.12446 tab. 3. Languages are split into seen and unseen groups by whether the pretraining data included their speech 2205.12446 §3.2. The per-group seen and unseen breakdown is not reproduced here, because the source commented it out of the results table. Card: FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech.

XLS-R applies the same pattern to speech in 128 languages. A wav2vec 2.0 model is pretrained on about 436K hours of unlabelled audio, in 0.3B, 1B and 2B parameter sizes 2111.09296 §3.1 2111.09296 tab. 2. Recognition then adds a linear CTC head 2111.09296 §4.3, and translation adds a decoder initialised from mBART 2111.09296 §4.2. On the 14-language VoxPopuli set, the 0.3B model averages 12.8% word error against 37.5% with no pretraining 2111.09296 tab. 10. Across 10 CommonVoice languages with one hour of labels each, the 1B model averages 5.1% phoneme error, against 7.6% for XLSR-53 2111.09296 tab. 8. On CoVoST-2 into English, the low-resource group rises from 5.1 BLEU at 0.3B to 11.7 at 1B and 15.1 at 2B 2111.09296 tab. 3. Card: XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale.

The same pattern applies to molecules. ChemBERTa-2 pre-trains a RoBERTa-style encoder on SMILES strings from 5M, 10M or 77M compounds 2209.01712 §3. It compares two unlabelled objectives: masked-token prediction, and regression on about 200 RDKit descriptors computed from each string [arxiv:2209.01712 §3.1]. Each MoleculeNet task is then fine-tuned on a scaffold split with an 80/10/10 ratio 2209.01712 §3.3. The paper reports that the descriptor objective usually fine-tunes better 2209.01712 §4.1. At 77M compounds, it reaches 0.728 ROC-AUC on BBBP against 0.698 for masked-token pretraining, and 0.817 on SR-p53 against 0.749 2209.01712 tab. 1. Table 1 gives one value per cell and no spread 2209.01712 tab. 1. Card: ChemBERTa-2: Towards Chemical Foundation Models.

Pre-train on a cleaned ImageNet-21K, one softmax per hierarchy depth

ImageNet-21K has about 21,000 classes, but each image carries one label at any level of the WordNet hierarchy. The authors drop rare classes, hold out 50 validation images per class and resize every image to 224 px. The result, which they call ImageNet-21K-P, has 11,221 classes 2104.10972 §2.1. They then train one softmax per WordNet depth. Each image is trained at every depth where its ancestor labels exist, so the same photo gives a target at several levels 2104.10972 §3.3. Against single-label and multi-label pretraining on the same backbone, this semantic softmax scores higher on six of seven downstream sets. Multi-label is better on CIFAR-100 2104.10972 tab. 2. Against ImageNet-1K pretraining, the paper's pretraining is higher in all 30 model-dataset cells it tests 2104.10972 tab. 4. Against the official ImageNet-21K weights for ViT-B-16 and Mixer-B-16, it is higher in all 14 cells 2104.10972 tab. 3. The paper also adds a semantic distillation term that improves every column of its semantic-softmax results [arxiv:2104.10972 tab. 2]. The authors' public release does not include that training code, so those distillation numbers cannot be reproduced from it. The cost is high: from ImageNet-1K weights, 80 epochs took about 54 hours on eight V100 GPUs 2104.10972 §3.1. Card: ImageNet-21K Pretraining for the Masses.

Further pre-training on in-domain text, then fine-tuning

For text, the BERT review describes retraining the pretrained model on text from the target domain before fine-tuning. It names three kinds of data: an intra-task dataset, an intra-subject corpus and a cross-subject corpus. It reports that retraining helps, and that intra-subject data helps more than cross-subject data 2103.11943 p. 3. The learning rate of the cited retraining study is about 1e-5, and 4e-4 did not converge there. Further training can also cause catastrophic forgetting, which a low learning rate reduces 2103.11943 p. 3.

Domain models follow the same idea in two ways. BioBERT continues training on PubMed and PMC text, then fine-tunes on biomedical tasks [arxiv:2103.11943 p. 10]. SciBERT follows the BERT training method on a scientific corpus, with a vocabulary built for that corpus, because the general and scientific vocabularies overlap by only 42% 2103.11943 p. 10–11. In the review's Table 1, averaged over its tasks, fine-tuned SciBERT scores 79.27 and fine-tuned BERT-Base 77.16. The review takes these numbers from the SciBERT paper, and they are averages over several random seeds 2103.11943 tab. 1. Card: BERT: A Review of Applications in Natural Language Processing and Understanding.

One model trained on many tasks, then fine-tuned on a new one

Train one sequence model on offline trajectories from 41 Atari games, then fine-tune it on five games it has not seen. The paper uses a decoder-only transformer over image patches, return, action and reward tokens 2205.15241 §3.1 2205.15241 §3.2. Pre-training runs for 10M steps 2205.15241 §4.1. Fine-tuning uses 1% of a held-out game's data, about 500,000 transitions, for 100k steps, and the paper reports the pre-trained model best on every held-out game against a CQL model trained from scratch on the same 1% 2205.15241 §4.5 2205.15241 app. B.2. The per-game values are in a figure only 2205.15241 fig. 6. Card: Multi-Game Decision Transformers.

The headline score is 126% of human-normalised IQM across the 41 training games, with the model size not named in the summary sentence 2205.15241 §1 2205.15241 fig. 1. A separate five-seed run of the 40M-parameter model gives IQM 0.95 with standard deviation 0.05 2205.15241 app. B.1. Gains over more than one order of magnitude of model size are reported for this method, and not for CQL 2205.15241 §4.4.

A survey of curriculum learning for reinforcement learning treats transfer as one of three parts of a curriculum, next to choosing intermediate tasks and ordering them. Transfer carries policies, value functions or other knowledge from one task to the next. The survey runs no experiments of its own, and it classifies published methods by the knowledge they transfer [arxiv:2003.04960 §3.2] 2003.04960 §4.3. Card: Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey.

Robot policies follow a related route, but by co-training rather than by fine-tuning in sequence. Open X-Embodiment pools 60 robot datasets from 22 robot types in one format, and trains RT-1-X on that mixture alone 2310.08864 §3.1 2310.08864 §4.3. Against each dataset's own original method, RT-1-X is better on 4 of 5 small-data domains, although the per-dataset numbers appear only in a figure 2310.08864 §5.1 2310.08864 fig. 3. The large domains go the other way: on Bridge with the IRIS setup, RT-1 trained on that dataset alone scores 40% and RT-1-X 27% 2310.08864 tab. 1. In the RT-2 ablation, the 55B model rises from 27.3% emergent-skill success with Google Robot data only to 75.8% with the pooled robotics data, and both rows are co-trained with web data 2310.08864 tab. 2. Card: Open X-Embodiment: Robotic Learning Datasets and RT-X Models.

Add a column per task and freeze the old ones

Keep the trained network, and add a new network, a column, for each new task. Earlier columns are frozen. Each new column reads their hidden activations through lateral connections, so the earlier tasks are not overwritten 1606.04671 §2 eq. 1. In practice a learned scalar scales these lateral inputs, which pass through a small adapter 1606.04671 §2 eq. 2. The experiments use A3C agents on Pong variants, Atari games and 3D mazes. Against full finetuning, two columns reach a mean transfer score of 209% on Pong Soup against 181%, and 491% on Labyrinth against 235%. On Atari the mean is 132% against 133%, and the medians are 112% and 110% 1606.04671 tab. 1. On Labyrinth the medians are close (115% against 112%), so that mean gap is mostly outliers 1606.04671 tab. 1 1606.04671 §5.2. The cost is size: parameters grow quadratically with the number of columns, and at test time the caller must choose the column 1606.04671 app. 2 1606.04671 §2. Card: Progressive Neural Networks.

A lifelong-learning review files progressive networks under dynamic architectures, next to regularisation methods and replay [arxiv:1802.07569 §3.2] 1802.07569 §3.3 1802.07569 §3.4. The review's comparisons are qualitative and come from cited work 1802.07569 §3.5, so the numbers above still come from the progressive-network paper. Card: Continual Lifelong Learning with Neural Networks: A Review.

Freeze the convolution, fine-tune the fully connected layers with an alignment penalty

FedHealth builds a cloud CNN from user updates. Each user's model is uploaded in encrypted form, and the server updates the cloud model from them 1907.09173 §3.3. A user then freezes the convolution and pooling layers and fine-tunes the fully connected layers 1907.09173 §3.4. The user objective is the classification loss plus η times a CORAL term, which compares second-order statistics of the cloud model and the user data, with η = 0.01 1907.09173 eq. 6 1907.09173 §4.2. On five isolated UCI Smartphone subjects, mean accuracy is 99.4% against 94.1% for the cloud model with no transfer 1907.09173 tab. 2. The five runs are averaged with no variance reported 1907.09173 §4.2. The paper does not name the encryption scheme or the server's aggregation rule 1907.09173 §3.3, so the federated steps cannot be rebuilt from the text, and the layer count is given as three in one section and two in another 1907.09173 §3.4 1907.09173 §4.2. Card: FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare.

Fine-tune with a feature-distribution penalty on unlabelled target data

The deep transfer network starts from a CNN trained on labelled source data. It then fine-tunes on unlabelled target data with the cross-entropy loss plus a weighted joint distribution adaptation (JDA) term. The JDA term is a maximum mean discrepancy (MMD) between source and target features, plus one MMD per class, which uses target pseudo-labels from the current network 1804.07265 §III-A 1804.07265 eq. 10 1804.07265 eq. 11. The SGD learning rate is 0.01, and the weight is searched over a grid 1804.07265 §IV-A. The batch size and the number of iterations are not stated 1804.07265 §III-C 1804.07265 alg. 1. On ten transfer tasks over three fault data sets, the average accuracy is 97.5% with JDA, 87.4% with the marginal term alone, and 80.0% for the plain CNN 1804.07265 tab. III. These are the authors' numbers. The paper does not say whether the weight was chosen on labelled target data 1804.07265 §IV-A. Card: Deep Transfer Network with Joint Distribution Adaptation: A New Intelligent Fault Diagnosis Framework for Industry Application.

Inflate a 2D checkpoint to 3D

Video Swin Transformer starts from ImageNet Swin weights and inflates them in two places 2106.13230 §3.4. The patch-embedding kernel is repeated over the two temporal slots and halved, which keeps the output mean and variance. The relative bias table is repeated over the temporal offsets, so it is the same in every frame. The released backbone implements only this duplicate path swin_transformer.py. The pretraining source matters: Swin-B reaches 82.7% top-1 on Kinetics-400 from ImageNet-21K, against 80.6% from ImageNet-1K, both with 4 × 3 views 2106.13230 tab. 1. The ablations differ by little. A centre initialisation of the patch embedding gives 78.8% top-1, the same as the inflated one with Swin-T 2106.13230 tab. 8. For fine-tuning, a backbone learning rate of 0.1 times the head's gave 82.6% against 82.0% at 1.0 times, with ImageNet-21K pretraining 2106.13230 tab. 7. The SSv2 model starts from a Kinetics-400 model rather than from images 2106.13230 §4.1. Card: Video Swin Transformer.

Fine-tune end to end on sampled fragments

Video quality models often freeze a pretrained backbone and train only a small regressor on its features. FAST-VQA instead trains a Video Swin-T end to end, initialised from Kinetics-400, and makes this affordable by changing the input. Each frame is cut into a 7 × 7 grid. One 32 × 32 patch is taken from every cell at full resolution, and the patches are spliced into a smaller canvas. The same cell offsets are reused in every frame 2207.02595 §3.1 2207.02595 tab. 2 2207.02595 §4.1.

The input cost does not depend on the frame size. It is 279 GFLOPs at 540P, 720P and 1080P, against 11,852 GFLOPs at 1080P for the full-resolution Swin-T baseline, which uses fixed features 2207.02595 tab. 4. On LSVQ 1080p, the PLCC is 0.814, against 0.753 for that baseline 2207.02595 tab. 3. The sampling choice matters on the same set. Bilinear resizing gives 0.786 PLCC, and random mini-patches without a grid give 0.790 [arxiv:2207.02595 tab. 6]. In the transfer table, each of five small datasets is split at random ten times, and FAST-VQA is ahead of the same-features baseline on all five in SRCC. For example, it reaches 0.849 against 0.799 on LIVE-VQC and 0.891 against 0.841 on KoNViD-1k 2207.02595 tab. 5.

The paper does not state the optimiser, learning rate, schedule, epoch count or augmentation, so an implementer has to choose them 2207.02595 §4.1. Card: FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling.

Freeze a language model and reprogram its inputs for a new modality

Time-LLM keeps a Llama-7B backbone frozen and trains only the layers around it 2310.01728 §3. Each channel is normalised, cut into patches, and each patch attends to learned text-prototype vectors that are a linear mix of the model's word embeddings 2310.01728 §3.1. A text prompt with dataset context, a task instruction and input statistics goes in front of the patches, and the outputs at the patch positions are projected to the forecast horizon 2310.01728 §3.1. The trainable reprogramming network has 6.39 million parameters at ETTh1 with horizon 96, against 3,404.53 million in the 32-layer backbone 2310.01728 §4.5. On long-term benchmarks the paper reports the lowest MSE on seven of eight datasets 2310.01728 §4.1. In few-shot forecasting on the first 10% of training steps, its ETTh1 MSE at horizon 96 is 0.556, against 0.590 for GPT4TS 2310.01728 §4.3. Removing the patch reprogramming raised average MSE by 9.2%, and removing the input statistics raised it by 10.2% 2310.01728 §4.5. The paper has no randomly initialised backbone of the same size, so it cannot separate the effect of pretraining from the architecture 2310.01728 §4.5. The released scripts do not match the paper's configuration table, so check the card before reproducing a number. Card: Time-LLM.

Retrain the last layers on a microcontroller

Prune and quantize an ImageNet-pretrained network, deploy it to a board, reset its last five layers to random values and retrain them on the device. The paper's transfer setup runs 20 epochs at learning rate 0.001 with batch size 48 2407.10734 §4.1. The backbone is pruned with an iterative L1-norm schedule whose rates and final sparsity are not stated 2407.10734 §4.1. Training runs in 8-bit on the device, with the same quantization scheme as inference, and the weight update is applied in floating point before the tensor is requantized 2407.10734 §3.1.

On seven datasets, uint8 retraining matched or exceeded the GPU baseline on the simpler sets. It was lower on flowers, CIFAR-100 and speech. The mixed setting, which keeps the classification head in float32, matched the float baseline 2407.10734 §4.1 fig. 4. The values are in the figure, and the card does not transcribe them. The authors used no data augmentation on the device, and they report more overfitting than the GPU baseline on speech and flowers 2407.10734 §4.1.

A survey of TinyML reports TinyTL, a memory-saving fine-tuning method for an edge device, with a 6.5 times lower memory overhead than full fine-tuning and a moderate accuracy loss 2303.13569 §III-C. The survey gives no setup for that figure, and TinyTL has no card on this site, so the figure is unverified here. The same survey lists transfer learning for model design as an open lesson for TinyML 2303.13569 §IV-B. Cards: On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers, TinyML: Tools, Applications, Challenges, and Future Research Directions.

When to use which

The foundation-model report names three factors for choosing an adaptation procedure: the compute budget, how much task-specific data exists, and how much access to the model's gradients is available 2108.07258 §4.3. The table uses them.

Implementation sketch

The steps below are the common core of the variants. Names are ours. The papers state most values but leave several open, and these are marked.

# Frozen-feature transfer: fixed backbone, linear classifier  [arxiv:1610.02055 §3.3]
# Input:  pretrained backbone; labelled target images
# Output: classifier on backbone features

def fit_frozen_feature_classifier(backbone, images, labels):
    # Backbone weights stay fixed; only the classifier is fitted  [§3.3]
    feats = [backbone.embed(img) for img in images]
    # Linear SVM with C = 1, as in the Places paper  [§3.3]
    return linear_svm(feats, labels, C=1.0)

# Replace-the-head transfer  [arxiv:2009.08369 §4.2] [fig. 4]
# Input:  backbone with ImageNet weights; num_classes = 2 in the paper
# Output: classifier producing one probability per target class

def build_transfer_classifier(backbone, num_classes):
    # Drop the source model's final classification layer  [§4.2]
    features = backbone.without_last_layer()
    # New head, applied in this order  [§4.2]
    pooled = average_pool(features, size=(5, 5))
    flat = flatten(pooled)
    hidden = dense(flat, units=128, activation="relu")
    hidden = dropout(hidden, rate=0.5)
    # Which backbone layers stay frozen is not stated  [§4.2]
    return dense(hidden, units=num_classes, activation="softmax")

def train_transfer_classifier(model, images, labels):
    # Fixed schedule: 80 epochs of 42 steps each  [§4.2]
    for epoch in range(80):
        for step in range(42):
            # Optimizer, learning rate and batch size are not stated  [§4.2]
            batch = sample_batch(images, labels)
            loss = classification_loss(model, batch)
            update_weights(model, loss)
    return model

# Linear probe, then fine-tune all layers (LP-FT)  [arxiv:2202.10054 §4.2]
# Input:  pretrained network g; labelled target train and validation sets
# Output: fine-tuned network

def lp_then_ft(network, train_set, val_set):
    # Stage 1: freeze the features and fit only the head  [§4.1]
    feats = network.features(train_set.inputs)
    head = fit_logistic_regression(feats, train_set.labels)
    # Choose the regularisation strength on in-distribution validation  [§4.1]
    head = select_by_val_accuracy(head, feats, val_set)
    # Stage 2: start all layers from the probed head  [§4.2]
    # The fine-tuning learning rate is chosen on validation too  [§4.1]
    network.head = head
    return fine_tune_all_parameters(network, train_set)

# Text-initialised head for an image-text model  [arxiv:2204.08790 §4.2] [tab. 2]
# Input:  class names (K), prompt templates, text encoder g
# Output: linear head with weight (P, K) and zero bias

def init_language_head(class_names, templates, text_encoder):
    # One column per class, built from its prompt embeddings  [§4.2]
    columns = []
    for name in class_names:
        # Encode each prompt that contains the class name  [§4.2]
        prompt_vectors = normalise(text_encoder(make_prompts(name, templates)))
        # Average the normalised prompt embeddings, then renormalise  [§4.2]
        columns.append(normalise(mean(prompt_vectors, axis=0)))
    # Bias starts at zero  [§4.2]
    return stack(columns, axis=1), zeros(len(class_names))

# Staged fine-tuning of a segmentation network  [arxiv:1706.06480 §4.3.3]
# Momentum 0.9, weight decay 5e-4, batch size 1 in every stage
# Iterations: 7000 is stated once; the per-stage count is not stated

def train_staged_fcn(patches, fcn32_imagenet):
    # Stage 1: FCN-32s, learning rate 1e-10  [§4.3.3]
    fcn32 = train(fcn32_imagenet, skips=[], learning_rate=1e-10)
    # Stage 2: add skip layer A, fine-tune from stage 1, rate 1e-11  [§4.3.3]
    fcn16 = train(fcn32, skips=["A"], learning_rate=1e-11)
    # Stage 3: add skip layer B, fine-tune from stage 2, rate 3e-12  [§4.3.3]
    return train(fcn16, skips=["A", "B"], learning_rate=3e-12)

# Interpolation after one fine-tune  [arxiv:2208.05592 §2]
# Input:  zero-shot encoder weights, fine-tuned encoder weights, alpha in [0, 1]
# Output: patched encoder weights

def interpolate(zeroshot, finetuned, alpha):
    # Choose alpha on held-out validation data of both tasks  [§2]
    # Mix every tensor with the same coefficient  [§2]
    return {name: (1 - alpha) * zeroshot[name] + alpha * finetuned[name]
            for name in zeroshot}

# Sparse mask from burn-in gradients (SAM)  [arxiv:2211.15583 §4] [eq. 10]
# Input:  pre-trained weights theta0, tunable fraction p, burn-in batches
# Output: binary mask with floor(m * p) ones

def select_sam_mask(theta0, batches, p, burn_in_steps):
    # Sum the loss gradient at theta0 over the burn-in batches  [§5.1]
    grad_sum = zeros_like(theta0)
    for batch in batches[:burn_in_steps]:
        grad_sum = grad_sum + gradient(theta0, batch)
    # Rank entries by squared gradient; the curvature term is dropped  [eq. 10]
    scores = grad_sum ** 2
    # Keep the top floor(m * p) entries; the rest stay fixed  [§4]
    mask = zeros_like(theta0)
    mask[top_k_indices(scores, floor(size(theta0) * p))] = 1
    return mask

# Fine-tuning with a JDA penalty on unlabelled target data  [arxiv:1804.07265 alg. 1]
# Input:  network trained on labelled source; unlabelled target inputs; weight lam
# Output: adapted network and target pseudo-labels

def adapt_with_jda(network, source, target_x, lam):
    # Pseudo-labels from the source-trained network  [alg. 1 line 3]
    target_labels = predict(network, target_x)
    # The paper does not define the convergence test  [§III-A]
    while not converged(target_labels):
        f_s = features(network, source.x)
        f_t = features(network, target_x)
        # Marginal MMD, then one MMD per class on pseudo-labels  [eq. 10]
        penalty = mmd(f_s, f_t)
        for c in classes(source.y):
            penalty = penalty + mmd(f_s[source.y == c], f_t[target_labels == c])
        # Source cross-entropy plus lam times the penalty  [eq. 11]
        network = sgd_step(network, cross_entropy(network, source) + lam * penalty)
        # Re-predict the target pseudo-labels  [alg. 1 line 8]
        target_labels = predict(network, target_x)
    return network, target_labels

Contradictions

Disagreements between papers, or between a paper's text and its own tables, with each side cited. Where two results come from different models or budgets, the bullet says so and does not call them a contradiction.

• FlauBERT is not best on every task. On PAWS-X, CamemBERT scores 90.14%, above FlauBERT-base (89.49%) and FlauBERT-large (89.34%) [arxiv:1912.05372 tab. 4]. On XNLI, XLM-R large scores 85.2% against 83.4% for FlauBERT-large 1912.05372 tab. 5. These two scores are the authors' reported numbers.

• The base model is not uniformly weaker than the large one. On constituency parsing, FlauBERT-base scores 89.05% test F1 and FlauBERT-large 88.63% 1912.05372 tab. 6. On verb sense disambiguation, FlauBERT-base scores 43.92% and mBERT 49.83% 1912.05372 tab. 8, while FlauBERT-base beats mBERT on dependency parsing 1912.05372 tab. 7.

• Learning rate for the object-level VGG16. The steel paper gives 0.001 in one sentence and 0.0001 in another 1706.06480 §4.3.1. The validation split used to choose it is not described.

• Face-mask counts. The text gives a 470-image test set, but the test confusion matrix totals 388 2009.08369 §3 2009.08369 fig. 5. The abstract gives 99.9% training accuracy and Table 1 gives 99.92% 2009.08369 tab. 1.

• LP-FT on OOD data. The text says LP-FT is best on all ten OOD datasets 2202.10054 §4.2. In the table, linear probing scores 63.2% on Entity-30 and LP-FT 62.3% 2202.10054 tab. 2, so the claim does not hold on that dataset. The gap is within the stated intervals.

• Feature-change ratio. The introduction gives "10 to 100 times less" feature change for LP-FT, while the tables give about 17 times for Living-17 and about 1000 times for DomainNet 2202.10054 §1 2202.10054 app. B.

• Fisher merge against gradient training. The paper reports merging as a cheaper alternative to intermediate-task training 2111.09832 §3.3. For an MNLI donor on RTE, sequential gradient training beats the merge on BERT-base 2111.09832 tab. A1 2111.09832 tab. A3. The Fisher-merge standard deviation on that pair is 5.1 2111.09832 tab. A1.

• Adapter against full fine-tuning. On Kinetics-400 with CLIP weights, the adapter is 0.3 points above full fine-tuning 2206.13559 tab. 1. With ImageNet-21K weights, it is 1.4 points below 2206.13559 tab. 1. The paper reports both, and does not treat the second as a failure of the method. On GLUE and SuperGLUE tasks with RoBERTa-base, an adapter averaged 76.03 against 74.78 for full fine-tuning 2211.15583 tab. 1. These are different models, tasks and budgets, so the two results do not settle the question between them.

• Random sparse masks against full tuning. The SAM paper's text says even the random mask beats full tuning 2211.15583 §5.2. Its own averages say otherwise: 74.55 for random and 74.78 for full tuning 2211.15583 tab. 1.

• CLOCS percentage claims. The paper says CMSC beats SimCLR by 15.8% in linear AUC on Chapman 2005.13249 §6.2. The table difference is 0.896 − 0.738 = 0.158 AUC points, not a relative gain 2005.13249 tab. 1. The paper's claim that all three CLOCS variants beat SimCLR in every linear setting does not hold for one PhysioNet 2020 cell 2005.13249 app. D.1.

• Speech baselines. The text says most gains from multimodal pre-training appear in three regions 2205.12446 §4.1. The table shows the speech- and-text model ahead in one region only 2205.12446 tab. 3.

• VADeR and Mask R-CNN. The text says VADeR consistently outperforms MoCo and the supervised baseline on Mask R-CNN 2011.05499 §4.3. Table 4 has supervised ImageNet at 59.8 AP50 for boxes against VADeR's 59.7 2011.05499 tab. 4. The detection experiments also use a different FPN from the one the paper describes 2011.05499 §4.3.

• One alpha against per-model optima. PAINT chooses alpha on validation data for each patch task 2208.05592 §2. Its appendix tables peak near 0.5 to 0.6 for ViT-B/32 and 0.65 to 0.70 for ViT-L/14 [arxiv:2208.05592 app. O]. A value copied from one backbone may not suit another.

• Stacked outputs and the 100% result. The rice paper reports XGBoost at 100% on the TOP (1,2,3,4) combination 2609.10524 tab. 6. The paper does not say which split that table uses, and the base models were selected on their own performance 2609.10524 §5. Its own text describes the meta-model as trained on base-model outputs, and its Figure 3 describes stacked feature sets 2609.10524 §4.4 2609.10524 fig. 3.

• Multi-game fine-tuning size. The paper says 1% of the held-out data is "1M steps", while the appendix gives about 500,000 transitions 2205.15241 §4.5 2205.15241 app. B.2. The appendix is consistent with a 50M-step dataset.

• Fault-diagnosis text against its own table. The text says DTN with JDA stays above 95% on all ten tasks 1804.07265 §IV-B. The table gives 94.3% for F→E 1804.07265 tab. III.

• DeepWeeds augmentation, paper against code. The paper describes random perspective transforms and an intensity shift of ±25 on all channels. The released code uses shear and a channel shift, with no perspective transform and no separate intensity shift 1810.05726 § Deep Learning. A rerun with the code therefore does not follow the paper's augmentation.

• Which layers stay frozen for ewaste classification. The paper says the network is frozen "for all layers except the last three" and also that the learnable weights are frozen "in the fully connected layer" [arxiv:2606.23286 §4, p. 5]. The two statements conflict, and the card gives no resolution.

• Positive transfer against a single-dataset model. The paper's headline is positive transfer from pooled robot data 2310.08864 §1. For large domains it says RT-1-X underfits and does not beat RT-1 trained on the dataset alone 2310.08864 §5.1, and the Bridge rows of its table show the same thing 2310.08864 tab. 1. Both results are reported.

• English-only against multilingual at low label counts. The XLS-R paper reports gains across languages, but on LibriSpeech with 10 minutes or 1 hour of labels, the English-only wav2vec 2.0 model at 0.3B beats XLS-R at 0.3B: test clean 32.1% against 34.1% at 10 minutes, and 13.7% against 16.8% at 1 hour 2111.09296 tab. 11. XLS-R at 1B is slightly better than the English-only model at 1 hour (13.1%) 2111.09296 tab. 11.

• Layer count in FedHealth. Section 3.4 describes two fully connected layers, with the alignment term replacing the second 1907.09173 §3.4. Section 4.2 describes three 1907.09173 §4.2. The position of the alignment term therefore depends on which section is followed.

• Best rate for Cars. The caption of the survey's BiT table gives 0.02 as the best rate for both less similar targets. The table lists 0.025, with decay every 5 steps at 0.96, as the best rate for Cars 2205.09904 tab. 3.

• Sarcasm: text against its own table. Section 5.5 calls the sentiment model the best pre-trained extractor on Dataset 3. Table 2 gives personality 74.41% and sentiment 73.50% there 1610.08815 §5.5 [arxiv:1610.08815 tab. 2]. Section 5.4 gives 87.00% for sentiment on Dataset 2, while Table 2 gives 86.50% for Dataset 2 and 87.00% for Dataset 1 1610.08815 §5.4 1610.08815 tab. 2.

• Does ImageNet pretraining help plant images? One weed study reports that ImageNet weights still reduce the number of training iterations for weed classes. Another argues that a CNN initialised without plant images would not work well, and builds a new network 2103.01415 §10.1.1 2103.01415 §10.1.2. The survey does not settle the question.

• Fine-tuning against frozen features in medical images. Rakhlin et al. say fine-tuning a pretrained network did not work well for them, with no numbers 1802.00752 §2.2. The steel paper reports a fine-tuned VGG16 at 66.50%, above a frozen VGG19 feature extractor with an SVM at 64.84% 1706.06480 tab. 1. The two studies use different images, networks and baselines, so they do not settle the question.

• Checkpoint choice in the benchmark. The AQUA20 paper keeps "the checkpoints that achieved the highest performance during evaluation" 2506.17455 §4.1. The released notebook keeps the lowest test loss and stops on it, so its test scores are not from data that the selection never saw (AQUA20 card, Pitfalls).

Papers

• A Taxonomy of Deep Convolutional Neural Nets for Computer Vision: a survey of convolutional networks and their fine-tuning advice, with no experiments.
• Progressive Neural Networks: one column per task, with frozen earlier columns and lateral inputs, tested with A3C agents on Pong, Atari and mazes.
• Places: An Image Database for Deep Scene Understanding: a scene dataset, and generic features from scene-trained networks compared with ImageNet features on eight benchmarks.
• A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks: sentiment, emotion and personality CNNs used as feature extractors and merged with a baseline sarcasm CNN, on three Twitter datasets.
• Video Salient Object Detection via Fully Convolutional Networks: a video saliency network trained partly on synthetic video made from annotated still images, with ImageNet VGG-16 initialisation for its static network.
• Advanced Steel Microstructural Classification by Deep Learning Methods: fine-tuned VGG16 and staged fine-tuning of a fully convolutional network for steel microstructure.
• Deep Learning to Improve Breast Cancer Early Detection on Screening Mammography: a patch classifier converted to a whole-image classifier, and transferred from film to digital mammograms.
• Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis: frozen ImageNet encoders with pooled crop features, classified by gradient-boosted trees, on 400 breast slides.
• Machine learning methods for histopathological image analysis: a mini-review of machine learning for whole-slide pathology images, with no experiments of its own.
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation: a U-Net whose VGG11 encoder starts from ImageNet weights, tested on one aerial building dataset.
• Deep Transfer Network with Joint Distribution Adaptation: A New Intelligent Fault Diagnosis Framework for Industry Application: a CNN fine-tuned on unlabelled target data with a marginal and class-wise MMD penalty, on three fault data sets.
• DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning: a field photo dataset of eight weed species, with ImageNet ResNet-50 and Inception-v3 baselines.
• Meta-Learning: A Survey: a survey that groups meta-learning by the meta-data it reuses, with transfer as one family, and no experiments of its own.
• Continual Lifelong Learning with Neural Networks: A Review: a review of catastrophic forgetting and its fixes (regularisation, dynamic architectures, replay), with no new experiments.
• Deep learning in agriculture: A survey: a review of 40 agricultural deep-learning studies, with their use of transfer learning and test data.
• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets: convex polytope clean-label poisons, crafted on substitute networks and tested on frozen and end-to-end victims.
• FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare: a federated cloud model, with frozen convolution layers and CORAL-aligned fine-tuning for each user.
• FlauBERT: Unsupervised Language Model Pre-training for French: masked-language-model pre-training on French text, evaluated on six French tasks.
• fastai: A Layered API for Deep Learning: a layered PyTorch library with one-call training, data blocks and callbacks, and no benchmark tables.
• Face Mask Detection using Transfer Learning of InceptionV3: an ImageNet-pretrained InceptionV3 with a new head, trained on augmented images of one simulated face dataset.
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey: a survey that splits curriculum learning for RL into task generation, sequencing and transfer, with no experiments.
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients: contrastive pre-training on ECG where positives share a patient, compared with SimCLR and BYOL.
• Unsupervised Learning of Dense Visual Representations: per-pixel contrastive pre-training with known view correspondences, evaluated with frozen features and fine-tuning.
• BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data: a wav2vec 2.0 style transformer pre-trained on unlabelled EEG, then fine-tuned on five public BCI and sleep tasks.
• A Survey of Deep Learning Techniques for Weed Detection from Images: a survey of 70 deep-learning weed-detection papers, with no experiments of its own.
• Pretrained Transformers as Universal Computation Engines: a frozen GPT-2 with trained input, output and normalisation layers, applied to non-text tasks.
• BERT: A Review of Applications in Natural Language Processing and Understanding: a 2021 review of BERT, its domain retraining, extractive annotation, BERTScore and adversarial attacks.
• Machine learning and deep learning: a conceptual review of machine learning and deep learning, which lists transfer learning as a practical challenge.
• ImageNet-21K Pretraining for the Masses: a cleaned ImageNet-21K with one softmax per WordNet depth, pretrained and transferred to seven downstream sets.
• MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space: a last-layer retrain with an "others" class per ImageNet group, scored for out-of-distribution inputs.
• Video Swin Transformer: a shifted-window transformer for video, initialised by inflating an image Swin checkpoint.
• On the Opportunities and Risks of Foundation Models: a report on foundation models, which frames adaptation choices by compute, task data and gradient access.
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale: wav2vec 2.0 pretrained on about 436K hours in 128 languages, then fine-tuned for speech tasks.
• Merging Models with Fisher-Weighted Averaging: a Fisher-weighted average of models that share an initialisation, for ensembling, robust fine-tuning and donor-task transfer.
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution: linear probing then fine-tuning, and when full fine-tuning loses to linear probing out of distribution.
• ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models: a benchmark of 20 classification and 35 detection datasets, and a text-initialised linear head.
• A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities: a survey of few-shot methods grouped by challenge, with copied benchmark tables and no experiments.
• Deep transfer learning for image classification: a survey: a survey that sorts transfer methods by source-target similarity and target size, with one fine-tuning experiment on five targets.
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech: a 102-language read-speech benchmark with baselines from two pretrained 600M speech models.
• Multi-Game Decision Transformers: one offline transformer for 41 Atari games, fine-tuned on five held-out games.
• ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning: adapters with a depth-wise 3D convolution inside a frozen image ViT, for video action recognition.
• FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling: a Video Swin-T trained end to end on grid mini-patch fragments, for video quality scoring.
• Patching open-vocabulary models by interpolating weights: PAINT, which interpolates between zero-shot and fine-tuned CLIP weights to patch one weak task.
• ChemBERTa-2: Towards Chemical Foundation Models: a RoBERTa-style SMILES encoder pretrained with masked tokens or descriptor regression, then fine-tuned on MoleculeNet.
• UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer: temporal and query-based modules added to a pretrained image ViT, for video recognition.
• On the Effectiveness of Parameter-Efficient Fine-Tuning: parameter-efficient methods as sparse fine-tuning, and the SAM gradient-based mask.
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models: 60 robot datasets pooled in one format, with RT-1-X and RT-2-X policies trained on them.
• TinyML: Tools, Applications, Challenges, and Future Research Directions: a survey of TinyML tools, boards, applications and open challenges, with no experiments of its own.
• RoMa: Robust Dense Feature Matching: a dense matcher with a frozen DINOv2 coarse encoder, trained fine features and a Transformer decoder.
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models: a frozen Llama-7B reprogrammed for forecasting, with trained input and output layers.
• Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook: a survey and taxonomy of large language and foundation models for temporal data, with no experiments.
• Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields: joint training of a neural CRF on a low-resource and a high-resource language, with a language vector.
• On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers: 8-bit training on three Cortex-M boards, with transfer-learning retraining of the last layers on seven datasets.
• AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions: 8,171 underwater images in 20 classes, with 13 ImageNet-pretrained networks fine-tuned on them.
• A Transfer Learning Evaluation of Deep Neural Networks for Image Classification: eleven ImageNet CNNs adapted to five image datasets under two tuning modes, compared by accuracy density.
• Transfer learning-based method for automated ewaste recycling in smart cities: an ImageNet-pretrained AlexNet fine-tuned to sort 12 smartphone classes from 6 brands.
• Precision in Rice Variety Classification using Stacking-Based Ensemble Learning: fine-tuned ImageNet CNNs combined by a stacked XGBoost meta-learner, for rice variety classification.