---
kind: concept
slug: transfer-learning
title: Transfer learning
summary: >-
  Starting from weights learned on a source task and adapting them to a target task: by training a head on frozen features, fine-tuning all layers, adding adapters or columns, merging weights, or pre-training on unlabelled data. Fifty-seven cards cover these routes. Gains are author-reported; freezing and learning-rate details are often not stated.
tags: [transfer-learning, pretrained-models, fine-tuning]
papers: ["1601.06615", "1606.04671", "1610.02055", "1610.08815", "1702.00871", "1706.06480", "1708.09427", "1709.00786", "1801.05746", "1802.00752", "1802.07569", "1804.07265", "1807.11809", "1810.03548", "1810.05726", "1905.05897", "1907.09173", "1912.05372", "2002.04688", "2003.04960", "2009.08369", "2005.13249", "2011.05499", "2101.12037", "2103.01415", "2103.05247", "2103.11943", "2104.05314", "2104.10972", "2105.01879", "2106.13230", "2108.07258", "2111.09296", "2111.09832", "2202.10054", "2204.08790", "2205.06743", "2205.09904", "2205.12446", "2205.15241", "2206.13559", "2207.02595", "2208.05592", "2209.01712", "2211.09552", "2211.15583", "2303.13569", "2305.15404", "2310.01728", "2310.08864", "2310.10196", "2404.09383", "2407.10734", "2506.17455", "2605.11989", "2606.23286", "2609.10524"]
related: [sentiment-analysis, few-shot-learning, linear-probing, domain-adaptation, data-augmentation, language-model-pretraining, parameter-efficient-fine-tuning, sparse-fine-tuning, model-merging, self-supervised-learning, contrastive-learning, named-entity-recognition, vision-transformer, vision-language-pretraining, zero-shot-learning, continual-learning, experience-replay, dynamic-network-expansion, ensemble-learning, stacked-generalization, multi-task-learning, return-conditioned-sequence-modeling, convolutional-neural-network, semantic-segmentation, self-attention, federated-learning, personalized-federated-learning, foundation-model, machine-learning, deep-learning, explainable-ai, imitation-learning, vision-language-action-model, shifted-window-attention, curriculum-learning, meta-learning, time-series-forecasting, prompt-engineering, learning-rate-schedule, knowledge-distillation, on-device-training, quantization-aware-training, weight-quantization, data-poisoning, out-of-distribution-detection, gradient-boosted-decision-trees, molecular-representation-learning, masked-language-modeling]
status: draft
updated_at: 2026-11-01
---

## Overview

Transfer learning starts from a network whose weights were learned on a source
task, often a large dataset, and adapts it to a target task with fewer labelled
examples. What every variant shares is the starting point: the target model is
not trained from random weights. The variants differ in how much of the source
network is kept fixed, what is added on top, and whether the source is reused
through weights, through shared parameters or through a merge.

The cards on this page show these routes:

- a fixed backbone whose features feed a linear classifier or an SVM
  [arxiv:1610.02055 §3.3] [arxiv:1706.06480 §4.3.2], or gradient-boosted trees
  over features pooled across augmented crops [arxiv:1802.00752 §2.5];
- several pretrained networks, each trained on a different source task, whose
  features are merged with a baseline model's features and classified by an
  SVM [arxiv:1610.08815 §5.2];
- a fixed image encoder whose frozen per-pixel features feed a linear dense
  head [arxiv:2011.05499 §4.1];
- a backbone with its last layer removed, a new head added, and the network
  trained on the target set [arxiv:2009.08369 §4.2];
- an encoder-decoder segmentation network whose encoder starts from ImageNet
  weights while its decoder starts from random values [arxiv:1801.05746 §3];
- a segmentation network fine-tuned in stages, each starting from the weights
  of the previous one [arxiv:1706.06480 §4.3.3];
- a linear probe first, then fine-tuning of all layers from that head
  [arxiv:2202.10054 §4.2];
- a frozen backbone whose last layer is retrained with an extra "others"
  logit per group of classes [arxiv:2105.01879 §3.1];
- a pretrained network with a new 20-way head, trained with the head alone and
  then with all layers [arxiv:2506.17455 §4.1];
- a classifier head for an image-text model initialised from the text
  encoder [arxiv:2204.08790 §4.2];
- a frozen image model with small trainable adapters inside each block, for
  video [arxiv:2206.13559 §3.2], or an image ViT with added temporal modules
  [arxiv:2211.09552 §3.1];
- a frozen pretrained transformer with only its input, output and
  normalisation parameters trained [arxiv:2103.05247 §2.2];
- a sparse subset of the weights, chosen from gradients, is tuned
  [arxiv:2211.15583 §4];
- a weighted average of parameters from models that share an initialisation
  [arxiv:2111.09832 §2.4], or an interpolation between zero-shot and
  fine-tuned weights [arxiv:2208.05592 §2];
- joint training on a labelled source language and a small target language,
  with a language vector [arxiv:2404.09383 §3.1];
- contrastive pre-training on unlabelled ECG, where positives are segments of
  the same patient [arxiv:2005.13249 §4.2];
- pre-training on a large unlabelled corpus, then fine-tuning on each
  downstream task [arxiv:1912.05372 §3.2];
- a SMILES transformer pre-trained on unlabelled molecule strings, then
  fine-tuned per property [arxiv:2209.01712 §3];
- a transformer pre-trained on unlabelled EEG with a contrastive masked-vector
  task, then fine-tuned on five BCI and sleep tasks [arxiv:2101.12037 §2.4];
- a frozen DINOv2 encoder that supplies coarse features to a dense matcher whose
  fine encoder and decoder are trained at their own learning rates
  [arxiv:2305.15404 §3.2] [arxiv:2305.15404 §4.2];
- one sequence model trained on many games, then fine-tuned on held-out games
  [arxiv:2205.15241 §4.5];
- a new column per task, with lateral inputs from frozen earlier columns
  [arxiv:1606.04671 §2];
- a convolutional trunk kept fixed while the fully connected layers are
  fine-tuned with an alignment penalty, across users [arxiv:1907.09173 §3.4];
- a 2D image checkpoint inflated to 3D video weights [arxiv:2106.13230 §3.4];
- a video backbone trained end to end on sampled fragments of high-resolution
  frames, initialised from Kinetics-400 [arxiv:2207.02595 §3.1] [arxiv:2207.02595 §4.1];
- one policy co-trained on pooled datasets from many robots
  [arxiv:2310.08864 §4.3];
- a network trained on labelled source data, then fine-tuned on unlabelled
  target data with a penalty that pulls the two feature distributions together
  [arxiv:1804.07265 §III-A];
- further training on in-domain text before fine-tuning on a task
  [arxiv:2103.11943 p. 3];
- a layered library's recipe: freeze or group the layers, fit a head, then
  train with a separate learning rate per group [arxiv:2002.04688 §4.1];
- pre-training on a cleaned ImageNet-21K with one softmax per WordNet depth,
  then transfer to other image tasks [arxiv:2104.10972 §3.3];
- a frozen 7B language model, with only its input and output layers trained,
  reprogrammed to forecast time series [arxiv:2310.01728 §3];
- the last layers of a pruned and quantized network reset and retrained on a
  microcontroller, with 8-bit tensors [arxiv:2407.10734 §4.1].

Thirteen cards are surveys or reviews and report no experiments of their own. They
are used for guidance and taxonomy only: [arxiv:1601.06615 §2.6],
[arxiv:2310.10196 §3], [arxiv:2108.07258 §4.3], [arxiv:2104.05314 p. 8–9],
[arxiv:1807.11809 §4.6], [arxiv:1709.00786 §4.2.3], [arxiv:2003.04960 §3.2],
[arxiv:1802.07569 §1], [arxiv:1810.03548 §1], [arxiv:2205.06743 §4],
[arxiv:2103.11943 p. 1], [arxiv:2303.13569 §IV-B] and [arxiv:2103.01415 §4]. Three cards are benchmarks or datasets: one compares pretrained CNNs
without naming a single winner [arxiv:2605.11989 §5], another is a
benchmark for language-augmented visual models [arxiv:2204.08790 §3], and the
third introduces a labelled underwater image set and fine-tunes 13 networks on
it [arxiv:2506.17455 §3.1] [arxiv:2506.17455 §3.2].

**Scope.** All cards are under CC BY, CC BY-SA or CC0 licences. Each
result comes from one paper, usually one dataset and one split, and the numbers
are author-reported. Treat them as evidence for a direction, not as a general
ranking of methods. Backbone rankings do not carry over between papers,
because each one uses its own datasets, metrics and training budget.

Two broader points come from the reviews. A survey of 40 agricultural
deep-learning studies found that most studies using popular architectures
applied transfer learning [arxiv:1807.11809 §4.6], but only 8 of 40 tested on
data different from their training data [arxiv:1807.11809 §4.10]. The review
of machine learning and deep learning lists transfer learning as a practical
challenge, and names backdoors in pretrained models as a risk, reported from
other work [arxiv:2104.05314 p. 8–9]. The cards do not measure either risk.

Transfer learning also has a risk on the data side. In a transfer victim the
feature extractor is frozen and only a linear head is fitted, on data that an
attacker can partly influence. A clean-label poison keeps its correct label,
and the convex polytope attack places such poisons so that a target image is
classed with them. The authors use 5 poisons among 500 fine-tuning images, which
is 1% [arxiv:1905.05897 §4]. They report success rates at or near 50% for
substitute-trained poisons, and say feature collision never exceeds 50% in the
same transfer setting [arxiv:1905.05897 §4.1]. Against end-to-end victims, the
last-layer version of the attack almost never succeeds, and a multi-layer
variant is needed [arxiv:1905.05897 §4.3]. The per-victim rates appear only in
a figure, so the values are estimates [arxiv:1905.05897 fig. 5]. Card:
[Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](/papers/1905.05897).

## Variants

### Frozen features with a linear classifier

Keep the source network fixed, take one of its internal layers as a feature
vector, and fit a simple classifier on those features. The Places paper uses
the fully connected layer fc7 for AlexNet and VGG, and the globally pooled
layer for GoogLeNet, with a linear SVM at C = 1 [arxiv:1610.02055 §3.3]. The
steel paper uses the layer before the classifier of a pretrained VGG19 with an
RBF-kernel SVM, and does not state C or gamma [arxiv:1706.06480 §4.3.2]. Cards:
[Places: An Image Database for Deep Scene Understanding](/papers/1610.02055),
[Advanced Steel Microstructural Classification by Deep Learning Methods](/papers/1706.06480).

The source domain matters for this variant. Features from a scene-trained
network scored higher than ImageNet features on scene benchmarks, and lower on
object benchmarks [arxiv:1610.02055 tab. 3]. A network trained on both the
ImageNet classes and the Places classes had the best average over the eight
benchmarks in that comparison [arxiv:1610.02055 §3.3] [arxiv:1610.02055 tab. 3].

The survey gives the same basic advice in general terms. Activations of a
trained network can serve as generic features, and for fine-tuning one should
start from the trained weights, use a much lower learning rate than the
original, and re-learn fewer layers when the new task is closer to the original
and data are scarcer [arxiv:1601.06615 §2.6.1] [arxiv:1601.06615 §2.6.2]. The
survey gives no values for the learning rate or the number of layers.

A few-shot survey describes the same baseline: pre-train a backbone on a large
labelled dataset, freeze most of its parameters, and train only the
classification layer on the few labelled target examples. It cites works where
fine-tuning improves 5-way 1-shot accuracy by 2% to 7% over this baseline, but
gives no table for that gain [arxiv:2205.06743 §4.1]. Card:
[A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](/papers/2205.06743).

Rakhlin et al. keep three ImageNet networks frozen (ResNet-50, InceptionV3 and
VGG-16) and classify breast histology slides with gradient-boosted trees
instead of a linear model. Each slide gets 50 colour-augmented copies, and 20
crops are taken from each copy. The crop features are pooled with a power mean
into one vector per copy [arxiv:1802.00752 §2.3]. The blend of models reaches
87.2 ± 2.6% four-class accuracy under 10-fold cross-validation, against 84.2 ±
4.2% for the best single encoder [arxiv:1802.00752 tab. 1]. All numbers come
from that one dataset of 400 images [arxiv:1802.00752 §2.1]. The authors say
fine-tuning did not work well for them, but give no numbers [arxiv:1802.00752
§2.2]. Card: [Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis](/papers/1802.00752).

The same frozen-backbone idea serves out-of-distribution (OOD) detection. MOS
retrains only the last fully connected layer of an ImageNet-1k BiT-S-R101x1
model. The 1,000 classes are split into 8 super-classes, and each group gets an
"others" logit [arxiv:2105.01879 §3.1] [arxiv:2105.01879 §4.2]. The score is
the negated smallest "others" probability across groups [arxiv:2105.01879
§3.2]. The backbone is pretrained on ImageNet-1k, which is also the
in-distribution set [arxiv:2105.01879 §4.1]. On four curated OOD sets, the
paper reports an average FPR95 of 39.97%, against 54.30% for KL matching and
76.96% for maximum softmax probability. Each figure is from a single run with
no variance [arxiv:2105.01879 tab. 1]. Card: [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](/papers/2105.01879).

### Pre-trained feature extractors from several source tasks

Train small networks on other labelled tasks, use each one as a feature
extractor on the target inputs, and merge their outputs with the target
model's own features. The sarcasm paper uses three sources: sentiment on
SemEval 2014 tweets, emotion on blog sentences, and personality on essays,
with one binary network per trait. Their fully connected layers give 100, 150
and 750 features [arxiv:1610.08815 §4.2] [arxiv:1610.08815 §4.3]
[arxiv:1610.08815 §4.4]. A baseline sarcasm CNN adds 100 more, and an SVM is
fitted to the 1,100 concatenated values [arxiv:1610.08815 §5.2]
[arxiv:1610.08815 §5.3]. The paper does not say whether the extractors'
weights are updated during sarcasm training, so frozen and fine-tuned
extraction are both open readings [arxiv:1610.08815 §4.2].

Within each dataset the gains are small. On Dataset 1 the all-feature SVM
reaches 97.71% F1 against 97.60% for the baseline features alone, and on
Dataset 3 it reaches 93.30% against 92.20% [arxiv:1610.08815 tab. 2]. The
pre-trained features without the baseline reach 84.43% on Dataset 3
[arxiv:1610.08815 tab. 2]. The cross-dataset result is larger. Trained on
Dataset 3 and tested on Dataset 1, the model reaches 76.78%, against 53.02%
for the authors' re-implementation of Ptacek et al. [arxiv:1610.08815 tab. 3].
The reverse direction falls to 33.05% [arxiv:1610.08815 §5.6]. Card:
[A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](/papers/1610.08815).

### Replace the head and fine-tune

Load an ImageNet-pretrained network, remove its last classification layer,
attach a new head and train on the target set. The face-mask paper uses
InceptionV3 with a head of average pooling, a dense ReLU layer, dropout and a
two-way softmax [arxiv:2009.08369 §4.2]. The card does not say which backbone
layers were frozen or fine-tuned. Its target set is small by design, because
access to real faces is restricted [arxiv:2009.08369 §4].

The steel paper also fine-tunes an ImageNet-pretrained VGG16 on single objects,
resized to 224 x 224 px, with SGD at momentum 0.9. Its learning rate is stated
as 0.001 in one sentence and as 0.0001 in another, so the value is open
[arxiv:1706.06480 §4.3.1]. Cards:
[Face Mask Detection using Transfer Learning of InceptionV3](/papers/2009.08369),
[Advanced Steel Microstructural Classification by Deep Learning Methods](/papers/1706.06480).

The DeepWeeds baselines take the same route on field photos of weeds. An
ImageNet-pretrained ResNet-50 or Inception-v3 loses its 1000-way head, which
is replaced by nine sigmoid units, one per class, so that an image can score
for more than one species [arxiv:1810.05726 § Deep Learning]. Training uses
Adam at learning rate 1e-4 with binary cross-entropy and batches of 32, and
the rate is halved after 16 epochs without a validation improvement
[arxiv:1810.05726 § Deep Learning]. The paper does not state whether layers
are frozen. The authors' code trains all layers. Averaged over five folds, the
weighted top-1 accuracy is 95.7% for ResNet-50 and 95.1% for Inception-v3
[arxiv:1810.05726 tab. 2]. Card:
[DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning](/papers/1810.05726).

A benchmark of eleven ImageNet CNNs adapted to five image datasets compares
the same recipe under two modes: only the classifier is retrained, or every
layer is retrained. It trains with Adam, batch size 10 and a learning-rate
grid of 1e-2, 1e-3 and 1e-4, and keeps the weights from the best validation
epoch [arxiv:2605.11989 §4]. Its scores are accuracy divided by parameter
count, and it reports rankings from figures only [arxiv:2605.11989 §5.1]. Card:
[A Transfer Learning Evaluation of Deep Neural Networks for Image Classification](/papers/2605.11989).

A rice-variety study fine-tunes ten ImageNet CNNs with Adam at learning rate
1e-4, batch size 32 and 25 epochs [arxiv:2609.10524 tab. 3]. The four best
reach 97.5% to 99.2% validation accuracy [arxiv:2609.10524 tab. 4]. Their
outputs are then combined by a meta-learner [arxiv:2609.10524 §4.4]. Card:
[Precision in Rice Variety Classification using Stacking-Based Ensemble Learning](/papers/2609.10524).

A mammography study moves the same recipe from a patch to a whole image. A
patch classifier is first trained with ImageNet weights on lesion-centred
patches, which beats random initialisation on the harder set (ResNet50: 0.89
against 0.63 accuracy) [arxiv:1708.09427 §3.1.3] [arxiv:1708.09427 tab. 1].
Its convolutional layers are then reused on whole images, with two new
convolutional blocks and global average pooling on top [arxiv:1708.09427 §2.2].
The network is fine-tuned on INbreast, a digital dataset, with Adam at learning
rate 1e-5 for 200 epochs and weight decay 0.01 [arxiv:1708.09427 §3.2.2]. On the
INbreast validation split, each transferred model reaches AUC 0.95 and the
ensemble 0.98 [arxiv:1708.09427 tab. 4] [arxiv:1708.09427 §3.2.2]. The paper
reports no baseline trained on INbreast alone, so these numbers do not show
what the transfer adds. Card:
[Deep Learning to Improve Breast Cancer Early Detection on Screening Mammography](/papers/1708.09427).

An ImageNet-pretrained AlexNet is fine-tuned on 650 photos of smartphone backs
in 12 classes. Its last three layers are replaced, and SGD with momentum at
learning rate 3e-4 reaches 98.35 ± 2.02% accuracy over seven trials, against
95.26 ± 3.04% for Adam and 93.20 ± 3.86% for RMSProp [arxiv:2606.23286 tab. 1].
These are validation numbers: the paper describes no separate test set, and it
is not stated whether augmentation was applied before the split
[arxiv:2606.23286 §4, p. 5]. Card:
[Transfer learning-based method for automated ewaste recycling in smart cities](/papers/2606.23286).

A fine-tuning benchmark follows the same two-step recipe. Thirteen
ImageNet-pretrained networks, from SqueezeNet to ConvNeXt and ViT-B/16, are
fine-tuned on 8,171 underwater images in 20 classes [arxiv:2506.17455 §3.2].
Each classifier is replaced by a 20-way output layer. Training runs in two
stages: first only the new head, then all layers [arxiv:2506.17455 §4.1]. The
paper reports that most models peak after the first stage, and that only
EfficientNetB0 improves in the second [arxiv:2506.17455 §4.1]. The best model,
ConvNeXt, reaches 90.69% top-1 accuracy and 88.92% macro-F1 on the test split
[arxiv:2506.17455 tab. 3]. The paper reports no repeats or spread
[arxiv:2506.17455 §4]. The text gives a 32×32 input size, which is not the
224×224 used in the released notebook, so the setting behind the numbers is
unclear [arxiv:2506.17455 §4.1]. Card: [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455).

### Fine-tune with settings chosen by source-target similarity

The deep transfer survey argues against one recipe for every target. It orders
source types from most to least preferred: a very similar and much larger
labelled source first, and supervised training on a less related target alone
last [arxiv:2205.09904 §4.2]. Its one experiment extends Big Transfer (BiT)
fine-tuning to less similar targets, with ResNet-50x1 pretrained on ImageNet-21k
[arxiv:2205.09904 §3.4] [arxiv:2205.09904 tab. 3]. The recommended BiT settings
are best or near best on Cifar10, Cifar100 and Oxford Pets. On Cars and
Aircraft, a learning rate of 0.02 raises the mean accuracy from 81.8 to 87.6,
and from 82.4 to 87.3. Changing only the decay leaves both near 81%
[arxiv:2205.09904 tab. 3]. The survey calls the extension to less related
targets a hypothesis [arxiv:2205.09904 §3.4].

The same paper's similarity measures (earth mover's distance, Wasserstein and
Fréchet distance) place Cifar10 and Cifar100 closest to the source, and Cars
and Aircraft furthest [arxiv:2205.09904 tab. 2]. The table caption names
ImageNet-1k as the source, while the text names ImageNet-21k for the same
measures [arxiv:2205.09904 §3.4]. The source is therefore unclear.

The survey also discusses penalties that keep weights near their pretrained
values. L2-SP pulls weights toward the pretrained values [arxiv:2205.09904
§6.2.1], and distillation-based penalties compare feature maps with the source
network [arxiv:2205.09904 §6.2.2]. It presents the guidance on when such
penalties help as prose, not as measured results [arxiv:2205.09904 §7.3]
[arxiv:2205.09904 §7.7]. A meta-learning survey notes that transfer from a large
pretrained image model can fail when the target task is not similar
[arxiv:1810.03548 §4.1]. Cards:
[Deep transfer learning for image classification: a survey](/papers/2205.09904),
[Meta-Learning: A Survey](/papers/1810.03548).

### Freeze the body, fit a head, then unfreeze with per-group learning rates

The fastai library packages this recipe in a few calls. Its learner splits the
model into parameter groups, which can be frozen or given their own learning
rates [arxiv:2002.04688 §4.1]. The Pets example replaces the head of an
ImageNet-pretrained ResNet-34 and runs one-cycle training for four epochs, with
batch size 64 and no learning rate stated [arxiv:2002.04688 §2.1]. The paper
reports that result as close to the state of the art, without a number
[arxiv:2002.04688 §2.1]. It has no baseline and no variance, so the example is
a recipe rather than evidence of a gain. The authors single out batch-norm
handling. In their tests it changed the error rate by up to 300% across
configurations, and they argue for keeping batch-norm layers trainable with
their statistics still updating [arxiv:2002.04688 §4.1]. Card:
[fastai: A Layered API for Deep Learning](/papers/2002.04688).

### Frozen coarse encoder inside a trained matcher

Keep one large encoder frozen and train the rest of the network, each part
with its own learning rate. RoMa takes the patch tokens of a frozen DINOv2
ViT-L/14 as coarse features, projects them to 512 dimensions, and trains a
Gaussian-process encoder and a Transformer decoder on top [arxiv:2305.15404
§3.2] [arxiv:2305.15404 §3.3]. A VGG19 pyramid supplies the fine features and
is trained. The paper gives a learning rate of 1e-4 for the decoder and 5e-6
for the encoders, at batch size 8 [arxiv:2305.15404 §4.2]. The paper keeps
DINOv2 frozen throughout training [arxiv:2305.15404 §3.2], and the cards do not
report a run that fine-tunes it inside RoMa.

The frozen-feature probe in the paper compares backbones with a linear layer
and a kernel nearest-neighbour matcher. DINOv2 gives an endpoint error of 27.1
on MegaDepth, against 60.2 for ResNet-50 and 87.6 for VGG19 [arxiv:2305.15404
tab. 1]. That probe does not train the matcher, so it is not evidence for the
full system. On WxBS, the full RoMa system scores 80.1 mAA@10px, against 58.9
for DKM [arxiv:2305.15404 tab. 4]. This result also includes the trained
parts, so it does not isolate the effect of the frozen backbone. Card:
[RoMa: Robust Dense Feature Matching](/papers/2305.15404).

### Pretrained encoder in a segmentation network

Keep the encoder of a U-Net-style network, initialise it from an ImageNet
classifier, and train the decoder from scratch. TernausNet uses a VGG11
encoder. On 30 validation images of one aerial building dataset, after 100
epochs, validation IoU is 0.686 with the pretrained encoder and 0.593 with
LeCun-uniform initialisation for every layer, a difference of 0.093
[arxiv:1801.05746 §3]. The paper runs one trial per setting and reports no
seeds or variance, so this is a single measurement. A network pretrained on
Carvana reaches 0.687, but the paper does not state the size, epochs or loss
of that pretraining [arxiv:1801.05746 §3]. Card:
[TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation](/papers/1801.05746).

A video saliency network follows the same pattern. Its static network takes
the first five convolution blocks of an ImageNet-trained VGG-16 and trains the
rest from random values. Its dynamic network also starts from VGG-16, except
for its first convolution, which is Gaussian-initialised because its input has
seven channels [arxiv:1702.00871 §3.2] [arxiv:1702.00871 §5.1]. The ablation
table does not isolate the ImageNet start, so its gain is not measured
[arxiv:1702.00871 tab. 2]. Card:
[Video Salient Object Detection via Fully Convolutional Networks](/papers/1702.00871).

### Linear probe, then fine-tune (LP-FT)

Train only a new head on frozen features first, then fine-tune all parameters
starting from that head. The paper calls this LP-FT. The reason given is that
fine-tuning from a random head changes the features that the training data
touches, which hurts out-of-distribution (OOD) accuracy [arxiv:2202.10054 §3.2].
Starting the fine-tuning from the probed head is the paper's remedy
[arxiv:2202.10054 §3.4] [arxiv:2202.10054 §4.2]. In the paper's experiments,
the learning rate and the early-stopping point are chosen on in-distribution
validation data [arxiv:2202.10054 §4.1].

The effect depends on two conditions. It appears when the pretrained features
are good and the shift is large. With a weaker MoCo-v1 checkpoint, fine-tuning
beat linear probing on the OOD set (71% against 66%) [arxiv:2202.10054 §4.3],
and on a small shift (CIFAR-10 to CIFAR-10.1) fine-tuning also beat linear
probing OOD [arxiv:2202.10054 §4.3]. Card:
[Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](/papers/2202.10054).

### Text-initialised head for image-text models

An image-text model such as CLIP classifies by comparing an image embedding
with the embeddings of prompts that name each class. The usual adaptation
replaces this with a linear head that starts from random weights. ELEVATER
instead starts the linear head from the text encoder's class vectors, with a
zero bias, and reports this as the better choice across settings [arxiv:2204.08790 §4.2] [arxiv:2204.08790 §5.1].

Averaged over 20 image-classification datasets, CLIP ViT-B/32 with a linear
probe at 5 shots reaches 58.09% with a random head and 65.35% with the
text-initialised head (Separate-Head) [arxiv:2204.08790 tab. 2]. Fine-tuning
at 5 shots gives 29.75% with a random head and 63.29% with the text-initialised
head [arxiv:2204.08790 tab. 2]. The authors report that with the initialised
head, few-shot results are above zero-shot results, and that fine-tuning
overtakes linear probing from about 20 shots [arxiv:2204.08790 §5.1]
[arxiv:2204.08790 tab. 2]. A knowledge-augmented CLIP, with WordNet and
Wiktionary text added to the prompts, gains 0.48 points over CLIP at 5 shots
with a linear probe [arxiv:2204.08790 tab. 4]. The same paper reports
object detection as well, but its code covers classification only, so the
detection numbers are not checked against code [arxiv:2204.08790 §4.1]. Card:
[ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](/papers/2204.08790).

### Frozen backbone with small trainable adapters

Keep the pretrained network frozen and insert a small bottleneck module into
each block. The module down-projects, applies a depth-wise 3D convolution over
neighbouring frames, and up-projects back, with a residual connection
[arxiv:2206.13559 §3.2]. Only the adapters and a linear classifier are
trained [arxiv:2206.13559 §4.1]. The paper uses this to turn an image ViT into a
video recogniser. Card:
[ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](/papers/2206.13559).

The paper's numbers depend on the backbone. With CLIP ViT-B/16 on Kinetics-400,
the adapter reached 82.0% top-1 against 81.7% for full fine-tuning, with 7.20M
trained parameters against 121.57M [arxiv:2206.13559 tab. 1]. With ImageNet-21K
pretraining on the same dataset it reached 76.6%, against 78.0% for full
fine-tuning [arxiv:2206.13559 tab. 1]. The authors also report that with a
Swin-B backbone it underperforms full fine-tuning on ImageNet-21K pretraining
[arxiv:2206.13559 tab. 10].

A related route keeps the image ViT's blocks and adds video modules to them.
UniFormerV2 puts a temporal relation module in front of each pretrained block
and adds a query-based cross-attention module on the last layers, so video
training starts from image weights without a separate video pretraining stage
[arxiv:2211.09552 §3.1] [arxiv:2211.09552 §3.2]. On Kinetics-400 with
CLIP-ViT-B/16 and 8 x 4 x 3 testing, the full model reaches 84.4% top-1
[arxiv:2211.09552 tab. 9]. The paper's best headline, 90.0% with ViT-L/14 at
336 px after Kinetics-710 post-pretraining, uses 64 frames and far more compute
[arxiv:2211.09552 tab. 1]. Card:
[UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](/papers/2211.09552).

### Freeze the blocks, train input, output and normalisation

A pretrained transformer can be reused on a different kind of data while
its attention and feed-forward blocks stay frozen. Only a new input layer, a
new output layer, the layer-norm parameters and the position embeddings are
trained [arxiv:2103.05247 §2.2]. The source model is a GPT-2 base trained on
natural language, and the targets are bit strings, MNIST and CIFAR-10 patches,
ListOps and protein sequences [arxiv:2103.05247 §2.1]. Card:
[Pretrained Transformers as Universal Computation Engines](/papers/2103.05247).

The layer-norm parameters carry much of the effect. On CIFAR-10, training only
the layer norms gives 54% and training only the output layer gives 25%
[arxiv:2103.05247 tab. 15]. Freezing the layer norms in the full recipe drops
CIFAR-10 from 68.2% to 61.5% [arxiv:2103.05247 tab. 16]. The frozen model
reaches 100% on bit memory and 98.0% on MNIST, against 99.1% for a fully
trained transformer on MNIST [arxiv:2103.05247 tab. 1]. Its gains over random
initialisation are large on bit memory, where a randomly initialised frozen
model reaches 75.8% [arxiv:2103.05247 tab. 2]. The paper reports these results
as author-measured, and the released code trains more parameters than the text
states (see the card's Paper vs code section).

### Sparse fine-tuning with a chosen subset

Several parameter-efficient methods, including random masks, rule-based masks
(BitFit, magnitude pruning) and adapters, can be written as sparse fine-tuning:
only a fixed subset of weights changes [arxiv:2211.15583 §2.2].
The open question is which subset to pick. The SAM method ranks each weight by
the square of its gradient, summed over a few burn-in batches at the
pre-trained weights, keeps the top fraction, and freezes the rest
[arxiv:2211.15583 §4] [arxiv:2211.15583 §5.1]. Card:
[On the Effectiveness of Parameter-Efficient Fine-Tuning](/papers/2211.15583).

On seven GLUE and SuperGLUE tasks with RoBERTa-base at a tunable fraction of
0.5%, SAM averages 77.12 ± 1.51, against 74.78 ± 2.60 for full fine-tuning and
76.03 ± 2.35 for an adapter [arxiv:2211.15583 tab. 1]. The paper links stability and performance
with a rank correlation of 0.4356 [arxiv:2211.15583 §5.2]. In its sparsity
sweep, the mean and standard deviation on most tasks fall as the tunable
fraction falls, until a threshold below which performance drops sharply and
training error rises [arxiv:2211.15583 §5.2] [arxiv:2211.15583 fig. 5]. Masking does not reduce compute here: SAM averages
2.02 hours of training, against 1.67 for full fine-tuning [arxiv:2211.15583 tab. 3].

### Merge or interpolate weights

Average the parameters of models that start from the same initialisation, but
weight each parameter by its diagonal Fisher information instead of taking a
plain mean [arxiv:2111.09832 §2.4] [eq. 4]. The paper uses the merge for
ensembles of fine-tuned checkpoints, for WiSE-FT robust fine-tuning, and for
moving skills from a donor task to a target task [arxiv:2111.09832 §3.1]
[arxiv:2111.09832 §3.2] [arxiv:2111.09832 §3.3]. The Fisher is estimated from
training examples, and the authors say its cost is about that of training on
the same number of examples [arxiv:2111.09832 §2.3]. Card:
[Merging Models with Fisher-Weighted Averaging](/papers/2111.09832).

For a donor merge into a GLUE target, the paper compares Fisher merging with
gradient-based intermediate-task training, and the authors estimate the merge
at about six times less compute than fine-tuning [arxiv:2111.09832 §3.3]. The
exception matters: with an MNLI donor on RTE, the gradient-trained model scores
77.4 ± 1.6 against 73.2 ± 5.1 for the merge [arxiv:2111.09832 tab. A1]
[arxiv:2111.09832 tab. A3].

PAINT is a different interpolation: it fine-tunes a zero-shot CLIP encoder on
one patch task and then mixes the two sets of weights with one coefficient
alpha in [0, 1] [arxiv:2208.05592 §2]. The text head is frozen, so the
interpolation touches only the visual encoder [arxiv:2208.05592 §3]. With
ViT-L/14, the authors report gains of 15 to 60 points on nine patch tasks, and
ImageNet accuracy changes by less than one point [arxiv:2208.05592 §1]
[arxiv:2208.05592 §4]. Joint patching of all nine tasks reaches a combined
accuracy gain of 15.8 points [arxiv:2208.05592 §5]. Card:
[Patching open-vocabulary models by interpolating weights](/papers/2208.05592).

PAINT's interpolation also helps sequential patching. With ViT-B/32, sequential
patching without interpolation lost 4.6 points of combined accuracy, and with
interpolation it gained 11 points [arxiv:2208.05592 §5]. Forgetting is reduced
but not removed [arxiv:2208.05592 §5].

### Joint training with a related high-resource language

Train one model on the labelled target data and the labelled data of a related
source language at the same time. The two languages share the character encoder,
the sentence BiLSTM, the tag layers and the transition weights. Each language
keeps its own word embeddings and a learned language vector
[arxiv:2404.09383 §3.1]. The objective is the target log-likelihood plus a
weighted source log-likelihood, and the weight is a trade-off parameter that
the paper does not give a value for [arxiv:2404.09383 §3.1] [eq. 7]. Card:
[Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields](/papers/2404.09383).

With 100 labelled target sentences, the transfer setting raised F1 over the
monolingual log-linear CRF in every listed pair, up to 9.80 points for Galician
with French as source [arxiv:2404.09383 tab. 2]. Without a source, the neural
model was below the log-linear baseline on Galician, 49.19 against 57.64 F1
[arxiv:2404.09383 tab. 2]. With 10,000 target sentences the transfer gains were
small, and positive in every row [arxiv:2404.09383 tab. 2]. The table
reports no seeds or variance [arxiv:2404.09383 tab. 2].

### Contrastive and self-supervised pre-training, then reuse

Pre-train an encoder on unlabelled data with a contrastive or self-supervised
loss, then use it frozen for linear evaluation or as the starting point for
fine-tuning.

For ECG, the CLOCS family defines positives as segments or leads that share a
patient, not only the same instance [arxiv:2005.13249 §4.2] [arxiv:2005.13249
§4.1]. On Chapman, with 4 leads, an embedding size of 128 and half the labels,
the best variant (CMSC) reached a linear AUC of 0.896, against 0.738 for SimCLR
[arxiv:2005.13249 tab. 1]. For fine-tuning from PhysioNet 2020 to Cardiology, the
best value was 0.714 for CMSC, against 0.658 for SimCLR and 0.678 for random
initialisation [arxiv:2005.13249 tab. 2]. These gains are reported in AUC points,
not percent of the baseline [arxiv:2005.13249 tab. 1]. Card:
[CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](/papers/2005.13249).

For dense prediction, VADeR learns one embedding per pixel. Two augmented
views are matched pixel to pixel using the known view geometry, and a
contrastive loss is applied per pixel [arxiv:2011.05499 §3.1] [arxiv:2011.05499 §3.2].
The encoder starts from MoCo v2 and the decoder from random weights
[arxiv:2011.05499 §3.2]. With frozen features and a linear 1 x 1 convolution on
top, it reaches 56.7% mIoU on VOC against 43.0% for MoCo and 54.4% for
supervised ImageNet [arxiv:2011.05499 tab. 1]. On NYU depth it reaches an RMSE
of 0.964, against 1.136 for MoCo and 0.994 for supervised ImageNet (lower is
better) [arxiv:2011.05499 tab. 1]. With 2% of the VOC labels in fine-tuning it
reaches 47.7% against 44.7% for MoCo [arxiv:2011.05499 app. A.3]. The gains
are not uniform: at 100% of the NYU labels, MoCo's RMSE is 0.640 and VADeR's
0.641 [arxiv:2011.05499 app. A.3]. Card:
[Unsupervised Learning of Dense Visual Representations](/papers/2011.05499).

For EEG, BENDR pre-trains a convolutional encoder and an 8-layer transformer
on 60-second windows of unlabelled recordings. The loss asks for the true
masked vector among 20 distractors [arxiv:2101.12037 §2.4.1] [arxiv:2101.12037
eq. 2]. The pre-training corpus is about 1.5 TB of EEG before preprocessing
[arxiv:2101.12037 §2.1]. For each downstream task the paper compares six
configurations: a new softmax on the start token, average pooling of the
sequence with a new classifier, and variants that keep the first stage fixed.
Two of the six, configurations 3 and 5, use random initialisation as the
baseline [arxiv:2101.12037 §2.4.2]. The pooled classifier was the
best configuration on four of five datasets, and the no-pre-training full model
was generally ineffective, except on sleep staging [arxiv:2101.12037 §3.2]. The
paper's own numbers are in [arxiv:2101.12037 tab. 2]. The official code's
linear-head setting also freezes the input projection and position convolution,
which the paper does not say, so an implementation should not assume the
paper's freezing scheme. See the card's Paper vs code section. Card:
[BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](/papers/2101.12037).

### Pre-training on a large corpus, then task fine-tuning

FlauBERT is a BERT-style encoder pre-trained with masked language modelling on
a 71 GB French corpus, in a base and a large size [arxiv:1912.05372 §3.1]
[arxiv:1912.05372 §3.2]. Each downstream task is then fine-tuned separately. For
text classification, the paper grid-searches four learning rates over 30
epochs with batch size 16 [arxiv:1912.05372 §5.1]. Some tasks need no
fine-tuning at all: verb sense disambiguation scores each sense by the cosine
similarity between a test embedding and the mean embedding of that sense's
examples [arxiv:1912.05372 §5.6]. Card:
[FlauBERT: Unsupervised Language Model Pre-training for French](/papers/1912.05372).

The same recipe appears for speech. The FLEURS benchmark fine-tunes two 600M
pretrained speech models on 102 languages, and reports a character error rate
of 14.1% for the speech-only model and 14.6% for the speech-and-text model
[arxiv:2205.12446 tab. 3]. Languages are split into seen and unseen groups by
whether the pretraining data included their speech [arxiv:2205.12446 §3.2].
The per-group seen and unseen breakdown is not reproduced here, because the
source commented it out of the results table. Card:
[FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](/papers/2205.12446).

XLS-R applies the same pattern to speech in 128 languages. A wav2vec 2.0 model
is pretrained on about 436K hours of unlabelled audio, in 0.3B, 1B and 2B
parameter sizes [arxiv:2111.09296 §3.1] [arxiv:2111.09296 tab. 2]. Recognition
then adds a linear CTC head [arxiv:2111.09296 §4.3], and translation adds a
decoder initialised from mBART [arxiv:2111.09296 §4.2]. On the 14-language
VoxPopuli set, the 0.3B model averages 12.8% word error against 37.5% with no
pretraining [arxiv:2111.09296 tab. 10]. Across 10 CommonVoice languages with one
hour of labels each, the 1B model averages 5.1% phoneme error, against 7.6% for
XLSR-53 [arxiv:2111.09296 tab. 8]. On CoVoST-2 into English, the low-resource
group rises from 5.1 BLEU at 0.3B to 11.7 at 1B and 15.1 at 2B
[arxiv:2111.09296 tab. 3]. Card:
[XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](/papers/2111.09296).

The same pattern applies to molecules. ChemBERTa-2 pre-trains a RoBERTa-style
encoder on SMILES strings from 5M, 10M or 77M compounds [arxiv:2209.01712 §3].
It compares two unlabelled objectives: masked-token prediction, and regression
on about 200 RDKit descriptors computed from each string [arxiv:2209.01712
§3.1]. Each MoleculeNet task is then fine-tuned on a scaffold split with an
80/10/10 ratio [arxiv:2209.01712 §3.3]. The paper reports that the descriptor
objective usually fine-tunes better [arxiv:2209.01712 §4.1]. At 77M compounds,
it reaches 0.728 ROC-AUC on BBBP against 0.698 for masked-token pretraining,
and 0.817 on SR-p53 against 0.749 [arxiv:2209.01712 tab. 1]. Table 1 gives one
value per cell and no spread [arxiv:2209.01712 tab. 1]. Card: [ChemBERTa-2: Towards Chemical Foundation Models](/papers/2209.01712).

### Pre-train on a cleaned ImageNet-21K, one softmax per hierarchy depth

ImageNet-21K has about 21,000 classes, but each image carries one label at any
level of the WordNet hierarchy. The authors drop rare classes, hold out 50
validation images per class and resize every image to 224 px. The result, which
they call ImageNet-21K-P, has 11,221 classes [arxiv:2104.10972 §2.1]. They then
train one softmax per WordNet depth. Each image is trained at every depth where
its ancestor labels exist, so the same photo gives a target at several levels
[arxiv:2104.10972 §3.3]. Against single-label and multi-label pretraining on
the same backbone, this semantic softmax scores higher on six of seven
downstream sets. Multi-label is better on CIFAR-100 [arxiv:2104.10972 tab. 2].
Against ImageNet-1K pretraining, the paper's pretraining is higher in all 30
model-dataset cells it tests [arxiv:2104.10972 tab. 4]. Against the official
ImageNet-21K weights for ViT-B-16 and Mixer-B-16, it is higher in all 14 cells
[arxiv:2104.10972 tab. 3]. The paper also adds a semantic distillation term
that improves every column of its semantic-softmax results [arxiv:2104.10972
tab. 2]. The authors' public release does not include that training code, so
those distillation numbers cannot be reproduced from it. The cost is high: from
ImageNet-1K weights, 80 epochs took about 54 hours on eight V100 GPUs
[arxiv:2104.10972 §3.1]. Card:
[ImageNet-21K Pretraining for the Masses](/papers/2104.10972).

### Further pre-training on in-domain text, then fine-tuning

For text, the BERT review describes retraining the pretrained model on text
from the target domain before fine-tuning. It names three kinds of data: an
intra-task dataset, an intra-subject corpus and a cross-subject corpus. It
reports that retraining helps, and that intra-subject data helps more than
cross-subject data [arxiv:2103.11943 p. 3]. The learning rate of the cited
retraining study is about 1e-5, and 4e-4 did not converge there. Further
training can also cause catastrophic forgetting, which a low learning rate
reduces [arxiv:2103.11943 p. 3].

Domain models follow the same idea in two ways. BioBERT continues training on
PubMed and PMC text, then fine-tunes on biomedical tasks [arxiv:2103.11943
p. 10]. SciBERT follows the BERT training method on a scientific corpus, with a
vocabulary built for that corpus, because the general and scientific
vocabularies overlap by only 42% [arxiv:2103.11943 p. 10–11]. In the review's
Table 1, averaged over its tasks, fine-tuned SciBERT scores 79.27 and
fine-tuned BERT-Base 77.16. The review takes these numbers from the SciBERT
paper, and they are averages over several random seeds [arxiv:2103.11943 tab. 1].
Card: [BERT: A Review of Applications in Natural Language Processing and Understanding](/papers/2103.11943).

### One model trained on many tasks, then fine-tuned on a new one

Train one sequence model on offline trajectories from 41 Atari games, then
fine-tune it on five games it has not seen. The paper uses a decoder-only
transformer over image patches, return, action and reward tokens
[arxiv:2205.15241 §3.1] [arxiv:2205.15241 §3.2]. Pre-training runs for 10M
steps [arxiv:2205.15241 §4.1]. Fine-tuning uses 1% of a held-out game's data,
about 500,000 transitions, for 100k steps, and the paper reports the pre-trained
model best on every held-out game against a CQL model trained from scratch on
the same 1% [arxiv:2205.15241 §4.5] [arxiv:2205.15241 app. B.2]. The per-game
values are in a figure only [arxiv:2205.15241 fig. 6]. Card:
[Multi-Game Decision Transformers](/papers/2205.15241).

The headline score is 126% of human-normalised IQM across the 41 training
games, with the model size not named in the summary sentence
[arxiv:2205.15241 §1] [arxiv:2205.15241 fig. 1]. A separate five-seed run of
the 40M-parameter model gives IQM 0.95 with standard deviation 0.05
[arxiv:2205.15241 app. B.1]. Gains over more than one order of magnitude of
model size are reported for this method, and not for CQL [arxiv:2205.15241 §4.4].

A survey of curriculum learning for reinforcement learning treats transfer as
one of three parts of a curriculum, next to choosing intermediate tasks and
ordering them. Transfer carries policies, value functions or other knowledge
from one task to the next. The survey runs no experiments of its own, and it
classifies published methods by the knowledge they transfer [arxiv:2003.04960
§3.2] [arxiv:2003.04960 §4.3]. Card:
[Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](/papers/2003.04960).

Robot policies follow a related route, but by co-training rather than by
fine-tuning in sequence. Open X-Embodiment pools 60 robot datasets from 22
robot types in one format, and trains RT-1-X on that mixture alone
[arxiv:2310.08864 §3.1] [arxiv:2310.08864 §4.3]. Against each dataset's own
original method, RT-1-X is better on 4 of 5 small-data domains, although the
per-dataset numbers appear only in a figure [arxiv:2310.08864 §5.1]
[arxiv:2310.08864 fig. 3]. The large domains go the other way: on Bridge with
the IRIS setup, RT-1 trained on that dataset alone scores 40% and RT-1-X 27%
[arxiv:2310.08864 tab. 1]. In the RT-2 ablation, the 55B model rises from 27.3%
emergent-skill success with Google Robot data only to 75.8% with the pooled
robotics data, and both rows are co-trained with web data [arxiv:2310.08864 tab. 2].
Card: [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](/papers/2310.08864).

### Add a column per task and freeze the old ones

Keep the trained network, and add a new network, a column, for each new task.
Earlier columns are frozen. Each new column reads their hidden activations
through lateral connections, so the earlier tasks are not overwritten
[arxiv:1606.04671 §2] [eq. 1]. In practice a learned scalar scales these
lateral inputs, which pass through a small adapter [arxiv:1606.04671 §2]
[eq. 2]. The experiments use A3C agents on Pong variants, Atari games and 3D
mazes. Against full finetuning, two columns reach a mean transfer score of 209%
on Pong Soup against 181%, and 491% on Labyrinth against 235%. On Atari the
mean is 132% against 133%, and the medians are 112% and 110%
[arxiv:1606.04671 tab. 1]. On Labyrinth the medians are close (115% against
112%), so that mean gap is mostly outliers [arxiv:1606.04671 tab. 1]
[arxiv:1606.04671 §5.2]. The cost is size: parameters grow quadratically with
the number of columns, and at test time the caller must choose the column
[arxiv:1606.04671 app. 2] [arxiv:1606.04671 §2]. Card:
[Progressive Neural Networks](/papers/1606.04671).

A lifelong-learning review files progressive networks under dynamic
architectures, next to regularisation methods and replay [arxiv:1802.07569
§3.2] [arxiv:1802.07569 §3.3] [arxiv:1802.07569 §3.4]. The review's comparisons
are qualitative and come from cited work [arxiv:1802.07569 §3.5], so the
numbers above still come from the progressive-network paper. Card:
[Continual Lifelong Learning with Neural Networks: A Review](/papers/1802.07569).

### Freeze the convolution, fine-tune the fully connected layers with an alignment penalty

FedHealth builds a cloud CNN from user updates. Each user's model is uploaded
in encrypted form, and the server updates the cloud model from them
[arxiv:1907.09173 §3.3]. A user then freezes the convolution and pooling layers
and fine-tunes the fully connected layers [arxiv:1907.09173 §3.4]. The user
objective is the classification loss plus η times a CORAL term, which compares
second-order statistics of the cloud model and the user data, with η = 0.01
[arxiv:1907.09173 eq. 6] [arxiv:1907.09173 §4.2]. On five isolated UCI
Smartphone subjects, mean accuracy is 99.4% against 94.1% for the cloud model
with no transfer [arxiv:1907.09173 tab. 2]. The five runs are averaged with no
variance reported [arxiv:1907.09173 §4.2]. The paper does not name the
encryption scheme or the server's aggregation rule [arxiv:1907.09173 §3.3], so
the federated steps cannot be rebuilt from the text, and the layer count is
given as three in one section and two in another [arxiv:1907.09173 §3.4]
[arxiv:1907.09173 §4.2]. Card:
[FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](/papers/1907.09173).

### Fine-tune with a feature-distribution penalty on unlabelled target data

The deep transfer network starts from a CNN trained on labelled source data.
It then fine-tunes on unlabelled target data with the cross-entropy loss plus
a weighted joint distribution adaptation (JDA) term. The JDA term is a maximum
mean discrepancy (MMD) between source and target features, plus one MMD per
class, which uses target pseudo-labels from the current network
[arxiv:1804.07265 §III-A] [arxiv:1804.07265 eq. 10] [arxiv:1804.07265 eq. 11].
The SGD learning rate is 0.01, and the weight is searched over a grid
[arxiv:1804.07265 §IV-A]. The batch size and the number of iterations are not
stated [arxiv:1804.07265 §III-C] [arxiv:1804.07265 alg. 1]. On ten transfer
tasks over three fault data sets, the average accuracy is 97.5% with JDA, 87.4%
with the marginal term alone, and 80.0% for the plain CNN [arxiv:1804.07265 tab. III].
These are the authors' numbers. The paper does not say whether the weight was
chosen on labelled target data [arxiv:1804.07265 §IV-A]. Card:
[Deep Transfer Network with Joint Distribution Adaptation: A New Intelligent Fault Diagnosis Framework for Industry Application](/papers/1804.07265).

### Inflate a 2D checkpoint to 3D

Video Swin Transformer starts from ImageNet Swin weights and inflates them in
two places [arxiv:2106.13230 §3.4]. The patch-embedding kernel is repeated over
the two temporal slots and halved, which keeps the output mean and variance.
The relative bias table is repeated over the temporal offsets, so it is the
same in every frame. The released backbone implements only this duplicate
path [code: mmaction/models/backbones/swin_transformer.py#L588-L608]. The
pretraining source matters: Swin-B reaches 82.7% top-1 on Kinetics-400 from
ImageNet-21K, against 80.6% from ImageNet-1K, both with 4 × 3 views
[arxiv:2106.13230 tab. 1]. The ablations differ by little. A centre
initialisation of the patch embedding gives 78.8% top-1, the same as the
inflated one with Swin-T [arxiv:2106.13230 tab. 8]. For fine-tuning, a backbone
learning rate of 0.1 times the head's gave 82.6% against 82.0% at 1.0 times,
with ImageNet-21K pretraining [arxiv:2106.13230 tab. 7]. The SSv2 model starts
from a Kinetics-400 model rather than from images [arxiv:2106.13230 §4.1]. Card:
[Video Swin Transformer](/papers/2106.13230).

### Fine-tune end to end on sampled fragments

Video quality models often freeze a pretrained backbone and train only a small
regressor on its features. FAST-VQA instead trains a Video Swin-T end to end,
initialised from Kinetics-400, and makes this affordable by changing the
input. Each frame is cut into a 7 × 7 grid. One 32 × 32 patch is taken from
every cell at full resolution, and the patches are spliced into a smaller
canvas. The same cell offsets are reused in every frame [arxiv:2207.02595 §3.1]
[arxiv:2207.02595 tab. 2] [arxiv:2207.02595 §4.1].

The input cost does not depend on the frame size. It is 279 GFLOPs at 540P,
720P and 1080P, against 11,852 GFLOPs at 1080P for the full-resolution Swin-T
baseline, which uses fixed features [arxiv:2207.02595 tab. 4]. On LSVQ 1080p,
the PLCC is 0.814, against 0.753 for that baseline [arxiv:2207.02595 tab. 3].
The sampling choice matters on the same set. Bilinear resizing gives 0.786
PLCC, and random mini-patches without a grid give 0.790 [arxiv:2207.02595
tab. 6]. In the transfer table, each of five small datasets is split at random
ten times, and FAST-VQA is ahead of the same-features baseline on all five in
SRCC. For example, it reaches 0.849 against 0.799 on LIVE-VQC and 0.891 against
0.841 on KoNViD-1k [arxiv:2207.02595 tab. 5].

The paper does not state the optimiser, learning rate, schedule, epoch count
or augmentation, so an implementer has to choose them [arxiv:2207.02595 §4.1].
Card: [FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling](/papers/2207.02595).

### Freeze a language model and reprogram its inputs for a new modality

Time-LLM keeps a Llama-7B backbone frozen and trains only the layers around it
[arxiv:2310.01728 §3]. Each channel is normalised, cut into patches, and each
patch attends to learned text-prototype vectors that are a linear mix of the
model's word embeddings [arxiv:2310.01728 §3.1]. A text prompt with dataset
context, a task instruction and input statistics goes in front of the patches,
and the outputs at the patch positions are projected to the forecast horizon
[arxiv:2310.01728 §3.1]. The trainable reprogramming network has 6.39 million
parameters at ETTh1 with horizon 96, against 3,404.53 million in the 32-layer
backbone [arxiv:2310.01728 §4.5]. On long-term benchmarks the paper reports the
lowest MSE on seven of eight datasets [arxiv:2310.01728 §4.1]. In few-shot
forecasting on the first 10% of training steps, its ETTh1 MSE at horizon 96 is
0.556, against 0.590 for GPT4TS [arxiv:2310.01728 §4.3]. Removing the patch
reprogramming raised average MSE by 9.2%, and removing the input statistics
raised it by 10.2% [arxiv:2310.01728 §4.5]. The paper has no randomly
initialised backbone of the same size, so it cannot separate the effect of
pretraining from the architecture [arxiv:2310.01728 §4.5]. The released scripts
do not match the paper's configuration table, so check the card before
reproducing a number. Card:
[Time-LLM](/papers/2310.01728).

### Retrain the last layers on a microcontroller

Prune and quantize an ImageNet-pretrained network, deploy it to a board, reset
its last five layers to random values and retrain them on the device. The
paper's transfer setup runs 20 epochs at learning rate 0.001 with batch size 48
[arxiv:2407.10734 §4.1]. The backbone is pruned with an iterative L1-norm
schedule whose rates and final sparsity are not stated [arxiv:2407.10734 §4.1].
Training runs in 8-bit on the device, with the same quantization scheme as
inference, and the weight update is applied in floating point before the
tensor is requantized [arxiv:2407.10734 §3.1].

On seven datasets, uint8 retraining matched or exceeded the GPU baseline on the
simpler sets. It was lower on flowers, CIFAR-100 and speech. The mixed setting,
which keeps the classification head in float32, matched the float baseline
[arxiv:2407.10734 §4.1] [fig. 4]. The values are in the figure, and the card
does not transcribe them. The authors used no data augmentation on the device,
and they report more overfitting than the GPU baseline on speech and flowers
[arxiv:2407.10734 §4.1].

A survey of TinyML reports TinyTL, a memory-saving fine-tuning method for an
edge device, with a 6.5 times lower memory overhead than full fine-tuning and a
moderate accuracy loss [arxiv:2303.13569 §III-C]. The survey gives no setup for
that figure, and TinyTL has no card on this site, so the figure is unverified
here. The same survey lists transfer learning for model design as an open
lesson for TinyML [arxiv:2303.13569 §IV-B]. Cards:
[On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734),
[TinyML: Tools, Applications, Challenges, and Future Research Directions](/papers/2303.13569).

## When to use which

The foundation-model report names three factors for choosing an adaptation
procedure: the compute budget, how much task-specific data exists, and how much
access to the model's gradients is available [arxiv:2108.07258 §4.3]. The table
uses them.

| Situation | Use | Why |
|---|---|---|
| Labelled target set is small, and a network trained on a large source dataset is available | Fine-tune a pretrained network, or train a head on its features | In the steel paper, a fine-tuned VGG16 reached 66.50% and a VGG19 feature extractor with an SVM reached 64.84%, both above a CIFAR-style network trained from scratch at 57.03% [arxiv:1706.06480 tab. 1]. The source here is natural images and the target is microscopy, and the baselines differ in architecture, not only in initialisation. |
| Compute is tight, and no fine-tuning run can be afforded | Frozen features with a linear or SVM classifier, or adapters | On the same steel task, the frozen-feature SVM was 1.7 points behind the fine-tuned VGG16 [arxiv:1706.06480 tab. 1]. On video, a linear probe trains 0.31M parameters and reached 76.6% on Kinetics-400, against 81.6% for the adapter with 6.77M [arxiv:2206.13559 tab. 1]. |
| Image-text model, few labels, class names can be written as text | Text-initialised linear head, not a random head | At 5 shots, the text-initialised head beat the random head for linear probing (65.35% against 58.09%) and for fine-tuning (63.29% against 29.75%) [arxiv:2204.08790 tab. 2]. Random-head fine-tuning falls far below linear probing at few shots [arxiv:2204.08790 tab. 2]. |
| Target is out of distribution, and the pretrained features are good | Linear probe, then fine-tune (LP-FT) | LP-FT averaged 68.9% OOD across ten shifts, against 59.3% for fine-tuning and 66.2% for linear probing [arxiv:2202.10054 tab. 2]. The same paper reports that the advantage depends on feature quality and shift size [arxiv:2202.10054 §4.3]. |
| Target is in distribution, with enough labels to fine-tune | Full fine-tuning | Fine-tuning averaged 85.1% ID, above linear probing at 82.9% [arxiv:2202.10054 tab. 1]. LP-FT was best on ID with 85.7% [arxiv:2202.10054 tab. 1]. |
| Many tasks share one backbone, and each needs its own copy is costly | Adapters on a frozen backbone | ST-Adapter stores 7.20M trained parameters per task, against 121.57M for full fine-tuning [arxiv:2206.13559 tab. 1]. Training took 23 GPU-hours on 8 V100s against 40 hours for full fine-tuning [arxiv:2206.13559 tab. 6]. Check the backbone first, because the gap to full fine-tuning changes with pretraining data [arxiv:2206.13559 tab. 1]. |
| Video task, image ViT available, video pretraining unavailable | Add temporal modules to the image ViT and train on video | UniFormerV2 trains from image weights and reports 84.4% on Kinetics-400 with CLIP-ViT-B/16 [arxiv:2211.09552 tab. 9]. Its best numbers need many frames and compute, so compare like with like [arxiv:2211.09552 tab. 1]. |
| Modality far from the pretraining data, and only a small trainable subset is affordable | Freeze the blocks, train input, output and layer norms | Layer norms alone reached 54% on CIFAR-10 against 25% for the output layer alone [arxiv:2103.05247 tab. 15]. Freezing the norms costs about 7 points on CIFAR-10 [arxiv:2103.05247 tab. 16]. |
| Full fine-tuning is affordable, storage per task is the constraint, and a gradient-based subset choice is acceptable | Sparse fine-tuning with SAM or a comparable subset | SAM averaged 77.12 against 74.78 for full fine-tuning on the RoBERTa tasks [arxiv:2211.15583 tab. 1]. It does not save training time: 2.02 hours against 1.67 [arxiv:2211.15583 tab. 3]. |
| Two or more fine-tuned checkpoints share an initialisation, and no training budget remains | Fisher-weighted merge | On domain adaptation, merging gave ChemProt 83.1 ± 0.4 against 82.7 ± 0.3 unmerged, and ACL-ARC 73.2 ± 1.7 against 70.5 ± 3.2 [arxiv:2111.09832 tab. 1]. The merge needs the same starting point [arxiv:2111.09832 §2]. |
| Model is already good on a supported task, and one weak task must be improved | Interpolate between zero-shot and fine-tuned weights, and choose alpha on validation data | ViT-L/14 gains of 15 to 60 points on nine tasks, with ImageNet down by less than one point [arxiv:2208.05592 §4]. The alpha was chosen on held-out data, and the value that works best differs by model size [arxiv:2208.05592 app. O]. |
| A donor task is close to the target, and gradient training is affordable | Gradient-based transfer, and compare it to a merge | With an MNLI donor, sequential gradient training beat the Fisher merge on RTE [arxiv:2111.09832 tab. A1] [arxiv:2111.09832 tab. A3]. |
| Labelled data exists for a related language, but the target has about 100 sentences | Joint training with the source language | With 100 target sentences, Galician with French as source reached 68.02 F1 against 58.22 for the log-linear CRF [arxiv:2404.09383 tab. 2]. The source was chosen by the authors, and the paper gives no general rule [arxiv:2404.09383 §5.1]. |
| Only unlabelled recordings are available, and a few labels will follow | Contrastive pre-training on the same signal type | On ECG, the CMSC encoder was above SimCLR in linear AUC at every label fraction the authors tested [arxiv:2005.13249 tab. 1] [arxiv:2005.13249 app. D.1]. The released code is for non-commercial use [arxiv:2005.13249 §5.1]. |
| EEG or other biosignal with many unlabelled recordings and few labelled trials | Contrastive pre-training on the raw signal, then a pooled head fine-tuned per task | Pooling with a new classifier was highest on four of five EEG datasets, with MMI at 86.7 balanced accuracy [arxiv:2101.12037 tab. 2] [arxiv:2101.12037 §3.2]. The paper's pre-training corpus is large, so this route assumes that much unlabelled data exists [arxiv:2101.12037 §2.1]. |
| Dense two-view matching under strong appearance change, with a large frozen image model available | Keep the foundation encoder frozen for coarse features, and train the fine encoder and decoder | With frozen DINOv2 features on MegaDepth, the endpoint error is 27.1, against 60.2 for ResNet-50 features [arxiv:2305.15404 tab. 1]. The full system's WxBS result is 80.1 against 58.9 for DKM [arxiv:2305.15404 tab. 4]. |
| Dense outputs (segmentation, depth) from frozen features, no fine-tuning run | A per-pixel self-supervised encoder, frozen | VADeR with a frozen encoder beats MoCo and supervised ImageNet on VOC and NYU depth [arxiv:2011.05499 tab. 1]. Its Mask R-CNN gains are small, and the text overstates them (see Contradictions). |
| Target domain is close to the source domain | Pick the source by domain | Scene-trained features beat ImageNet features on scene benchmarks (SUN397 63.24% against 48.29%), and lose on object benchmarks (Caltech101 67.63% against 88.42%) [arxiv:1610.02055 tab. 3]. |
| Large, unlabelled corpus in the target language is available | Pretrain on it, then fine-tune each task | FlauBERT-large reached 91.61% UAS on French dependency parsing, against 88.92% with no pre-training [arxiv:1912.05372 tab. 7]. The large run used 128 GPUs for 390 hours [arxiv:1912.05372 §3.2], so the cost is high. |
| Many related tasks arrive over time, and each has little data | One model trained on many tasks, then fine-tuned on each new task | The multi-game transformer was best after fine-tuning on every held-out game, against a CQL model trained from scratch on 1% of the data [arxiv:2205.15241 §4.5]. Per-game numbers are in a figure only [arxiv:2205.15241 fig. 6]. |
| Choosing a pretrained CNN for a small image dataset | Benchmark several backbones under the two tuning modes you can afford | Under full tuning SqueezeNet has the highest accuracy density, and under classifier-only tuning ResNet-18 does [arxiv:2605.11989 §5.1] [arxiv:2605.11989 §5.2]. The results are figure-only, and the ranking is by a density score the authors define. |
| Segmenting large images, where the input cannot be resized | Staged fine-tuning with very small learning rates | The steel paper resizes nothing and trains on 1000 x 1000 px patches, which is limited by GPU memory [arxiv:1706.06480 §4.2]. Training took about 4.5 days on one GPU [arxiv:1706.06480 §4.3.3]. |
| Dense labels are few, and a classification-pretrained encoder exists for the same image type | Initialise the encoder from ImageNet weights, train the decoder from scratch | On 30 validation aerial images, IoU is 0.686 with the pretrained encoder against 0.593 with random initialisation [arxiv:1801.05746 §3]. One run per setting, so the gap has no measured spread. |
| Pathology slides, with ImageNet as the only pretrained source | Use early CNN layers as feature extractors, then fine-tune | The review says ImageNet-pretrained networks are common because no large pathology pretrained network is available [arxiv:1709.00786 §4.2.3]. It reports no experiment, so it gives no evidence of how well this works. |
| Real-world photographs as the target | Not supported by this evidence yet | All results of the face-mask paper come from one simulated dataset, with no real-world test [arxiv:2009.08369 §3]. |
| Labelled source data, unlabelled target data from a shifted domain, and the same label set | Fine-tune with a feature-distribution penalty (JDA), using the target pseudo-labels | On ten fault-diagnosis tasks, 97.5% with JDA, 87.4% with the marginal term only, 80.0% for the plain CNN [arxiv:1804.07265 tab. III]. The selection data for the weight is not stated, so treat the gap as possibly optimistic [arxiv:1804.07265 §IV-A]. |
| Domain text is far from the pretraining corpus, and unlabelled in-domain text is available | Further pre-train on in-domain text, with a low learning rate, then fine-tune | The review reports that retraining helps, and that intra-subject data helps more than cross-subject data [arxiv:2103.11943 p. 3]. Fine-tuned SciBERT averages 79.27 against 77.16 for fine-tuned BERT-Base, on the SciBERT paper's numbers [arxiv:2103.11943 tab. 1]. |
| Field photos of a few classes on an ImageNet backbone, and one image may show several classes | Replace the head with one sigmoid unit per class, fine-tune, and augment heavily | ResNet-50 reached 95.7% and Inception-v3 95.1% weighted top-1 over five folds [arxiv:1810.05726 tab. 2]. The paper does not state freezing, and its augmentation list differs from the code (see Contradictions). |
| Reproducing a published number exactly | Not possible from the cards alone | Optimizer, learning rate, batch size and frozen layers are missing from the face-mask paper [arxiv:2009.08369 §4.2]. The steel paper gives two conflicting learning rates for its object CNNs [arxiv:1706.06480 §4.3.1]. The LP-FT card leaves the optimiser and weight decay unstated [arxiv:2202.10054 §4.1]. The benchmark of eleven CNNs gives no numeric results at all [arxiv:2605.11989 §5]. |
| Language other than the pretraining language | Not covered by these results | FlauBERT was pretrained on French only, and the comparisons cover French tasks only [arxiv:1912.05372 §4]. FLEURS divides languages into seen and unseen by pretraining data [arxiv:2205.12446 §3.2]. |
| Each new task must keep earlier tasks intact, and extra model size per task is affordable | Progressive network: one column per task, lateral inputs from frozen columns | On Pong Soup, two columns reach a 209% mean against 181% for full finetuning, but on Atari they match it (132% against 133%) [arxiv:1606.04671 tab. 1]. Parameters grow quadratically with the number of columns [arxiv:1606.04671 app. 2]. |
| Each user needs a personal model, and the user data must stay with its owner | Freeze the convolution, fine-tune the fully connected layers with an alignment penalty | On five held-out subjects, 99.4% against 94.1% for the cloud model with no transfer [arxiv:1907.09173 tab. 2]. This is one dataset with five test subjects and no variance, and the federated steps are underspecified [arxiv:1907.09173 §4.2] [arxiv:1907.09173 §3.3]. |
| Video task, and only an image checkpoint with a hierarchical backbone is available | Inflate the 2D checkpoint, with ImageNet-21K weights where possible | Swin-B reaches 82.7% top-1 on Kinetics-400 from ImageNet-21K, against 80.6% from ImageNet-1K [arxiv:2106.13230 tab. 1]. The inflation ablations give the same top-1 for either initialisation [arxiv:2106.13230 tab. 8] [arxiv:2106.13230 tab. 9]. |
| Speech in many languages, little labelled data per language, and some unlabelled audio | Multilingual self-supervised pretraining, then fine-tune with CTC or a decoder | On VoxPopuli, 14 languages average 12.8% word error at 0.3B, against 37.5% with no pretraining [arxiv:2111.09296 tab. 10]. On LibriSpeech with 10 minutes of English labels, the English-only wav2vec 2.0 at 0.3B is better than XLS-R at 0.3B (test clean 32.1% against 34.1%) [arxiv:2111.09296 tab. 11]. |
| Several robots or datasets with little data each, and one policy is wanted | Co-train one policy on the pooled datasets, and check the large-data domains against a model trained alone | RT-1-X beats each dataset's own method on 4 of 5 small-data domains [arxiv:2310.08864 §5.1], but on Bridge (IRIS) it scores 27%, against 40% for RT-1 trained on that dataset alone [arxiv:2310.08864 tab. 1]. |
| Image task, and a large labelled source set is available to pretrain on before the target | Pretrain on a cleaned ImageNet-21K with semantic softmax, then transfer | Higher than ImageNet-1K pretraining in all 30 model-dataset cells tested [arxiv:2104.10972 tab. 4]. Pretraining from ImageNet-1K weights took about 54 hours on eight V100 GPUs [arxiv:2104.10972 §3.1]. The results are the authors' own, and the distillation part is not reproducible from their release. |
| Time series with few labelled examples, and a 7B-class language model can be served | Freeze the language model, train input and output layers, and prepend a text prompt | Time-LLM's few-shot ETTh1 MSE at horizon 96 is 0.556, against 0.590 for GPT4TS [arxiv:2310.01728 §4.3]. Its trainable reprogramming network is 6.39 million parameters [arxiv:2310.01728 §4.5]. The paper compares against no randomly initialised model of the same size [arxiv:2310.01728 §4.5]. |
| A working transfer baseline is needed quickly, and no benchmark claim is required | Layered library defaults: freeze or group layers, fit a head, then unfreeze with per-group rates | The paper has no benchmark tables, so it gives no accuracy comparison [arxiv:2002.04688 §8]. Batch-norm handling is the setting the authors report as most sensitive [arxiv:2002.04688 §4.1]. |
| Target is less similar to the source, for example Cars or Aircraft with an ImageNet-21k backbone | Try a learning rate of 0.02 next to the recommended setting, and choose on validation data | On Cars and Aircraft, this rate raised the mean from 81.8 to 87.6 and from 82.4 to 87.3 [arxiv:2205.09904 tab. 3]. Aircraft's spread under this setting is ±4.750, and the number of runs is not stated [arxiv:2205.09904 tab. 3]. |
| Small target set, and a very similar source is available | Fine-tune with a low learning rate, and transfer fewer layers than the whole network | The survey gives these as prose rules with no numbers [arxiv:2205.09904 §7.5] [arxiv:2205.09904 §7.6]. They are not measured in the paper. |
| Target data is collected on a microcontroller, and the model must be adapted on the device | Retrain the last layers in 8-bit on the device, with the classification head kept in float32 if accuracy falls | On seven datasets, uint8 retraining matched or exceeded the GPU baseline on the simpler sets, and was lower on flowers, CIFAR-100 and speech. The mixed setting matched float [arxiv:2407.10734 §4.1] [fig. 4]. A TinyML survey cites a 6.5 times memory saving for TinyTL, from a source it does not detail [arxiv:2303.13569 §III-C]. |
| Several source tasks have labelled data, and the target set is small | Pre-trained extractors from each source task, merged with the target model's own features | Sarcasm detection: the merged features reached 93.30% F1 on Dataset 3 against 92.20% for the baseline alone, and 97.71% against 97.60% on Dataset 1 [arxiv:1610.08815 tab. 2]. The paper does not say whether the extractors were frozen. |
| Histology or another specialised image set of a few hundred images, with ImageNet networks available | Frozen ImageNet features from several encoders, pooled over augmented crops, then a tree ensemble | The blend reaches 87.2 ± 2.6% four-class accuracy, against 84.2 ± 4.2% for the best single encoder [arxiv:1802.00752 tab. 1]. All numbers come from 10-fold cross-validation on one dataset [arxiv:1802.00752 §2.1]. |
| Out-of-distribution detection over a large label space, with a pretrained backbone whose last layer can be retrained | Grouped "others" head with the MOS score | Average FPR95 is 39.97% on four OOD sets, against 54.30% for KL matching and 76.96% for maximum softmax probability [arxiv:2105.01879 tab. 1]. Test time is 3.2 minutes, against 20.6 for KL matching [arxiv:2105.01879 tab. 1]. Single runs, author-reported. |
| Fine-tuning a frozen backbone on data that an attacker can add to | Treat the training data as attackable; the attack needs no access to the victim's weights or outputs [arxiv:1905.05897 §2] | Transferred clean-label poisons reach success rates at or near 50% with 1% of the set poisoned, and feature collision never exceeds 50% in the same setting [arxiv:1905.05897 §4.1]. The end-to-end result needs a multi-layer variant [arxiv:1905.05897 §4.3]. |
| Small labelled molecular property set, with many unlabelled SMILES strings | Pretrain a SMILES encoder with descriptor regression, then fine-tune | At 77M compounds, descriptor regression beats masked-token pretraining on BBBP (0.728 against 0.698 ROC-AUC) and SR-p53 (0.817 against 0.749) [arxiv:2209.01712 tab. 1]. The table gives one value per cell and no spread [arxiv:2209.01712 tab. 1]. |
| Underwater or other specialised photos with a few classes, and an ImageNet backbone | Two stages: train the new head, then all layers, and check the input size used | ConvNeXt reaches 90.69% top-1 on the test split [arxiv:2506.17455 tab. 3]. Most models peak after the first stage [arxiv:2506.17455 §4.1]. The notebook's input size and checkpoint rule differ from the text (see Contradictions). |
| Crop or weed images with a classification-pretrained backbone, and the benefit of ImageNet pretraining is in doubt | Fine-tune the ImageNet backbone, and run a model trained from scratch on your own images as a baseline | The weed survey cites one study where ImageNet weights still cut training iterations for weed classes, and another that argues a CNN without plant-image initialisation would not work well [arxiv:2103.01415 §10.1.1] [arxiv:2103.01415 §10.1.2]. The survey tests neither claim. |
| High-resolution video, where a frozen backbone's features are judged too coarse for the task | Fine-tune a video backbone end to end, on sampled fragments that keep raw-resolution detail | On LSVQ 1080p, FAST-VQA reached PLCC 0.814 against 0.753 for full-resolution Swin-T features [arxiv:2207.02595 tab. 3]. Its cost is 279 GFLOPs at every resolution, against 11,852 for the baseline at 1080P [arxiv:2207.02595 tab. 4]. The paper does not state the training recipe [arxiv:2207.02595 §4.1]. |

## Implementation sketch

The steps below are the common core of the variants. Names are ours. The
papers state most values but leave several open, and these are marked.

```pseudocode
# Frozen-feature transfer: fixed backbone, linear classifier  [arxiv:1610.02055 §3.3]
# Input:  pretrained backbone; labelled target images
# Output: classifier on backbone features

def fit_frozen_feature_classifier(backbone, images, labels):
    # Backbone weights stay fixed; only the classifier is fitted  [§3.3]
    feats = [backbone.embed(img) for img in images]
    # Linear SVM with C = 1, as in the Places paper  [§3.3]
    return linear_svm(feats, labels, C=1.0)
```

```pseudocode
# Replace-the-head transfer  [arxiv:2009.08369 §4.2] [fig. 4]
# Input:  backbone with ImageNet weights; num_classes = 2 in the paper
# Output: classifier producing one probability per target class

def build_transfer_classifier(backbone, num_classes):
    # Drop the source model's final classification layer  [§4.2]
    features = backbone.without_last_layer()
    # New head, applied in this order  [§4.2]
    pooled = average_pool(features, size=(5, 5))
    flat = flatten(pooled)
    hidden = dense(flat, units=128, activation="relu")
    hidden = dropout(hidden, rate=0.5)
    # Which backbone layers stay frozen is not stated  [§4.2]
    return dense(hidden, units=num_classes, activation="softmax")

def train_transfer_classifier(model, images, labels):
    # Fixed schedule: 80 epochs of 42 steps each  [§4.2]
    for epoch in range(80):
        for step in range(42):
            # Optimizer, learning rate and batch size are not stated  [§4.2]
            batch = sample_batch(images, labels)
            loss = classification_loss(model, batch)
            update_weights(model, loss)
    return model
```

```pseudocode
# Linear probe, then fine-tune all layers (LP-FT)  [arxiv:2202.10054 §4.2]
# Input:  pretrained network g; labelled target train and validation sets
# Output: fine-tuned network

def lp_then_ft(network, train_set, val_set):
    # Stage 1: freeze the features and fit only the head  [§4.1]
    feats = network.features(train_set.inputs)
    head = fit_logistic_regression(feats, train_set.labels)
    # Choose the regularisation strength on in-distribution validation  [§4.1]
    head = select_by_val_accuracy(head, feats, val_set)
    # Stage 2: start all layers from the probed head  [§4.2]
    # The fine-tuning learning rate is chosen on validation too  [§4.1]
    network.head = head
    return fine_tune_all_parameters(network, train_set)
```

```pseudocode
# Text-initialised head for an image-text model  [arxiv:2204.08790 §4.2] [tab. 2]
# Input:  class names (K), prompt templates, text encoder g
# Output: linear head with weight (P, K) and zero bias

def init_language_head(class_names, templates, text_encoder):
    # One column per class, built from its prompt embeddings  [§4.2]
    columns = []
    for name in class_names:
        # Encode each prompt that contains the class name  [§4.2]
        prompt_vectors = normalise(text_encoder(make_prompts(name, templates)))
        # Average the normalised prompt embeddings, then renormalise  [§4.2]
        columns.append(normalise(mean(prompt_vectors, axis=0)))
    # Bias starts at zero  [§4.2]
    return stack(columns, axis=1), zeros(len(class_names))
```

```pseudocode
# Staged fine-tuning of a segmentation network  [arxiv:1706.06480 §4.3.3]
# Momentum 0.9, weight decay 5e-4, batch size 1 in every stage
# Iterations: 7000 is stated once; the per-stage count is not stated

def train_staged_fcn(patches, fcn32_imagenet):
    # Stage 1: FCN-32s, learning rate 1e-10  [§4.3.3]
    fcn32 = train(fcn32_imagenet, skips=[], learning_rate=1e-10)
    # Stage 2: add skip layer A, fine-tune from stage 1, rate 1e-11  [§4.3.3]
    fcn16 = train(fcn32, skips=["A"], learning_rate=1e-11)
    # Stage 3: add skip layer B, fine-tune from stage 2, rate 3e-12  [§4.3.3]
    return train(fcn16, skips=["A", "B"], learning_rate=3e-12)
```

```pseudocode
# Interpolation after one fine-tune  [arxiv:2208.05592 §2]
# Input:  zero-shot encoder weights, fine-tuned encoder weights, alpha in [0, 1]
# Output: patched encoder weights

def interpolate(zeroshot, finetuned, alpha):
    # Choose alpha on held-out validation data of both tasks  [§2]
    # Mix every tensor with the same coefficient  [§2]
    return {name: (1 - alpha) * zeroshot[name] + alpha * finetuned[name]
            for name in zeroshot}
```

```pseudocode
# Sparse mask from burn-in gradients (SAM)  [arxiv:2211.15583 §4] [eq. 10]
# Input:  pre-trained weights theta0, tunable fraction p, burn-in batches
# Output: binary mask with floor(m * p) ones

def select_sam_mask(theta0, batches, p, burn_in_steps):
    # Sum the loss gradient at theta0 over the burn-in batches  [§5.1]
    grad_sum = zeros_like(theta0)
    for batch in batches[:burn_in_steps]:
        grad_sum = grad_sum + gradient(theta0, batch)
    # Rank entries by squared gradient; the curvature term is dropped  [eq. 10]
    scores = grad_sum ** 2
    # Keep the top floor(m * p) entries; the rest stay fixed  [§4]
    mask = zeros_like(theta0)
    mask[top_k_indices(scores, floor(size(theta0) * p))] = 1
    return mask
```

```pseudocode
# Fine-tuning with a JDA penalty on unlabelled target data  [arxiv:1804.07265 alg. 1]
# Input:  network trained on labelled source; unlabelled target inputs; weight lam
# Output: adapted network and target pseudo-labels

def adapt_with_jda(network, source, target_x, lam):
    # Pseudo-labels from the source-trained network  [alg. 1 line 3]
    target_labels = predict(network, target_x)
    # The paper does not define the convergence test  [§III-A]
    while not converged(target_labels):
        f_s = features(network, source.x)
        f_t = features(network, target_x)
        # Marginal MMD, then one MMD per class on pseudo-labels  [eq. 10]
        penalty = mmd(f_s, f_t)
        for c in classes(source.y):
            penalty = penalty + mmd(f_s[source.y == c], f_t[target_labels == c])
        # Source cross-entropy plus lam times the penalty  [eq. 11]
        network = sgd_step(network, cross_entropy(network, source) + lam * penalty)
        # Re-predict the target pseudo-labels  [alg. 1 line 8]
        target_labels = predict(network, target_x)
    return network, target_labels
```

## Contradictions

Disagreements between papers, or between a paper's text and its own tables,
with each side cited. Where two results come from different models or
budgets, the bullet says so and does not call them a contradiction.

- **FlauBERT is not best on every task.** On PAWS-X, CamemBERT scores 90.14%,
  above FlauBERT-base (89.49%) and FlauBERT-large (89.34%) [arxiv:1912.05372
  tab. 4]. On XNLI, XLM-R large scores 85.2% against 83.4% for FlauBERT-large
  [arxiv:1912.05372 tab. 5]. These two scores are the authors' reported numbers.
- **The base model is not uniformly weaker than the large one.** On
  constituency parsing, FlauBERT-base scores 89.05% test F1 and FlauBERT-large
  88.63% [arxiv:1912.05372 tab. 6]. On verb sense disambiguation, FlauBERT-base
  scores 43.92% and mBERT 49.83% [arxiv:1912.05372 tab. 8], while FlauBERT-base
  beats mBERT on dependency parsing [arxiv:1912.05372 tab. 7].
- **Learning rate for the object-level VGG16.** The steel paper gives 0.001 in
  one sentence and 0.0001 in another [arxiv:1706.06480 §4.3.1]. The validation
  split used to choose it is not described.
- **Face-mask counts.** The text gives a 470-image test set, but the test
  confusion matrix totals 388 [arxiv:2009.08369 §3] [arxiv:2009.08369 fig. 5].
  The abstract gives 99.9% training accuracy and Table 1 gives 99.92%
  [arxiv:2009.08369 tab. 1].
- **LP-FT on OOD data.** The text says LP-FT is best on all ten OOD datasets
  [arxiv:2202.10054 §4.2]. In the table, linear probing scores 63.2% on
  Entity-30 and LP-FT 62.3% [arxiv:2202.10054 tab. 2], so the claim does not
  hold on that dataset. The gap is within the stated intervals.
- **Feature-change ratio.** The introduction gives "10 to 100 times less" feature
  change for LP-FT, while the tables give about 17 times for Living-17 and about
  1000 times for DomainNet [arxiv:2202.10054 §1] [arxiv:2202.10054 app. B].
- **Fisher merge against gradient training.** The paper reports merging as a
  cheaper alternative to intermediate-task training [arxiv:2111.09832 §3.3].
  For an MNLI donor on RTE, sequential gradient training beats the merge on
  BERT-base [arxiv:2111.09832 tab. A1] [arxiv:2111.09832 tab. A3]. The
  Fisher-merge standard deviation on that pair is 5.1 [arxiv:2111.09832 tab. A1].
- **Adapter against full fine-tuning.** On Kinetics-400 with CLIP weights, the
  adapter is 0.3 points above full fine-tuning [arxiv:2206.13559 tab. 1]. With
  ImageNet-21K weights, it is 1.4 points below [arxiv:2206.13559 tab. 1]. The
  paper reports both, and does not treat the second as a failure of the method.
  On GLUE and SuperGLUE tasks with RoBERTa-base, an adapter averaged 76.03
  against 74.78 for full fine-tuning [arxiv:2211.15583 tab. 1]. These are
  different models, tasks and budgets, so the two results do not settle the
  question between them.
- **Random sparse masks against full tuning.** The SAM paper's text says even
  the random mask beats full tuning [arxiv:2211.15583 §5.2]. Its own averages
  say otherwise: 74.55 for random and 74.78 for full tuning [arxiv:2211.15583 tab. 1].
- **CLOCS percentage claims.** The paper says CMSC beats SimCLR by 15.8% in
  linear AUC on Chapman [arxiv:2005.13249 §6.2]. The table difference is
  0.896 − 0.738 = 0.158 AUC points, not a relative gain [arxiv:2005.13249 tab. 1].
  The paper's claim that all three CLOCS variants beat SimCLR in every linear
  setting does not hold for one PhysioNet 2020 cell [arxiv:2005.13249 app. D.1].
- **Speech baselines.** The text says most gains from multimodal pre-training
  appear in three regions [arxiv:2205.12446 §4.1]. The table shows the speech-
  and-text model ahead in one region only [arxiv:2205.12446 tab. 3].
- **VADeR and Mask R-CNN.** The text says VADeR consistently outperforms MoCo
  and the supervised baseline on Mask R-CNN [arxiv:2011.05499 §4.3]. Table 4
  has supervised ImageNet at 59.8 AP50 for boxes against VADeR's 59.7
  [arxiv:2011.05499 tab. 4]. The detection experiments also use a different
  FPN from the one the paper describes [arxiv:2011.05499 §4.3].
- **One alpha against per-model optima.** PAINT chooses alpha on validation
  data for each patch task [arxiv:2208.05592 §2]. Its appendix tables peak near
  0.5 to 0.6 for ViT-B/32 and 0.65 to 0.70 for ViT-L/14 [arxiv:2208.05592
  app. O]. A value copied from one backbone may not suit another.
- **Stacked outputs and the 100% result.** The rice paper reports XGBoost at
  100% on the TOP (1,2,3,4) combination [arxiv:2609.10524 tab. 6]. The paper
  does not say which split that table uses, and the base models were selected
  on their own performance [arxiv:2609.10524 §5]. Its own text describes the
  meta-model as trained on base-model outputs, and its Figure 3 describes
  stacked feature sets [arxiv:2609.10524 §4.4] [arxiv:2609.10524 fig. 3].
- **Multi-game fine-tuning size.** The paper says 1% of the held-out data is
  "1M steps", while the appendix gives about 500,000 transitions
  [arxiv:2205.15241 §4.5] [arxiv:2205.15241 app. B.2]. The appendix is consistent
  with a 50M-step dataset.
- **Fault-diagnosis text against its own table.** The text says DTN with JDA
  stays above 95% on all ten tasks [arxiv:1804.07265 §IV-B]. The table gives
  94.3% for F→E [arxiv:1804.07265 tab. III].
- **DeepWeeds augmentation, paper against code.** The paper describes random
  perspective transforms and an intensity shift of ±25 on all channels. The
  released code uses shear and a channel shift, with no perspective transform
  and no separate intensity shift [arxiv:1810.05726 § Deep Learning]. A
  rerun with the code therefore does not follow the paper's augmentation.

- **Which layers stay frozen for ewaste classification.** The paper says the
  network is frozen "for all layers except the last three" and also that the
  learnable weights are frozen "in the fully connected layer" [arxiv:2606.23286
  §4, p. 5]. The two statements conflict, and the card gives no resolution.
- **Positive transfer against a single-dataset model.** The paper's headline is
  positive transfer from pooled robot data [arxiv:2310.08864 §1]. For large
  domains it says RT-1-X underfits and does not beat RT-1 trained on the
  dataset alone [arxiv:2310.08864 §5.1], and the Bridge rows of its table show
  the same thing [arxiv:2310.08864 tab. 1]. Both results are reported.
- **English-only against multilingual at low label counts.** The XLS-R paper
  reports gains across languages, but on LibriSpeech with 10 minutes or 1 hour
  of labels, the English-only wav2vec 2.0 model at 0.3B beats XLS-R at 0.3B:
  test clean 32.1% against 34.1% at 10 minutes, and 13.7% against 16.8% at 1
  hour [arxiv:2111.09296 tab. 11]. XLS-R at 1B is slightly better than the
  English-only model at 1 hour (13.1%) [arxiv:2111.09296 tab. 11].
- **Layer count in FedHealth.** Section 3.4 describes two fully connected layers,
  with the alignment term replacing the second [arxiv:1907.09173 §3.4]. Section
  4.2 describes three [arxiv:1907.09173 §4.2]. The position of the alignment term
  therefore depends on which section is followed.
- **Best rate for Cars.** The caption of the survey's BiT table gives 0.02 as the
  best rate for both less similar targets. The table lists 0.025, with decay
  every 5 steps at 0.96, as the best rate for Cars [arxiv:2205.09904 tab. 3].
- **Sarcasm: text against its own table.** Section 5.5 calls the sentiment
  model the best pre-trained extractor on Dataset 3. Table 2 gives personality
  74.41% and sentiment 73.50% there [arxiv:1610.08815 §5.5] [arxiv:1610.08815
  tab. 2]. Section 5.4 gives 87.00% for sentiment on Dataset 2, while Table 2
  gives 86.50% for Dataset 2 and 87.00% for Dataset 1 [arxiv:1610.08815 §5.4]
  [arxiv:1610.08815 tab. 2].
- **Does ImageNet pretraining help plant images?** One weed study reports that
  ImageNet weights still reduce the number of training iterations for weed
  classes. Another argues that a CNN initialised without plant images would not
  work well, and builds a new network [arxiv:2103.01415 §10.1.1]
  [arxiv:2103.01415 §10.1.2]. The survey does not settle the question.
- **Fine-tuning against frozen features in medical images.** Rakhlin et al.
  say fine-tuning a pretrained network did not work well for them, with no
  numbers [arxiv:1802.00752 §2.2]. The steel paper reports a fine-tuned VGG16
  at 66.50%, above a frozen VGG19 feature extractor with an SVM at 64.84%
  [arxiv:1706.06480 tab. 1]. The two studies use different images, networks
  and baselines, so they do not settle the question.
- **Checkpoint choice in the benchmark.** The AQUA20 paper keeps "the
  checkpoints that achieved the highest performance during evaluation"
  [arxiv:2506.17455 §4.1]. The released notebook keeps the lowest test loss and
  stops on it, so its test scores are not from data that the selection never
  saw ([AQUA20 card](/papers/2506.17455), Pitfalls).

## Papers

- [A Taxonomy of Deep Convolutional Neural Nets for Computer Vision](/papers/1601.06615): a survey of convolutional networks and their fine-tuning advice, with no experiments.
- [Progressive Neural Networks](/papers/1606.04671): one column per task, with frozen earlier columns and lateral inputs, tested with A3C agents on Pong, Atari and mazes.
- [Places: An Image Database for Deep Scene Understanding](/papers/1610.02055): a scene dataset, and generic features from scene-trained networks compared with ImageNet features on eight benchmarks.
- [A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](/papers/1610.08815): sentiment, emotion and personality CNNs used as feature extractors and merged with a baseline sarcasm CNN, on three Twitter datasets.
- [Video Salient Object Detection via Fully Convolutional Networks](/papers/1702.00871): a video saliency network trained partly on synthetic video made from annotated still images, with ImageNet VGG-16 initialisation for its static network.
- [Advanced Steel Microstructural Classification by Deep Learning Methods](/papers/1706.06480): fine-tuned VGG16 and staged fine-tuning of a fully convolutional network for steel microstructure.
- [Deep Learning to Improve Breast Cancer Early Detection on Screening Mammography](/papers/1708.09427): a patch classifier converted to a whole-image classifier, and transferred from film to digital mammograms.
- [Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis](/papers/1802.00752): frozen ImageNet encoders with pooled crop features, classified by gradient-boosted trees, on 400 breast slides.
- [Machine learning methods for histopathological image analysis](/papers/1709.00786): a mini-review of machine learning for whole-slide pathology images, with no experiments of its own.
- [TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation](/papers/1801.05746): a U-Net whose VGG11 encoder starts from ImageNet weights, tested on one aerial building dataset.
- [Deep Transfer Network with Joint Distribution Adaptation: A New Intelligent Fault Diagnosis Framework for Industry Application](/papers/1804.07265): a CNN fine-tuned on unlabelled target data with a marginal and class-wise MMD penalty, on three fault data sets.
- [DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning](/papers/1810.05726): a field photo dataset of eight weed species, with ImageNet ResNet-50 and Inception-v3 baselines.
- [Meta-Learning: A Survey](/papers/1810.03548): a survey that groups meta-learning by the meta-data it reuses, with transfer as one family, and no experiments of its own.
- [Continual Lifelong Learning with Neural Networks: A Review](/papers/1802.07569): a review of catastrophic forgetting and its fixes (regularisation, dynamic architectures, replay), with no new experiments.
- [Deep learning in agriculture: A survey](/papers/1807.11809): a review of 40 agricultural deep-learning studies, with their use of transfer learning and test data.
- [Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](/papers/1905.05897): convex polytope clean-label poisons, crafted on substitute networks and tested on frozen and end-to-end victims.
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](/papers/1907.09173): a federated cloud model, with frozen convolution layers and CORAL-aligned fine-tuning for each user.
- [FlauBERT: Unsupervised Language Model Pre-training for French](/papers/1912.05372): masked-language-model pre-training on French text, evaluated on six French tasks.
- [fastai: A Layered API for Deep Learning](/papers/2002.04688): a layered PyTorch library with one-call training, data blocks and callbacks, and no benchmark tables.
- [Face Mask Detection using Transfer Learning of InceptionV3](/papers/2009.08369): an ImageNet-pretrained InceptionV3 with a new head, trained on augmented images of one simulated face dataset.
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](/papers/2003.04960): a survey that splits curriculum learning for RL into task generation, sequencing and transfer, with no experiments.
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](/papers/2005.13249): contrastive pre-training on ECG where positives share a patient, compared with SimCLR and BYOL.
- [Unsupervised Learning of Dense Visual Representations](/papers/2011.05499): per-pixel contrastive pre-training with known view correspondences, evaluated with frozen features and fine-tuning.
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](/papers/2101.12037): a wav2vec 2.0 style transformer pre-trained on unlabelled EEG, then fine-tuned on five public BCI and sleep tasks.
- [A Survey of Deep Learning Techniques for Weed Detection from Images](/papers/2103.01415): a survey of 70 deep-learning weed-detection papers, with no experiments of its own.
- [Pretrained Transformers as Universal Computation Engines](/papers/2103.05247): a frozen GPT-2 with trained input, output and normalisation layers, applied to non-text tasks.
- [BERT: A Review of Applications in Natural Language Processing and Understanding](/papers/2103.11943): a 2021 review of BERT, its domain retraining, extractive annotation, BERTScore and adversarial attacks.
- [Machine learning and deep learning](/papers/2104.05314): a conceptual review of machine learning and deep learning, which lists transfer learning as a practical challenge.
- [ImageNet-21K Pretraining for the Masses](/papers/2104.10972): a cleaned ImageNet-21K with one softmax per WordNet depth, pretrained and transferred to seven downstream sets.
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](/papers/2105.01879): a last-layer retrain with an "others" class per ImageNet group, scored for out-of-distribution inputs.
- [Video Swin Transformer](/papers/2106.13230): a shifted-window transformer for video, initialised by inflating an image Swin checkpoint.
- [On the Opportunities and Risks of Foundation Models](/papers/2108.07258): a report on foundation models, which frames adaptation choices by compute, task data and gradient access.
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](/papers/2111.09296): wav2vec 2.0 pretrained on about 436K hours in 128 languages, then fine-tuned for speech tasks.
- [Merging Models with Fisher-Weighted Averaging](/papers/2111.09832): a Fisher-weighted average of models that share an initialisation, for ensembling, robust fine-tuning and donor-task transfer.
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](/papers/2202.10054): linear probing then fine-tuning, and when full fine-tuning loses to linear probing out of distribution.
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](/papers/2204.08790): a benchmark of 20 classification and 35 detection datasets, and a text-initialised linear head.
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](/papers/2205.06743): a survey of few-shot methods grouped by challenge, with copied benchmark tables and no experiments.
- [Deep transfer learning for image classification: a survey](/papers/2205.09904): a survey that sorts transfer methods by source-target similarity and target size, with one fine-tuning experiment on five targets.
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](/papers/2205.12446): a 102-language read-speech benchmark with baselines from two pretrained 600M speech models.
- [Multi-Game Decision Transformers](/papers/2205.15241): one offline transformer for 41 Atari games, fine-tuned on five held-out games.
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](/papers/2206.13559): adapters with a depth-wise 3D convolution inside a frozen image ViT, for video action recognition.
- [FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling](/papers/2207.02595): a Video Swin-T trained end to end on grid mini-patch fragments, for video quality scoring.
- [Patching open-vocabulary models by interpolating weights](/papers/2208.05592): PAINT, which interpolates between zero-shot and fine-tuned CLIP weights to patch one weak task.
- [ChemBERTa-2: Towards Chemical Foundation Models](/papers/2209.01712): a RoBERTa-style SMILES encoder pretrained with masked tokens or descriptor regression, then fine-tuned on MoleculeNet.
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](/papers/2211.09552): temporal and query-based modules added to a pretrained image ViT, for video recognition.
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](/papers/2211.15583): parameter-efficient methods as sparse fine-tuning, and the SAM gradient-based mask.
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](/papers/2310.08864): 60 robot datasets pooled in one format, with RT-1-X and RT-2-X policies trained on them.
- [TinyML: Tools, Applications, Challenges, and Future Research Directions](/papers/2303.13569): a survey of TinyML tools, boards, applications and open challenges, with no experiments of its own.
- [RoMa: Robust Dense Feature Matching](/papers/2305.15404): a dense matcher with a frozen DINOv2 coarse encoder, trained fine features and a Transformer decoder.
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](/papers/2310.01728): a frozen Llama-7B reprogrammed for forecasting, with trained input and output layers.
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](/papers/2310.10196): a survey and taxonomy of large language and foundation models for temporal data, with no experiments.
- [Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields](/papers/2404.09383): joint training of a neural CRF on a low-resource and a high-resource language, with a language vector.
- [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](/papers/2407.10734): 8-bit training on three Cortex-M boards, with transfer-learning retraining of the last layers on seven datasets.
- [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455): 8,171 underwater images in 20 classes, with 13 ImageNet-pretrained networks fine-tuned on them.
- [A Transfer Learning Evaluation of Deep Neural Networks for Image Classification](/papers/2605.11989): eleven ImageNet CNNs adapted to five image datasets under two tuning modes, compared by accuracy density.
- [Transfer learning-based method for automated ewaste recycling in smart cities](/papers/2606.23286): an ImageNet-pretrained AlexNet fine-tuned to sort 12 smartphone classes from 6 brands.
- [Precision in Rice Variety Classification using Stacking-Based Ensemble Learning](/papers/2609.10524): fine-tuned ImageNet CNNs combined by a stacked XGBoost meta-learner, for rice variety classification.