Deep transfer learning for image classification: a survey

In plain words

Training a good image classifier usually takes a lot of labelled pictures. When you do not have them, a common shortcut is to start from a model that someone already trained on a big dataset and adapt it to your task. This survey looks at how that shortcut works for image classification and when it does not.

The main point is that the answer depends on two things: how similar the big original dataset is to yours, and how many labelled images you have. The authors argue that most failures come from using settings that suit a large, very similar target for a small, different one. Instead of asking "transfer or not", they suggest asking how much to transfer and how far the weights should be allowed to move.

The survey defines the problem, groups the existing methods, and gives rules of thumb for choosing a setup. It has no new model and no code.

TL;DR

A survey of deep transfer learning for image classification, written as a guide for choosing settings. It proposes a taxonomy based on source-target similarity and target dataset size, reviews methods in five groups, and argues against treating transfer as a yes-or-no choice. Its only own experiment shows Big Transfer's fine-tuning settings work for similar targets but not for Cars and Aircraft. Most recommendations rest on cited work, and several are marked as hypotheses.

Problem

Deep image classifiers need large labelled datasets to train well from random initialisation. Many real settings (medical, robotics, agriculture, rare classes) lack that data §1. Larger models can overfit even ImageNet-1k, which is why some very large models show lower accuracy fig. 1 §3.

Transfer learning addresses this by starting from pretrained weights. The authors treat this as a constraint on the parameters that reduces the effect of an unreliable empirical loss on small data §3.1 eq. 1:

L = \varepsilon_{bias} + \varepsilon_{var} eq. 1

Here the bias term comes from the chosen architecture and the variance term from fitting a finite training set §3.1. Pretraining can hurt when the source is unrelated and the target is large enough to train alone. This is called negative transfer §3.5.

Method

This is a survey, so there is no algorithm to implement. Its contribution is the definitions, the taxonomy, and the recommendations below.

Definition. The authors refine the definition of deep transfer learning: the task is deep when the target function is approximated by a deep network and a network trained on the source dataset is used to improve that approximation. Fine-tuning means continuing to train transferred weights on the target. Freezing means training only the new weights §3.2.

Taxonomy of dataset relationships. Two axes replace the older homogeneous versus heterogeneous and label-based splits §3.7:

• Similarity: similar versus dissimilar source and target. Similarity is continuous and can be estimated visually or through transferability measures §3.3 §9.
• Size: small versus large target dataset. Small is around 10,000 labelled examples and large is a million or more. The transition is gradual, roughly 50,000 to 200,000 examples, and the number of model parameters also matters fig. 5.

Taxonomy of solutions. The survey removes relational approaches and generative adversarial training as a category, since GAT is placed under feature-based methods or under data augmentation §3.7. It then groups methods as follows:

• Instance-based: choosing or weighting source data §4.
• Feature-based: aligning source and target feature spaces, for example with MMD plus cross-entropy, or adversarial alignment §5.
• Parameter-based: parameter-efficient tuning (prompts, adapters, selective bias or layer-norm tuning, low-rank residuals) §6.1; regularisation §6.2; normalisation changes §6.3; ensembles and model soups §6.4.
• Best practice: pretraining data, fine-tuning hyperparameters, and measures of transferability §7.

Regularisation toward the pretrained weights. The survey writes the regularised loss as the empirical loss plus a weighted penalty term §6.2. The L2-SP penalty pulls the current weights toward the pretrained values §6.2.1:

\Omega(w) = \lVert w - w^{0} \rVert_2^2 §6.2.1

Knowledge distillation instead penalises differences between feature maps of the source and target networks §6.2.2. DELTA weights those differences per filter, and AFA adds spatial weights §6.2.2.

Source ranking. The survey ranks source types from most to least preferred §4.2:

1. Supervised training on a very similar labelled dataset that is much larger than the target.
2. Supervised training on a closely related dataset with uncurated or pseudo labels, much larger than the target.
3. Self-supervised training on a closely related unlabelled dataset, much larger than the target.
4. Self-training on the target dataset itself, when no closely related source exists.
5. Self-supervised pretraining on a less related source.
6. Supervised training on a less related target dataset alone.

Multi-step fine-tuning, with an interim dataset, is said to help in many cases, but is not part of this ranking §4.2.

Decision guide. Section 7 and the conclusion combine the points above into advice. This sketch restates them and is not an algorithm from the paper.

# Choose a transfer plan from source-target similarity and target size
# Inputs: similarity in [0, 1] (visual judgement or a transferability score)
# Inputs: target_size (number of labelled target images)  [§3.7] [fig. 5]

def plan_transfer(similarity, target_size):
    # Rank candidate sources: a similar, labelled, much larger source first;
    # otherwise self-train on the target  [§4.2] [§7.2]
    source = pick_source(similarity, target_size)

    # Small targets are fragile: transfer fewer layers than all of them,
    # even when the source is very similar  [§7.5] [§7.6]
    # With a less similar source, transfer lower layers only  [§7.7] [§9]
    layers_to_transfer = choose_depth(similarity, target_size)

    # Similar source: low learning rate, decayed quickly  [§7.3] [§7.6]
    # Less similar source: high initial rate and momentum, then decay  [§7.4]
    learning_rate = choose_rate(similarity, target_size)

    # L2-SP-style penalties help when the source is similar; on all layers
    # they can hinder a less related source  [§6.2] [§7.3] [§7.7]
    penalty = choose_penalty(similarity, target_size)

    return source, layers_to_transfer, learning_rate, penalty

Measures of transferability. Source-free estimators (H-Score, LEEP, negative conditional entropy, FERM, LogME, KITE) need no source data. Source- dependent ones (optimal-transport variants, gradient-based PGE) use it. The survey calls these an open question: they do not yet reliably pick hyperparameters across datasets, and the studies testing them are limited §7.1 §7.2.

Hyperparameters

The survey reports only one experiment's settings §3.4 tab. 3. Values come from the caption and the table of Big Transfer (BiT) fine-tuning.

Recommended hyperparameter rules for other settings (learning rate by target size, decay schedules, penalty strength) are given in prose and are not quantified in this paper §7.3 §7.5 §7.6.

Results

Author-reported. The only new experiment is an extension of BiT to less similar targets, on Cifar10, Cifar100, Oxford Pets, Cars and Aircraft, with ResNet-50x1 pretrained on ImageNet-21k §3.4 tab. 3. Mean ± standard deviation, accuracy in %. "Less similar combined" uses the best lr and decay for less similar targets.

tab. 3. Headline for the survey: on the two less similar targets, Cars and Aircraft, the less-similar lr setting raises the mean from 81.8 to 87.6 and from 82.4 to 87.3 tab. 3. On the similar targets, the recommended settings are best or near best tab. 3 §3.4.

Similarity and distance of five targets to the source (ImageNet-1k in the caption; see Pitfalls), author-computed tab. 2:

tab. 2. Cifar10 and Cifar100 are the most similar in this table, and Cars and Aircraft the least, which matches the BiT split in tab. 3.

Other results are summaries of cited work, not new measurements. Examples: self-training on the target beat ImageNet pretraining for COCO detection in the cited El-Nouby et al. work §4.1.1, and a global ensemble helps when the target is less similar §6.4.1. Check the cited paper before using these claims.

Paper vs code

No official code. The paper is a survey and links no code or project page §1 tab. 1. A GitHub search for the title and for the authors' names returned nothing from the authors. The survey has no implementation to compare.

Pitfalls

• Few own experiments. The only experiment is the BiT extension §3.4 tab. 3. Most recommendations summarise cited work, so check the source before implementing one.
• Hypotheses, not findings. Some statements are marked as the authors' hypothesis or as needing more work: the BiT rule extending to less related targets ("We hypothesise") §3.4, lower-layer penalties §6.2, and gradual freezing §7.2. Treat these as open.
• Source name conflict. The tab. 2 caption names ImageNet-1k as the source, while the text of §3.4 names ImageNet-21k for the same similarity measures. Which source the similarity numbers use is unclear.
• Column meaning in tab. 3. The text says the decay schedule barely changes mean accuracy and mainly reduces variance §3.4. In the table, the "less similar lr" column has the higher mean for Cars (87.6 vs 81.8) and Aircraft (87.3 vs 81.4). Confirm the meaning of each column against the BiT setup before using the numbers.
• Best hyperparameters for Cars. The tab. 3 caption gives lr 0.02 as the best for both less similar targets. The table lists lr 0.025, decay steps 5, rate 0.96 for Cars.
• Unknown run count. The ± values in tab. 3 are not explained. Aircraft's less-similar-lr column has ±4.750, a large spread for a single setting.
• "Recommended" values are not in the paper. The recommended column comes from BiT §3.4 tab. 3, and its numbers are not stated here.
• Overstated conclusion. The claim that negative transfer "can be eliminated" by thinking of transfer as a continuum §9 is the authors' proposal. It is not tested in the paper.
• Taxonomy is approximate. The size and similarity categories are continuous, and their cut-offs come from the figure caption fig. 5. Similarity is often a visual judgement §9. Categories from older taxonomies (Tan 2018, Iman 2023, Yu 2022) do not map one to one §3.7.
• Training setup missing. The BiT runs do not state optimiser, momentum, batch size, epochs or warmup. An implementer must take these from the BiT paper or choose them.

Lineage

• Builds on: the definition of transfer learning and the terminology of Pan and Yang's survey §3.2. The 2018, 2022 and 2023 deep transfer learning surveys (Tan et al.; Yu et al.; Iman et al.) are the taxonomies this paper extends §1 tab. 1.
• Compared against: the broader and narrower transfer learning surveys in tab. 1, and the taxonomy of general transfer learning fig. 3.
• Extends: Big Transfer (BiT), with its fine-tuning rules tested on less similar targets §3.4 tab. 3.
• Superseded by: not known from this wiki.