A Transfer Learning Evaluation of Deep Neural Networks for Image Classification

In plain words

Many image classifiers start from a model that was already trained on a large public image collection, and then get adapted to a new task. This paper asks which of eleven such pre-trained models is the best starting point. The authors adapt each one to five small image sets, such as photos of ants and bees, of handwritten digits, and of smartphone backs.

They try two ways of adapting. In one, only the final classification layer is retrained and everything else stays frozen. In the other, every layer is retrained. They also compare a short training run, where each image is seen once, with a longer one, where each image is seen ten times. For each model they record accuracy, training time and model size, and they combine accuracy with model size into a single "accuracy density" score.

The paper's answer is a set of rankings rather than one winner. Under full retraining, SqueezeNet scores best on accuracy density. When only the classifier is retrained, ResNet-18 scores best on that density. The authors add that the best choice depends on the constraints of the project, such as accuracy, training time or memory.

TL;DR

• Eleven ImageNet-pretrained PyTorch CNNs are adapted to five image datasets, in two tuning modes and two episode settings §1 §4 §5.
• The headline score is accuracy divided by parameter count eq. 1 §4.1.
• Author-reported: SqueezeNet has the highest average accuracy density under full tuning, and ResNet-18 the highest under classifier-only tuning §5.1.1 §5.1.2 fig. 3.
• The text gives no numeric accuracy, density or time table. All values appear only in plotted figures, several without value labels fig. 3 fig. 4 fig. 5 fig. 6.
• No official code was found. Several setup details (epochs, splits, schedule) are not stated, so exact reproduction is not possible from the paper alone §4.

Problem

Choosing a pre-trained model for transfer learning is hard, because the usual comparisons rank models on ImageNet accuracy alone. The authors argue that target-task accuracy, training time, memory and model size also matter, and that earlier benchmarks either used reported literature values or ImageNet only §1. The paper asks which pre-trained model fits a target task, under two adaptation strategies and two training-length settings §1 §4 §5.

Method

Models. The study uses eleven PyTorch vision models, from AlexNet (2012) to MnasNet (2019). GoogLeNet is called Inception-V1 in the implementation section §4 tab. 1. The weights are the ImageNet-pretrained ones shipped with the PyTorch vision models §4. The paper gives depth and the main design idea of each model in a table; the MnasNet depth entry says it has no linear sequence tab. 1.

Datasets. Five datasets are used §3.3.

• CIFAR-10, described as 60K images §3.3.
• MNIST, described as 70K images §3.3.
• Hymenoptera (ants and bees): 245 training and 153 test images §3.3.1.
• Smartphones: 654 RGB images in twelve classes, from six brands, collected from search engines with a focus on the back of the phone §3.3.2 fig. 2.
• Augmented smartphones: each original image is rotated by 45, 135, 225 or 315 degrees, and noise is added at 10%, 25% or 50%; the augmented set has 8,502 images, including the 654 originals §3.3.3.

Tuning modes. Both modes start from the same ImageNet weights §4.

• Classifier-only: the feature-extraction layers are frozen, and the task-specific classifier is re-initialised with random values. For models without an explicit classifier part (the paper names ResNet-18), only the last fully connected layer is trained §4.
• Full: every parameter is trained §4.

Training. Each model is trained on each dataset with Adam, a batch size of 10, and a grid over learning rates {1e-2, 1e-3, 1e-4} §4. Early stopping keeps the weights from the epoch with the best validation accuracy, and a later epoch with lower validation accuracy does not replace them §4. The plots show only the top-scoring learning rate for each model §4.

Episodes. In the one-episode setting each sample is presented once. In the ten-episode setting each sample is presented ten times §5. Section 5.2 also says that ten independent trials were run and averaged §5.2. See Pitfalls.

Scores.

• Accuracy density is accuracy divided by the number of parameters eq. 1 §4.1. A higher value means more accuracy per parameter §4.1.
• Training time is measured in seconds on one GPU, and model size as the number of learning parameters in MB §4.2.

# Fine-tune one model in one tuning mode  [§4]
# Input:  pretrained (ImageNet weights); mode; learning_rate; train, val sets
# Output: best model by validation accuracy

def finetune(pretrained, mode, learning_rate, train, val):
    model = copy(pretrained)
    # Classifier-only: freeze all weights, then re-initialise the classifier
    # with random values; the initialisation scheme is not stated  [§4]
    if mode == "classifier_only":
        freeze_all_parameters(model)
        reinitialise_random(classifier_part(model))
    # Full tuning keeps the ImageNet weights and trains every layer  [§4]
    optimizer = Adam(trainable_parameters(model), learning_rate)
    best_model = None
    best_val_accuracy = minus_infinity
    # Epoch count, patience and the LR schedule are not stated  [§4]
    for each epoch:
        train_one_epoch(model, train, optimizer, batch_size=10)
        val_accuracy = evaluate(model, val)
        # Keep weights only when validation accuracy rises  [§4]
        if val_accuracy > best_val_accuracy:
            best_val_accuracy = val_accuracy
            best_model = copy(model)
    return best_model

# Accuracy density  [eq. 1] [§4.1]
# Input:  accuracy in percent [fig. 3]; num_params (count, see Pitfalls)
# Output: accuracy per parameter

def accuracy_density(accuracy, num_params):
    # Higher means more accuracy per parameter  [§4.1]
    return accuracy / num_params

# Smartphone augmentation: 4 rotations x 3 noise levels  [§3.3.3]
# Input:  image (H, W, 3)
# Output: 12 augmented images

def augment_smartphone(image):
    # Rotations of 45, 135, 225 and 315 degrees  [§3.3.3]
    for angle in [45, 135, 225, 315]:
        rotated = rotate(image, angle)
        for percent in [0.10, 0.25, 0.50]:
            # Noise values drawn from a discrete uniform on {0, ..., 255 * p}
            # Whether they are added to or replace pixels is not stated
            # [§3.3.3]
            noise = uniform_int(0, 255 * percent, shape=image.shape)
            yield add_noise(rotated, noise)

The paper has one numbered equation (density) eq. 1. It has one numbered table, which describes the models and not the results tab. 1. Its algorithms are given in prose only; no algorithm box is present.

Hyperparameters

Results

Author-reported. Setup: eleven pre-trained PyTorch models, five datasets, two tuning modes, two episode settings, one GPU §4 §5. The paper reports averages over datasets and, for ten episodes, over ten trials §5.1 §5.2. No error bars or ranges are shown in the plotted averages fig. 3.

The text gives qualitative rankings only. Absolute accuracies, densities and training times are not in the text.

Figure reading, not numeric values. In fig. 3, the full-tuning panel lists VGG-16 with the lowest density and SqueezeNet with the highest, and the classifier-only panel shows ResNet-18 highest. In the classifier-only panel, no bars are drawn for AlexNet or VGG-16; the text does not explain this fig. 3. The two panels use different axis scales fig. 3. The per-task scatter plots of accuracy against training time and model size are in fig. 5, fig. 6 and app. C and app. D. This card does not read values from them.

Paper vs code

No official code.

The paper has no code or data link. Its data availability statement says no new data were created §6. The search covered:

• the paper text and the arXiv abstract page, which lists no code or project page for this work;
• GitHub repository search for Zengeler, transfer learning smartphone recycling, Abou Baker transfer and Handmann transfer learning, all of which returned no results.

Pitfalls

• No numeric results in the text. All accuracy, density and time values appear only in plotted figures, and the bars in fig. 3 carry no value labels. Implementers cannot compare their numbers with exact values from the paper. The card has not read values from the plots.
• "Ten episodes" is defined two ways. Section 5 says each sample is shown ten times §5. Section 5.2 says ten independent trials were run and averaged §5.2. The paper does not say whether both hold at once.
• Epoch count, patience and schedule are not stated. Early stopping is described but its patience and the number of epochs are not §4. The one-episode setting shows each sample once §5, so early stopping on a single pass is unclear.
• Validation and test sets are not defined for CIFAR-10, MNIST and the smartphone sets §3.3. The paper does not say whether the reported accuracy is on the validation or the test split. Early stopping uses validation accuracy §4.
• Learning-rate selection. The plots show only the top learning rate per model §4. The text does not say whether that rate was chosen per dataset, per model, or on validation or test data.
• Parameter count in accuracy density. Eq. 1 uses "#parameters" eq. 1. The paper does not say whether this counts all parameters or only the trainable ones. In classifier-only mode these differ, and the panels of fig. 3 are on different scales.
• Units. The paper's section on model size says "number of learning parameters in MB" §4.2. It does not say whether MB means a parameter count or bytes, or at what precision.
• Which layer is the classifier. Only ResNet-18's last fully connected layer is named §4. For the other models the paper refers to "the PyTorch vision model implementations" §4. Models such as SqueezeNet are not described here.
• Augmentation details. The noise sentence says pixels are added "from a discrete uniform distribution {0 … 255·p}" §3.3.3. It does not say whether this is added to or replaces each pixel. The augmented count is 8,502 images, which equals 654 originals plus 12 augmentations of each (7,848), so the total is 13 times the original set. The text says the count is "multiplied by 12" §3.3.3. It also does not say whether augmentation happens before or after the train/validation split. If before, copies of the same photo can land in both.
• Ambiguous wording of the classifier-only mode. The related-work section says classifier-only fine-tuning "keeps the entire model's weight constant, excluding the last layer" §2.4. The implementation section is the operative definition §4.
• Small test sets. Hymenoptera has 153 test images §3.3.1. The paper reports no variance, and the ten-trial average is the only repetition §5.2. Differences between models of a few points may be within noise.
• Hardware-bound timings. Training times come from one GTX 1080 Ti with a batch size of 10 §4. Ranking by time on other hardware may differ.
• Conclusion and results wording. Several conclusions name a model for a claim that the text does not tie to a figure, such as AlexNet for "the shortest training time" §6. Check the scatter figures before relying on them.
• Missing AlexNet and VGG-16 bars. The classifier-only panel in fig. 3 has no bars for these two models, and the text does not explain why fig. 3.
• Model naming. The implementation section uses "Inception-V1 (GoogLeNet)" §4, and the tables use "GoogLeNet" tab. 1. These are the same model in the text.

Lineage

• Builds on: ImageNet-pretrained PyTorch vision models §4; the transfer learning by fine-tuning and the classifier-only split as described in §2.4 and §4.
• Compared against: the model benchmarks of Canziani et al. (2016) and Bianco et al. (2018), which the paper says covered ImageNet only §1.
• Uses an earlier dataset: the smartphone set comes from the authors' earlier e-waste recycling work §3.3.2.
• Future work: more evaluation metrics and the use of metadata to estimate transfer-learning hyperparameters §6.
• Superseded: not stated.