Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

In plain words

A common way to reuse a pretrained image model is to put a new output layer on top and then train the whole network on your task. This works well on data that looks like your training data. The paper finds that on data that looks quite different, the same recipe can do worse than a cheaper option: freeze the pretrained network and train only the new output layer.

The reason is that when the new layer starts out random, its early errors push large changes into the layers underneath, and those changes damage the general features the pretrained model had learned. The fix the authors propose is a two-step recipe. First train only the output layer, then train everything. The first step gives the second a sensible starting point, so the features move less. In their tests this gives the best average accuracy of the three methods, both on the data they were trained on and on the shifted data.

The theory is worked out for a very simple linear model. The large-scale experiments are the evidence that the effect shows up in practice.

TL;DR

Full fine-tuning of a pretrained network gives higher in-distribution (ID) accuracy than linear probing but lower out-of-distribution (OOD) accuracy when the pretrained features are good and the shift is large §4.1. The authors explain this by feature distortion: fine-tuning from a random head changes the features that training data touches, which hurts the directions OOD data uses §3.2. Linear probing then fine-tuning (LP-FT) gives the best average ID and OOD accuracy on the experiments run §4.2. Author-reported; the paper does not compare against other robust fine-tuning methods on the main benchmarks.

Problem

Fine-tuning (FT) updates all parameters starting from a pretrained network. It usually beats linear probing (LP), which freezes the feature extractor and trains only the head, on the same distribution as the training data §1 §4.1. Prior work on out-of-distribution accuracy usually fine-tuned all parameters, so the trade-off was not studied in depth §1. The paper asks under what conditions FT underperforms LP OOD §3. Its answer assumes good pretrained features (small distance between the pretrained and the ideal feature extractor) and a large distribution shift in which OOD inputs leave the span of the training inputs §3.1.

Theory assumptions: squared loss, a two-layer linear network with a linear feature extractor, overparameterised training (the training span has dimension m with 1 ≤ m < d − k), an invertible OOD second-moment matrix, and gradient flow (vanishing step size) §3.1. The analysis does not cover deep networks; those results are empirical §3.1 §4.

Method

Setting. A predictor is f(x) = v^T g_B(x), with a feature extractor g_B (parameters B) and a linear head v. LP fixes B at the pretrained value B_0 and fits v. FT starts from B_0 and updates both v and B by gradient descent §2.

Theoretical models. With linear g_B(x) = Bx, gradient flows are written for the training loss L̂(v, B) = ‖X B^T v − Y‖² eq. 3 eq. 4 §3.1. The head starts from a random Gaussian or zero initialisation; the paper's results hold for any such standard choice §3.1. The LP and FT limits are defined in eq. 5 eq. 6 §3.1.

Core observation 1 (feature distortion). The gradient with respect to B is 2 v (Y − X B v)^T X eq. 7 §3.2.1. It is zero along directions orthogonal to the training span, so features change only inside that span. Features in the span move while orthogonal features stay fixed §3.2.1.

Core observation 2. The lower bound on FT's OOD error is eq. 8 §3.2.2:

\sqrt{\mathcal{L}_{ood}(v_t, B_t)} \ge \sqrt{\sigma_{min}(\Sigma)} \Big( \frac{\angle(R_{init}, \mathcal{S}^\perp)}{\sqrt{k}} \frac{\min(\varphi, \varphi^2 / \|w^*\|_2)}{(1 + \|w^*\|_2)^2} - \epsilon \Big)

Here φ² = |(v_init^T v*)² − (v*^T v*)²| is the initial head alignment error, ε bounds the feature extractor error d(B_init, B*), and the bound holds at all times t when the principal angle between R_init and the training-span complement is positive §3.2.2 eq. 8. The bound grows with φ, so a random head start is what hurts §3.2.2.

Comparison with LP. Under non-degeneracy conditions, the ratio of LP's OOD loss to FT's OOD loss goes to 0 as B_init → B* eq. 10 §3.3. The ID comparison goes the other way: with imperfect features FT can fit the training data better than LP [Prop. 1] §3.3.

LP-FT (the method). Train the head by linear probing with the features frozen, then fine-tune all parameters starting from that head §4.2. In the theory, for perfect features, FT with a random head distorts features, but initialising the head at the linear-probe solution gives zero OOD error for all times t [Prop. 2] eq. 11 eq. 12 §3.4. The authors state that the analysis for imperfect features is not closed-form and that LP-FT is a first step rather than a state-of-the-art method §3.4 §1 app. B.

# LP-FT: linear probe first, then fine-tune all layers  [§4.2] [§3.4]
# Input:  pretrained network g (features g(x) in R^k), train set, val set
# Output: fine-tuned network f = head(g(.))

def lp_then_ft(network, train_set, val_set):
    # Stage 1: freeze the features and fit only the head  [§4.1]
    features = network.features(train_set.inputs)     # (N, k)
    head = fit_l2_logistic_regression(features, train_set.labels)
    # Choose the l2 strength by ID validation accuracy  [§4.1]
    head = select_by_val_accuracy(head, features, val_set)

    # Stage 2: fine-tune all parameters, starting from the probed head  [§4.2]
    # Learning rate chosen by ID validation accuracy  [§4.2]
    network.head = head
    fine_tuned = fine_tune_all_parameters(network, train_set)
    return fine_tuned

Hyperparameters

Results

Author-reported. Setup: 10 distribution shifts. Living-17, Entity-30, DomainNet and FMoW use ResNet-50 except ImageNet, which uses CLIP ViT-B/16 §4.1. Values are means over 3 runs with 90% confidence intervals in parentheses tab. 1 tab. 2.

ID accuracy tab. 1:

OOD accuracy tab. 2:

Headline statements, author-reported:

• FT beats LP ID by about 2 points on average (85.1 vs 82.9) and loses OOD by about 7 points (59.3 vs 66.2) §1 §4.1.
• LP beats FT on 8 of 10 OOD datasets; FT wins on CIFAR-10.1 and ImageNetV2 §4.1 tab. 2.
• LP-FT's average is 85.7 ID and 68.9 OOD §4.2 tab. 1 tab. 2.
• Feature change (Living-17, ID examples): mean feature distance before and after training is 0.0188 for FT and 0.0011 for LP-FT. The tables report these multiplied by 100 (1.88 and 0.11) app. B. LP-FT moves features far less than FT on every dataset with a feature-change table app. B.
• Early stopping on OOD data does not close the gap: LP averages 67.1 OOD and FT averages 61.3 §4.3 app. B.
• Effective robustness: LP and LP-FT sit above FT when OOD accuracy is plotted against ID accuracy app. B fig. 3.
• Feature-quality ablation: with an earlier MoCo-v1 checkpoint (ImageNet accuracy 10% lower), FT beats LP on Living-17 OOD (71% vs 66%) §4.3.
• ID ≈ OOD ablation on CIFAR-10 → CIFAR-10.1: FT 92.3% vs LP 82.7% OOD, LP-FT 93.5% §4.3.

ImageNet: the four shifts (ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch) are reported in tab. 2. LP and LP-FT were run without augmentation; the paper ran FT without augmentation as an ablation and found it worse ID and OOD app. B.

Paper vs code

The paper names no code link. The first author's public repository AnanyaKumar/transfer_learning is used, at commit 13c3e2bbb8a34bbb781e28cee96ddf20bed5f76f. Its README states it is the code for this paper README.md. The repository has no LICENSE file, so code_license is recorded as not stated. The code is not quoted here. All citations to it are for reading the logic.

• LP-FT: the second stage loads a linear-probe checkpoint as the starting head baseline_train.py. This matches the method description §4.2.
• Learning rate grid: the paper says six rates were swept for FT and LP-FT on non-ImageNet datasets §4.1. The default grid in the sweep script has ten values run_adaptation_experiments.py. The repository does not show which six were used in the paper.
• ImageNet: the FT sweep (1e-4, 3e-4, 1e-3), LP-FT second-stage sweep (1e-5, 3e-5, 1e-4) and LP sweep (0.01, 0.03, 0.1) match the appendix run_adaptation_experiments.py app. B.
• Schedule and epochs: CosineAnnealingLR with T_max equal to the epoch count living17.yaml [code: configs/adaptation/imagenet.yaml#L7-L18]. Epoch counts match app. B.
• Batch size: 64 by default resnet50_transfer.yaml. The ImageNet config uses 128 imagenet.yaml (batch size not read in detail; the paper states 128 app. B).
• Linear probing: a logistic regression over a log-spaced grid of inverse regularisation strengths (C) from 1e-7 to 1e2 with max_iter 200 log_reg_sk.py. The paper does not state this grid.
• Augmentation: RandomResizedCrop(224) with random horizontal flip appears as a training transform connectivity_utils.py. The excerpt read does not tie it to the ImageNet FT run, so the paper's statement app. B is not independently verified against code.
• Paper vs paper: see Pitfalls for numbers that disagree within the paper.

Pitfalls

• The theory is for linear models with gradient flow. The lower bound eq. 8 is for a two-layer linear net with infinitesimal step size. The authors say it does not cover neural tangent or deep networks §3.1 §3.2.2. Do not read it as a guarantee for a fine-tuned ResNet or ViT.
• The bound can vanish. It contains a −ε term that can cancel the positive term. The authors note that worse pretrained features may not make the bound worse §3.2.2. It also needs the principal angle between R_init and the training span complement to be positive. The paper proves that this angle is non-zero almost surely for random subspaces §3.3 app. A.
• Head initialisation drives the effect. The mechanism depends on a random or zero head. If the head started at the optimum, the authors say fine-tuning updates may not increase OOD error §3.2.2. In practice, a random head is standard §4.3.
• LP-FT is conditional. Gains require good pretrained features and a large shift. With weaker features (MoCo-v1) or a small shift (CIFAR-10 vs CIFAR-10.1), FT is better OOD than LP §4.3. The authors say LP-FT is not meant as state of the art §1.
• The "10/10 OOD" claim does not hold on Entity-30. The text says LP-FT is best on all ten OOD datasets §4.2, and the table caption says it matches or exceeds both methods on all ten tab. 2. In the table, LP is 63.2 on Entity-30 and LP-FT is 62.3, and LP is bolded there tab. 2. The difference is within the 90% intervals, but the claim as written is wrong.
• Abstract rounding. The abstract says LP-FT is "1% better ID". The averages give 85.7 − 85.1 = 0.6 §4.1 tab. 1. The OOD claim "10% better" matches 68.9 − 59.3 = 9.6 §4.1.
• Feature-change ratio disagrees between sections. The introduction says LP-FT changes features "10×–100× less" §1. The feature-change tables give roughly 17× for Living-17 (1.88 vs 0.11), 9× for FMoW (4.87 vs 0.57), and roughly 1000× for DomainNet (207.6 vs 0.19) app. B. Section 4.3 says "20×" for Living-17 §4.3; 0.0188 / 0.0011 is about 17×. Use the tables, not the prose ratios.
• Scratch baseline conflicts. The appendix text gives training from scratch as 89.3% ID on Living-17 app. B; the table reports 92.4% ID for the same setting app. B. Both are attributed to a prior paper. The text also gives FT OOD on Living-17 as 77.8% (the same as the main table), while another table gives 77.7% app. B.
• LP-FT's ImageNet ID result ties FT. Both score 81.7 tab. 1. The caption says LP-FT is best "on all except FMoW", which counts this as a win; the text counts 5 of 6 wins §4.2.
• Compute matching. LP-FT is compared with FT at similar cost: the linear stage is cheap. On ImageNet the stages are split 5 + 5 epochs against 10 epochs for FT app. B. On other datasets the paper says the extra compute is small without giving the numbers §4.2.
• ImageNet augmentation confound. LP and LP-FT on ImageNet use no augmentation, while FT uses augmentation. The paper ran FT without augmentation and it was worse, but the comparison is not perfectly matched app. B.
• Learning-rate selection uses ID data. Learning rate and early stopping are chosen on ID validation data, which is the protocol for all three methods §4.1. With OOD-based selection, the gap narrows but LP still wins §4.3.
• Unstated details for reimplementation. Optimizer, momentum, weight decay, warm-up, the exact six learning rates for the non-ImageNet sweep and the augmentations for non-ImageNet FT are not stated in the paper §4.1 app. B.
• Reproducing from code. The repository has no LICENSE file. Its paper claim is the authors' own README.md, but the code's license is not stated.

Lineage

• Builds on: linear probing and fine-tuning as standard transfer methods, with implicit regularisation of initialisation §1. Prior LP-FT use is cited to levine2016end and kanavati2021transfusion, layerwise fine-tuning to howard2018universal, and larger head learning rates to prabhu2021sentry, all not for robustness §1.
• Compared against: linear probing; full fine-tuning; a 10× learning rate on the head; L2-SP style regularisation toward pretrained weights (li2018explicit); side-tuning (zhang2020sidetuning); training from scratch app. B.
• Superseded by: not stated in this wiki.