---
kind: concept
slug: weight-decay
title: Weight decay
summary: >-
  A penalty on the size of the weights, added to the training loss. Covers pulling weights toward zero on a backbone or classifier, alone and with MaxNorm in long-tailed classification, decay on each feature net of an additive tabular model, and a squared anchor to source weights in source-free adaptation. Numbers are author-reported.
tags: [regularisation, optimisation, long-tail, interpretability, domain-adaptation]
papers: ["2004.13912", "2203.14197", "2502.19316"]
related: [long-tailed-recognition, class-balanced-loss, domain-adaptation, interpretable-machine-learning, dropout]
status: draft
updated_at: 2026-10-24
---

## Overview

Weight decay adds a penalty on the size of the weights to the training
objective. The penalty grows with the weights, so training trades some fit for
smaller weights. The card's penalty for a classifier with one weight vector
$\theta_k$ per class is $\lambda \sum_k \lVert \theta_k \rVert_2^2$, added to
the cross-entropy loss [arxiv:2203.14197 eq. 3] [arxiv:2203.14197 §3.1]. The
authors say the penalty bites harder on large weights and keeps small ones
small [arxiv:2203.14197 §3.1] [arxiv:2203.14197 fig. 2c].

The shared core across all variants is the coefficient $\lambda$, the set of
parameters it applies to, and the point the penalty pulls toward. Decay pulls
toward zero. The adaptation variant below pulls toward a fixed source model.
Each choice changes the result, so a variant is defined by them. In "Long-Tailed
Recognition via Weight Balancing", weight decay is one of three
weight-balancing tools for the classifier [arxiv:2203.14197 §3.1]. In "Model
Adaptation", a squared-distance term to the source weights plays a similar role
[arxiv:2502.19316 §3.2] [arxiv:2502.19316 eq. 6]. In "Neural Additive Models",
the same kind of L2 term is applied to each per-feature network, averaged over
the networks [arxiv:2004.13912 app. A.3] [arxiv:2004.13912 eq. 3].

## Variants

### Whole-network weight decay in the first stage

Stage 1 trains the backbone and linear head with cross-entropy and a single
$\lambda$ on all network parameters. The authors say tuning $\lambda$ is enough
to get a good backbone [arxiv:2203.14197 §3.3]. On CIFAR100-LT with imbalance
factor 100, tuned weight decay raises class-averaged accuracy from 38.38% for
training with no decay to 46.08% [arxiv:2203.14197 tab. 2]. Card:
[Long-Tailed Recognition via Weight Balancing](/papers/2203.14197).

### Classifier-only weight decay in the second stage

Stage 2 freezes the features and retrains only the classifier weight and bias.
Weight decay is applied there together with a class-balanced loss and a MaxNorm
projection [arxiv:2203.14197 §3.3]. In the paper's ablation, adding decay to
the class-balanced loss raises accuracy from 47.09% to 52.42%, and adding the
MaxNorm projection on top gives 53.35% [arxiv:2203.14197 tab. 1]. The paper
does not report the second-stage $\lambda$ value.

### Weight decay as a step toward MaxNorm

The paper also reads weight decay as one step in solving a MaxNorm problem
written with a Lagrange multiplier [arxiv:2203.14197 §3.2] [arxiv:2203.14197
eq. 6]. In its experiments, decay and MaxNorm together beat either alone
[arxiv:2203.14197 §3.2]. The main table lists decay plus MaxNorm at 50.24%
[arxiv:2203.14197 tab. 2]. Adding the class-balanced loss and decay on top
gives 53.35% [arxiv:2203.14197 tab. 1].

### Anchor to source weights in source-free adaptation

When a source-trained classifier is adapted without source data, the loss
$\ell_{wReg}$ is the squared distance between the current classifier weights
and the frozen source weights. It keeps the classifier near the source model
[arxiv:2502.19316 §3.2] [arxiv:2502.19316 eq. 6]. If the source weights are set
to zero, the term becomes ordinary L2 weight decay [arxiv:2502.19316 §3.2]. The
paper uses $\lambda_w = 10^{-4}$ [arxiv:2502.19316 §4.1]. In its digit
ablation, which uses a small LeNet-style classifier, adding this term to the
generated-sample loss raises accuracy from 97.9% to 98.4% on SVHN → MNIST, 94.5%
to 95.4% on MNIST → USPS and 91.8% to 94.2% on MNIST → MNIST-M. USPS → MNIST goes
from 98.2% to 98.3% [arxiv:2502.19316 tab. 4]. The ablation covers digits only.
Card: [Model Adaptation: Unsupervised Domain Adaptation without Source
Data](/papers/2502.19316).

### Per-feature decay in additive tabular models

A neural additive model gives each input column its own small network and sums
their outputs. Its training loss has three terms: the task loss, an output
penalty on the feature-net outputs, and a weight decay on the feature-net
weights, each with its own coefficient [arxiv:2004.13912 app. A.3]
[arxiv:2004.13912 eq. 3]. The decay is the mean over the feature nets of their
L2 penalty, so it does not grow with the number of columns
[arxiv:2004.13912 app. A.3]. The coefficient $\lambda_2$ is searched in
$[10^{-6}, 10^{-4})$ on each cross-validation fold with Bayesian optimisation
[arxiv:2004.13912 app. A.5] [arxiv:2004.13912 app. A.6]. The tuned values
range from 0.0 on COMPAS to $9.6 \times 10^{-5}$ on MIMIC-II, and sit at
$10^{-6}$ on the housing data [arxiv:2004.13912 app. A.6]. The results the card
records do not compare runs with and without this term. Card:
[Neural Additive Models: Interpretable Machine Learning with Neural Nets](/papers/2004.13912).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Training a long-tailed classifier from scratch with a backbone | Tuned whole-network decay in stage 1 | 38.38% to 46.08% over no decay on CIFAR100-LT, IF 100 [arxiv:2203.14197 tab. 2] |
| You can afford a second pass over the last layer | Decay on the classifier in stage 2, with a class-balanced loss and MaxNorm | Best reported number, 53.35% [arxiv:2203.14197 tab. 1] |
| You cannot retrain anything after stage 1 | A post-hoc rescaling of the classifier, not decay | On the tuned model, the post-hoc tau-norm reaches 51.31%, above decay with MaxNorm alone (50.24%) [arxiv:2203.14197 tab. 1] [arxiv:2203.14197 tab. 2]. The paper's tau-norm is a separate step, see [Long-Tailed Recognition](/concepts/long-tailed-recognition) |
| Large datasets (ImageNet-LT, iNaturalist2018) | Decay plus MaxNorm, checked per dataset | Best of the paper's own variants at 53.9% and 70.2% [arxiv:2203.14197 tab. 3] |
| Choosing $\lambda$ for a new dataset | Tune it on a held-out split | The paper says the best $\lambda$ varies with dataset size and that large values destabilise training [arxiv:2203.14197 §3.2] [arxiv:2203.14197 §4.2] [arxiv:2203.14197 fig. 4] |
| Adapting a source-trained classifier with no source data, and you want to stay near its weights | Squared anchor to the source weights, with a tuned $\lambda_w$ | Gains of 0.1 to 2.4 points over the generated-sample loss alone in the digit ablation, not on Office-31 or VisDA17 [arxiv:2502.19316 tab. 4] |
| A tabular model where each input needs its own readable curve | Decay on each feature net, with $\lambda_2$ tuned per dataset in $[10^{-6}, 10^{-4})$ | Tuned values differ by dataset, from 0.0 to $9.6 \times 10^{-5}$ [arxiv:2004.13912 app. A.6] |

Post-hoc L2-normalisation is a related weight-balancing step, but it is not weight
decay. It helped on CIFAR100-LT, yet on iNaturalist it dropped from 65.4% to
51.3% when added to decay [arxiv:2203.14197 tab. 3]. Check it per dataset
before using it.

## Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants
differ. The first block is the whole-network decay of stage 1. The second is
the classifier retraining of stage 2, which adds the MaxNorm projection. Both
use SGD with momentum; the paper does not state the stage-2 learning rate.

```pseudocode
# Stage 1: decay on every parameter, plain cross-entropy  [arxiv:2203.14197 §4.1]
# lam is tuned; the paper gives no general value  [arxiv:2203.14197 §4.2]
def train_backbone(model, train_set, lam, epochs):
    optimizer = sgd(model.parameters(), lr=0.01, momentum=0.9,
                    weight_decay=lam)
    for epoch in range(epochs):
        for images, labels in train_set:
            loss = cross_entropy(model(images), labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
    return model
```

```pseudocode
# Stage 2: decay on the classifier only, then MaxNorm projection  [arxiv:2203.14197 §3.3]
# Freeze everything but the classifier weight and bias  [arxiv:2203.14197 §3.3]
def retrain_classifier(model, train_set, lam, radius, epochs):
    optimizer = sgd([model.fc.weight, model.fc.bias], lr=0.01,
                    momentum=0.9, weight_decay=lam)
    for epoch in range(epochs):
        for images, labels in train_set:
            loss = class_balanced_loss(model(images), labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        # Rescale any class row above the radius  [arxiv:2203.14197 eq. 5]
        model.fc.weight.data = project_max_norm(model.fc.weight.data, radius)
    return model
```

Where variants differ:

- **Decay scope.** Stage 1 applies decay to all network parameters; stage 2
  applies it to the classifier. The paper does not state which layers are
  trainable in stage 2, so the exact scope there is a release detail
  [arxiv:2203.14197 §3.3].
- **Scale of $\lambda$.** The optimiser argument is not the same quantity as
  the penalty in eq. 3; see Contradictions.
- **MaxNorm timing and radius.** The paper projects after each batch; the
  released code projects once per epoch. The radius is a single value the paper
  leaves unspecified [arxiv:2203.14197 eq. 4] [arxiv:2203.14197 app. A].
- **Reference point.** Decay pulls weights toward zero. The anchor pulls them
  toward a frozen copy of the source weights, taken before adaptation
  [arxiv:2502.19316 §3.2] [arxiv:2502.19316 alg. 1]. The two terms have the same
  form when the copy is zero, so one implementation can serve both.
- **Averaging over sub-networks.** In the additive model, the L2 sum is divided
  by the number of feature nets [arxiv:2004.13912 app. A.3]. Each net's pull is
  therefore scaled by the column count, so a $\lambda_2$ tuned on one dataset
  does not transfer directly to one with a different number of columns.

## Contradictions

- **Penalty scale.** The paper's penalty in eq. 3 is $\lambda \sum_k
  \lVert \theta_k \rVert_2^2$. The card states that PyTorch's `weight_decay`
  adds $\text{weight\_decay} \cdot w$ to the gradient, and that the paper's
  $\lambda$ is therefore about half the code's value for the same objective.
  This is the card's inference from the optimiser's documented behaviour, not
  something the repository states [arxiv:2203.14197 eq. 3] [arxiv:2203.14197
  §3.1]. Check the convention before copying a $\lambda$ between the two.
- **Released value vs paper value.** The paper gives no stage-1 $\lambda$ for
  CIFAR100-LT; the release uses $5\times10^{-3}$ [arxiv:2203.14197 fig. 4]. The
  stage-2 value is "tuned" in the paper and 0.1 in the release [arxiv:2203.14197
  §3.3] [arxiv:2203.14197 tab. 2].
- **Reproduced numbers.** The paper reports 46.08% for decay alone on CIFAR100-LT,
  IF 100 [arxiv:2203.14197 tab. 2]. The release's own logs record 47.18%
  best-epoch accuracy for that setting. Its stage-2 log records 51.98%
  best-epoch and 51.83% final, against 53.35% in the paper
  [arxiv:2203.14197 tab. 1]. The release's logs are not part of the paper, so
  these figures are from the release only.
- **Label conflict in the paper.** The main text gives decay alone as 46.09% in
  one place and 46.08% in the tables [arxiv:2203.14197 §4.3] [arxiv:2203.14197
  tab. 2]. The "+" labels in the main table also disagree with the caption and
  the ablation [arxiv:2203.14197 tab. 2] [arxiv:2203.14197 tab. 1].
- **Which parameters decay, in the additive model.** The paper applies the
  penalty to the feature-net weights [arxiv:2004.13912 app. A.3]. The released
  code applies it to all trainable variables, including the centres of the
  ExU units and the output bias [arxiv:2004.13912 app. A.3]. The code uses
  TensorFlow's L2 loss, which is half the sum of squares, and the paper does
  not say which convention its $\lambda_2$ uses. Match the scope and the scale
  to the reference you are reproducing.

## Pitfalls

- **Selecting on the test split.** The released training loop picks the best
  epoch on the evaluation loader, so the reported best-epoch numbers are
  selected on that split [arxiv:2203.14197 §4.1]. Budget for lower accuracy when
  you select on a held-out split.
- **No general $\lambda$.** The paper gives no value for the general case, and
  the released value applies only to CIFAR100-LT with imbalance factor 100
  [arxiv:2203.14197 §3.2] [arxiv:2203.14197 fig. 4].
- **Decay on normalisation layers.** The paper says decay covers all network
  parameters in stage 1 [arxiv:2203.14197 §3.3]. The release applies it to
  normalisation layers too, which the paper does not mention. Check whether your
  setup should exclude them.
- **Anchor is not decay.** Copying a decay coefficient from the long-tail
  setup into an adaptation setup changes the reference point, not just the
  strength. The paper's $\lambda_w$ is for its own digit and Office-31 runs
  and is not stated as a general default [arxiv:2502.19316 §4.1]. The paper
  also leaves the start of adaptation and whether the anchor is delayed
  unspecified [arxiv:2502.19316 §3.4] [arxiv:2502.19316 alg. 1].
- **Balanced evaluation is assumed.** The paper evaluates on balanced test and
  validation sets and does not discuss other test distributions
  [arxiv:2203.14197 §4.1].
- **Tuned value at the search edge.** The housing value of $10^{-6}$ is the
  lower bound of the searched range [arxiv:2004.13912 app. A.6]. The optimum
  may lie outside that range, so widen the search before you rely on the value.

## Lineage

- Used as a component in the paper's stage-2 pipeline, alongside the
  class-balanced loss [arxiv:2203.14197 §3.3]. See [Class-balanced loss](/concepts/class-balanced-loss).
- The paper's motivation is that the classifier norms grow for common classes
  [arxiv:2203.14197 §1] [arxiv:2203.14197 fig. 1].
- The source-weight anchor in "Model Adaptation" is credited to two
  separate-model adaptation methods, Beyond Sharing Weights and ADDA
  [arxiv:2502.19316 §3.2]. Neither has a card here, so they are named only.
  See [Domain adaptation](/concepts/domain-adaptation).
- In the additive-model paper, decay is one of three regularisers on each
  feature net, with an output penalty and feature dropout [arxiv:2004.13912
  app. A.3]. See [Interpretable machine learning](/concepts/interpretable-machine-learning)
  and [Dropout](/concepts/dropout).

## Papers

- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](/papers/2004.13912):
  puts an L2 penalty on each per-feature network's weights, averaged over the
  networks, in a tabular model that stays readable per input column.
- [Long-Tailed Recognition via Weight Balancing](/papers/2203.14197): uses weight
  decay on the backbone and on the classifier, and combines it with a MaxNorm
  projection on long-tailed image classification.
- [Model Adaptation: Unsupervised Domain Adaptation without Source Data](/papers/2502.19316):
  anchors the classifier to its source weights with a squared-distance term
  during source-free adaptation, and notes this reduces to L2 decay when the
  anchor is zero.