Weight decay

Overview

Weight decay adds a penalty on the size of the weights to the training objective. The penalty grows with the weights, so training trades some fit for smaller weights. The card's penalty for a classifier with one weight vector \theta_k per class is \lambda \sum_k \lVert \theta_k \rVert_2^2, added to the cross-entropy loss 2203.14197 eq. 3 2203.14197 §3.1. The authors say the penalty bites harder on large weights and keeps small ones small 2203.14197 §3.1 2203.14197 fig. 2c.

The shared core across all variants is the coefficient \lambda, the set of parameters it applies to, and the point the penalty pulls toward. Decay pulls toward zero. The adaptation variant below pulls toward a fixed source model. Each choice changes the result, so a variant is defined by them. In "Long-Tailed Recognition via Weight Balancing", weight decay is one of three weight-balancing tools for the classifier 2203.14197 §3.1. In "Model Adaptation", a squared-distance term to the source weights plays a similar role 2502.19316 §3.2 2502.19316 eq. 6. In "Neural Additive Models", the same kind of L2 term is applied to each per-feature network, averaged over the networks 2004.13912 app. A.3 2004.13912 eq. 3.

Variants

Whole-network weight decay in the first stage

Stage 1 trains the backbone and linear head with cross-entropy and a single \lambda on all network parameters. The authors say tuning \lambda is enough to get a good backbone 2203.14197 §3.3. On CIFAR100-LT with imbalance factor 100, tuned weight decay raises class-averaged accuracy from 38.38% for training with no decay to 46.08% 2203.14197 tab. 2. Card: Long-Tailed Recognition via Weight Balancing.

Classifier-only weight decay in the second stage

Stage 2 freezes the features and retrains only the classifier weight and bias. Weight decay is applied there together with a class-balanced loss and a MaxNorm projection 2203.14197 §3.3. In the paper's ablation, adding decay to the class-balanced loss raises accuracy from 47.09% to 52.42%, and adding the MaxNorm projection on top gives 53.35% 2203.14197 tab. 1. The paper does not report the second-stage \lambda value.

Weight decay as a step toward MaxNorm

The paper also reads weight decay as one step in solving a MaxNorm problem written with a Lagrange multiplier 2203.14197 §3.2 [arxiv:2203.14197 eq. 6]. In its experiments, decay and MaxNorm together beat either alone 2203.14197 §3.2. The main table lists decay plus MaxNorm at 50.24% 2203.14197 tab. 2. Adding the class-balanced loss and decay on top gives 53.35% 2203.14197 tab. 1.

Anchor to source weights in source-free adaptation

When a source-trained classifier is adapted without source data, the loss \ell_{wReg} is the squared distance between the current classifier weights and the frozen source weights. It keeps the classifier near the source model 2502.19316 §3.2 2502.19316 eq. 6. If the source weights are set to zero, the term becomes ordinary L2 weight decay 2502.19316 §3.2. The paper uses \lambda_w = 10^{-4} 2502.19316 §4.1. In its digit ablation, which uses a small LeNet-style classifier, adding this term to the generated-sample loss raises accuracy from 97.9% to 98.4% on SVHN → MNIST, 94.5% to 95.4% on MNIST → USPS and 91.8% to 94.2% on MNIST → MNIST-M. USPS → MNIST goes from 98.2% to 98.3% 2502.19316 tab. 4. The ablation covers digits only. Card: Model Adaptation: Unsupervised Domain Adaptation without Source Data.

Per-feature decay in additive tabular models

A neural additive model gives each input column its own small network and sums their outputs. Its training loss has three terms: the task loss, an output penalty on the feature-net outputs, and a weight decay on the feature-net weights, each with its own coefficient 2004.13912 app. A.3 2004.13912 eq. 3. The decay is the mean over the feature nets of their L2 penalty, so it does not grow with the number of columns 2004.13912 app. A.3. The coefficient \lambda_2 is searched in [10^{-6}, 10^{-4}) on each cross-validation fold with Bayesian optimisation 2004.13912 app. A.5 2004.13912 app. A.6. The tuned values range from 0.0 on COMPAS to 9.6 \times 10^{-5} on MIMIC-II, and sit at 10^{-6} on the housing data 2004.13912 app. A.6. The results the card records do not compare runs with and without this term. Card: Neural Additive Models: Interpretable Machine Learning with Neural Nets.

When to use which

Post-hoc L2-normalisation is a related weight-balancing step, but it is not weight decay. It helped on CIFAR100-LT, yet on iNaturalist it dropped from 65.4% to 51.3% when added to decay 2203.14197 tab. 3. Check it per dataset before using it.

Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants differ. The first block is the whole-network decay of stage 1. The second is the classifier retraining of stage 2, which adds the MaxNorm projection. Both use SGD with momentum; the paper does not state the stage-2 learning rate.

# Stage 1: decay on every parameter, plain cross-entropy  [arxiv:2203.14197 §4.1]
# lam is tuned; the paper gives no general value  [arxiv:2203.14197 §4.2]
def train_backbone(model, train_set, lam, epochs):
    optimizer = sgd(model.parameters(), lr=0.01, momentum=0.9,
                    weight_decay=lam)
    for epoch in range(epochs):
        for images, labels in train_set:
            loss = cross_entropy(model(images), labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
    return model

# Stage 2: decay on the classifier only, then MaxNorm projection  [arxiv:2203.14197 §3.3]
# Freeze everything but the classifier weight and bias  [arxiv:2203.14197 §3.3]
def retrain_classifier(model, train_set, lam, radius, epochs):
    optimizer = sgd([model.fc.weight, model.fc.bias], lr=0.01,
                    momentum=0.9, weight_decay=lam)
    for epoch in range(epochs):
        for images, labels in train_set:
            loss = class_balanced_loss(model(images), labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        # Rescale any class row above the radius  [arxiv:2203.14197 eq. 5]
        model.fc.weight.data = project_max_norm(model.fc.weight.data, radius)
    return model

Where variants differ:

• Decay scope. Stage 1 applies decay to all network parameters; stage 2 applies it to the classifier. The paper does not state which layers are trainable in stage 2, so the exact scope there is a release detail 2203.14197 §3.3.
• Scale of \lambda. The optimiser argument is not the same quantity as the penalty in eq. 3; see Contradictions.
• MaxNorm timing and radius. The paper projects after each batch; the released code projects once per epoch. The radius is a single value the paper leaves unspecified 2203.14197 eq. 4 2203.14197 app. A.
• Reference point. Decay pulls weights toward zero. The anchor pulls them toward a frozen copy of the source weights, taken before adaptation 2502.19316 §3.2 2502.19316 alg. 1. The two terms have the same form when the copy is zero, so one implementation can serve both.
• Averaging over sub-networks. In the additive model, the L2 sum is divided by the number of feature nets 2004.13912 app. A.3. Each net's pull is therefore scaled by the column count, so a \lambda_2 tuned on one dataset does not transfer directly to one with a different number of columns.

Contradictions

• Penalty scale. The paper's penalty in eq. 3 is \lambda \sum_k \lVert \theta_k \rVert_2^2. The card states that PyTorch's weight_decay adds \text{weight\_decay} \cdot w to the gradient, and that the paper's \lambda is therefore about half the code's value for the same objective. This is the card's inference from the optimiser's documented behaviour, not something the repository states 2203.14197 eq. 3 [arxiv:2203.14197 §3.1]. Check the convention before copying a \lambda between the two.
• Released value vs paper value. The paper gives no stage-1 \lambda for CIFAR100-LT; the release uses 5\times10^{-3} 2203.14197 fig. 4. The stage-2 value is "tuned" in the paper and 0.1 in the release [arxiv:2203.14197 §3.3] 2203.14197 tab. 2.
• Reproduced numbers. The paper reports 46.08% for decay alone on CIFAR100-LT, IF 100 2203.14197 tab. 2. The release's own logs record 47.18% best-epoch accuracy for that setting. Its stage-2 log records 51.98% best-epoch and 51.83% final, against 53.35% in the paper 2203.14197 tab. 1. The release's logs are not part of the paper, so these figures are from the release only.
• Label conflict in the paper. The main text gives decay alone as 46.09% in one place and 46.08% in the tables 2203.14197 §4.3 [arxiv:2203.14197 tab. 2]. The "+" labels in the main table also disagree with the caption and the ablation 2203.14197 tab. 2 2203.14197 tab. 1.
• Which parameters decay, in the additive model. The paper applies the penalty to the feature-net weights 2004.13912 app. A.3. The released code applies it to all trainable variables, including the centres of the ExU units and the output bias 2004.13912 app. A.3. The code uses TensorFlow's L2 loss, which is half the sum of squares, and the paper does not say which convention its \lambda_2 uses. Match the scope and the scale to the reference you are reproducing.

Pitfalls

• Selecting on the test split. The released training loop picks the best epoch on the evaluation loader, so the reported best-epoch numbers are selected on that split 2203.14197 §4.1. Budget for lower accuracy when you select on a held-out split.
• No general \lambda. The paper gives no value for the general case, and the released value applies only to CIFAR100-LT with imbalance factor 100 2203.14197 §3.2 2203.14197 fig. 4.
• Decay on normalisation layers. The paper says decay covers all network parameters in stage 1 2203.14197 §3.3. The release applies it to normalisation layers too, which the paper does not mention. Check whether your setup should exclude them.
• Anchor is not decay. Copying a decay coefficient from the long-tail setup into an adaptation setup changes the reference point, not just the strength. The paper's \lambda_w is for its own digit and Office-31 runs and is not stated as a general default 2502.19316 §4.1. The paper also leaves the start of adaptation and whether the anchor is delayed unspecified 2502.19316 §3.4 2502.19316 alg. 1.
• Balanced evaluation is assumed. The paper evaluates on balanced test and validation sets and does not discuss other test distributions 2203.14197 §4.1.
• Tuned value at the search edge. The housing value of 10^{-6} is the lower bound of the searched range 2004.13912 app. A.6. The optimum may lie outside that range, so widen the search before you rely on the value.

Lineage

• Used as a component in the paper's stage-2 pipeline, alongside the class-balanced loss 2203.14197 §3.3. See Class-balanced loss.
• The paper's motivation is that the classifier norms grow for common classes 2203.14197 §1 2203.14197 fig. 1.
• The source-weight anchor in "Model Adaptation" is credited to two separate-model adaptation methods, Beyond Sharing Weights and ADDA 2502.19316 §3.2. Neither has a card here, so they are named only. See Domain adaptation.
• In the additive-model paper, decay is one of three regularisers on each feature net, with an output penalty and feature dropout [arxiv:2004.13912 app. A.3]. See Interpretable machine learning and Dropout.

Papers

• Neural Additive Models: Interpretable Machine Learning with Neural Nets: puts an L2 penalty on each per-feature network's weights, averaged over the networks, in a tabular model that stays readable per input column.
• Long-Tailed Recognition via Weight Balancing: uses weight decay on the backbone and on the classifier, and combines it with a MaxNorm projection on long-tailed image classification.
• Model Adaptation: Unsupervised Domain Adaptation without Source Data: anchors the classifier to its source weights with a squared-distance term during source-free adaptation, and notes this reduces to L2 decay when the anchor is zero.