Adaptive loss weighting

Overview

A training loss is often a sum of terms, or a mean over many points. A plain mean lets a small set of points with large errors go unnoticed while the total loss keeps falling 2307.00379 §2.2. Adaptive loss weighting replaces the plain mean, or a hand-set sum, with weights set by a rule. The rules covered here read one of three inputs:

• Per-point state built from residuals, for collocation points in a physics-informed network 2307.00379 §2.2 2403.18494 §2.5.
• Per-task training rates for several losses summed in one network. The rule reads either gradient norms on a shared layer 2606.02979 §3.3, or how much each loss has fallen over the last epochs 2303.14968 §3.2.
• The noise level of each sample in diffusion training. P2 is included because it re-weights the same loss, but its weight does not change during training. It is a fixed function of the step 2204.00227 §3.2.

Variants

Fixed global weights

One constant weight per loss term, chosen by hand before training. The residual-based paper uses this as its reference setup, for example a weight of 100 on the initial-condition term and 1 on the PDE term for 1D Allen-Cahn 2307.00379 §3.1. The weights do not respond to the errors during training.

Fixed weights by noise level (P2)

Diffusion training sums a denoising loss over about a thousand noise levels. P2 multiplies the simple per-step loss by 1 / (k + SNR(t))^gamma, which down-weights the low-noise steps where the model learns imperceptible detail 2204.00227 §3.2 eq. 8. The authors set k = 1 and gamma to 0.5 or 1 2204.00227 §3.4. They report that gamma above 2 causes noise artifacts, because almost no weight is left for the low-noise steps 2204.00227 §3.4. The weight depends only on the step, so the training state does not enter it. Card: Perception Prioritized Training of Diffusion Models.

The authors' reading of the noise levels is that light noise leaves almost the whole image visible, so the model learns only detail it could guess from its neighbours, while heavy noise forces it to learn content [arxiv:2204.00227 §3.1]. Author-reported FID-50k on FFHQ falls from 7.86 to 6.92 at 1000 steps, and on MetFaces from 44.34 to 36.80 at 250 steps 2204.00227 tab. 1. Across five model configurations, P2 gives a lower FID-10k each time, by 0.25 to 6.83 2204.00227 tab. 3. The paper says this needs no extra model 2204.00227 §5.1.

Residual-based attention (RBA)

Each collocation point keeps a multiplier that is a decaying sum of its normalised PDE residual. The residual is divided by the largest residual magnitude in the current set, so each new term lies in [0, 1]. The multipliers are state, not parameters: they are held constant in the gradient and are not trained by gradient descent 2307.00379 §2.2 eq. 12. Card: Residual-based attention in physics-informed neural networks.

Because the decay factor is below 1, the multipliers have a bounded range. The paper gives an upper bound of eta* / (1 - gamma) 2307.00379 eq. 13. The authors report that RBA makes a vanilla physics-informed network converge about one order of magnitude faster, even without the global loss weights 2307.00379 §3.4.

A second paper by the same authors uses the same update but ties the two constants: gamma = 1 - eta* 2403.18494 §2.5. With that tie, the bound in eq. 13 is 1, so every multiplier lies in [0, 1]. The paper reports about 10k against 100k iterations to reach 10% relative L2 error on Allen-Cahn 2403.18494 fig. 4, and a final error of about 6% on a lid-driven cavity, where vanilla training stalls near 60% 2403.18494 fig. 5. It also finds that vanilla training is best on Burgers, with RBA "a bit worse" 2403.18494 app. C. Card: Learning in PINNs: Phase transition, diffusion equilibrium, and generalization.

Self-adaptive weights (named, not carded)

The card describes self-adaptive weights as the closest method. They train one multiplier per point by gradient ascent, and the authors compare against them throughout 2307.00379 §1 2307.00379 §3.4. Their reported numbers are in the card's results. No card in this wiki describes them in depth yet.

Gradient-norm balancing (modified GradNorm)

Used when several task losses are summed and one task can dominate. The weight of each task is nudged so that the gradient norm of its weighted loss, measured on one shared layer, moves toward a target. The target depends on how fast that task's loss has been falling. The paper adapts GradNorm (Chen et al.; no card here) in three ways: the weights change only at the last step of each epoch, the shared layer is the first layer of each bottleneck, and the asymmetry is 1.5 2606.02979 §2.3 2606.02979 §3.3 eq. 10 eq. 16. After each update the weights are renormalised to sum to the number of tasks eq. 17. The authors update once per epoch to reduce computation 2606.02979 §3.3. Card: Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion.

The paper's ablation compares the same 15-channel model with and without these weights. The total metric falls on every test set: from 1.448 to 1.393 on set A, 1.131 to 1.086 on B, 1.050 to 0.999 on C, and 1.079 to 1.069 on nuScenes-lidarseg 2606.02979 tab. 2 §5.2. The gains are small in absolute terms. The combined single-task baselines score 1.430, 1.160, 0.976 and 1.124 on the same sets 2606.02979 tab. 2 §5.4, so the weighted model beats them on A, B and nuScenes-lidarseg, but not on C.

Dynamic weight averaging (DWA)

The task weights come from loss descent, not gradients. Each weight is a softmax over the ratio of a task's mean loss in the last epoch to its mean loss in the epoch before, with a temperature tau. A higher ratio, meaning slower descent, gets more weight 2303.14968 §3.2 eq. 11. The paper does not give tau. Its code uses 2.0 and sets the weights to 1 for the first two epochs, and the paper mentions neither 2303.14968 §3.2; see Paper vs code on the card. Card: Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective.

The weighted losses are the quality, scene and distortion terms of LIQE, a CLIP-based quality model trained on six datasets 2303.14968 §3.2. The ablation replaces these weights with equal ones. Mean SRCC over six datasets is 0.920 with equal weights and 0.922 with DWA, a gap of 0.002. The paper reports no spread for the ablation 2303.14968 tab. 4.

When to use which

The cards' ablations are author-reported and use best seeds for some rows, so do not read a single ablation row as a general effect. Use the five-seed means for comparisons 2307.00379 §3.1.1 2307.00379 §3.2.1. The LIQE ablation reports no spread at all 2303.14968 tab. 4.

Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants differ. Same style as paper cards: Python-flavoured, lines of at most 80 characters, a # comment line above each step (with its citation), short trailing comments only for shapes or units.

# Residual-based attention: one training step  [arxiv:2307.00379 eq. 12]
# lam is per-point state, zero at the start; it is not a trained parameter
def rba_step(theta, lam, x_points):
    # Residuals of the PDE at each point [arxiv:2307.00379 §2.1]
    residual = pde_residual(theta, x_points)            # (N,)

    # Scale by the largest residual magnitude, so new terms lie in [0, 1]
    normalised = abs(residual) / max(abs(residual))     # (N,)

    # Decaying sum; gamma < 1 keeps lam bounded  [arxiv:2307.00379 eq. 13]
    lam = gamma * lam + eta_star * normalised

    # Weighted loss; lam is held constant in the gradient
    loss = mean((lam * residual) ** 2)

    # Update the network only; the experiments used Adam  [arxiv:2307.00379 §3.1]
    theta = optimiser_step(theta, loss)
    return theta, lam

Fixed global weights replace the update of lam with a constant per term. Shared multipliers replace the per-point lam with one value per group of points. Setting gamma = 1 - eta_star, as the second paper does, keeps lam in [0, 1] 2403.18494 §2.5.

Gradient-norm balancing has a different state: one weight per task. It is updated once per epoch from the gradient norms of the weighted losses on one shared layer, a loss ratio against the first step, and a gradient-norm loss, then renormalised to sum to the number of tasks 2606.02979 eq. 11 2606.02979 eq. 13 2606.02979 eq. 17. The card has the full pseudocode.

DWA also keeps one weight per task, but reads only the losses from the last two epochs 2303.14968 eq. 11:

# Dynamic weight averaging: task weights from loss descent  [arxiv:2303.14968 eq. 11]
# losses_last, losses_before: (T,) mean loss per task, last two epochs
# tau: softmax temperature; not given in the paper (the code uses 2.0)
def dwa_weights(losses_last, losses_before, tau, n_tasks):
    # Ratio of mean losses; a higher ratio means slower descent  [eq. 11]
    ratio = losses_last / losses_before                 # (T,)
    # Softmax over tasks, so slowly falling tasks get more weight  [eq. 11]
    w = softmax(ratio / tau)                            # (T,)
    # The code scales the weights to sum to T; eq. 11 has no such factor
    return w * n_tasks

P2 keeps the simple MSE loss and multiplies each sample's loss by 1 / (k + snr) ** gamma, with snr = alpha / (1 - alpha) [arxiv:2204.00227 eq. 4] eq. 8. The baseline lambda_t is already inside the simple loss, so do not multiply by it again 2204.00227 eq. 7.

Implementation notes:

• Algorithm 1 of the first paper is written for plain gradient descent, but the experiments used Adam 2307.00379 §3.1.
• The RBA scale is set by the single largest residual. One outlier shrinks the new term for every other point 2307.00379 eq. 12.
• Check your chosen eta* and gamma against the bound in eq. 13 before you run. The two papers' values do not always satisfy the same relation (see Contradictions).
• The gradient-norm update has no clamp on the weights. Eq. 17 only renormalises, so a weight can become negative, and the paper does not say what to do then 2606.02979 eq. 17.
• The driving code's loss-weight gradient also includes the total-loss term, ∂L_total/∂w_i, not only the gradient-norm loss of eq. 15 2606.02979 eq. 15. The code's L_i(0) is one batch, taken at the first step of each epoch, so the ratio in eq. 13 is noisy. The card's Paper vs code section lists these and other differences.
• P2 applies the factor to the simple MSE term only. The paper does not say which term of its hybrid objective receives the weight; the released code applies it to the MSE term and not to the learned-variance term 2204.00227 §2.2. The card's Paper vs code section has the details.
• P2 with k = 1 gives a factor of (1 − alpha_t)^gamma. Under the linear schedule the first step has a very small weight at gamma = 1 (derived from the formula, not stated by the authors), so the lowest noise levels are nearly ignored 2204.00227 §3.2.
• DWA's temperature is not in the paper. The code's choice of 2.0 and its unit weights for the first two epochs are not mentioned in the paper 2303.14968 §3.2.

Contradictions

The two residual-based cards disagree on parameters and on where the gain holds. The two task-weighting cards (driving and LIQE) use different methods and metrics, so they are compared only on size. The P2 card is a different kind of weight, fixed per noise level, and is not compared with the others. Each side is cited.

• Decay and bound. The first paper treats eta* and gamma as independent, with bound eta* / (1 - gamma) 2307.00379 eq. 13. For Allen-Cahn it states eta* = 1e-2 with gamma = 0.999 2307.00379 app. A.2, which gives a bound of 10. The second paper sets gamma = 1 - eta*, which gives a bound of 1 2403.18494 §2.5. The two agree only when eta* = 1 - gamma. The first paper's code uses eta* = 1e-3 with gamma = 0.999, which does agree. The card's Paper vs code section records this.
• Size of the speed-up. Both papers report about a tenfold gain. The first measures convergence speed 2307.00379 §3.4. The second measures iterations to 10% error 2403.18494 fig. 4. The two come from the same authors and the same method family, so they are not independent checks. The second card records no seed spread for its numbers 2403.18494 fig. 4.
• Ablations against the headline. In the first paper's Allen-Cahn ablation, RBA with Fourier features gives 1.0e-4, the same as mMLP with Fourier features and no RBA 2307.00379 tab. 2. In the Helmholtz ablation, RBA with Fourier features gives 5.57e-5, against 1.50e-5 for mMLP with Fourier features 2307.00379 tab. 4. The headline five-seed RBA result on the same case is 8.21e-6 2307.00379 tab. 3. The Allen-Cahn ablation uses the single best seed 2307.00379 §3.1.1. The Helmholtz full-model row, 5.18e-6, is also a best seed 2307.00379 §3.2.1. Use the five-seed means for comparisons.
• Where RBA loses. The second paper finds vanilla training best on Burgers 2403.18494 app. C. The first paper reports gains on Allen-Cahn, Helmholtz and its AIV problem 2307.00379 tab. 1 tab. 3 tab. 5. Neither paper covers the other's full set of problems.
• Size of task-weighting gains. The driving paper's total metric is lower with its adaptive weights on all four test sets, by 0.055 on A, 0.045 on B, 0.051 on C and 0.010 on nuScenes-lidarseg 2606.02979 tab. 2. The LIQE ablation finds equal weights only 0.002 mean SRCC below DWA, with no spread 2303.14968 tab. 4. The metrics differ (a combined total metric against a rank correlation), so the sizes are not comparable. Both are author-reported.
• Trade-off inside the driving paper. The 15-channel model without the adaptive weights has a lower or equal depth error, and a higher segmentation IoU, on every test set. On nuScenes-lidarseg the depth errors tie at 0.123. The total metric favours the weighted model. The authors call this a trade-off between two groups of tasks 2606.02979 §5.2 tab. 2. A single total metric hides it.

Papers

• Perception Prioritized Training of Diffusion Models: fixed per-noise-level factor 1/(k+SNR)^gamma on the simple diffusion loss; author-reported FID-50k gains on every dataset in its main table.
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective: dynamic weight averaging over three fidelity losses in LIQE; the equal-weight ablation is within 0.002 mean SRCC.
• Residual-based attention in physics-informed neural networks: per-point decaying multipliers from normalised PDE residuals, with no gradients or extra networks; author-reported gains on Allen-Cahn and Helmholtz.
• Learning in PINNs: Phase transition, diffusion equilibrium, and generalization: RBA with gamma = 1 - eta*, studied through a gradient signal-to-noise ratio; author-reported about 10x on Allen-Cahn and a lower cavity error, with vanilla training best on Burgers.
• Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion: modified GradNorm weights for four perception tasks, updated once per epoch; author-reported lower total metric than separate baselines on three of four test sets.