---
kind: concept
slug: adaptive-loss-weighting
title: Adaptive loss weighting
summary: >-
  Re-weighting loss terms, points or noise levels so that hard or slow parts of training count for more. Covers per-point residual attention (no extra network), task weights from gradient norms or loss descent, and a fixed noise-level weight for diffusion (P2). Five cards; the numbers are author-reported, and paper and code disagree on several values.
tags: [loss-function, loss-weighting, training-dynamics]
papers: ["2204.00227", "2303.14968", "2307.00379", "2403.18494", "2606.02979"]
related: [multi-task-learning, class-balanced-loss, focal-loss, importance-weighting, physics-informed-neural-network, gradient-descent-stability, diffusion-models, noise-schedule]
status: draft
updated_at: 2026-10-31
---

## Overview

A training loss is often a sum of terms, or a mean over many points. A plain
mean lets a small set of points with large errors go unnoticed while the total
loss keeps falling [arxiv:2307.00379 §2.2]. Adaptive loss weighting replaces the
plain mean, or a hand-set sum, with weights set by a rule. The rules covered
here read one of three inputs:

- **Per-point state** built from residuals, for collocation points in a
  physics-informed network [arxiv:2307.00379 §2.2] [arxiv:2403.18494 §2.5].
- **Per-task training rates** for several losses summed in one network. The
  rule reads either gradient norms on a shared layer [arxiv:2606.02979 §3.3],
  or how much each loss has fallen over the last epochs
  [arxiv:2303.14968 §3.2].
- **The noise level of each sample** in diffusion training. P2 is included
  because it re-weights the same loss, but its weight does not change during
  training. It is a fixed function of the step [arxiv:2204.00227 §3.2].

## Variants

### Fixed global weights

One constant weight per loss term, chosen by hand before training. The
residual-based paper uses this as its reference setup, for example a weight of
100 on the initial-condition term and 1 on the PDE term for 1D Allen-Cahn
[arxiv:2307.00379 §3.1]. The weights do not respond to the errors during
training.

### Fixed weights by noise level (P2)

Diffusion training sums a denoising loss over about a thousand noise levels.
P2 multiplies the simple per-step loss by `1 / (k + SNR(t))^gamma`, which
down-weights the low-noise steps where the model learns imperceptible detail
[arxiv:2204.00227 §3.2] [eq. 8]. The authors set k = 1 and gamma to 0.5 or 1
[arxiv:2204.00227 §3.4]. They report that gamma above 2 causes noise artifacts,
because almost no weight is left for the low-noise steps [arxiv:2204.00227 §3.4].
The weight depends only on the step, so the training state does not enter it.
Card: [Perception Prioritized Training of Diffusion Models](/papers/2204.00227).

The authors' reading of the noise levels is that light noise leaves almost the
whole image visible, so the model learns only detail it could guess from its
neighbours, while heavy noise forces it to learn content [arxiv:2204.00227
§3.1]. Author-reported FID-50k on FFHQ falls from 7.86 to 6.92 at 1000 steps,
and on MetFaces from 44.34 to 36.80 at 250 steps [arxiv:2204.00227 tab. 1].
Across five model configurations, P2 gives a lower FID-10k each time, by 0.25
to 6.83 [arxiv:2204.00227 tab. 3]. The paper says this needs no extra model
[arxiv:2204.00227 §5.1].

### Residual-based attention (RBA)

Each collocation point keeps a multiplier that is a decaying sum of its
normalised PDE residual. The residual is divided by the largest residual
magnitude in the current set, so each new term lies in [0, 1]. The multipliers
are state, not parameters: they are held constant in the gradient and are not
trained by gradient descent [arxiv:2307.00379 §2.2] [eq. 12]. Card:
[Residual-based attention in physics-informed neural networks](/papers/2307.00379).

Because the decay factor is below 1, the multipliers have a bounded range.
The paper gives an upper bound of eta* / (1 - gamma) [arxiv:2307.00379 eq. 13].
The authors report that RBA makes a vanilla physics-informed network converge
about one order of magnitude faster, even without the global loss weights
[arxiv:2307.00379 §3.4].

A second paper by the same authors uses the same update but ties the two
constants: gamma = 1 - eta* [arxiv:2403.18494 §2.5]. With that tie, the bound
in eq. 13 is 1, so every multiplier lies in [0, 1]. The paper reports about
10k against 100k iterations to reach 10% relative L2 error on Allen-Cahn
[arxiv:2403.18494 fig. 4], and a final error of about 6% on a lid-driven
cavity, where vanilla training stalls near 60% [arxiv:2403.18494 fig. 5]. It
also finds that vanilla training is best on Burgers, with RBA "a bit worse"
[arxiv:2403.18494 app. C]. Card:
[Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494).

### Self-adaptive weights (named, not carded)

The card describes self-adaptive weights as the closest method. They train one
multiplier per point by gradient ascent, and the authors compare against them
throughout [arxiv:2307.00379 §1] [arxiv:2307.00379 §3.4]. Their reported
numbers are in the card's results. No card in this wiki describes them in
depth yet.

### Gradient-norm balancing (modified GradNorm)

Used when several task losses are summed and one task can dominate. The
weight of each task is nudged so that the gradient norm of its weighted loss,
measured on one shared layer, moves toward a target. The target depends on how
fast that task's loss has been falling. The paper adapts GradNorm (Chen et
al.; no card here) in three ways: the weights change only at the last step of
each epoch, the shared layer is the first layer of each bottleneck, and the
asymmetry is 1.5 [arxiv:2606.02979 §2.3] [arxiv:2606.02979 §3.3] [eq. 10]
[eq. 16]. After each update the weights are renormalised to sum to the number
of tasks [eq. 17]. The authors update once per epoch to reduce computation
[arxiv:2606.02979 §3.3]. Card:
[Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion](/papers/2606.02979).

The paper's ablation compares the same 15-channel model with and without these
weights. The total metric falls on every test set: from
1.448 to 1.393 on set A, 1.131 to 1.086 on B, 1.050 to 0.999 on C, and 1.079 to
1.069 on nuScenes-lidarseg [arxiv:2606.02979 tab. 2] [§5.2]. The gains are
small in absolute terms. The combined single-task baselines score 1.430, 1.160,
0.976 and 1.124 on the same sets [arxiv:2606.02979 tab. 2] [§5.4], so the
weighted model beats them on A, B and nuScenes-lidarseg, but not on C.

### Dynamic weight averaging (DWA)

The task weights come from loss descent, not gradients. Each weight is a
softmax over the ratio of a task's mean loss in the last epoch to its mean
loss in the epoch before, with a temperature tau. A higher ratio, meaning
slower descent, gets more weight [arxiv:2303.14968 §3.2] [eq. 11]. The paper
does not give tau. Its code uses 2.0 and sets the weights to 1 for the first
two epochs, and the paper mentions neither [arxiv:2303.14968 §3.2]; see Paper
vs code on the card. Card:
[Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](/papers/2303.14968).

The weighted losses are the quality, scene and distortion terms of LIQE, a
CLIP-based quality model trained on six datasets [arxiv:2303.14968 §3.2]. The
ablation replaces these weights with equal ones. Mean SRCC over six datasets is
0.920 with equal weights and 0.922 with DWA, a gap of 0.002. The paper reports
no spread for the ablation [arxiv:2303.14968 tab. 4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| The right per-term weights are known and the loss has few terms | Fixed global weights | The reference runs of the first paper use fixed weights per term, with values chosen in advance [arxiv:2307.00379 §3.1] |
| Hard regions are small and a plain mean hides them, as with PINN collocation points | Residual-based attention | Points with recent large error gain weight without a second network [arxiv:2307.00379 §2.2]. The second paper reports the gain as iterations to a set error [arxiv:2403.18494 fig. 4] |
| No room for an extra network or for gradients of the weights | Residual-based attention | The multipliers are updated from the residuals already computed, not trained [arxiv:2307.00379 §2.2] |
| Several task losses are summed, and you want a rule that reads only the losses | Dynamic weight averaging | Eq. 11 uses only epoch-mean losses, not gradient norms [arxiv:2303.14968 §3.2] [eq. 11]. Its ablation gap to equal weights is 0.002 mean SRCC, so test equal weights first [arxiv:2303.14968 tab. 4] |
| Several task losses are summed and the tasks learn at different speeds, and gradient norms on one shared layer are affordable | Gradient-norm balancing | The update moves each task's gradient norm toward a target on a shared layer [arxiv:2606.02979 §3.3]. Its total metric was lower than the same model without it on every test set [arxiv:2606.02979 tab. 2] |
| Compute per step is tight | Update the task weights once per epoch (GradNorm variant or DWA) | The driving authors update once per epoch "to reduce computation" [arxiv:2606.02979 §3.3]. The card gives no timing for the saving |
| Mini-batches of points, so per-point state must follow the batch | Residual-based attention, with shared multipliers | For the AIV problem the authors share multipliers within training groups and use ordered mini-batches [arxiv:2307.00379 §3.3] |
| Memory per point is a concern | Check first | RBA stores one multiplier per collocation point, so memory grows with the number of points [arxiv:2307.00379 §2.2] |
| Diffusion training, sample quality is the goal, and no extra model is wanted | P2 weighting | One closed-form factor on the simple loss [arxiv:2204.00227 §3.2]. At 250 steps, P2 beats the baseline at 500 steps [arxiv:2204.00227 tab. 4] |
| The weight must follow training progress | RBA, GradNorm or DWA, not P2 | P2's weight depends only on the noise level of the sampled step [arxiv:2204.00227 §3.2] |
| Plain training may already be strong, as on Burgers | Compare against unweighted training | The second paper found vanilla training best on that problem [arxiv:2403.18494 app. C] |

The cards' ablations are author-reported and use best seeds for some rows, so
do not read a single ablation row as a general effect. Use the five-seed means
for comparisons [arxiv:2307.00379 §3.1.1] [arxiv:2307.00379 §3.2.1]. The LIQE
ablation reports no spread at all [arxiv:2303.14968 tab. 4].

## Implementation sketch

Language-agnostic pseudocode of the common core, with notes on where variants
differ. Same style as paper cards: Python-flavoured, lines of at most 80
characters, a `# comment` line above each step (with its citation), short
trailing comments only for shapes or units.

```pseudocode
# Residual-based attention: one training step  [arxiv:2307.00379 eq. 12]
# lam is per-point state, zero at the start; it is not a trained parameter
def rba_step(theta, lam, x_points):
    # Residuals of the PDE at each point [arxiv:2307.00379 §2.1]
    residual = pde_residual(theta, x_points)            # (N,)

    # Scale by the largest residual magnitude, so new terms lie in [0, 1]
    normalised = abs(residual) / max(abs(residual))     # (N,)

    # Decaying sum; gamma < 1 keeps lam bounded  [arxiv:2307.00379 eq. 13]
    lam = gamma * lam + eta_star * normalised

    # Weighted loss; lam is held constant in the gradient
    loss = mean((lam * residual) ** 2)

    # Update the network only; the experiments used Adam  [arxiv:2307.00379 §3.1]
    theta = optimiser_step(theta, loss)
    return theta, lam
```

Fixed global weights replace the update of `lam` with a constant per term.
Shared multipliers replace the per-point `lam` with one value per group of
points. Setting `gamma = 1 - eta_star`, as the second paper does, keeps `lam`
in [0, 1] [arxiv:2403.18494 §2.5].

Gradient-norm balancing has a different state: one weight per task. It is
updated once per epoch from the gradient norms of the weighted losses on one
shared layer, a loss ratio against the first step, and a gradient-norm loss,
then renormalised to sum to the number of tasks [arxiv:2606.02979 eq. 11]
[arxiv:2606.02979 eq. 13] [arxiv:2606.02979 eq. 17]. The card has the full
pseudocode.

DWA also keeps one weight per task, but reads only the losses from the last
two epochs [arxiv:2303.14968 eq. 11]:

```pseudocode
# Dynamic weight averaging: task weights from loss descent  [arxiv:2303.14968 eq. 11]
# losses_last, losses_before: (T,) mean loss per task, last two epochs
# tau: softmax temperature; not given in the paper (the code uses 2.0)
def dwa_weights(losses_last, losses_before, tau, n_tasks):
    # Ratio of mean losses; a higher ratio means slower descent  [eq. 11]
    ratio = losses_last / losses_before                 # (T,)
    # Softmax over tasks, so slowly falling tasks get more weight  [eq. 11]
    w = softmax(ratio / tau)                            # (T,)
    # The code scales the weights to sum to T; eq. 11 has no such factor
    return w * n_tasks
```

P2 keeps the simple MSE loss and multiplies each sample's loss by
`1 / (k + snr) ** gamma`, with `snr = alpha / (1 - alpha)` [arxiv:2204.00227
eq. 4] [eq. 8]. The baseline `lambda_t` is already inside the simple loss, so do
not multiply by it again [arxiv:2204.00227 eq. 7].

Implementation notes:

- Algorithm 1 of the first paper is written for plain gradient descent, but the
  experiments used Adam [arxiv:2307.00379 §3.1].
- The RBA scale is set by the single largest residual. One outlier shrinks the
  new term for every other point [arxiv:2307.00379 eq. 12].
- Check your chosen eta* and gamma against the bound in eq. 13 before you run.
  The two papers' values do not always satisfy the same relation (see
  Contradictions).
- The gradient-norm update has no clamp on the weights. Eq. 17 only
  renormalises, so a weight can become negative, and the paper does not say
  what to do then [arxiv:2606.02979 eq. 17].
- The driving code's loss-weight gradient also includes the total-loss term,
  ∂L_total/∂w_i, not only the gradient-norm loss of eq. 15
  [arxiv:2606.02979 eq. 15]. The code's L_i(0) is one batch, taken at the
  first step of each epoch, so the ratio in eq. 13 is noisy. The card's Paper
  vs code section lists these and other differences.
- P2 applies the factor to the simple MSE term only. The paper does not say
  which term of its hybrid objective receives the weight; the released code
  applies it to the MSE term and not to the learned-variance term
  [arxiv:2204.00227 §2.2]. The card's Paper vs code section has the details.
- P2 with k = 1 gives a factor of (1 − alpha_t)^gamma. Under the linear schedule
  the first step has a very small weight at gamma = 1 (derived from the formula,
  not stated by the authors), so the lowest noise levels are nearly ignored
  [arxiv:2204.00227 §3.2].
- DWA's temperature is not in the paper. The code's choice of 2.0 and its
  unit weights for the first two epochs are not mentioned in the paper
  [arxiv:2303.14968 §3.2].

## Contradictions

The two residual-based cards disagree on parameters and on where the gain
holds. The two task-weighting cards (driving and LIQE) use different methods
and metrics, so they are compared only on size. The P2 card is a different kind
of weight, fixed per noise level, and is not compared with the others. Each
side is cited.

- **Decay and bound.** The first paper treats eta* and gamma as independent,
  with bound eta* / (1 - gamma) [arxiv:2307.00379 eq. 13]. For Allen-Cahn it
  states eta* = 1e-2 with gamma = 0.999 [arxiv:2307.00379 app. A.2], which
  gives a bound of 10. The second paper sets gamma = 1 - eta*, which gives a
  bound of 1 [arxiv:2403.18494 §2.5]. The two agree only when eta* = 1 - gamma.
  The first paper's code uses eta* = 1e-3 with gamma = 0.999, which does agree.
  The card's Paper vs code section records this.
- **Size of the speed-up.** Both papers report about a tenfold gain. The first
  measures convergence speed [arxiv:2307.00379 §3.4]. The second measures
  iterations to 10% error [arxiv:2403.18494 fig. 4]. The two come from the same
  authors and the same method family, so they are not independent checks. The
  second card records no seed spread for its numbers [arxiv:2403.18494 fig. 4].
- **Ablations against the headline.** In the first paper's Allen-Cahn ablation,
  RBA with Fourier features gives 1.0e-4, the same as mMLP with Fourier
  features and no RBA [arxiv:2307.00379 tab. 2]. In the Helmholtz ablation,
  RBA with Fourier features gives 5.57e-5, against 1.50e-5 for mMLP with
  Fourier features [arxiv:2307.00379 tab. 4]. The headline five-seed RBA result
  on the same case is 8.21e-6 [arxiv:2307.00379 tab. 3]. The Allen-Cahn
  ablation uses the single best seed [arxiv:2307.00379 §3.1.1]. The Helmholtz
  full-model row, 5.18e-6, is also a best seed [arxiv:2307.00379 §3.2.1].
  Use the five-seed means for comparisons.
- **Where RBA loses.** The second paper finds vanilla training best on Burgers
  [arxiv:2403.18494 app. C]. The first paper reports gains on Allen-Cahn,
  Helmholtz and its AIV problem [arxiv:2307.00379 tab. 1] [tab. 3] [tab. 5].
  Neither paper covers the other's full set of problems.
- **Size of task-weighting gains.** The driving paper's total metric is lower
  with its adaptive weights on all four test sets, by 0.055 on A, 0.045 on B,
  0.051 on C and 0.010 on nuScenes-lidarseg [arxiv:2606.02979 tab. 2]. The LIQE
  ablation finds equal weights only 0.002 mean SRCC below DWA, with no spread
  [arxiv:2303.14968 tab. 4]. The metrics differ (a combined total metric against
  a rank correlation), so the sizes are not comparable. Both are author-reported.
- **Trade-off inside the driving paper.** The 15-channel model without the
  adaptive weights has a lower or equal depth error, and a higher segmentation
  IoU, on every test set. On nuScenes-lidarseg the depth errors tie at 0.123.
  The total metric favours the weighted model. The authors call this a
  trade-off between two groups of tasks [arxiv:2606.02979 §5.2] [tab. 2]. A
  single total metric hides it.

## Papers

- [Perception Prioritized Training of Diffusion Models](/papers/2204.00227): fixed per-noise-level factor 1/(k+SNR)^gamma on the simple diffusion loss; author-reported FID-50k gains on every dataset in its main table.
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](/papers/2303.14968): dynamic weight averaging over three fidelity losses in LIQE; the equal-weight ablation is within 0.002 mean SRCC.
- [Residual-based attention in physics-informed neural networks](/papers/2307.00379): per-point decaying multipliers from normalised PDE residuals, with no gradients or extra networks; author-reported gains on Allen-Cahn and Helmholtz.
- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494): RBA with gamma = 1 - eta*, studied through a gradient signal-to-noise ratio; author-reported about 10x on Allen-Cahn and a lower cavity error, with vanilla training best on Burgers.
- [Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion](/papers/2606.02979): modified GradNorm weights for four perception tasks, updated once per epoch; author-reported lower total metric than separate baselines on three of four test sets.