---
kind: concept
slug: physics-informed-neural-network
title: Physics-informed neural network
summary: >-
  A neural network trained so that its output satisfies a differential equation: the loss adds the equation residual and the boundary conditions. Some variants build a thermodynamic law into the output. Variants differ in point weighting, derivative estimation, boundary enforcement, penalty form and error bars. An SNR diagnostic tracks stalled training. Numbers are author-reported.
tags: [pinn, pde, loss-weighting, boundary-conditions, equilibrium-penalty, training-dynamics, uncertainty-quantification, thermodynamics, numerical-differentiation]
papers: ["2307.00379", "2507.05291", "1903.12394", "2403.18494", "2201.07766", "2005.12183", "2110.15832"]
related: [adaptive-loss-weighting, fourier-features, neural-operator, graph-neural-network, finite-element-method, informed-machine-learning, gradient-descent-stability, information-bottleneck, uncertainty-estimation, calibration, bayesian-neural-network, ensembles, automatic-differentiation, finite-difference-method, surrogate-modeling]
status: draft
updated_at: 2026-11-01
---

## Overview

A physics-informed neural network (PINN) is a network trained to satisfy a
differential equation. Its loss is a weighted sum of terms: the equation
residual at collocation points, the initial and boundary conditions, and any
measured data [arxiv:2307.00379 §2.1] [arxiv:2307.00379 eq. 6]. Each term is a
mean over its points [arxiv:2307.00379 eq. 10]. The weakness that the
reweighting variants respond to is that a plain mean lets a small region with
large residuals disappear into the average while the total loss keeps falling.
That can hurt multiscale problems and can block information from reaching the
interior of the domain from the boundary [arxiv:2307.00379 §2.2].

A 2024 study by the same authors gives a second account of the stall. With a
quadratic loss, the batch-averaged gradient can point in a steady direction
while individual point gradients cancel. Some points then stay unresolved while
the loss looks settled [arxiv:2403.18494 §2.1] [arxiv:2403.18494 §2.3]. Its
remedy is a reweighting of the same kind, described below.

A third response leaves the weights alone and changes how the derivatives in
the residual are estimated. With sparse points the network can match the
autodiff slopes at every point and still be wrong between them. CAN-PINN mixes
each autodiff slope with finite-difference-style values from neighbouring points
[arxiv:2110.15832 §1] [arxiv:2110.15832 §2.3].

Most variants share one invariant: a physics penalty is added to a training
loss, and the network weights take gradient steps on it. These variants differ
in three choices: how the penalty is formed (a derivative of the network output
at sample points, a derivative built from neighbouring network values, or a
discrete operator on a mesh), how the terms or points are
weighted, and how boundary conditions are enforced. One family does not use a
penalty at all. It puts the law into the output: the network predicts an energy,
and the stress is that energy's derivative, so the law holds whatever the
weights are [arxiv:2005.12183 §3.3]. Architecture and input encoding are a
further, separate choice. An optional layer attaches error bars to the
prediction, which a plain PINN does not provide. It treats the network weights
as random and averages over posterior samples or over an ensemble
[arxiv:2201.07766 §3.1] [arxiv:2201.07766 §4.1] [arxiv:2201.07766 §4.2].

The name is used loosely. The survey that classifies informed machine learning
lists physics-informed deep learning, physics-guided neural networks and
semantic-based regularisation as overlapping names for related methods
[arxiv:1903.12394 §1]. It finds that scientific knowledge is mostly given as
equations or simulation results, and that equations most often change the
learning algorithm, which here means the loss [arxiv:1903.12394 §3.2.2]
[arxiv:1903.12394 §3.2.3].

## Variants

### Fixed global weights (vanilla PINN)

One constant weight per loss term, set before training. The reference setup
for 1D Allen-Cahn uses 100 on the initial-condition term and 1 on the PDE
residual [arxiv:2307.00379 §3.1]. The weights do not respond to the errors.
The card's comparison tables give the vanilla baseline as 4.98e-1 relative L2
on Allen-Cahn and 8.14e-2 on 2D Helmholtz. The authors took these baseline
values from the literature [arxiv:2307.00379 tab. 1] [arxiv:2307.00379 tab. 3].
On Allen-Cahn, the vanilla network converges only with a scheduled learning
rate [arxiv:2307.00379 §3.4]. Card:
[Residual-based attention in physics-informed neural networks](/papers/2307.00379).

### Residual-based attention (RBA) on the PDE residual

Each collocation point keeps a multiplier. The multiplier decays by a factor
gamma each step and adds the point's residual, divided by the largest residual
magnitude in the current set, scaled by a learning rate eta*
[arxiv:2307.00379 eq. 12]. The normalisation keeps each new term in [0, 1].
The multipliers are state: they are held constant in the loss and are never
trained by gradient descent, so the method needs no auxiliary network
[arxiv:2307.00379 §2.2]. With gamma below 1, the multipliers stay below
eta* / (1 - gamma) [arxiv:2307.00379 eq. 13].

Reported results, all author-reported and five-seed means:

- 1D Allen-Cahn: 5.8e-5 relative L2, against 1.11e-4 for self-adaptive
  weights re-run by the authors [arxiv:2307.00379 tab. 1].
- 2D Helmholtz with Fourier features: 8.21e-6, against 4.80e-5 for
  self-adaptive weights [arxiv:2307.00379 tab. 3].
- Early convergence about one order of magnitude faster than the vanilla
  PINN, even without the global loss weights [arxiv:2307.00379 §3.4].

The ablations are less favourable. On Allen-Cahn, mMLP with Fourier features
and no RBA gives 1.0e-4, the same as RBA with Fourier features
[arxiv:2307.00379 tab. 2]. On Helmholtz, mMLP with Fourier features and no RBA
gives 1.50e-5, and RBA with Fourier features gives 5.57e-5
[arxiv:2307.00379 tab. 4]. These
ablation rows are single-seed, so the effect of RBA alone is not settled by
this card. Card: [Residual-based attention in physics-informed neural networks](/papers/2307.00379).

### RBA with the decay tied to the rate (gamma = 1 - eta*)

The 2024 study uses the same normalised moving average but sets the decay to
gamma = 1 - eta* [arxiv:2403.18494 §2.5] [arxiv:2403.18494 eq. 26]. The
multipliers then stay in [0, 1], so the weighted residual term is never larger
than the unweighted one. The released Allen-Cahn and cavity code uses eta* =
0.001 and gamma = 0.999 [code: AC/AC.py#L171-L172]. The initial-condition term
keeps weight 1 in the 2024 Allen-Cahn setup [arxiv:2403.18494 eq. 25].

Reported results are read from figures, from single runs:

- Allen-Cahn: 10% relative L2 error in about 10k iterations with RBA, against
  about 100k for the vanilla PINN [arxiv:2403.18494 fig. 4].
- Lid-driven cavity at Re = 1000: vanilla training stalls near 60% error, and
  RBA reaches about 6% [arxiv:2403.18494 fig. 5].
- Burgers: the paper reports vanilla as the best of the two, with RBA slightly
  worse [arxiv:2403.18494 app. C].

The paper reports no seed spread, so treat these as single-run results.
Card: [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494).

### RBA with shared multipliers for inverse problems with mini-batches

For the 3D Navier-Stokes inverse problem, reconstructing a flow from particle
tracking velocimetry (PTV) data, the authors give each data point its own
multiplier but share one multiplier per training group for boundary and
collocation points. Training uses ordered mini-batches of one point per group
[arxiv:2307.00379 §3.3] [arxiv:2307.00379 fig. 6]. The validation errors are
lower than for self-adaptive weights and for global weights alone, but the
split and iteration count are inconsistent in the paper (see Pitfalls).
This setting is paper-only: the repository has no code or data for it
[arxiv:2307.00379 §3.3].

| Method (validation set) | Relative L2, u | Relative L2, v |
|---|---|---|
| Learning-rate annealing (from the literature) | 4.21e-1 | 2.11e-1 |
| PINN with global weights | 3.31e-1 | 2.67e-1 |
| Self-adaptive weights | 2.94e-1 +/- 2.5e-3 | 2.48e-1 +/- 2.3e-3 |
| RBA weights | 2.43e-1 +/- 1.6e-3 | 2.10e-1 +/- 1.5e-3 |

[arxiv:2307.00379 tab. 5]

### Exact Dirichlet conditions through an approximate distance function

The solution is written as a fixed term that matches the boundary data plus a
network term multiplied by a function that vanishes on the boundary
[arxiv:2307.00379 §2.3.3] [arxiv:2307.00379 eq. 20] [arxiv:2307.00379 eq. 21].
The boundary is then satisfied by construction and its loss term is dropped.
On Helmholtz with the distance function (x^2 - 1)(y^2 - 1), the RBA result is
7.73e-5 relative L2, against 8.21e-6 with the Fourier variant
[arxiv:2307.00379 tab. 3] [arxiv:2307.00379 eq. 32]. The two runs differ in
boundary handling as well as in encoding, so the gap does not isolate either.

### Exact periodic conditions through a period-matched Fourier embedding

Input coordinates are mapped through sines and cosines whose period equals the
domain length, which makes the periodicity exact [arxiv:2307.00379 §2.3.3]
[arxiv:2307.00379 eq. 22]. The embedding details and the Helmholtz truncation
are in [Fourier features](/concepts/fourier-features).

### Energy potential with stress as its derivative (thermodynamics-based network)

No penalty is added here. The network predicts a stored-energy function and the
changes of a few internal variables. The stress and the dissipation are then
derivatives of that energy, computed by automatic differentiation, so the first
law holds by construction [arxiv:2005.12183 §3.3] [arxiv:2005.12183 eq. 13]. The
second law is not built in. It holds in the examples because the training data
have non-negative dissipation, and consistency on unseen inputs is shown only by
those examples [arxiv:2005.12183 §5.1.2] [arxiv:2005.12183 §5.1.3]. The
temperature sub-network can be dropped for isothermal processes, which the
reported cases are [arxiv:2005.12183 §3.3] [arxiv:2005.12183 fig. 7b].

Because the stress is a derivative of the network, the loss compares
derivatives, so the activation's second derivative must not vanish. The authors
show this with a toy map from x to x² and 2x. On that toy problem, ReLU applied
to z ends at an error figure L of 1521.2 × 10⁻⁴, while the energy network's
activation (written ELU_{z²} in the paper) ends at 0.13 × 10⁻⁴
[arxiv:2005.12183 §3.2] [arxiv:2005.12183 tab. 2]. The reported elasto-plastic
results are author-reported and use data generated from the analytic reference
model, not measurements [arxiv:2005.12183 §4.2]. A standard ANN with a matched
hyperparameter count gives the wrong energy and dissipation on the 1D cases, and
generalises worse on cyclic and random paths [arxiv:2005.12183 §5.1.4]. In 3D,
the network agrees with the reference model for uniaxial cyclic loading across
increment sizes, with less accuracy at the largest increments. The authors also
report larger errors for it in the mean and deviatoric stress
[arxiv:2005.12183 §5.2.2] [arxiv:2005.12183 §5.2.3].
Card: [Thermodynamics-based Artificial Neural Networks for constitutive modeling](/papers/2005.12183).

### Discrete equilibrium penalty on a graph network

This variant does not differentiate the network output. A graph neural network
predicts the stress at every node of a periodic micro-structure mesh. The
training loss adds, to a normalised mean-squared error on the stress, a weight
times the mean squared discrete divergence of the predicted stress. The
divergence is built from the finite-element shape-function derivatives, so it
acts on the mesh operator, not on autodiff gradients [arxiv:2507.05291 §3.3]
[arxiv:2507.05291 eq. 41] [arxiv:2507.05291 eq. 42]. Hole and outer-boundary
nodes contribute zero to the penalty [arxiv:2507.05291 §4.3]. Periodicity is
built into the graph, as extra edges between opposite faces, and not as a loss
term [arxiv:2507.05291 §3.1] [arxiv:2507.05291 fig. 2]. The paper states that
enforcing equilibrium on GNN stress predictions was unresolved before this work
[arxiv:2507.05291 §1.3].

Reported results, author-reported and single values with no spread across seeds:

- Linear elastic mesh: NMSE 1.08e-2, against 1.13e-2 for the same network
  without the penalty. Mean divergence 3.23e-4, against 1.11e-3 for that
  network [arxiv:2507.05291 tab. 1].
- Hyperelastic mesh: NMSE 2.18e-2, against 5.36e-2 for the network without the
  penalty [arxiv:2507.05291 tab. 2].
- Inference on the hyperelastic set is about 500 times faster than finite
  elements on CPU. The comparison is GPU inference against CPU finite elements
  and excludes training [arxiv:2507.05291 §5.3.1].

The linear NMSE gain is small. Card:
[Physics-Informed Graph Neural Networks to Reconstruct Local Fields Considering Finite Strain Hyperelasticity](/papers/2507.05291).

### Coupled automatic and numerical differentiation (CAN-PINN)

The penalty is still a residual at sample points, but each derivative in it is
built differently. A face value is a linear mix of the network value and its
autodiff slope. The mixing weights come from Taylor expansions about the face,
which give a1 = 1 and a2 = Δx/2 [arxiv:2110.15832 eq. 6a] [arxiv:2110.15832 eq. 7a]
[arxiv:2110.15832 eq. 7b]. For the upwind convection term, the paper's
second-order scheme, can(uw2), takes the east face value as û(x) plus Δx/2 times
the slope at x [arxiv:2110.15832 eq. 8a]. The pressure gradient uses a central
difference, can(cd), whose face values are corrected with the autodiff slopes at
the centre and at the neighbour [arxiv:2110.15832 §2.3.2] [arxiv:2110.15832 eq. 12a].
In a Fourier analysis against first- and second-order upwind schemes, both can
schemes have smaller dispersion error. can(uw2) has smaller dissipation than
those upwind schemes, and can(cd) is non-dissipative [arxiv:2110.15832 §2.4]
[arxiv:2110.15832 fig. 3].

Reported results, all author-reported:

- Lid-driven cavity, Re = 400, 2,601 points, 50 runs: the AD-only PINN's u and v
  MSE is above 1e-2. can-PINN is about two orders lower and the
  numerical-differentiation PINN about one order lower
  [arxiv:2110.15832 §3.3.1] [arxiv:2110.15832 fig. 10a].
- The same cavity with 40,401 points: the AD-only PINN reaches about 2e-4 and
  can-PINN goes below 1e-5 [arxiv:2110.15832 §3.3.2] [arxiv:2110.15832 fig. 11a].
- Backward-facing step, Re = 200, 16,000 points, 25 runs: reattachment point x1
  against a ground truth of 2.608 gives 4.761 for the AD-only PINN, 2.601 for
  n-PINN and 2.604 for can-PINN [arxiv:2110.15832 tab. 3].

These tables compare against AD-only and numerical-differentiation PINNs only,
not against the reweighting variants above. Card:
[CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method](/papers/2110.15832).

### Batch-wise gradient SNR (diagnostic, no loss change)

This is a monitor, not a training method. At fixed parameters, the paper takes
the gradient of each batch and computes the norm of the mean batch gradient
divided by the norm of the batch standard deviation, over the weights
[arxiv:2403.18494 §2.1] [arxiv:2403.18494 eq. 7]. For full-batch Adam on PINNs,
it names three phases [arxiv:2403.18494 §2.1] [arxiv:2403.18494 fig. 2]:

- Fitting: high SNR and a fast decline.
- Diffusion: SNR falls while the error barely moves.
- Diffusion equilibrium (DE): a sudden rise to a plateau where SNR is flat,
  after which the error drops steeply.

The paper links DE to residuals that are homogeneous across collocation points
[arxiv:2403.18494 §2.4]. An SNR built from Adam's moment estimates tracks the
batch SNR and reaches its equilibrium at DE [arxiv:2403.18494 §2.2]
[arxiv:2403.18494 fig. 7]. For the order parameter phi, the paper writes
SNR = phi / sqrt(1 - phi^2), which needs the population standard deviation
[arxiv:2403.18494 app. B]. Card:
[Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494).

### Network components used with RBA

- **Modified MLP (mMLP)**: two encoders, each a tanh layer on the input, mixed
  into every hidden layer by a learned gate [arxiv:2307.00379 §2.3.1]
  [arxiv:2307.00379 eq. 14] [arxiv:2307.00379 eq. 17].
- **Weight normalisation**: each weight vector is split into a length and a
  direction [arxiv:2307.00379 §2.3.2] [arxiv:2307.00379 eq. 18].

On Allen-Cahn, the ablation gives 1.58e-3 with RBA and mMLP, and 4.57e-5 with
RBA, mMLP and Fourier features [arxiv:2307.00379 tab. 2]. On AIV, weight
normalisation alone gives 2.71e-1 on v and RBA with it gives 2.08e-1
[arxiv:2307.00379 tab. 6]. These are single-seed or validation-split numbers.

### Error bars from posterior samples (U-PINN)

The predictive mean is the average of the network over M posterior samples. The
total variance adds a noise variance to the spread of those samples
[arxiv:2201.07766 §3.1]. The paper calls the PINN version U-PINN. Its posterior
methods are HMC, Langevin dynamics, mean-field variational inference, MC dropout
and a Laplace fit at the trained mode. Its ensemble methods are independent
runs, snapshots from one cyclic-learning-rate run, and SWAG
[arxiv:2201.07766 §4.1] [arxiv:2201.07766 §4.2].

On a function-approximation test with 32 noisy points, the mean's relative L2
error ranges from 21.9 to 29.2 (×10²) across the methods shown. The uncertainty
RMSCE differs more: 7.1 for LA, 7.2 for both HMC and SEns, 13.6 for SWAG and
15.5 for LD [arxiv:2201.07766 §6.1].

A functional prior replaces the weight prior with a generator pre-trained on
historical solutions, and inference runs on the generator's low-dimensional
input [arxiv:2201.07766 §4.3]. On the diffusion–reaction case, HMC with this
learned prior has a relative L2 error of 1.6 (×10²) on u, against 3.7 for HMC
with the standard prior [arxiv:2201.07766 §6.2]. That is one case among several.
For stochastic PDEs the paper adds PI-GAN variants that sample the generator
and two polynomial-chaos variants, NNPC and NNPC+ [arxiv:2201.07766 §4.4].
Card: [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](/papers/2201.07766).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Few loss terms and the right per-term weights are known in advance | Fixed global weights | Simplest option; the card's reference runs use this [arxiv:2307.00379 §3.1] |
| Sharp or multiscale features that a plain mean hides | RBA on the PDE residual | Points with recent large residuals gain weight, with no second network [arxiv:2307.00379 §2.2] [arxiv:2307.00379 tab. 1] |
| Training stalls with a few points unresolved, e.g. a cavity-type flow | RBA with the tied decay | One cavity run went from about 60% to about 6% error; single run [arxiv:2403.18494 fig. 5] |
| Multipliers must never exceed 1 | Tied decay, gamma = 1 - eta* | The bound in the update is 1 [arxiv:2403.18494 §2.5] |
| Gradient-trained per-point weights are not wanted | RBA, not self-adaptive weights | RBA updates the multipliers from residuals already computed; the authors compare it against self-adaptive weights [arxiv:2307.00379 §1] [arxiv:2307.00379 §3.4] |
| Need to know whether a run is still in diffusion or has reached DE | Batch-wise SNR | Shows the phase, not a fix [arxiv:2403.18494 §2.1] [arxiv:2403.18494 fig. 2] |
| SNR logging must be cheap | Log at the released code's rate | One backward pass per batch every 100 iterations, and every 10 before iteration 300 [code: utils/snr.py#L21-L33] |
| Dirichlet boundaries that must hold exactly | Approximate distance function | Removes the boundary loss term [arxiv:2307.00379 §2.3.3] |
| Periodic boundaries on a known domain | Period-matched Fourier embedding | Exact by construction; the period must match the domain [arxiv:2307.00379 §2.3.3] |
| Large point sets with mini-batches | Shared multipliers per group | Per-point state must follow the batch; the authors used this for AIV [arxiv:2307.00379 §3.3] |
| Memory per point is tight | Check first | RBA stores one multiplier per point, which is O(N) memory [arxiv:2307.00379 §2.2] |
| Need a reproducible comparison | Five-seed means | The ablations are single-seed or best-seed [arxiv:2307.00379 §3.1.1] [arxiv:2307.00379 §3.2.1] |
| A learned field must balance on an unstructured mesh, and the model is a GNN | Discrete equilibrium penalty | The penalty lowered mean divergence against the same network without it [arxiv:2507.05291 tab. 1] |
| Boundary-node divergence is needed for training | Leave boundary nodes out of the penalty | The paper calls extrapolated boundary divergence not accurate enough for training [arxiv:2507.05291 §4.3] |
| Sparse collocation points, where the AD residual can look small while the flow is wrong | Coupled AD and numerical differentiation | At 2,601 points the AD-only PINN fails while can-PINN works [arxiv:2110.15832 §3.3.1] [arxiv:2110.15832 §3.3.2] |
| Dense points, and per-iteration wall-clock is the limit with few CPUs | Plain AD or numerical-differentiation PINN | The paper reports can-PINN as the slowest per iteration with 4 CPUs per PINN [arxiv:2110.15832 fig. 11c] |
| Dispersion error in the convective term matters | can(uw2) for convection, can(cd) for pressure | The paper's Fourier analysis favours both can schemes over the upwind schemes [arxiv:2110.15832 §2.4] |
| Inverse problem with the Reynolds number unknown and few observations | Coupled residual | The paper infers Re within about 10% at Re = 400 from 10 observations [arxiv:2110.15832 §3.3.3] |
| Released code must be reused | Reimplement from the equations | The repository has no licence file and says all rights are reserved [code: README.md#L13-L15] |
| Prior equations are available but the loss weights are unknown | Knowledge term in the loss | The survey lists how to weight knowledge against labels as an open problem [arxiv:1903.12394 §7]; it gives no method for it |
| Need error bars and a fixed compute budget | Snapshot ensemble (SEns) | RMSCE 7.2, the same as HMC, at CPU time 31.9 s against 131.5 s for HMC on the function example [arxiv:2201.07766 §6.1] [arxiv:2201.07766 §6.6] |
| Need a sampling-based posterior on a small problem and time is not the limit | HMC | RMSCE 7.2 on the function example; 50 leapfrog steps cost about 4 to 5 times standard training [arxiv:2201.07766 §6.1] [arxiv:2201.07766 §6.6] |
| Error bars must not be over-confident on inputs unlike the training data | Check OOD behaviour; do not rely on MC dropout alone | MC dropout's uncertainty did not increase on out-of-distribution data [arxiv:2201.07766 §6.1] |
| Error bars need checking, and a left-out set is available | Post-training calibration (variance scaling or CDF correction) | The paper describes both methods and reports RMSCE falling about threefold in one case with 2 to 10 left-out realizations [arxiv:2201.07766 §5.2] [arxiv:2201.07766 §6.3] |
| Prior or noise scale is unknown | Learn them during inference; consider a functional prior | Cheap methods match expensive ones mainly when these are learned [arxiv:2201.07766 §7]; a learned prior improves one case from 3.7 to 1.6 (×10²) [arxiv:2201.07766 §6.2] |
| The output must obey a thermodynamic law by construction, and the model is a material law | Energy potential with stress as its derivative | The first law holds whatever the weights are [arxiv:2005.12183 §3.3]; the paper tests only isothermal 1D and 3D elasto-plasticity [arxiv:2005.12183 §5] |
| Training data come only from measurements | Not shown for this variant | The paper generates all data from the analytic reference model and shows no measured data [arxiv:2005.12183 §4.2] |
| Dissipation must stay non-negative on unseen inputs | Check the dissipation on your own test paths | The architecture does not enforce the second law; it holds in the examples because the training dissipation is non-negative [arxiv:2005.12183 §5.1.2] [arxiv:2005.12183 §5.1.3] |

The card gives no run that compares RBA against a tuned vanilla PINN with a
learning-rate schedule and the same architecture, apart from the convergence
figure [arxiv:2307.00379 §3.4]. The 2024 study compares only against vanilla
PINN training with uniform residual weights [arxiv:2403.18494 §3.1]. Treat the
reported gains as the authors' best case. No card compares the discrete penalty
with an autodiff residual on the same problem, so the two cannot be ranked from
this page. The error-bar comparisons come from the UQ paper's own prototype
problems and baselines, so they do not rank the residual-weighting variants.

## Implementation sketch

The loss with one weight per term, following the card's Allen-Cahn setup; a
per-point multiplier `lam` can replace the scalar weights, as in
[adaptive-loss-weighting](/concepts/adaptive-loss-weighting).

```pseudocode
# PINN loss: weighted mean residual plus boundary and initial terms
# [§2.1] [eq. 6] [eq. 10]
# theta: network weights; x_r: collocation points; x_b, u_b: boundary data
def pinn_loss(theta, x_r, x_b, u_b, w_r, w_b):
    # PDE residual at each collocation point  [§2.1]
    r = pde_residual(theta, x_r)                 # (N_r,)

    # Mismatch on the boundary and initial points  [§2.1]
    e_b = net(theta, x_b) - u_b                  # (N_b,)

    # Squared residuals, mean over each term, one fixed weight per term
    # [§3.1] (w_b = 100, w_r = 1 on Allen-Cahn)
    return w_r * mean(r ** 2) + w_b * mean(e_b ** 2)
```

For RBA, replace `w_r` with a per-point multiplier `lam` updated by eq. 12
before the loss is formed, and hold `lam` constant when taking the gradient
[arxiv:2307.00379 eq. 12] [arxiv:2307.00379 §2.2]. For a distance-function
boundary, drop the `w_b` term and build the boundary into the output
[arxiv:2307.00379 §3.2]. For the tied-decay form, set gamma = 1 - eta* so that
the multiplier stays at most 1 [arxiv:2403.18494 §2.5].

For an energy-potential network, do not output the stress. Output the energy and
the internal-variable increments, then take the stress and the dissipation as
derivatives of the energy with automatic differentiation [arxiv:2005.12183 §3.3]
[arxiv:2005.12183 eq. 13]. The paper's stress derivative is a partial one with
respect to the energy's own inputs; the card reads it that way (see Pitfalls).

For a mesh-based penalty, the residual is a discrete operator applied to the
predicted nodal stress rather than an autodiff term. The P-DivGNN loss has
this form: a normalised mean-squared error plus a weight times the mean squared
divergence over nodes [arxiv:2507.05291 eq. 41] [arxiv:2507.05291 eq. 42].

For monitoring, compute the batch SNR of eq. 7 from per-batch gradients at
fixed parameters, with no optimiser step [arxiv:2403.18494 §2.1].

For error bars, run the network at each of the M posterior samples or ensemble
members. Take the mean of the predictions as the prediction, and the spread of
the predictions plus the noise variance as the total variance. Divide the spread
by M, as the paper does [arxiv:2201.07766 §3.1].

For CAN-PINN, build an upwind face value from the network value and its autodiff
slope rather than from the network value alone. For positive flow, the east face
is û(x) plus Δx/2 times the slope, and the gradient is the difference of the two
faces over Δx [arxiv:2110.15832 eq. 8a] [arxiv:2110.15832 eq. 9]. The negative-flow
branch and the choice of upwind side are not in the paper; see Pitfalls. The
pressure face value is corrected in the same way, with a Δx/8 factor on the
difference of the two slopes [arxiv:2110.15832 eq. 12a].

## Pitfalls

- **Multiplier bounds.** The upper bound is eta* / (1 - gamma), plus any
  offset [arxiv:2307.00379 eq. 13]. Check the values you pick against it. The
  paper's Allen-Cahn value of eta* = 1e-2 with gamma = 0.999 gives a bound of
  10 (derived by the card, not stated in the paper).
- **Normalisation by the maximum.** The divisor is the largest residual in the
  current set, so one large outlier shrinks every other new term
  [arxiv:2307.00379 §3.3]. The 2024 form divides by the same maximum
  [arxiv:2403.18494 eq. 26].
- **Warm-up under the tied decay.** With eta* = 0.001 and gamma = 0.999 the
  multipliers approach their steady value with a time constant of about 1000
  iterations. For the first few hundred iterations the residual term is almost
  switched off. This is derived from the update, not stated in the paper
  [arxiv:2403.18494 §2.5] [code: AC/AC.py#L282-L287].
- **The residual term is only down-weighted.** With the multipliers at most 1,
  the data term keeps weight 1, so the balance between the two shifts over
  training. This is inferred, not stated [arxiv:2403.18494 §2.5]
  [code: AC/AC.py#L287-L293].
- **Do not differentiate the multipliers.** They are state. The released
  notebook detaches them [code: Code/Pytorch/allen-cahn_rba-sa.ipynb#L292].
- **Adam, not plain gradient descent.** Algorithm 1 is written for gradient
  descent, but the experiments used Adam [arxiv:2307.00379 §3.1]
  [arxiv:2307.00379 alg. 1].
- **Fixed Fourier periods.** The embedding is tied to the domain length, so a
  different domain needs a new period [arxiv:2307.00379 §2.3.3].
- **Ablation numbers.** The Allen-Cahn ablation uses the best seed of the
  main table, and the Helmholtz ablation's full-model number is its best seed
  [arxiv:2307.00379 §3.1.1] [arxiv:2307.00379 §3.2.1]. Compare five-seed means
  in the main tables.
- **Baselines.** Apart from the self-adaptive rows, the baselines in the
  Allen-Cahn and Helmholtz tables were not re-run [arxiv:2307.00379 §3.1]
  [arxiv:2307.00379 §3.2].
- **AIV inconsistencies.** The text gives a 60/40 training/validation split for
  PTV data, but the appendix table gives 9.4e3 training and 1.4e4 validation
  points, which is the reverse [arxiv:2307.00379 §3.3] [arxiv:2307.00379 app. A.3.3].
  The text gives 2.5e5 iterations, and the ablation figure caption gives 2.5e4
  [arxiv:2307.00379 §3.3] [arxiv:2307.00379 fig. 7]. The paper does not say
  which value is correct.
- **Timing.** The authors say the code is not tuned for speed, and the three
  cases ran on different GPUs, so the timings are not comparable across cases
  [arxiv:2307.00379 app. A.2] [arxiv:2307.00379 app. A.3.3].
- **The stress scale sets the meaning of the penalty weight.** In P-DivGNN the
  divergence is computed on standardised predictions. The NMSE does not depend
  on scale, but the penalty does, so a change of units or normalisation needs
  a new weight [arxiv:2507.05291 §4.2] [code: scripts/gnn_train.py#L162-L167].
- **Early epochs can mislead.** The mean stress is uniform over the nodes, so
  the divergence starts near zero. The paper's NMSE drops later with the
  penalty, around epoch 60 rather than 40 [arxiv:2507.05291 §4.2]
  [arxiv:2507.05291 fig. 5]. Do not judge a run by its first epochs.
- **Nodal stress needs averaging.** Element stresses are constant or
  discontinuous, so the paper averages interpolated stress from the surrounding
  elements at each node before the divergence is taken [arxiv:2507.05291 §4.1]
  [arxiv:2507.05291 §2.3].
- **Test-set checkpoint selection.** The P-DivGNN training script keeps the
  checkpoint with the lowest test loss. The paper says the test set is used for
  results only, so its reported numbers are selected on that same data
  [arxiv:2507.05291 §4.1] [code: scripts/gnn_train.py#L284-L292]. Hold out a
  validation split before comparing with it.
- **Coverage and loss form in the survey.** The survey does not say how it
  chose its papers, and it gives no count [arxiv:1903.12394 §3.1]. Its
  knowledge term in eq. 1 has no sum over samples, and the survey does not say
  which functional measures violation [arxiv:1903.12394 eq. 1]
  [arxiv:1903.12394 §5.1]. Read eq. 1 as a form, not a recipe.
- **DE onsets are found by eye.** The 2024 paper gives no detector, and it
  says the first-order nature of the onset is unexplained [arxiv:2403.18494 §4].
  Do not treat a DE label as a measured quantity.
- **SNR batch layout.** The paper gives a batch-independent SNR, SNR / sqrt(m),
  and effective batch sizes that account for symmetry. The released code does
  not implement these [arxiv:2403.18494 app. A].
- **Architecture is fixed in the 2024 study.** All cases use 6 x 128 tanh
  networks with no sweep, and the DE onset may depend on architecture and
  stiffness [arxiv:2403.18494 §3.1].
- **Single runs in the 2024 study.** The 10x speed-up on Allen-Cahn and the
  cavity result are single runs with no spread [arxiv:2403.18494 fig. 4]
  [arxiv:2403.18494 fig. 5].
- **Error bars depend on the prior and noise model.** Cheap methods match
  expensive ones mainly when the prior or the noise scale is learned during
  inference [arxiv:2201.07766 §7]. With fixed priors, the paper says the
  epistemic part does not grow reliably with less data or more noise
  [arxiv:2201.07766 §6.1].
- **Calibration and out-of-distribution inputs.** Post-training calibration uses
  data from the training distribution, and the paper warns that performance may
  get worse out of distribution [arxiv:2201.07766 §5.2]. The combined uncertainty
  of deep ensembles did not always cover the errors in the stochastic statistics
  [arxiv:2201.07766 §7].
- **Gaussian predictive assumption.** The comparison metrics assume Gaussian
  predictive distributions, often fitted to posterior samples
  [arxiv:2201.07766 §3.1] [arxiv:2201.07766 §5.1].
- **Error-bar cost is indicative.** The paper's wall-clock table is for one
  function example and is not a benchmark. HMC with 50 leapfrog steps costs
  about 4 to 5 times standard training, and a serial deep ensemble costs M times
  standard training [arxiv:2201.07766 §6.6].
- **Hyperparameters are per experiment.** Dropout rate, weight decay, learning
  rate and cycle length were tuned for each experiment. The paper gives only
  cross-validation and meta-learning as options [arxiv:2201.07766 §6.1]
  [arxiv:2201.07766 §4.1] [arxiv:2201.07766 §4.2].
- **Partial or total derivative.** Stress and dissipation are written as partial
  derivatives of the energy network with respect to its strain and internal
  variable inputs. The internal variables also depend on strain through their
  own network, so a total derivative would add that path and change the tangent.
  The paper does not say which it uses. The card reads the partial form, which
  the step list supports; this is an interpretation [arxiv:2005.12183 §3.3].
- **Mixed units in the energy-potential reference.** The 1D yield function
  combines a modulus in GPa with stress and yield values in MPa, and the paper
  gives no unit system. Choose one and convert every input and target
  [arxiv:2005.12183 eq. 25] [arxiv:2005.12183 tab. 3].
- **Activation rule for the leaky-ReLU network.** The second-derivative rule of
  §3.2 is shown for the energy network. The paper does not say whether it also
  applies to the leaky-ReLU network that predicts the internal variables
  [arxiv:2005.12183 §5.1.1].
- **Energy-potential training details are missing.** Not stated: learning rate
  for the applications, epochs, the loss weights, the leaky-ReLU slope and the
  3D batch size [arxiv:2005.12183 §5.1.1] [arxiv:2005.12183 §5.2.1]. The
  learning rate of 1e-5 is stated for the toy example only [arxiv:2005.12183
  §3.2.1].
- **Out-of-range accuracy drops.** Strain increments beyond the training range
  reduce accuracy, and the tangent claim outside that range rests on plots only
  [arxiv:2005.12183 §5.1.3] [arxiv:2005.12183 §6].
- **Extensions are untested.** The non-isothermal architecture and the claimed
  extension to damage or viscosity are not tested [arxiv:2005.12183 fig. 7a]
  [arxiv:2005.12183 §5 intro].
- **Real materials need data the paper does not provide.** Each training pair
  needs the free energy and dissipation at the next step. The examples take both
  from analytic models; for real materials they must come from micromechanical
  simulation or experiment, which the paper does not show [arxiv:2005.12183 §1]
  [arxiv:2005.12183 §4.2].
- **Upwind branch for CAN-PINN.** The paper gives the positive-flow face formulas
  only [arxiv:2110.15832 eq. 8a] [arxiv:2110.15832 eq. 8b]. The negative branch
  and the rule for choosing the upwind side are not stated. The released code
  uses a mirror formula and the average of the two face velocities
  [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18]. Say which rule you
  use.
- **Residual points per batch.** The sampler draws the batch from all points,
  boundary points included, so a batch yields fewer PDE residuals than its
  nominal size [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-16]
  [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18].
- **Corner double count.** Left and bottom boundary terms both include the
  bottom-left corner, so it counts twice in the boundary loss
  [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18].
- **Pressure is fixed only up to a constant.** The boundary loss enforces u and
  v alone, and the paper states no pressure condition [arxiv:2110.15832 §3.3].
- **Forward-only code.** The released notebooks cover the Re = 400 cavity
  forward case. They do not reproduce the inverse or the other reported
  problems [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18].

## Contradictions

The two RBA cards describe the same moving-average update with different
constants and bounds. The 2023 card uses an independent gamma and gives
Allen-Cahn eta* = 1e-2 with gamma = 0.999, so the multiplier can reach 10
[arxiv:2307.00379 eq. 13] [arxiv:2307.00379 app. A.2]. The 2024 card sets
gamma = 1 - eta*, so the multiplier stays at most 1 [arxiv:2403.18494 §2.5]. Its
Allen-Cahn setup also keeps weight 1 on the initial term, where the 2023 card
uses 100 [arxiv:2307.00379 §3.1] [arxiv:2403.18494 eq. 25]. The 2024 card
credits RBA to an earlier paper by the same authors and does not link it by
arXiv id [arxiv:2403.18494 §2.5]. The two cards do not reconcile these
settings, so do not assume the 2024 constants transfer to the 2023 loss.

Within [arxiv:2307.00379](/papers/2307.00379); its code is at commit c8ca4cb5
under the MIT licence:

- **Allen-Cahn RBA learning rate.** The paper gives eta* = 1e-2
  [arxiv:2307.00379 app. A.2]; the code uses 1e-3 and explains that the
  initial-condition term is scaled to compensate
  [code: Code/Pytorch/allen-cahn_enhanced.ipynb#L291-L292].
- **Helmholtz loss form.** The paper writes the squared, weighted residual
  [arxiv:2307.00379 eq. 10]. The PyTorch code matches it; the JAX code
  multiplies the squared residual by the weight, which is a different loss for
  weights above zero [code: Code/Jax/helmholtz_fourier.ipynb#L74-L75].
- **Optimiser.** Algorithm 1 is plain gradient descent; all runs used Adam
  [arxiv:2307.00379 alg. 1] [arxiv:2307.00379 §3.1].
- **Helmholtz residual points.** The paper gives 1201; the code uses a 101 x 101
  grid, which is 10201 points [arxiv:2307.00379 app. A.2]
  [code: Code/Jax/helmholtz_fourier.ipynb#L184-L185].
- **Helmholtz L-BFGS.** The paper gives 2e3 iterations; the code runs at most
  1005 [arxiv:2307.00379 §3.2] [code: Code/Jax/helmholtz_fourier.ipynb#L166-L167].
- **mMLP gate order.** The PyTorch code matches the paper's Eq. 17; the JAX
  code swaps the roles of the two encoders. Both train
  [arxiv:2307.00379 eq. 17] [code: Code/Jax/helmholtz_fourier.ipynb#L111].
- **Helmholtz initialisation.** The paper says Xavier; the code uses Glorot
  normal with zero bias [arxiv:2307.00379 §3.2] [code: Code/Jax/helmholtz_fourier.ipynb#L79-L81].

Within [arxiv:2403.18494](/papers/2403.18494); its code is at commit 312b830d
under the MIT licence:

- **SNR parameters.** The paper uses all network parameters [arxiv:2403.18494 §2.1].
  The code uses weight matrices only, with biases excluded
  [code: utils/snr.py#L27] [code: utils/func.py#L41].
- **Adam SNR.** The paper gives a simplified update without bias correction
  [arxiv:2403.18494 §2.2]. The code uses bias-corrected moments and includes
  biases [code: utils/snr.py#L51-L62]. The diagnostic is off by default.
- **Smoothing window.** The paper uses 100 past iterations [arxiv:2403.18494 §3.1].
  The notebook uses a window of 60 stored rows, and the code stores a row every
  100 iterations, and every 10 before iteration 300, so the window covers
  thousands of iterations [code: snr_plots/Paper_plots.ipynb#L221]
  [code: utils/snr.py#L21].
- **RBA normalisation.** The paper divides by the largest residual magnitude and
  bounds the multipliers in [0, 1] [arxiv:2403.18494 §2.5] [arxiv:2403.18494 eq. 26].
  The Helmholtz code first subtracts the minimum and divides by the range, and
  the shifted weights enter the training loss [code: HM/HM.py#L293-L299]. The
  appendix calls the shift cosmetic [arxiv:2403.18494 app. D].
- **Cavity collocation count.** The paper gives 25,600 points
  [arxiv:2403.18494 app. D]; the code uses 10,000 [code: CV/CV.py#L469].

Within [arxiv:2507.05291](/papers/2507.05291); its code is at commit 7847c02e
under GPL-3.0-or-later, and the card paraphrases it without quoting:

- **Learning rate.** The paper uses 1e-2 for both datasets
  [arxiv:2507.05291 §4.2] [arxiv:2507.05291 §5.2]. The released configs set
  0.001 [code: scripts/configs_train/config_train_div.yml#L6].
- **Hyperelastic divergence weight.** The paper sets 1e-2 for hyperelastic
  data [arxiv:2507.05291 §5.2]. The hyperelastic config sets 10
  [code: scripts/configs_train/config_train_div_hyperelast.yml#L9]. The paper
  also says the weight was not tuned [arxiv:2507.05291 §5.3].
- **Early stopping.** The paper uses patience 20 on validation NMSE
  [arxiv:2507.05291 §4.2]. The released configs turn early stopping off
  [code: scripts/configs_train/config_train_div.yml#L12].
- **Linear divergence numbers.** Section 4.3 gives 4.56e-4 for P-DivGNN and
  9.43e-4 for FEM [arxiv:2507.05291 §4.3]. Table 1 gives 3.23e-4 and 9.18e-4
  [arxiv:2507.05291 tab. 1]. The paper does not explain the difference.
- **Periodic edge feature.** The paper sets the feature to zero
  [arxiv:2507.05291 §3.1]. The code standardises all edge features, so these
  edges do not enter as zero [code: gnn_local_stress/models.py#L154-L162].

Within [arxiv:2005.12183](/papers/2005.12183); the card records no official code,
only a related repository by the first author that does not contain this
paper's training setup:

- **Early stopping.** The application section stops training when the test error
  rises [arxiv:2005.12183 §5]. The text also uses a validation set for early
  stopping [arxiv:2005.12183 §3.2.1] [arxiv:2005.12183 §5.1.1]. Use validation
  for stopping and keep the test set for final numbers.

Within [arxiv:2201.07766](/papers/2201.07766); the card records no code release:

- **Calibration spread.** The text says post-training calibration makes all
  methods roughly equally calibrated [arxiv:2201.07766 §6.1], while the
  calibrated RMSCE column of the same table ranges from 3.8 for MFVI to 8.0 for
  MCD [arxiv:2201.07766 §6.1]. Do not take either as settled.

Within [arxiv:2110.15832](/papers/2110.15832); the code is at commit a92594e6
with no licence file, so it is cited by path only:

- **Momentum residual.** The paper's conservative momentum form has no extra
  divergence term [arxiv:2110.15832 eq. 27b]. The code's momentum residual adds
  minus u times the discrete divergence, and the same in v
  [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18].
- **Boundary weight.** The paper weights the boundary and initial terms by
  λ_BC and λ_IC [arxiv:2110.15832 eq. 2a]. The code divides the PDE loss by λ
  and adds the boundary loss with no separate weight
  [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18].
- **Weight initialisation.** The paper gives N(0, σ²); the code uses a truncated
  normal with σ = 1 [code: d01_can-pinn_lid_driven_cavity_re400.ipynb#cell-18].

## Papers

- [Residual-based attention in physics-informed neural networks](/papers/2307.00379): decaying per-point multipliers from normalised PDE residuals, with no gradients or extra networks; author-reported gains on Allen-Cahn, Helmholtz and an inverse flow problem.
- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](/papers/2403.18494): full-batch Adam on PINNs, a batch-gradient SNR with three training phases, and RBA with the decay tied to the rate; author-reported single-run gains on Allen-Cahn and a cavity flow.
- [Physics-Informed Graph Neural Networks to Reconstruct Local Fields Considering Finite Strain Hyperelasticity](/papers/2507.05291): a graph network for periodic micro-structure meshes with a penalty on the discrete stress divergence; author-reported gains on linear and hyperelastic meshes.
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](/papers/1903.12394): a taxonomy of prior knowledge in learning, including equations in the loss and the overlapping names for physics-informed methods; a survey with no experiments of its own.
- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](/papers/2201.07766): posterior and ensemble error bars for PINNs and DeepONets, functional GAN priors and post-training calibration; author-reported comparisons on prototype problems, no code released.
- [Thermodynamics-based Artificial Neural Networks for constitutive modeling](/papers/2005.12183): a network predicts a free-energy potential and internal-variable increments, and stress follows by differentiation so the first law holds by construction; author-reported on isothermal 1D and 3D elasto-plasticity, with data from the analytic models.
- [CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method](/papers/2110.15832): couples autodiff slopes with finite-difference-style neighbour terms for sparse collocation points; author-reported gains over AD-only and numerical-differentiation PINNs on flow problems, with code for the forward cavity case only.