Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

In plain words

Neural networks that learn from data and from physics equations give a single answer, but they rarely say how sure they are. This paper asks how to attach honest error bars to such models. It separates two kinds of doubt: noise in the measurements that no model can remove, and doubt caused by having too little data or an uncertain network.

The authors build one framework around two model families, physics-informed networks that solve equations and networks that learn a whole solution mapping (operators). They then plug in several standard uncertainty techniques, add a few of their own, and test them on toy problems with known answers. They also measure how well the error bars match reality and how to correct them after training with a small leftover dataset.

The practical message is that a cheaper method often gets close to an expensive one, that the choice of prior and noise model matters, and that a calibration step can make the error bars more trustworthy.

TL;DR

• Defines total predictive uncertainty as aleatoric (data noise) plus epistemic (model uncertainty), estimated from posterior samples §3.1.
• Applies Bayesian methods (HMC, Langevin, mean-field VI, MC dropout, Laplace) and ensembles (deep, snapshot, SWAG) to PINNs and DeepONets §4.1, §4.2.
• Adds functional priors from pre-trained GANs, a physics-agnostic DeepONet variant for noisy inference data, and an extension of NNPC for stochastic PDEs §4.3, §4.4.
• Author-reported comparisons on prototype problems suggest cheap methods (LA, MFVI, SEns) can match expensive ones when priors are learned, and that small-sample calibration can cut calibration error §6.1, §7.

Problem

Standard neural networks give point predictions with no error bars. In scientific ML, data are noisy and sparse, the physics may be partly unknown, and network size and optimisation add further doubt. Standard training ignores epistemic uncertainty §3.1. The paper targets three problem classes: forward and mixed deterministic PDEs, mixed stochastic PDEs, and operator learning with noisy inference data §1.2. It assumes the data-generating process is u(x) = u_c(x) + ε_u with additive noise, and that the likelihood, prior and network architecture are fixed by the user, collectively written H §3.1.

Method

Problem formulation

A PDE F_{λ,ξ}[u(x;ξ)] = f(x;ξ) with boundary operator B_{λ,ξ}[u] = b on the boundary Γ, where λ are problem parameters and ξ indexes randomness §1.2. Four scenarios are defined: (1) forward deterministic PDE with noisy f and b data; (2) mixed deterministic PDE with noisy u and λ data as well; (3) mixed stochastic PDE with N noisy paired realizations; (4) operator learning with clean pre-training data and noisy, limited inference data §1.2.

Background: PINN and DeepONet

A PINN uses a network u_θ(x), differentiates it through the PDE operators to get f_θ and b_θ, and minimises a weighted sum of mean squared errors over the four data categories (f, b, u, λ) with weights w_f, w_b, w_u, w_λ §2.1. A DeepONet has a branch net on the sampled input function λ and a trunk net on the query point x; the output is their inner product. It is trained by MSE on the solution data §2.2.

Uncertainty modelling

1. Assume a Gaussian likelihood p(u|x,θ) = N(u | u_θ(x), Σ_u²) with a constant noise variance σ_u² (homoscedastic), or with a location-dependent variance predicted by an extra network output (heteroscedastic) §3.1.
2. Put a prior on θ and form the posterior p(θ|D) by Bayes' rule §3.1.
3. Approximate the predictive density by a Monte Carlo mixture over M posterior samples θ̂_j §3.1.
4. Read off the predictive mean and the total variance, which splits into an aleatoric and an epistemic part §3.1.
5. For PINNs, the data likelihood factorises over the four data categories §3.2. For DeepONets, it is over the solution data only, with clean λ inputs §3.3.

Posterior inference

Each method produces M samples θ̂_j. The paper calls the Bayesian versions "U-PINN" and "U-DeepONet" (U = uncertain) §4.1.

• Bayesian methods [§4.1; details in app. "Supplementary material to Section 4"]:
  • HMC: Hamiltonian dynamics with leapfrog steps and rejection.
  • LD: Langevin dynamics, one step per update, no rejection.
  • MFVI: factorised Gaussian variational posterior on the network weights.
  • MCD: standard training with dropout, then dropout at inference.
  • LA: Gaussian fitted at the trained mode, width from the Hessian.
• Ensembles §4.2:
  • DEns: M independent trainings from different initialisations.
  • SEns: snapshots along one training run with a cyclic learning rate.
  • SWAG: SEns plus a Gaussian fitted to the snapshots.
• Heteroscedastic variants (h-X) add one network output for the noise. The noise is then part of the likelihood, so all inference methods use it §4.1.
• Noise scale and priors: online (alternating updates) or offline (cross validation, meta-learning) identification is described §4.1, §4.2.

Functional priors

A prior on the network weights may not reflect the function space well. The paper uses a generator from a GAN pre-trained on historical realizations as the model, so posterior inference runs on a low-dimensional latent input θ §4.3.

• GAN-FP: generator u_θ(x) pre-trained on historical data U_i; posterior inference on θ with any method above, denoted X+FP §4.3.
• PI-GAN-FP: the same generator used inside a PINN for mixed PDEs [§4.3, §4.5.1].
• PA-BNN-FP and PA-GAN-FP: a DeepONet pre-trained on clean data, plus a BNN or GAN prior. These treat noisy and limited inference data for f, u and λ §4.3, §4.5.2.

Stochastic PDEs

• PI-GAN: a GAN generator takes random θ from a known distribution and is trained on stochastic realizations; new samples of u come from new θ draws §4.4.
• U-PI-GAN: PI-GAN combined with posterior samples of the generator parameters β §4.4.
• GP+PI-GAN: a GP is fitted to noisy f data, its realizations drive a PI-GAN, which propagates uncertainty to u. Described as new §4.4.
• NNPC and NNPC+: a polynomial-chaos expansion of u with a network for the stochastic modes; NNPC+ adds a decoder network for f [§4.4, app. "Solving SPDEs with U-NNPC(+)"].

For the total variance of a stochastic model, the paper combines the aleatoric, epistemic and stochastic parts with M posterior samples and M′ realizations §4.4.

Evaluation metrics

All metrics are computed on test data. Let ū(x) be the predictive mean and σ̄²(x) the total variance §5.1.

• RL2E: relative L2 error of the mean over test points §5.1.
• MPL: mean predictive likelihood, the Gaussian pdf at each test value §5.1.
• RMSCE: root mean squared calibration error over a grid of probabilities p, comparing the fraction of test values inside the p-quantile of a Gaussian predictive with p §5.1.
• Secondary metrics: PIW (prediction interval width) and SDCV (spread of the predicted standard deviation over x), in app. "Additional evaluation metrics". Gold-standard metrics NIP-G and KL-G need a reference solution §5.1, app. "Additional evaluation metrics".

Post-training calibration

Given a left-out calibration set {x_i, u_i}, three methods are described §5.2:

• Variance scaling (Levi et al.): multiply σ̄² by a constant s chosen by optimising one metric. The mean is unchanged §5.2.
• CDF correction (Kuleshov et al.): fit a map Q approximating the observed calibration curve and apply it to all predictive CDFs. This changes mean and variance §5.2, app. "CDF modification by fitting an auxiliary model".
• CRUDE (Zelikman et al.): fit an empirical distribution to the scaled residuals (u − μ̄)/σ̄ of the calibration set §5.2, app. "CRUDE".

Pseudocode: predictive mean and variance from posterior samples

# Predictive mean and total variance at one point x  [§3.1]
# Input:  samples (M parameter sets), point x, noise variance sigma_a2
# Output: mean (scalar), variance (scalar)

def total_moments(samples, x, sigma_a2):
    # Mean prediction of each posterior sample  [§3.1]
    preds = [net(theta, x) for theta in samples]
    # Average over samples gives the predictive mean  [§3.1]
    mean = sum(preds) / len(samples)
    # Epistemic part: mean squared deviation, divided by M  [§3.1]
    epistemic = sum((p - mean) ** 2 for p in preds) / len(samples)
    # Total variance adds the aleatoric noise variance  [§3.1]
    variance = sigma_a2 + epistemic
    return mean, variance

For heteroscedastic noise, sigma_a2 is itself an average over samples of a per-sample noise output §3.1.

Pseudocode: PINN objective §2.1

# Weighted PINN loss for the mixed problem  [§2.1]
# Input:  weights (w_f, w_b, w_u, w_lam), batches of (x, value) per category
# Output: scalar loss

def pinn_loss(theta, data, weights):
    # Residual of the PDE at the f-points, via automatic differentiation  [§2.1]
    residual_f = F_op(net_u, theta, data.x_f) - data.f
    # Boundary residual at the b-points  [§2.1]
    residual_b = B_op(net_u, theta, data.x_b) - data.b
    # Data mismatch for u and lambda  [§2.1]
    residual_u = net_u(theta, data.x_u) - data.u
    residual_lam = net_lam(theta, data.x_lam) - data.lam
    # Weighted mean squared sum  [§2.1, eq. for L(θ)]
    loss = (weights.f * mean(residual_f ** 2)
            + weights.b * mean(residual_b ** 2)
            + weights.u * mean(residual_u ** 2)
            + weights.lam * mean(residual_lam ** 2))
    return loss

Unified view

The paper lists which posterior method, prior, and physics model combine into each named method §4.5, Fig. "all techniques". Neural PDEs (PINN-based) and neural operators (DeepONet-based) share the same posterior machinery. The model M is a BNN, a GAN generator, or absent §4.5.

Hyperparameters

Values are as printed in the paper's appendix tables. The paper states that TensorFlow Adam was the default optimiser with learning rate ε [§6, app. "Hyperparameters used in the comparative study"]. Architectures for the function example: 2 hidden layers with 50 units, tanh activation; the prior on each weight and bias is N(0, 1) §6.1, app. "Neural network architectures".

Function approximation, known homoscedastic noise §6.1

Noise N(0, 0.1²). Training data N = 32 points on [−0.8, −0.2] ∪ [0.2, 0.8].

Function approximation, unknown noise §6.1

Mixed deterministic PDE §6.2

Problem: the diffusion–reaction equation u_t = D u_xx − λ(x) u³ + f(x) with D = 0.01 on t ∈ [0, 1], x ∈ [−1, 1]. Data: N_f = 15, N_λ = 6, N_u = 66 points (11 in x, 6 in t). Reference by the MATLAB PDE toolbox on a 101 × 101 grid.

Architecture: 2 hidden layers of 50 units, tanh, for u and λ [§6.2, app. architectures]. The additional cases (heteroscedastic noise, large noise, extrapolation, steep boundaries) use HMC with ε between 10⁻³ and 10⁻², T of 30 or 50, and M = 10³ §6.2, app. additional-case table. The h-HMC row's burn-in is printed as 5×10⁻³, which looks like a typo for the step count; check the source before use.

Mixed stochastic elliptic problem §6.3

Equation: −(1/10) d/dx (λ(x;ξ) du/dx) = f(x;ξ), with f a GP with mean 1/2 and kernel (9/400) exp(−(x₁−x₂)²), λ = exp((1/5) sin((3/2)π(x+1)) + λ̃), and λ̃ a GP with kernel (1/100) exp(−(x₁−x₂)²) §6.3. Data: N = 1,000 realizations; N_f = 13, N_u = 9, N_λ = 5 sensors. Noisy case: homoscedastic noise σ_u = σ_f = σ_λ = 0.01. Reference: finite differences. Evaluation: 500 unseen realizations, 201 grid points §6.3.

GAN architectures for §6.3: generator 128 × 2 tanh, discriminator 128 × 3 Leaky ReLU, K = 20, 10⁵ steps app. GAN architecture table.

Forward deterministic PDE §6.4

Operator learning, Darcy flow §6.5

Problem: ∇·(λ ∇u) = 50 on the unit square, u = 1 at x = 0, u = 0 at x = 1, no-flux on y = 0 and y = 1. λ = exp(λ̃), with λ̃ a truncated 100-term Karhunen–Loève expansion of a zero-mean GP with kernel exp(−|x−x′|²/(2l²) − |y−y′|²/(2l²)), l = 0.25 §6.5. Pre-training data: N = 1,000 realizations, log λ and u standardised pointwise §6.5. Inference: N_λ′ = 20 and N_u′ = 10 noisy points, or 40 × 40 clean λ on the training grid §6.5.

GAN for PA-GAN-FP: K = 100 latent dimensions, 10⁵ training steps [app. GAN architecture table].

Results

All numbers are author-reported on the paper's own prototype problems. Metrics are as defined in §5.1; × 10² means the table values were scaled by 100. Table numbers in the paper are not reproduced here; each row cites its section.

Function approximation, known noise, in-distribution §6.1

Setup: N = 32 noisy points, noise N(0, 0.1²), ID test set; the GP is an analogue GP for a BNN with N(0, 1) prior, used as reference §6.1. The paper says post-training calibration makes all methods roughly equally calibrated §6.1, table caption.

Mixed deterministic PDE, standard case §6.2

Metric ×10² for u(t,x) over all test points: RL2E HMC 3.7, MFVI 6.1, MCD 13.5, DEns 4.1, HMC+FP 1.6 §6.2, table a. For f: RL2E 8.3, 10.0, 10.9, 8.2, 4.8 §6.2, table b. For λ: RL2E 17.3, 15.3, 19.5, 14.8, 2.6 §6.2, table c. The paper reports learned noise σ_u (reference 0.05) at 0.051 for HMC, 0.052 for MFVI, 0.041 for DEns §6.2, table d. The learned-prior method HMC+FP is best on all three quantities in this case §6.2.

Mixed stochastic elliptic problem, noisy data §6.3

Metric ×10² (test data clean), 1,000 training realizations:

Source: §6.3, table. The paper calls U-NNPC+ robust to noise. Calibration on 2–10 left-out realizations reduced RMSCE by about a factor of 3 in one case §6.3.

Operator learning, Darcy flow, limited noisy inference data §6.5

Metric ×10² with ID and OOD inference data. R2LE for λ and u uses clean reference data; MPL and RMSCE use noisy data §6.5, table caption.

The paper states that the BNN variant costs nothing extra and is better calibrated for u, while the GAN variant is more accurate §6.5, §7.

Cost §6.6

Indicative wall-clock for posterior inference on the function example (Table in §6.6; CPU: 2 Xeon E-2643 at 3.3 GHz, GPU: TITAN Xp; not a benchmark):

The paper states HMC with 50 leapfrog steps costs about 4–5 times standard training. DEns costs M times standard training if run serially §6.6.

Paper vs code

No official code.

The paper says the UQ library "will be made available" and that code will be released in a public GitHub repository, with an upcoming manual [abstract, §6.6]. Searches with the WebFetch GitHub API at the time of compiling:

• "uncertainty quantification scientific machine learning" within the PredictiveIntelligenceLab organisation: no results.
• "uq pinn deeponet" within the same organisation: no results.
• "UQ-SciML": one hit, ZongrenZou/HJPDE4UQSciML, for a viscous Hamilton–Jacobi PDE project with no license. It does not match this paper and is not used.
• "UQ SciML DeepONet uncertainty": one hit, atticusdrex/elyza, a third-party JAX library; not used.
• A listing of PredictiveIntelligenceLab repositories was truncated before the end, so the search was not exhaustive.

Pitfalls

• Equation and table numbers. This card cites sections rather than equation or table numbers, because the numbers in the compiled PDF were not verified. Look up the equations by section.
• Epistemic term normalisation. The paper's variance uses 1/M, not 1/(M−1); the two differ by the factor M/(M−1) §3.1.
• Gaussian approximation. Predictive distributions are often fitted by a Gaussian (or Student-t) to the samples. The comparative metrics assume a Gaussian §3.1, §5.1.
• Priors and noise must be learned. Cheap methods match expensive ones mainly when the prior or noise scale is learned during inference §7. With fixed priors, the paper says epistemic uncertainty does not grow reliably with less data or more noise §6.1.
• Over-confidence on OOD data. MC dropout gave good ID metrics but its uncertainty did not increase for OOD data, which the paper flags as risky §6.1. The SDCV diagnostic is used to detect this §6.1.
• GP+PI-GAN. It can outperform U-PINN in some forward cases but may be over-confident because the PDE is not used to fit f §7.
• Calibration may hurt out of distribution. Calibration uses data from the training distribution; performance may deteriorate for OOD tests §5.2.
• Stochastic methods need realizations. U-NNPC and U-NNPC+ cannot sample new realizations from a known distribution, so new f and λ draws are needed for new u samples §6.3.
• Noisy pre-training inputs are not supported. DeepONet requires clean λ in pre-training; noisy λ is listed as future work §3.3, §6.5.
• Deep ensembles may under-cover. Combined epistemic uncertainty did not always cover errors in the stochastic statistics §7.
• Hidden costs. HMC is about 4–5 times standard training for 50 steps. For HMC and LD the CPU was faster than the GPU in the authors' setup; for the Adam-trained methods the GPU was usually preferable on larger data §6.6.
• Hyperparameter tuning. Dropout rate, weight decay, learning rate and cycle length were tuned per experiment, and the paper gives no general selection rule beyond cross validation and meta-learning as options [§6.1, §4.1, §4.2].
• Physics-agnostic inference. PA-BNN-FP costs nothing extra and is better calibrated for u, while PA-GAN-FP is more accurate, including on OOD inputs §6.5, §7.

Lineage

• Builds on the PINN §2.1 and DeepONet §2.2 frameworks, cited to their original authors in the paper.
• Uses the GAN-based functional prior of Meng et al. §4.3, PI-GAN (Yang et al.) and NNPC (Zhang et al.) §4.4, and the calibration methods of Levi et al., Kuleshov et al. and Zelikman et al. §5.2.
• Introduces GP+PI-GAN, PA-BNN-FP, PA-GAN-FP, U-NNPC+ and three OOD detection metrics as new contributions §1.3, §7.