Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial

In plain words

A neural network gives an answer, but not how sure it is. In engineering, an answer like "this battery lasts 900 cycles" is only useful if the model also says how much to trust it. This paper is a guided tour of several ways to attach that "how sure" number to a neural network's prediction.

The paper contrasts two kinds of doubt. Some doubt is noise in the data that more data cannot remove. Some doubt comes from having seen too little data, which more data can reduce. The methods try to separate or estimate these parts. One simple idea is to train several copies of the same network with different random starts and see where they disagree. Another is to keep randomly switching neurons off at prediction time and look at the spread. Gaussian process regression is a classic approach that gets less uncertain close to its training data and more uncertain far from it.

The paper also checks whether the stated uncertainty is honest: when a model says it is 95% sure, the true answer should fall inside that range about 95% of the time. Two case studies apply these methods to predicting battery life and aircraft engine remaining life, and the authors share code for both.

TL;DR

• Covers four UQ approaches for neural networks: Gaussian process regression (GPR), Bayesian neural networks (with MC dropout as the scalable variant), neural network ensembles with Gaussian output heads, and deterministic SNGP.
• Gives a split of total predictive variance into aleatory and epistemic parts §2.2, applied to the ensemble §3.3.
• Evaluates calibration with reliability curves, expected calibration error (ECE), NLL and RMSE on two prognostics case studies §6.1, §6.2.
• Results are author-reported. The paper states it does not claim its methods beat the literature §6.2.
• The card flags a table caption/label mismatch, an epoch count that differs between paper and code, and a text claim that the authors' own table does not support.

Problem

Neural networks and other ML models give point predictions and often do not indicate when they are unreliable §1. Out-of-distribution inputs can produce confident but wrong outputs §1. In safety-critical settings (the paper names healthcare, aviation and power grids) decisions need a usable uncertainty estimate §1. The paper also notes that most benchmarks score accuracy only and rarely score the quality of the stated uncertainty §1.

Aleatory uncertainty is irreducible noise in observations. Epistemic uncertainty is reducible and comes from limited data or model choices §2. Total predictive variance mixes both, and the paper argues the split matters for deciding whether more data or a different model will help §2.1, §2.2.

Method

The paper is a tutorial and has no single new method. Its four covered families are summarized below.

Variance decomposition (§2.2). For a model with parameters θ and a posterior p(θ|D), the law of total variance gives

Var(y|x,D) = E_{\theta}[Var(y|x,\theta)] + Var_{\theta}[E(y|x,\theta)] §2.2

The first term is the aleatory part and the second is the epistemic part. The paper illustrates this on a 2-input linear model, where epistemic variance depends on parameter posterior variances and covariances and on the input values, and not on the noise variance §2.2.

Gaussian process regression (§3.1).

1. Place a GP prior over the function, with zero mean and a squared-exponential kernel, optionally with one length scale per input dimension (ARD) §3.1.
2. Observe y = f(x) + ε with Gaussian noise ε of variance σ_ε² §3.1.
3. The posterior mean and variance at test points X* are §3.1:

\bar f_* = K_{t*}^T (K_{tt} + \sigma_\varepsilon^2 I)^{-1} y_t

cov(f_*) = K_{**} - K_{t*}^T (K_{tt} + \sigma_\varepsilon^2 I)^{-1} K_{t*}

4. Hyperparameters (length scale, signal amplitude, noise standard deviation) are set by maximizing the log marginal likelihood, a point (MAP-style) estimate §3.1.
5. The posterior variance rises toward its prior maximum far from the training inputs. This gives distance-aware uncertainty §3.1.

Cost: training and the marginal likelihood scale as O(N³) in the number of training points, and storage as O(N²) §3.1. The paper's stated remedies are sparse and low-rank approximations, and dimension reduction for high-dimensional inputs §3.1.

Bayesian neural networks and MC dropout (§3.2). A Bayesian neural network puts a prior on weights and computes a posterior predictive by marginalizing over the weights §3.2. The paper covers MCMC, variational inference, and MC dropout. This card condenses only MC dropout. The MCMC and variational subsections of §3.2 are not summarized here.

MC dropout keeps dropout active at prediction time §3.2, subsubsection "MC dropout":

# MC dropout prediction  [§3.2 MC dropout]
# Input:  trained network with dropout layers, new input x
# Output: predictive mean and spread over T stochastic passes

def mc_dropout_predict(network, x, passes):
    # Keep dropout active at prediction time  [§3.2 MC dropout]
    # One random sparse sub-network per forward pass  [§3.2 MC dropout]
    outputs = []
    for t in range(passes):
        outputs.append(network(x, dropout_active=True))
    # Mean and spread over the passes  [§3.2 MC dropout]
    mean = average(outputs)
    spread = standard_deviation(outputs)
    return mean, spread

The paper interprets the dropout training loss as a single-sample approximation of a variational objective whose approximating family is a mixture of two Gaussians. It then warns that the result depends strongly on the dropout rate, number of dropout layers and training length §3.2.

Neural network ensembles with Gaussian heads (§3.3). Each member predicts a mean μ̂(x) and variance σ̂²(x) §3.3. Members are trained on the Gaussian negative log-likelihood, with the constant dropped §3.3, NLL loss. Members differ through random initialization and mini-batch order §3.3. The M members are combined as:

\bar\mu(x) = \frac{1}{M}\sum_{m=1}^{M}\hat\mu_m(x)

\bar\sigma^2(x) = \frac{1}{M}\sum_{m=1}^{M}\left(\hat\sigma_m^2(x) + \hat\mu_m^2(x)\right) - \bar\mu^2(x) §3.3

The average of σ̂_m² is read as aleatory uncertainty. The spread of the μ̂_m is read as epistemic uncertainty §3.3.

# Gaussian-head ensemble  [§3.3]
# Input:  training pairs (x_i, y_i), M members
# Output: members that predict (mean, variance) per input

def train_member(data, seed):
    # Random initialization and shuffled mini-batches differ per seed  [§3.3]
    network = build_network_with_gaussian_head(seed)
    # Minimize the Gaussian negative log-likelihood  [§3.3, NLL loss]
    fit(network, data, loss=gaussian_nll)
    return network

def predict_ensemble(members, x):
    # Collect (mean, variance) from each member  [§3.3]
    pairs = [member(x) for member in members]
    # Moment-match the mixture to one Gaussian  [§3.3, mixture formulas]
    return mixture_mean_and_variance(pairs)

Deterministic methods, SNGP (§3.4). A single forward pass produces the output §3.4. The paper's example is SNGP. It keeps distance information in the hidden layers and replaces the output layer with a Gaussian process §3.4. The steps are:

1. Use residual hidden layers and spectral normalization so each hidden weight matrix has spectral norm at most γ (the spectral norm upper bound) §3.4, spectral normalization.
2. Replace the output layer with a GP on the hidden features, using a squared-exponential kernel §3.4, GP output layer.
3. Approximate the GP with random Fourier features and a Laplace approximation of the posterior §3.4.

The paper states that SNGP and similar methods cannot separate aleatory and epistemic uncertainty §3.4, discussion.

Not covered in this card: the toy regression example (§3.5), the summary comparison table (§3.6), the sparsification and NLL metric subsections (§4.2, §4.3), the prognostics and engineering-design reviews (§5, appendix), and the appendices on GPR kernels and MC dropout instability.

Hyperparameters

Values as used in the case studies. Where the paper and code differ, both are given. Case study 1 uses one ResNet-style backbone for ensemble, MC dropout and SNGP §6.1.

Results

Author-reported. The paper states its case studies compare methods at similar accuracy and that it does not claim to beat the published literature §6.2. Values are mean ± standard deviation over 10 runs §6.1, §6.2. Table headers are reproduced as printed (see Pitfalls for the RMSE/MSE issue).

Case study 1: battery early life (169 LFP cells, §6.1, results table).

Source: §6.1, results table. Setup: ResNet backbone for NNE, MC and SNGP; GPR on the same features. Baseline: the other methods in the table.

Case study 2: turbofan RUL (N-CMAPSS DS02, §6.2, results table).

Source: §6.2, results table. GPR is excluded because of its scaling §6.2.

Qualitative findings the paper reports §6.1, §6.2:

• GPR fits the battery training set exactly (RMSE 0) but generalizes poorly, and its uncertainty is high on the test sets §6.1.
• SNGP tends to give large uncertainties, so it is underconfident and has high NLL and ECE on the battery data §6.1.
• MC dropout is overconfident, and its run-to-run variation is largest of the group §6.1, §6.2.
• NNE and SNGP are underconfident on the battery data; the paper calls underconfidence the safer error for battery early-life prediction §6.1.
• On turbofan data all three methods are overconfident for some units, most clearly unit 11 §6.2.

Paper vs code

Official code: the paper links https://github.com/VNemani14/UQ_ML_Review §6.1, §6.2. The owner matches a paper author. Cloned at commit 85a0ae4b19700bcf96194c3f469c6532ba8fbc4a, Apache-2.0 licensed (LICENSE file). The repository has notebooks for each case study and an SNGP toy example.

Discrepancies:

• Epochs. Paper: 300 epochs per ensemble member §6.1. The battery notebook calls the training function with epochs = 6000 in one cell and 8000 in others UQ_models_train_evaluate_Final-Resnet.ipynb UQ_models_train_evaluate_Final-Resnet.ipynb. Which setting produced the published table is not determined.
• Ensemble size. Paper: 15 members, chosen by elbow §6.1. The notebook sets 10 members in one cell UQ_models_train_evaluate_Final-Resnet.ipynb, 80 members with an elbow sweep up to 30 in another UQ_models_train_evaluate_Final-Resnet.ipynb, and 20 members with tanh in a third UQ_models_train_evaluate_Final-Resnet.ipynb. Not determined which feeds the table.
• MC dropout members. Paper: 15 passes §6.1. The notebook uses 5 and 3 members in different cells UQ_models_train_evaluate_Final-Resnet.ipynb UQ_models_train_evaluate_Final-Resnet.ipynb.
• SNGP activation. Paper: ReLU §6.1. The SNGP notebook has a sigmoid run UQ_models_train_evaluate_FinalSNGP.ipynb and a ReLU run UQ_models_train_evaluate_FinalSNGP.ipynb.
• GPR hyperparameters. Paper: grid search §6.1. The code calls .fit on the scikit-learn model without a visible grid UQ_models_train_evaluate_Final-Resnet.ipynb. Not verified further.
• NLL loss. Paper's loss omits the constant §3.3, NLL loss. The code adds 1e-6 to the loss UQ_models_train_evaluate_Final-Resnet.ipynb. This is a numerical guard, not in the paper.
• Variance head. Paper and code agree that variance is positive via softplus-type transform with a small offset §6.1 UQ_models_train_evaluate_Final-Resnet.ipynb.
• Dropout placement. Paper's listing and the notebook both put dropout after each residual block §6.1 UQ_models_train_evaluate_Final-Resnet.ipynb.

Pitfalls

• Table caption vs labels. The battery error table caption says "MSE ± standard deviation", but its row labels say "RMSE (cycles)" §6.1, results table. Treat the numbers as RMSE, as the labels say, and check against the spreadsheet in the repository.
• Text claim not supported by the table. The turbofan text says NNE has low ECE §6.2. On unit 15 NNE's ECE is 14.9%, above SNGP's 8.5% and MC's 11.5% §6.2, results table.
• Battery ECE claim. The text says GPR gives the least ECE on the other datasets §6.1. On the secondary test set MC dropout's 9.5% is lower than GPR's 10.4% §6.1, results table.
• Scope of the numbers. Each method uses the same simple backbone, not tuned per method. The paper says other architectures could do better §6.1.
• MC dropout sensitivity. The result depends on dropout rate, number of dropout layers and training length. The paper reports instability and overconfidence at low dropout rates §3.2, §6.1.
• Ensemble epistemic term. It is the spread of member means, so its size depends on how diverse the seeds and mini-batch orders make the members §3.3.
• Deterministic methods. SNGP needs spectral normalization and a bound γ. The paper says γ must be tuned against accuracy and distance awareness §3.4. In its discussion it reports that SNGP-type methods gave higher ECE than MC dropout and ensembles in other benchmarks §3.4, discussion.
• GPR cost. Exact GPR is O(N³) in training points, which rules it out for the turbofan data §3.1, §6.2.
• Numerical sensitivity. Inverting an ill-conditioned covariance needs a nugget term or a pseudoinverse §3.1.
• Unstated values. Learning rate for the ensemble, number of epochs in the final runs, and the choice between sigmoid and ReLU for SNGP are not settled by the text. Check the notebooks before reusing values.
• Author-reported. The paper says its implementations were verified against high-quality implementations §1. This card has not checked that claim.

Lineage

• Builds on: the Bayesian neural network literature and MC dropout §3.2; deep ensembles with Gaussian heads §3.3; spectral normalization and SNGP §3.4.
• Compared against: GPR, MC dropout and the Gaussian-head ensemble in the two case studies §6.1, §6.2.
• Superseded by: not determined from content in this wiki.