Multi-fidelity Bayesian Neural Networks: Algorithms and Applications

In plain words

Some measurements are cheap and plentiful but rough, such as a coarse simulation or a satellite image. Others are accurate but few and expensive, such as a buoy in the water. This method learns from both. A first network fits the cheap data. A second, "Bayesian" network learns how the accurate readings relate to the rough ones, and it reports how sure it is about each answer, not just a single guess. A third part can be told a physical law, written as an equation, so the answer must also respect it. The same machinery can estimate an unknown constant in the equation from a few noisy readings. When the model is unsure, it can say where a new measurement would help most, so the next reading is taken there. The authors test it on made-up functions, on two simple diffusion-reaction problems and on sea surface temperature.

TL;DR

• Use it when you have much low-fidelity data, few high-fidelity points, and need error bars.
• The low-fidelity network is trained once by MAP; the Bayesian part is sampled with HMC and given a prior fitted by mean-field VI.
• Author-reported results are small-scale (one- to four-dimensional functions, one- and two-dimensional PDEs, one SST day). No official code.

Problem

Single-fidelity Bayesian models use only the few accurate points and become inaccurate with little data §1. Multi-fidelity Gaussian-process regression uses both data sources, but its training cost scales as O(N³) in the total number of points and it mainly captures linear relations between fidelities §1. A composite deterministic network handles nonlinear relations and large datasets, but does not give uncertainty §1. This paper targets the gap: uncertainty, nonlinear and linear fidelity relations, and scalability to large low-fidelity sets §1.

The method assumes the high-fidelity measurements have known, independent Gaussian noise standard deviations §2.1. The low-fidelity data are assumed to be plentiful; the authors use MAP training on them to keep the cost low §2.1.

Method

Three networks, the last optional §2.1 fig. 1:

1. A DNN \tilde{u}_L fits the low-fidelity data by MAP, with an L2 penalty. Its weights are point estimates, not distributions §2.1 eq. 1.
2. A BNN takes the input location \bm{x}_H and the low-fidelity prediction \tilde{u}_L(\bm{x}_H) and predicts the high-fidelity value \tilde{u}. Its weights and biases \bm{\theta} have a Gaussian prior whose scale \sigma is set by data §2.2. Its posterior is sampled with HMC §2.1.
3. Optionally, a physics-informed term. For inverse PDE problems, the unknown constants \bm{\lambda} are added to \bm{\theta}, and the PDE residual and boundary operator are evaluated on the BNN output §2.1 eq. 3 eq. 5 eq. 6.

The prediction is \tilde{u}(\bm{x}, \tilde{u}_L; \bm{\theta}). The noisy high-fidelity data \mathcal{D} enter through a product of Gaussian likelihoods §2.1 eq. 4 eq. 7, which gives an unnormalised posterior eq. 8.

Low-fidelity loss (MAP). The DNN weights \bm{w}_L minimise the mean squared error on the low-fidelity data plus an L2 term with weight \alpha §2.1 eq. 1.

\mathcal{L}_L = \frac{1}{N_L}\lvert\bm{u}_L - \tilde{\bm{u}}_L\rvert^2 + \alpha\lvert\bm{w}_L\rvert^2 \qquad [\text{eq. 1}]

Prior. Every hidden layer is a tanh layer §2.2 fig. 1. Weights and biases are independent zero-mean Gaussians, with the weight scale of layer l set to \sigma/\sqrt{N_l} and the bias scale fixed at 1 §2.2. The one free prior parameter is \sigma.

Prior fitting (mean-field VI). Each parameter has an independent Gaussian q with mean \zeta_\mu and standard deviation \log(1+\exp(\zeta_\rho)) §2.2 eq. 10. The variational parameters and \sigma are chosen together by maximising the evidence lower bound §2.2 eq. 11. The paper writes the bound as a value to maximise, and its right-hand side is the expectation of \log Q - \log P(\bm{\theta}) - \log P(\mathcal{D}\mid\bm{\theta}), which is to be minimised. The authors use Adam for this step §2.2.

Posterior sampling (HMC). Starting from the VI prior, HMC with an adaptive time step draws samples of \bm{\theta} §3 alg. 1. The predictive mean and standard deviation at any \bm{x} are computed from the posterior samples §2.1. The same holds for \bm{\lambda} §2.1.

# Multi-fidelity BNN (MBNN)  [alg. 1]
# Input:  low-fidelity (x_L, u_L); high-fidelity data D; optional PDE terms
# Output: posterior samples of theta; predictive mean and std at any x

def train_mbnn(x_L, u_L, x_H, D, num_samples):
    # 1. Fit the low-fidelity DNN by MAP with an L2 penalty  [§2.1] [eq. 1]
    w_L = map_fit(x_L, u_L, alpha)                      # alpha: [§3]
    # 2. Predict the low-fidelity field at the high-fidelity inputs  [§2.1]
    u_L_at_H = dnn_L(x_H, w_L)
    # 3. Fit the prior scale sigma by maximising the ELBO with mean-field VI
    #    [§2.2] [eq. 10] [eq. 11]
    sigma = fit_vi_prior(x_H, u_L_at_H, D)
    # 4. Draw posterior samples of theta with HMC, started from the VI prior
    #    [§2.1] [§3]
    theta_samples = hmc(log_posterior(theta, D, sigma), burn_in=10000)
    # 5. Predictive mean and std from the posterior samples  [§2.1]
    mean, std = moments(bnn(x, u_L_at_H, theta_samples))
    return mean, std

Active learning. The authors add high-fidelity measurements one at a time, each at the location where the posterior variance is largest, following the strategy the paper takes from Raissi et al. (2017) §3.3. Iteration stops when the predicted variance drops below 0.05^2 §3.3. The candidate set and the exact variance measure are not stated §3.3.

Key equations. Measurement model for function data §2.1 eq. 2. Posterior up to a constant §2.1 eq. 8. Mean-field posterior approximation §2.2 eq. 10. Evidence lower bound as used §2.2 eq. 11.

Hyperparameters

Global settings, from §3 unless noted. Case-specific architectures and prior scales are in the second table.

Per-case settings. Layer sizes are the number of hidden layers × neurons.

Results

Author-reported. Setup: synthetic functions and PDEs with simulated noise, one SST day from satellite and in-situ data. Baselines: the same BNN trained on high-fidelity data only (single-fidelity), and, for SST only, a multi-fidelity Gaussian process §3.1.3. No dataset or model size is reported beyond the architectures above.

Coverage, from tab. 1. PICP is the fraction of exact values within ±2 predictive standard deviations; the ideal is 1 §3.1.2 eq. 18.

Inverse PDE, estimated k (exact value 1 per the table caption), mean ± std.

SST, predicted temperature at two held-out stations, mean ± std tab. 2:

The paper says the MF-BNN and MF-GP are comparable, and that MF-BNN is more accurate than SF-BNN only at x_A §3.1.3. At x_B, SF-BNN is closer to the measured value than MF-BNN tab. 2.

Qualitative results without tabulated numbers:

• 1D function: predicted means agree with the exact curve and errors are mostly within two standard deviations §3.1.1 fig. 2. Single-fidelity training is much worse fig. 2c.
• Noisier low-fidelity data (ε_L ~ N(0, 0.3²)): more low-fidelity points improve the low-fidelity fit and the high-fidelity fit §3.1.1 fig. 3.
• Biased low-fidelity data (u_L and u_H differ by a shift): means agree and errors are mostly within two standard deviations §3.1.1 fig. 4.
• 2D inverse PDE: errors are about one order of magnitude larger with single-fidelity data §3.2.2 fig. 9.
• Active learning: the high-fidelity profile improves after adding observations, for both function approximation and the 1D inverse PDE §3.3 fig. 10 fig. 11. The error E falls with each iteration; the values are only shown in fig. 11b, not in the text.

Paper vs code

No official code.

Searched: the LaTeX source has no repository, project-page or code link (it contains only LaTeX-template and bibliography URLs). GitHub repository search returned nothing under the authors' organisation for "multifidelity" or "multi-fidelity", and none for "MF-BNN" from the authors. The only multi-fidelity BNN repository found, CKunRen/MFBNN-IL-Reproducibility, is not from the authors and has no license, so it is not used.

Pitfalls

• Prior width formula is ambiguous. The text gives \sigma_{w,l} = \sigma/\sqrt{N_l} §2.2, with N_l not defined for the input layer or the output. Choose the fan-in of layer l and check the implementation against your own initialisation; the paper does not say which it used.
• The prior scale is learned, not fixed. σ is found by maximising the ELBO together with the VI parameters §2.2 eq. 11. The published σ values (table above) come from that fit for each case. They are not transferable constants; a different architecture, noise level or number of points gives a different σ.
• ELBO sign. eq. 11 writes the bound as something to maximise, then gives an expression that is to be minimised. Implement the minimisation of \mathbb{E}_Q[\log Q - \log P(\bm\theta) - \log P(\mathcal{D}\mid\bm\theta)].
• VI details missing. Learning rate, number of steps, initialisation of \zeta, and any minibatching are not stated §2.2. The summary says VI scales to big data with minibatches §4, but the experiments do not state any minibatching §3.
• HMC settings are underspecified. "Leapfrog step L = 50\,\delta_t" §3 can mean 50 leapfrog steps of size 0.1, or a trajectory length of 5. The step-size adaptation rule, the mass matrix, the target acceptance rate, thinning and whether the 1,000 samples follow burn-in are not stated. If L counts steps, each HMC proposal needs 50 gradient evaluations of the BNN and of the PDE residual, which is the main cost the abstract does not show.
• Stochastic HMC is not used. The conclusion says stochastic HMC could handle large data §4; all experiments use full-batch HMC §3.
• Algorithm text typo. alg. 1 says "mean-fidelity VI"; the body means mean-field VI §2.2.
• Low-fidelity function changes between sections. §3.1.1 uses u_L = \sin(8x) §3.1.1; §3.2.1 uses u_L = \sin(8\pi x) and says it is "the same as in Sec. 1dfunc" §3.2.1. Use the one stated in the section where you implement.
• Test noise and PICP. It is not stated whether the predictive standard deviation includes the known measurement noise \sigma_u, \sigma_f, which matters for PICP §3.1.2 eq. 18. The PICP definition has a typo: c_i and C_i are used for the same indicator §3.1.2.
• Unknown k is not fixed in the 2D text. The 2D inverse section does not state the true reaction rate in the body; the table caption gives k = 1 tab. 4.
• The k prior is a standard normal chosen by hand, not learned §3.2.1. Results depend on it, and the paper does not test other priors.
• The 1D inverse PDE equation is written as \frac{1}{192\pi^2}u_{xx} - \frac{k}{24\pi}uu_x = f §3.2.1 eq. 19. Check the constants against your derivation; the paper's derivation is not shown.
• Active-learning stop rule is vague. "Predicted variance smaller than 0.05^2" does not say which variance, which point, or over which set §3.3. The candidate set for new points is not stated.
• Single-fidelity baselines use the same BNN, which the authors say may be suboptimal for that baseline §3.1.1. Compare with that in mind.
• Biased and 0.3-noise cases lack complete settings. The biased case gives no network sizes §3.1.1; the 0.3-noise study gives the architecture only by reference to the previous case §3.1.1.
• Typos in the figures and text: "fideilty" (abstract); "obtaoned" §1; "\mathcal(0, 0.05²)" fig. 9 caption.
• Scope. Everything is one or two dimensions except the 4D function, and SST uses one day with 12 training stations. Results do not show behaviour on larger or noisier datasets.

Lineage

• Builds on: the composite NN for multi-fidelity data, Meng and Karniadakis (Journal of Computational Physics, 2020; cited as meng2020composite) §1 §2.1. It uses the same MAP-trained low-fidelity network; the Bayesian high-fidelity part is new here §2.1.
• Builds on: B-PINNs, Yang, Meng and Karniadakis, arXiv:2003.06097 §2.1 §3. The HMC and VI prior settings follow it; the multi-fidelity input structure is new here.
• Compared against: multi-fidelity GP regression §1 §3.1.3; single-fidelity BNN trained on high-fidelity data only §3.1; the composite NN, which has no uncertainty §1.
• Superseded by: not known from this wiki.