Bayesian Neural Networks: An Introduction and Survey

In plain words

A normal neural network gives one answer and no sense of how sure it is. A Bayesian neural network treats its weights as uncertain instead: it keeps a range of plausible weight settings rather than one best setting. Its predictions then come with a spread, which matters where a confident wrong answer is costly.

The hard part is that the exact calculation over all those weight settings is out of reach for real networks, so practical methods approximate it. This survey explains the main approaches and why they were developed, from early work in the 1990s to methods that train with gradient descent on large data sets. It also runs two of them on small problems and lists open questions.

TL;DR

A survey and tutorial, not a new method. It puts a prior on the weights, infers the posterior and averages predictions over it §2.2. It walks through early variational inference, Hamiltonian Monte Carlo, the Gaussian-process limit and the gradient-trained variational methods Bayes by Backprop and MC dropout §2.3 to §2.5. Its two experiments are small: a toy regression, and a Bayesian LeNet on MNIST at 98.99% against 99.92% for a vanilla network §3, §3.1. Use it as an orientation and a list of pitfalls, not as evidence for a method.

Problem

Standard networks give point estimates, and they can sound confident on inputs unlike their training data. The paper frames this as a barrier for high-risk uses such as medical diagnosis and driving §1. Exact Bayesian inference needs the marginal likelihood, which is intractable for non-linear networks, so an approximation is always needed §2.2. Early Bayesian work used Laplace approximations, or closed-form variational treatments limited to one hidden layer §2.3, §2.3.1. Modern work relies on stochastic-gradient variational inference, which scales to deep networks §2.4. The paper argues that the usual factorised-Gaussian variational approach underestimates predictive variance §2.6.

Method

This is the paper's own summary, in our notation: ω for weights, D for data, q_θ for the approximate posterior. Equation numbers are not shown; cite by section.

Set-up §2.2

• Prior p(ω). For regression the likelihood is Gaussian around the network output with noise variance σ². Independent data points give a product over the data.
• Posterior p(ω | D) is proportional to prior times likelihood. Every predictive quantity (mean, variance, interval) is an expectation over this posterior §2.2.
• Unknown noise or prior parameters can get their own hyper-prior and be integrated out §2.2.

Variational inference (VI) §2.2, §2.3.1, §2.4

• Choose a family q_θ(ω) and minimise KL(q_θ || p(ω | D)). Equivalently, maximise the evidence lower bound F[q] = E_q[log p(D | ω)] − KL(q || p(ω)). The paper's cost is L = −F §2.2, §2.4.
• Mean-field: a factorised Gaussian over every weight, with an analytic solution for one hidden layer §2.3.1, Hinton and van Camp.
• Full-covariance Gaussian for one hidden layer with a sigmoid, with a low-rank-plus-diagonal covariance C = diag(d²) + Σ_i s_i s_iᵀ. This form scales linearly with the number of hidden units §2.3.1, Barber and Bishop.
• Minibatch objective: the likelihood term is scaled by N/M and summed over a minibatch, plus the KL term §2.4.

Gradients for VI §2.4

• Score-function estimator: unbiased but high-variance. Control variates reduce the variance.
• Reparameterisation trick: write ω = μ + σ ⊙ ε with ε ~ N(0, I). Gradients then flow through μ and σ (pathwise estimator). The paper says this has lower variance than the score-function estimator.
• Local reparameterisation (Kingma et al.): sample the pre-activation instead of the weights. For a factorised Gaussian, the pre-activation is Gaussian with mean γ_j = Σ_i x_i μ_ij and variance δ_j² = Σ_i x_i² σ_ij² §2.4.

Bayes by Backprop §2.4, alg. 1

• Fully factorised Gaussian. The variational parameters are μ and ρ, with σ = softplus(ρ) §2.4.
• One weight sample is ω = μ + softplus(ρ) ⊙ ε. The cost term per sample is log q(ω | θ) − log p(ω) − log p(D | ω), summed over weights where needed §2.4.
• The algorithm averages this over sample draws and takes one gradient step on μ and ρ alg. 1.

# Bayes by Backprop, one gradient step  [alg. 1] [§2.4]
# Input:  mu, rho (variational parameters); data; alpha (learning rate)
# Output: updated mu, rho

def bayes_by_backprop_step(mu, rho, data, alpha, sample_count):
    # Standard deviation from rho; softplus keeps it positive  [§2.4]
    sigma = softplus(rho)                          # (P,)
    # Cost starts at zero, as in the algorithm  [alg. 1]
    cost = 0.0
    for i in range(sample_count):
        # Draw standard-normal noise  [alg. 1] [§2.4]
        epsilon = sample_standard_normal(shape=mu.shape)
        # Reparameterised weight sample  [§2.4]
        weights = mu + sigma * epsilon
        # Log density of the approximate posterior at this sample  [§2.4]
        log_q = sum(log_normal(weights, mu, sigma))
        # Log prior and log likelihood at this sample  [§2.4] [alg. 1]
        log_prior = sum(log_prior_density(weights))
        log_lik = log_likelihood(data, weights)
        # Average over the draws, as the algorithm does  [alg. 1]
        cost += (log_q - log_prior - log_lik) / sample_count
    # Gradient step on both variational parameters  [alg. 1]
    mu = mu - alpha * gradient(cost, mu)
    rho = rho - alpha * gradient(cost, rho)
    return mu, rho

Sequential approximate inference (PBP) §2.4

• Probabilistic Backpropagation (Hernandez et al.) updates the posterior one data point at a time with assumed density filtering and moment matching. The paper's update rules for the mean and variance are given in the text §2.4.
• The paper calls the online cost prohibitive for large data sets §2.4.

MC dropout §2.4, §3

• Dropout is read as variational inference. A Bernoulli variable ρ ~ Bernoulli(p) multiplies the input of each unit, which is equivalent to scaling a row of the weight matrix: W_ρ = diag(ρ) W. One dropout variable is shared across a row §2.4.
• The KL term is approximated by treating the scaled Bernoulli posterior as a mixture of two Gaussians with very small variance §2.4. The values are not stated.
• The paper also describes Gaussian dropout, which replaces the Bernoulli variable with a Gaussian one §2.4, Kingma et al..
• At test time, the mean and second moment are estimated by averaging N stochastic forward passes §3.

# Predictive mean and second moment by Monte Carlo  [§3]
# Input:  weight_samples: N draws from q; x_new: one test input
# Output: mean (K,); second moment (K, K); noise_variance is sigma squared

def predictive_moments(weight_samples, x_new, noise_variance):
    # Network output for each posterior draw  [§3]
    outputs = [network(x_new, w) for w in weight_samples]
    # First moment: average of the draws  [§3]
    mean = sum(outputs) / len(outputs)
    # Second moment: noise term plus averaged outer products  [§3]
    squares = [outer(o, o) for o in outputs]
    second = noise_variance * identity(len(mean)) + sum(squares) / len(squares)
    # Variance is not written in the paper: second minus outer(mean, mean)
    return mean, second

Hamiltonian Monte Carlo (HMC) §2.3.2

• Gradient-based MCMC that avoids the random-walk behaviour of simple samplers. Position is the weight vector ω, and an auxiliary momentum v is added. The potential energy is U(ω) = −log(p(ω) p(D | ω)) and the kinetic energy is K(v) = vᵀ M⁻¹ v §2.3.2.
• Discretised with leapfrog steps (step size ε, L steps), followed by a Metropolis accept or reject step §2.3.2.
• Neal's BNN puts hyper-priors on the prior and likelihood precisions, samples them with Gibbs steps, and samples the weights with HMC §2.3.2.
• Stochastic-gradient MCMC uses minibatches. The paper says naive subsampling biases the samples away from the posterior §2.6.

Gaussian-process link §2.5

• A single hidden layer with Gaussian weight priors gives a GP prior over the outputs as the number of hidden units grows §2.5, Neal. The paper's Figure 6 shows this numerically fig. 6.
• Williams (1997) analyses the covariance for different activations §2.5.
• Deep networks extend this to deep GPs, and convolutional networks to convolutional GPs §2.5.

Convolutional BNN (the paper's own experiment) §3.1

• The convolution is written as u(Xᵀ ∗ W), where ∗ is a "convolution-like" operator that the paper says is not standard convolution §3.1.
• Every convolution weight gets an independent (factorised) Gaussian, and Bayes by Backprop is applied to the whole network §3.1.
• Likelihood: softmax over classes §3.1.
• Architecture: the cited LeNet (layer sizes not given in the paper) §3.1.
• Predictive mean from the moment formula in §3. The 95% credible intervals come from Monte Carlo samples of the class probabilities §3.1, fig. 8.

Hyperparameters

The paper gives no training settings for its two experiments. Every value below is either stated in the text or marked not stated.

Results

Author-reported, from single runs; the paper does not state repeat counts or variance across seeds. The paper runs no benchmark against published baselines.

Derived from the table above: the Bayesian LeNet is about 0.9 points below the vanilla network on accuracy. The paper calls that comparable performance and values the uncertainty output more §3.1, §4.

Paper vs code

No official code. The paper links no repository. Its one code mention is a footnote pointing to a third party's Deep GP tutorial §2.5, which is not the authors' code and is not used here. Searched: the paper text, and the GitHub repository search for "Goan Fookes" and for "bayesian neural networks introduction survey goan". Neither gave results.

Pitfalls

• No experiment details. The toy regression and the BCNN have no learning rate, epochs, batch size, optimiser, sample count, width or prior values, and no seeds. Reproducing either figure means choosing these yourself.
• One accuracy number. The MNIST gap comes from one reported run. No variance, and no comparison of calibration beyond the qualitative figures §3.1.
• Activation placement in the MLP equations. The per-unit equations place the activation inside the sum over inputs. The text and the matrix form Φ = a(XᵀW¹) describe an affine sum followed by the activation. Implement the matrix form §2.1.
• Predictive variance. The paper gives the first moment and an uncentred second moment that includes σ²I. The variance is not written. Compute it as the second moment minus the outer product of the mean with itself §3.
• Monte Carlo cost averaging. The approximation of the cost is written as a sum over N samples §2.4, while the algorithm divides the sum by N. Use the mean. Using the sum scales the gradient by N.
• Minibatch scaling. The minibatch objective has a −N/M prefactor but sums over indices 1 to N, not over the M points in the batch. The standard estimator sums over the batch §2.4. Algorithm 1 does not show minibatch scaling, and the text does not say whether the BCNN used full batches.
• Hidden layers vs hidden units. The GP argument says the output becomes Gaussian as "the number of hidden layers" grows. The derivation and the Figure 6 caption are about the number of hidden units in one layer [§2.5, fig. 6]. Read the wider-layer version as the correct claim.
• Dropout description. The text says dropout multiplies each weight element-wise, but the equation multiplies the inputs. The text later says the noise is in input space, which matches the equation §2.4.
• Overloaded symbols. N is used for data size, for the Monte Carlo sample count and for hidden units in different places. θ is used for both the variational parameters and the activation in the dropout equation §2.4.
• Hellinger constant. The paper says the α-divergence at α = 1/2 gives the squared Hellinger distance. With the formula as written, α = 1/2 gives 4 times the squared Hellinger distance it defines. This is our calculation and the paper does not note the factor §2.6.
• HMC kinetic energy. K(v) = vᵀM⁻¹v has no ½ factor, which is the common convention. Its derivatives carry a factor of 2 that the text does not show. Keep K and its derivatives consistent in any implementation §2.3.2.
• PBP variance update. The σ update does not say whether σ is a standard deviation or a variance. Check the source before implementing §2.4.
• Convolution-like operator. The operator is not standard convolution, and the paper gives no kernel size, stride or padding. Do not map it to a library convolution without checking §3.1.
• Factorised posteriors. The paper's claims that factorised VI underestimates variance and that the reverse KL overestimates it are cited to the textbook and not derived here §2.6.
• Claims about MC dropout. That it is less sensitive to the prior and needs fewer samples come from one toy regression and are not tested more widely §3.
• Credible intervals on classes. The interval method for the BCNN, including how many samples are drawn, is not stated §3.1, fig. 8.
• Scope. This is a survey. It does not establish which method is best for any given task. The conclusion calls for more research on accurate inference in large networks §4.

Lineage

• Builds on (cited in the paper, not compiled as cards here): Neal (HMC for BNNs and the GP limit), Hinton and van Camp (factorised VI), Barber and Bishop (structured covariance), Graves (gradient-based factorised VI), Hernandez et al. (PBP), Blundell et al. (Bayes by Backprop), Gal and Ghahramani (MC dropout as VI), Kingma et al. (local reparameterisation and Gaussian dropout) §2.3 to §2.5.
• Compared against: Bayes by Backprop vs MC dropout vs GP on toy regression §3.
• Superseded by: not stated.