Bayesian neural network

Overview

A Bayesian neural network puts a prior on the weights, combines it with the likelihood of the data to get a posterior, and predicts by averaging over that posterior. The mean, variance or interval of a prediction is an expectation over the posterior 2006.12024 §2.2. This is the shared invariant of all variants below: a range of weight settings is kept, not one best setting.

The exact posterior needs a marginal likelihood that is intractable for non-linear networks, so every practical variant approximates it [arxiv:2006.12024 §2.2]. The variants differ in which approximation they use and therefore in cost, memory and accuracy.

Variants

Mean-field variational inference with reparameterisation (Bayes by Backprop)

Choose a tractable family for the weights and minimise the KL divergence to the posterior, which is the same as maximising the evidence lower bound. The standard choice is a fully factorised Gaussian with σ = softplus(ρ). Each weight is sampled as μ + σ ⊙ ε, so gradients flow through μ and ρ [arxiv:2006.12024 §2.4, alg. 1]. The method scales to deep networks because it is trained with stochastic gradients 2006.12024 §2.4. The paper gives no optimiser, learning rate, epoch count or batch size for its experiments [arxiv:2006.12024 §3, §3.1]. Card: Bayesian Neural Networks: An Introduction and Survey.

Local reparameterisation

Instead of sampling every weight, sample the pre-activation. For a factorised Gaussian the pre-activation is Gaussian with a mean and a variance that come directly from the layer's inputs and the variational parameters [arxiv:2006.12024 §2.4]. The paper credits this to Kingma et al. and does not compare its variance with the plain sampler. Card: Bayesian Neural Networks: An Introduction and Survey.

Full-covariance Gaussian for one hidden layer

A closed-form variational treatment exists for a network with one sigmoid hidden layer. Its covariance is a diagonal part plus a low-rank part, which the paper says scales linearly with the number of hidden units 2006.12024 §2.3.1. The paper frames this as an early method and notes that the mean-field approach is limited to one hidden layer in its closed form 2006.12024 §2.3, §2.3.1. Card: Bayesian Neural Networks: An Introduction and Survey.

Monte Carlo dropout

Dropout is read as variational inference. Keeping dropout active at test time and averaging N stochastic forward passes gives the predictive mean and second moment 2006.12024 §2.4, §3. The KL term is approximated, and its variances and the keep probability are not stated 2006.12024 §2.4. Unlike Bayes by Backprop, it adds no parameters and has the same inference time as a vanilla network 2006.12024 §3. Card: Bayesian Neural Networks: An Introduction and Survey.

Hamiltonian Monte Carlo (HMC)

A gradient-based Markov chain Monte Carlo sampler. It adds a momentum variable, integrates with leapfrog steps (step size ε, L steps) and applies a Metropolis accept or reject step. The potential energy is the negative log of prior times likelihood 2006.12024 §2.3.2. The paper gives ε and L only as symbols. Stochastic-gradient MCMC uses minibatches, and the paper warns that naive subsampling biases the samples away from the posterior 2006.12024 §2.6. Card: Bayesian Neural Networks: An Introduction and Survey.

Sequential approximate inference (Probabilistic Backpropagation)

Updates the posterior one data point at a time, using assumed-density filtering and moment matching 2006.12024 §2.4. The paper calls the online cost prohibitive for large data sets 2006.12024 §2.4. Card: Bayesian Neural Networks: An Introduction and Survey.

Gaussian-process limit

A single hidden layer with Gaussian weight priors gives a Gaussian-process prior over the outputs as the number of hidden units grows [arxiv:2006.12024 §2.5, fig. 6]. This is a reference point rather than a training method. The paper's toy regression uses a GP with a Matérn 5/2 kernel as its baseline, with kernel values not stated 2006.12024 §3, fig. 7. Card: Bayesian Neural Networks: An Introduction and Survey.

When to use which

The source gives no benchmark across methods, so these rows follow from the constraints each variant's cost implies, plus the paper's own toy results.

Accuracy cost: in the paper's one MNIST run, a Bayesian LeNet with Bayes by Backprop on the convolution layers reached 98.99% against 99.92% for a vanilla LeNet on 10,000 test images 2006.12024 §3.1. The paper reads this as comparable performance and values the uncertainty output more. It is one run with no variance reported 2006.12024 §3.1.

Implementation sketch

Every variant above ends in the same prediction step: draw weights from the approximate posterior, run the network, and average. The sketch below is that step, for regression with a Gaussian noise term 2006.12024 §2.2, §3. Training differs by variant and is not shown. The paper states no training settings for its experiments, so the sample count, the prior and the optimiser are choices the implementer must make.

# Predictive mean and variance by Monte Carlo over the posterior  [§2.2] [§3]
# Input:  sample_weights(): one draw from the approximate posterior
#         x_new: one test input; noise_var: sigma squared; n: number of draws
# Output: mean (K,); variance (K,)

def bayesian_predict(sample_weights, x_new, noise_var, n):
    # One network output per posterior draw  [§2.2]
    outputs = [network(x_new, sample_weights()) for _ in range(n)]
    # Predictive mean is the average over the draws  [§3]
    mean = sum(outputs) / n
    # Spread across draws, around the mean  [§3]
    spread = sum((o - mean) ** 2 for o in outputs) / n
    # Add the Gaussian noise term from the likelihood  [§2.2] [§3]
    return mean, spread + noise_var

Notes on the sketch:

• The variance is the second moment minus the outer product of the mean with itself. The paper gives the uncentred second moment, so this step is needed 2006.12024 §3. The noise term is for regression only.
• For Bayes by Backprop, the per-step cost averages over several draws and divides by the draw count, not by a sum 2006.12024 §2.4, alg. 1. Using the sum would scale the gradient.
• Choose N and the prior yourself. The paper gives neither [arxiv:2006.12024 §2.4, §3].

Contradictions

None known. Only one card covers this slug, and it contains no disagreement between papers. Within the one survey, the two MC-based and VI-based methods err in opposite directions on the toy regression, as noted in the table above 2006.12024 §3, fig. 7. The paper does not claim these results contradict each other.

Papers

• Bayesian Neural Networks: An Introduction and Survey: a survey and tutorial covering the prior, posterior and predictive view, the approximation families, and two small experiments. Not a new method.