---
kind: concept
slug: bayesian-neural-network
title: Bayesian neural network
summary: >-
  A neural network whose weights are random variables, so each prediction is an average over plausible weight settings and comes with a spread. Exact inference is intractable, so practical methods approximate it: variational inference, Hamiltonian Monte Carlo, MC dropout or the Gaussian-process limit. They trade accuracy, memory and compute. One survey card covers this slug.
tags: [bayesian-learning, uncertainty]
papers: ["2006.12024"]
related: [variational-inference, monte-carlo-dropout, uncertainty-estimation]
status: draft
updated_at: 2026-10-19
---

## Overview

A Bayesian neural network puts a prior on the weights, combines it with the
likelihood of the data to get a posterior, and predicts by averaging over that
posterior. The mean, variance or interval of a prediction is an expectation
over the posterior [arxiv:2006.12024 §2.2]. This is the shared invariant of all
variants below: a range of weight settings is kept, not one best setting.

The exact posterior needs a marginal likelihood that is intractable for
non-linear networks, so every practical variant approximates it [arxiv:2006.12024
§2.2]. The variants differ in which approximation they use and therefore in
cost, memory and accuracy.

## Variants

### Mean-field variational inference with reparameterisation (Bayes by Backprop)

Choose a tractable family for the weights and minimise the KL divergence to the
posterior, which is the same as maximising the evidence lower bound. The
standard choice is a fully factorised Gaussian with σ = softplus(ρ). Each
weight is sampled as μ + σ ⊙ ε, so gradients flow through μ and ρ [arxiv:2006.12024
§2.4, alg. 1]. The method scales to deep networks because it is trained with
stochastic gradients [arxiv:2006.12024 §2.4]. The paper gives no optimiser,
learning rate, epoch count or batch size for its experiments [arxiv:2006.12024
§3, §3.1]. Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

### Local reparameterisation

Instead of sampling every weight, sample the pre-activation. For a factorised
Gaussian the pre-activation is Gaussian with a mean and a variance that come
directly from the layer's inputs and the variational parameters [arxiv:2006.12024
§2.4]. The paper credits this to Kingma et al. and does not compare its variance
with the plain sampler. Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

### Full-covariance Gaussian for one hidden layer

A closed-form variational treatment exists for a network with one sigmoid hidden
layer. Its covariance is a diagonal part plus a low-rank part, which the paper
says scales linearly with the number of hidden units [arxiv:2006.12024 §2.3.1].
The paper frames this as an early method and notes that the mean-field approach
is limited to one hidden layer in its closed form [arxiv:2006.12024 §2.3, §2.3.1].
Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

### Monte Carlo dropout

Dropout is read as variational inference. Keeping dropout active at test time
and averaging N stochastic forward passes gives the predictive mean and second
moment [arxiv:2006.12024 §2.4, §3]. The KL term is approximated, and its
variances and the keep probability are not stated [arxiv:2006.12024 §2.4]. Unlike
Bayes by Backprop, it adds no parameters and has the same inference time as a
vanilla network [arxiv:2006.12024 §3]. Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

### Hamiltonian Monte Carlo (HMC)

A gradient-based Markov chain Monte Carlo sampler. It adds a momentum variable,
integrates with leapfrog steps (step size ε, L steps) and applies a Metropolis
accept or reject step. The potential energy is the negative log of prior times
likelihood [arxiv:2006.12024 §2.3.2]. The paper gives ε and L only as symbols.
Stochastic-gradient MCMC uses minibatches, and the paper warns that naive
subsampling biases the samples away from the posterior [arxiv:2006.12024 §2.6].
Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

### Sequential approximate inference (Probabilistic Backpropagation)

Updates the posterior one data point at a time, using assumed-density filtering
and moment matching [arxiv:2006.12024 §2.4]. The paper calls the online cost
prohibitive for large data sets [arxiv:2006.12024 §2.4]. Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

### Gaussian-process limit

A single hidden layer with Gaussian weight priors gives a Gaussian-process prior
over the outputs as the number of hidden units grows [arxiv:2006.12024 §2.5,
fig. 6]. This is a reference point rather than a training method. The paper's
toy regression uses a GP with a Matérn 5/2 kernel as its baseline, with kernel
values not stated [arxiv:2006.12024 §3, fig. 7]. Card: [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024).

## When to use which

The source gives no benchmark across methods, so these rows follow from the
constraints each variant's cost implies, plus the paper's own toy results.

| Situation | Use | Why |
|---|---|---|
| Deep network, large data, gradient training already in place | Mean-field VI (Bayes by Backprop) or Monte Carlo dropout | Both are trained with stochastic gradients [arxiv:2006.12024 §2.4]. Bayes by Backprop doubles the parameter count, while MC dropout adds none [arxiv:2006.12024 §3] |
| Tight memory or latency budget at inference | Monte Carlo dropout | No extra parameters and the same per-pass cost as a vanilla network. Each prediction still needs N passes, and N is not stated [arxiv:2006.12024 §3] |
| Need the sampled posterior to be accurate, and compute is no constraint | HMC | Gradient-based sampling avoids the random walk of simple samplers [arxiv:2006.12024 §2.3.2]. Stochastic variants need care with subsampling [arxiv:2006.12024 §2.6] |
| Small data, regression, and a GP-like reference is wanted | Gaussian-process limit | A wide single hidden layer with Gaussian weights gives a GP prior [arxiv:2006.12024 §2.5]. The paper uses a GP as its baseline [arxiv:2006.12024 §3] |
| Data arrives one point at a time and the data set is small | Probabilistic Backpropagation | Sequential updates; the paper calls it prohibitive for large data sets [arxiv:2006.12024 §2.4] |
| Uncertainty must be trusted on inputs far from the training data | Check before choosing | In one toy regression the paper reports that Bayes by Backprop under-estimates variance outside the data range, and MC dropout gives variance that is too large both outside and inside it [arxiv:2006.12024 §3, fig. 7]. The two errors go in opposite directions, so neither is a safe default |

Accuracy cost: in the paper's one MNIST run, a Bayesian LeNet with Bayes by
Backprop on the convolution layers reached 98.99% against 99.92% for a vanilla
LeNet on 10,000 test images [arxiv:2006.12024 §3.1]. The paper reads this as
comparable performance and values the uncertainty output more. It is one run
with no variance reported [arxiv:2006.12024 §3.1].

## Implementation sketch

Every variant above ends in the same prediction step: draw weights from the
approximate posterior, run the network, and average. The sketch below is that
step, for regression with a Gaussian noise term [arxiv:2006.12024 §2.2, §3].
Training differs by variant and is not shown. The paper states no training
settings for its experiments, so the sample count, the prior and the optimiser
are choices the implementer must make.

```pseudocode
# Predictive mean and variance by Monte Carlo over the posterior  [§2.2] [§3]
# Input:  sample_weights(): one draw from the approximate posterior
#         x_new: one test input; noise_var: sigma squared; n: number of draws
# Output: mean (K,); variance (K,)

def bayesian_predict(sample_weights, x_new, noise_var, n):
    # One network output per posterior draw  [§2.2]
    outputs = [network(x_new, sample_weights()) for _ in range(n)]
    # Predictive mean is the average over the draws  [§3]
    mean = sum(outputs) / n
    # Spread across draws, around the mean  [§3]
    spread = sum((o - mean) ** 2 for o in outputs) / n
    # Add the Gaussian noise term from the likelihood  [§2.2] [§3]
    return mean, spread + noise_var
```

Notes on the sketch:
- The variance is the second moment minus the outer product of the mean with
  itself. The paper gives the uncentred second moment, so this step is needed
  [arxiv:2006.12024 §3]. The noise term is for regression only.
- For Bayes by Backprop, the per-step cost averages over several draws and
  divides by the draw count, not by a sum [arxiv:2006.12024 §2.4, alg. 1].
  Using the sum would scale the gradient.
- Choose N and the prior yourself. The paper gives neither [arxiv:2006.12024
  §2.4, §3].

## Contradictions

None known. Only one card covers this slug, and it contains no disagreement
between papers. Within the one survey, the two MC-based and VI-based methods
err in opposite directions on the toy regression, as noted in the table above
[arxiv:2006.12024 §3, fig. 7]. The paper does not claim these results
contradict each other.

## Papers

- [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024): a survey and tutorial covering the prior, posterior and predictive view, the approximation families, and two small experiments. Not a new method.