---
kind: concept
slug: monte-carlo-dropout
title: Monte Carlo dropout
summary: >-
  Keep dropout on at prediction time and average several stochastic forward passes; their spread stands in for model uncertainty without retraining. Active learning uses it for BALD-style scores. Trade-offs: several passes per sample, overconfidence in a regression tutorial, too-large variance inside the data in a toy regression, and no guaranteed win over softmax confidence.
tags: [uncertainty, bayesian-deep-learning, dropout, active-learning, calibration]
papers: ["2009.00236", "2107.03342", "2305.04933", "2006.12024", "2110.05719"]
related: [active-learning, uncertainty-estimation, calibration, ensembles, bayesian-neural-network, variational-inference]
status: draft
updated_at: 2026-10-19
---

## Overview

Monte Carlo (MC) dropout turns a network trained with dropout into a rough
Bayesian model at prediction time. Dropout stays on, each forward pass drops a
different random set of units, and the collection of outputs gives a spread
that is read as uncertainty.

The reading behind it is that dropout training is a form of variational
inference, so the passes can be treated as samples from an approximate
posterior over the weights. The active-learning survey states this as an
equivalence [arxiv:2009.00236 §3.1.3]. The uncertainty survey calls MC dropout
a way of treating dropout layers as variational inference [arxiv:2107.03342
§3.2]. The Bayesian neural network survey gives the mechanics: a Bernoulli
variable multiplies the input of each unit, one variable is shared across a
row of the weight matrix, and the KL term is approximated by treating the
scaled Bernoulli posterior as a mixture of two Gaussians with very small
variance. The values of those variances are not stated [arxiv:2006.12024
§2.4]. A Gaussian variant of dropout replaces the Bernoulli variable with a
Gaussian one [arxiv:2006.12024 §2.4].

The uncertainty-quantification tutorial treats MC dropout as the scalable
variant of Bayesian neural networks. It keeps dropout active, runs several
passes, and reports the mean and standard deviation of the outputs
[arxiv:2305.04933 §3.2]. It reads the dropout training loss as a single-sample
approximation of a variational objective whose approximating family is a
mixture of two Gaussians [arxiv:2305.04933 §3.2].

Five cards discuss MC dropout. Two surveys place it among other methods: the
[survey of uncertainty in deep neural networks](/papers/2107.03342) and the
[Bayesian neural network survey](/papers/2006.12024), which also runs a small
toy regression. The [survey of deep active learning](/papers/2009.00236) has
no experiments and mentions MC dropout mainly in its active-learning
discussion. The [tutorial by Nemani et al.](/papers/2305.04933) compares MC
dropout with other methods on two regression case studies. The
[paper by Davani et al.](/papers/2110.05719) uses MC dropout as a baseline
for annotator-disagreement uncertainty on text classification. The primary papers
the surveys cite (for example the Bayesian CNN for active learning and
BatchBALD) have no cards yet.

Caveats that apply to every variant:

- The result depends strongly on the dropout rate, the number of dropout layers
  and the training length [arxiv:2305.04933 §3.2, §6.1].
- The active-learning survey does not state the number of passes or the dropout
  rate [arxiv:2009.00236 §3.1.3]. The tutorial uses 15 passes and a dropout rate
  of 0.10 in its battery case study [arxiv:2305.04933 §6.1].
- The Bayesian neural network survey's toy regression is its only test of MC
  dropout. Its claims that MC dropout is less sensitive to the prior and needs
  fewer samples rest on that one experiment [arxiv:2006.12024 §3]. It does not
  state the number of samples it used [arxiv:2006.12024 §2.4].
- Its statement that MC dropout adds no parameters and runs at the same
  inference time as a vanilla network is relative to Bayes by Backprop
  [arxiv:2006.12024 §3]. It does not change the fact that each prediction needs
  several passes, and the survey does not state how many.
- MC dropout is not automatically better than softmax confidence
  [arxiv:2009.00236 §3.1.3].
- The number of passes is not stated in the Davani et al. paper, which uses
  MC dropout only as a comparison baseline [arxiv:2110.05719 §4.3.2].
- Ensembles are reported to give better-calibrated uncertainty
  [arxiv:2009.00236 §3.1.4; arxiv:2107.03342 §3.3], and in the tutorial they had
  lower NLL than MC dropout in every reported row [arxiv:2305.04933 §6.1, §6.2].

## Variants

### BALD over dropout samples

Each unlabeled sample is scored by the mutual information between its label and
the weights, the BALD score. The expectation over the posterior is replaced by
dropout passes [arxiv:2009.00236 §3.1.3, eq. 3]. The survey describes this as
the DBAL approach of Gal et al., a Bayesian CNN with a Gaussian prior on the
weights and BALD as the acquisition function [arxiv:2009.00236 §3.1.3]. Card:
[A Survey of Deep Active Learning](/papers/2009.00236).

Because each sample is scored on its own, the top-K samples can be near
duplicates of one another. The survey's remedy is a joint batch score
(BatchBALD, eq. 4) rather than an extra diversity term [arxiv:2009.00236 §3.1.1].

### Mean and spread for regression

The tutorial applies MC dropout to regression of remaining life. The prediction
is the mean over the passes and the uncertainty is their standard deviation
[arxiv:2305.04933 §3.2]. Its case studies use a ResNet-style backbone with
dropout after each residual block, at rate 0.10 [arxiv:2305.04933 §6.1]. Card:
[Nemani et al., UQ for engineering and prognostics](/papers/2305.04933).

The tutorial reports that MC dropout is overconfident and that its run-to-run
variation is the largest of the methods it compares [arxiv:2305.04933 §6.1,
§6.2]. Its RMSE is also higher than the ensemble's in every turbofan row
[arxiv:2305.04933 §6.2, results table].

The Bayesian neural network survey takes the same mean-and-moments route with a
noise term. Its predictive mean is the average over N passes, and its second
moment adds the noise variance to the averaged outer products of the outputs.
The variance is the second moment minus the outer product of the mean
[arxiv:2006.12024 §3]. In its toy 1-D regression on three input sets, MC dropout
gave a variance that was larger outside the training range and too large inside
the data, compared with a Gaussian process [arxiv:2006.12024 §3, fig. 7]. The
card does not give the sample count or the training settings for that run.
Card: [Goan and Fookes, Bayesian Neural Networks](/papers/2006.12024).

### Dropout spread as a baseline for annotator disagreement

Davani et al. score each instance by the variance of its labels, which measures
annotator disagreement [arxiv:2110.05719 eq. 1]. For a model without annotator
heads, they take the same kind of score from the spread of MC dropout passes
(or from softmax probability) and correlate it with the disagreement
[arxiv:2110.05719 §4.3.2]. This is a comparison, not a recommended method, and
the paper does not give the pass count [arxiv:2110.05719 §4.3.2].

On the hate-speech data (GHC), the authors report that their three
multi-annotator methods correlate better with disagreement than softmax and MC
dropout, and that the ensemble is slightly highest [arxiv:2110.05719 §4.3.2].
On GoEmotions, the multi-task correlation is higher than both baselines for all
six emotions in the subset data, and in the full data for all but joy
[arxiv:2110.05719 §4.4]. These are author-reported results on one comparison
and on subjective labels. Card: [Davani et al., Dealing with
Disagreements](/papers/2110.05719).

### Dropout samples combined with an ensemble

DEBAL combines ensembles with MC dropout. The survey presents it as a response
to overconfidence that it attributes to mode collapse in variational inference
[arxiv:2009.00236 §3.1.3]. The card gives no further detail on how the two are
combined, so no procedure is described here.

## When to use which

The surveys give no benchmark that ranks these options. The tutorial compares
them on two regression case studies, with mean and standard deviation over 10
runs [arxiv:2305.04933 §6.1, §6.2]. The Bayesian neural network survey compares
MC dropout only with Bayes by Backprop and a Gaussian process, on one toy
regression [arxiv:2006.12024 §3]. The table follows the stated conditions and
the reported comparisons, not measured results on your data.

| Situation | Use | Why |
|---|---|---|
| A network is already trained with dropout, classification or active learning, and several forward passes per sample are affordable | BALD over dropout samples | The survey presents dropout as the cheap route to posterior samples [arxiv:2009.00236 §3.1.3]; its pass count is not stated |
| Regression with an uncertainty estimate, and the network already has dropout | MC dropout with checks against a baseline | Cheap to add, but the tutorial found it overconfident with the largest run-to-run variation [arxiv:2305.04933 §6.1, §6.2] |
| A Bayesian-style spread is wanted but a second set of weight parameters is not affordable | MC dropout, compared with Bayes by Backprop on your task | Relative to Bayes by Backprop, the survey reports no extra parameters [arxiv:2006.12024 §3]; the comparison is one toy regression |
| Calibrated uncertainty or NLL matters more than cost, and several models can be trained | Ensemble, or ensemble with dropout (DEBAL) | Ensembles are reported as at least as good as MC dropout on accuracy, calibration and out-of-distribution detection, and as more reliable in one comparison [arxiv:2107.03342 §3.3]; they are reported to give better-calibrated uncertainty than MC dropout in active learning [arxiv:2009.00236 §3.1.4]; the tutorial's ensemble had lower NLL in every reported row [arxiv:2305.04933 §6.1, §6.2]; DEBAL targets overconfidence [arxiv:2009.00236 §3.1.3] |
| Softmax confidence is the baseline already in use | Measure MC dropout against softmax response on your own data before adopting it | An analysis the survey cites finds MC-dropout uncertainty about as good as softmax response [arxiv:2009.00236 §3.1.3] |
| Large batches or near-duplicate candidates | Joint batch scoring (BatchBALD) instead of top-K scores | Independent scores repeat near-duplicates [arxiv:2009.00236 §3.1.1] |
| Several annotators label each instance and their disagreement matters | Per-annotator output heads, scored by the spread of their predictions, with MC dropout as the baseline to beat | The authors report that this spread tracks annotator disagreement better than softmax or MC dropout on their data [arxiv:2110.05719 §4.3.2, §4.4]; the comparison is on subjective labels only |

## Implementation sketch

The score below is the BALD estimate from dropout passes. The querying loop
around it is in [active-learning](/concepts/active-learning). The survey does
not define an interface, so the names are illustrative. The regression variant
uses the same passes and takes the mean and standard deviation of the outputs
instead [arxiv:2305.04933 §3.2].

```pseudocode
# Run T passes with dropout on at prediction time [arxiv:2009.00236 §3.1.3]
def mc_dropout_probs(model, x, T):
    # Each pass draws a new dropout mask; T is not stated in the survey
    passes = []
    for t in range(T):
        # Softmax per pass, so each entry is a class distribution
        passes.append(softmax(model(x, dropout=True)))
    return passes  # T x k
```

```pseudocode
# BALD score for one sample from its T class distributions [arxiv:2009.00236 eq. 3, eq. 9]
def bald_score(probs):
    # Average the softmax outputs over passes, not the logits [arxiv:2009.00236 §3.1.3]
    mean_p = mean(probs, axis=0)
    # Entropy of the averaged prediction: H(y | x, D_train) [arxiv:2009.00236 eq. 3]
    total = entropy(mean_p)
    # Average per-pass entropy: E over the posterior of H(y | x, omega) [arxiv:2009.00236 eq. 3]
    expected = mean([entropy(p) for p in probs])
    # Mutual information: total minus expected [arxiv:2009.00236 eq. 3]
    return total - expected
```

Selecting the batch is `top_k(U, bald_score, k)`, which is the independent
scoring the survey warns about [arxiv:2009.00236 §3.1.1]. The survey writes the
posterior predictive as an expectation of the network output [arxiv:2009.00236 eq. 9]
while describing the likelihood as a softmax. Averaging softmax outputs follows the
softmax likelihood and is an interpretation of the notation, not a step the
survey spells out.

The Bayesian neural network survey writes the dropout mask on the inputs of
each unit. Its text says the mask multiplies the weights element-wise. Its
equation and its later remarks put the noise in input space [arxiv:2006.12024
§2.4]. Implement the input-side mask, and check the framework's dropout
placement against it.

## Contradictions

- **MC dropout against softmax response.** One analysis the survey cites finds
  MC-dropout uncertainty about as good as softmax response [arxiv:2009.00236 §3.1.3].
  The survey does not report a result for this comparison of its own.
- **Direction of the error in MC dropout.** The tutorial reports MC dropout as
  overconfident in its battery and turbofan case studies [arxiv:2305.04933
  §6.1, §6.2]. The Bayesian neural network survey's toy regression reports
  MC-dropout variance as too large inside the training data [arxiv:2006.12024
  §3, fig. 7]. These are different tasks, metrics and training settings, and
  neither source resolves the other. Check both in your own setting.
- **Calibration of ensembles against MC dropout.** The active-learning survey,
  reporting Beluch et al., says ensembles give better-calibrated uncertainty
  [arxiv:2009.00236 §3.1.4]. The uncertainty survey reports the same for
  Beluch et al. and gives further cited results where ensembles were more
  reliable or more robust than MC dropout [arxiv:2107.03342 §3.3]. The
  tutorial finds ECE mixed between the two: the ensemble is lower on some
  battery and turbofan rows, MC dropout on others [arxiv:2305.04933 §6.1,
  §6.2]. Its lower NLL for the ensemble is consistent across rows.
- **Dropout spread against annotator disagreement.** Davani et al. report that
  MC dropout correlates less with annotator disagreement than their
  multi-annotator methods do [arxiv:2110.05719 §4.3.2, §4.4]. The tutorial
  measures calibration and NLL instead, and finds MC dropout overconfident
  [arxiv:2305.04933 §6.1]. The targets differ, so these results do not settle
  each other.
- **Meaning of the dropout objective.** The active-learning survey states that
  dropout training is equivalent to variational inference [arxiv:2009.00236
  §3.1.3]. The uncertainty survey calls it a treatment of dropout layers as
  variational inference [arxiv:2107.03342 §3.2]. The Bayesian neural network
  survey gives an approximation of the KL term with a mixture of two narrow
  Gaussians, with the variances not stated [arxiv:2006.12024 §2.4]. The tutorial
  reads the loss as a single-sample approximation of a variational objective
  with a mixture-of-two-Gaussians family [arxiv:2305.04933 §3.2]. The sources
  differ on whether the link is an equivalence or an approximation, and none of
  them proves it.
- **Suitability for large datasets.** The core-set paper calls DBAL unsuitable
  for large datasets because of batch sampling [arxiv:2009.00236 §3.1.3].
- **Reported numbers across papers.** Random-sampling baselines and some
  method results differ between papers at the same setting [arxiv:2009.00236 §5].
  Compare dropout-based and other scores under one training recipe.
- **Survey summaries of ensemble results.** The uncertainty survey runs no
  experiments of its own. Its ensemble-versus-MC-dropout claims are summaries
  of cited work [arxiv:2107.03342 §3.3]. Check the original papers before
  relying on them.

## Papers

- [A Survey of Deep Active Learning](/papers/2009.00236): taxonomy of deep active learning with a section on Bayesian acquisition, where MC dropout is discussed; no experiments of its own.
- [A Survey of Uncertainty in Deep Neural Networks](/papers/2107.03342): survey of uncertainty estimation that covers MC dropout as one Bayesian-style method and summarises ensemble comparisons; no experiments of its own.
- [Nemani et al., UQ for engineering and prognostics](/papers/2305.04933): tutorial comparing MC dropout with Gaussian process regression, a Gaussian-head ensemble and SNGP on battery and turbofan regression tasks.
- [Goan and Fookes, Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024): survey of Bayesian neural networks with the MC dropout mechanics and one toy regression comparing MC dropout with Bayes by Backprop and a Gaussian process.
- [Davani et al., Dealing with Disagreements](/papers/2110.05719): multi-head BERT for subjective annotations, which uses MC dropout as a baseline for disagreement-based uncertainty.