Monte Carlo dropout

Overview

Monte Carlo (MC) dropout turns a network trained with dropout into a rough Bayesian model at prediction time. Dropout stays on, each forward pass drops a different random set of units, and the collection of outputs gives a spread that is read as uncertainty.

The reading behind it is that dropout training is a form of variational inference, so the passes can be treated as samples from an approximate posterior over the weights. The active-learning survey states this as an equivalence 2009.00236 §3.1.3. The uncertainty survey calls MC dropout a way of treating dropout layers as variational inference [arxiv:2107.03342 §3.2]. The Bayesian neural network survey gives the mechanics: a Bernoulli variable multiplies the input of each unit, one variable is shared across a row of the weight matrix, and the KL term is approximated by treating the scaled Bernoulli posterior as a mixture of two Gaussians with very small variance. The values of those variances are not stated [arxiv:2006.12024 §2.4]. A Gaussian variant of dropout replaces the Bernoulli variable with a Gaussian one 2006.12024 §2.4.

The uncertainty-quantification tutorial treats MC dropout as the scalable variant of Bayesian neural networks. It keeps dropout active, runs several passes, and reports the mean and standard deviation of the outputs 2305.04933 §3.2. It reads the dropout training loss as a single-sample approximation of a variational objective whose approximating family is a mixture of two Gaussians 2305.04933 §3.2.

Five cards discuss MC dropout. Two surveys place it among other methods: the survey of uncertainty in deep neural networks and the Bayesian neural network survey, which also runs a small toy regression. The survey of deep active learning has no experiments and mentions MC dropout mainly in its active-learning discussion. The tutorial by Nemani et al. compares MC dropout with other methods on two regression case studies. The paper by Davani et al. uses MC dropout as a baseline for annotator-disagreement uncertainty on text classification. The primary papers the surveys cite (for example the Bayesian CNN for active learning and BatchBALD) have no cards yet.

Caveats that apply to every variant:

• The result depends strongly on the dropout rate, the number of dropout layers and the training length 2305.04933 §3.2, §6.1.
• The active-learning survey does not state the number of passes or the dropout rate 2009.00236 §3.1.3. The tutorial uses 15 passes and a dropout rate of 0.10 in its battery case study 2305.04933 §6.1.
• The Bayesian neural network survey's toy regression is its only test of MC dropout. Its claims that MC dropout is less sensitive to the prior and needs fewer samples rest on that one experiment 2006.12024 §3. It does not state the number of samples it used 2006.12024 §2.4.
• Its statement that MC dropout adds no parameters and runs at the same inference time as a vanilla network is relative to Bayes by Backprop 2006.12024 §3. It does not change the fact that each prediction needs several passes, and the survey does not state how many.
• MC dropout is not automatically better than softmax confidence 2009.00236 §3.1.3.
• The number of passes is not stated in the Davani et al. paper, which uses MC dropout only as a comparison baseline 2110.05719 §4.3.2.
• Ensembles are reported to give better-calibrated uncertainty 2009.00236 §3.1.4; arxiv:2107.03342 §3.3, and in the tutorial they had lower NLL than MC dropout in every reported row 2305.04933 §6.1, §6.2.

Variants

BALD over dropout samples

Each unlabeled sample is scored by the mutual information between its label and the weights, the BALD score. The expectation over the posterior is replaced by dropout passes 2009.00236 §3.1.3, eq. 3. The survey describes this as the DBAL approach of Gal et al., a Bayesian CNN with a Gaussian prior on the weights and BALD as the acquisition function 2009.00236 §3.1.3. Card: A Survey of Deep Active Learning.

Because each sample is scored on its own, the top-K samples can be near duplicates of one another. The survey's remedy is a joint batch score (BatchBALD, eq. 4) rather than an extra diversity term 2009.00236 §3.1.1.

Mean and spread for regression

The tutorial applies MC dropout to regression of remaining life. The prediction is the mean over the passes and the uncertainty is their standard deviation 2305.04933 §3.2. Its case studies use a ResNet-style backbone with dropout after each residual block, at rate 0.10 2305.04933 §6.1. Card: Nemani et al., UQ for engineering and prognostics.

The tutorial reports that MC dropout is overconfident and that its run-to-run variation is the largest of the methods it compares [arxiv:2305.04933 §6.1, §6.2]. Its RMSE is also higher than the ensemble's in every turbofan row 2305.04933 §6.2, results table.

The Bayesian neural network survey takes the same mean-and-moments route with a noise term. Its predictive mean is the average over N passes, and its second moment adds the noise variance to the averaged outer products of the outputs. The variance is the second moment minus the outer product of the mean 2006.12024 §3. In its toy 1-D regression on three input sets, MC dropout gave a variance that was larger outside the training range and too large inside the data, compared with a Gaussian process 2006.12024 §3, fig. 7. The card does not give the sample count or the training settings for that run. Card: Goan and Fookes, Bayesian Neural Networks.

Dropout spread as a baseline for annotator disagreement

Davani et al. score each instance by the variance of its labels, which measures annotator disagreement 2110.05719 eq. 1. For a model without annotator heads, they take the same kind of score from the spread of MC dropout passes (or from softmax probability) and correlate it with the disagreement 2110.05719 §4.3.2. This is a comparison, not a recommended method, and the paper does not give the pass count 2110.05719 §4.3.2.

On the hate-speech data (GHC), the authors report that their three multi-annotator methods correlate better with disagreement than softmax and MC dropout, and that the ensemble is slightly highest 2110.05719 §4.3.2. On GoEmotions, the multi-task correlation is higher than both baselines for all six emotions in the subset data, and in the full data for all but joy 2110.05719 §4.4. These are author-reported results on one comparison and on subjective labels. Card: Davani et al., Dealing with Disagreements.

Dropout samples combined with an ensemble

DEBAL combines ensembles with MC dropout. The survey presents it as a response to overconfidence that it attributes to mode collapse in variational inference 2009.00236 §3.1.3. The card gives no further detail on how the two are combined, so no procedure is described here.

When to use which

The surveys give no benchmark that ranks these options. The tutorial compares them on two regression case studies, with mean and standard deviation over 10 runs 2305.04933 §6.1, §6.2. The Bayesian neural network survey compares MC dropout only with Bayes by Backprop and a Gaussian process, on one toy regression 2006.12024 §3. The table follows the stated conditions and the reported comparisons, not measured results on your data.

Implementation sketch

The score below is the BALD estimate from dropout passes. The querying loop around it is in active-learning. The survey does not define an interface, so the names are illustrative. The regression variant uses the same passes and takes the mean and standard deviation of the outputs instead 2305.04933 §3.2.

# Run T passes with dropout on at prediction time [arxiv:2009.00236 §3.1.3]
def mc_dropout_probs(model, x, T):
    # Each pass draws a new dropout mask; T is not stated in the survey
    passes = []
    for t in range(T):
        # Softmax per pass, so each entry is a class distribution
        passes.append(softmax(model(x, dropout=True)))
    return passes  # T x k

# BALD score for one sample from its T class distributions [arxiv:2009.00236 eq. 3, eq. 9]
def bald_score(probs):
    # Average the softmax outputs over passes, not the logits [arxiv:2009.00236 §3.1.3]
    mean_p = mean(probs, axis=0)
    # Entropy of the averaged prediction: H(y | x, D_train) [arxiv:2009.00236 eq. 3]
    total = entropy(mean_p)
    # Average per-pass entropy: E over the posterior of H(y | x, omega) [arxiv:2009.00236 eq. 3]
    expected = mean([entropy(p) for p in probs])
    # Mutual information: total minus expected [arxiv:2009.00236 eq. 3]
    return total - expected

Selecting the batch is top_k(U, bald_score, k), which is the independent scoring the survey warns about 2009.00236 §3.1.1. The survey writes the posterior predictive as an expectation of the network output 2009.00236 eq. 9 while describing the likelihood as a softmax. Averaging softmax outputs follows the softmax likelihood and is an interpretation of the notation, not a step the survey spells out.

The Bayesian neural network survey writes the dropout mask on the inputs of each unit. Its text says the mask multiplies the weights element-wise. Its equation and its later remarks put the noise in input space [arxiv:2006.12024 §2.4]. Implement the input-side mask, and check the framework's dropout placement against it.

Contradictions

• MC dropout against softmax response. One analysis the survey cites finds MC-dropout uncertainty about as good as softmax response 2009.00236 §3.1.3. The survey does not report a result for this comparison of its own.
• Direction of the error in MC dropout. The tutorial reports MC dropout as overconfident in its battery and turbofan case studies [arxiv:2305.04933 §6.1, §6.2]. The Bayesian neural network survey's toy regression reports MC-dropout variance as too large inside the training data [arxiv:2006.12024 §3, fig. 7]. These are different tasks, metrics and training settings, and neither source resolves the other. Check both in your own setting.
• Calibration of ensembles against MC dropout. The active-learning survey, reporting Beluch et al., says ensembles give better-calibrated uncertainty 2009.00236 §3.1.4. The uncertainty survey reports the same for Beluch et al. and gives further cited results where ensembles were more reliable or more robust than MC dropout 2107.03342 §3.3. The tutorial finds ECE mixed between the two: the ensemble is lower on some battery and turbofan rows, MC dropout on others [arxiv:2305.04933 §6.1, §6.2]. Its lower NLL for the ensemble is consistent across rows.
• Dropout spread against annotator disagreement. Davani et al. report that MC dropout correlates less with annotator disagreement than their multi-annotator methods do 2110.05719 §4.3.2, §4.4. The tutorial measures calibration and NLL instead, and finds MC dropout overconfident 2305.04933 §6.1. The targets differ, so these results do not settle each other.
• Meaning of the dropout objective. The active-learning survey states that dropout training is equivalent to variational inference [arxiv:2009.00236 §3.1.3]. The uncertainty survey calls it a treatment of dropout layers as variational inference 2107.03342 §3.2. The Bayesian neural network survey gives an approximation of the KL term with a mixture of two narrow Gaussians, with the variances not stated 2006.12024 §2.4. The tutorial reads the loss as a single-sample approximation of a variational objective with a mixture-of-two-Gaussians family 2305.04933 §3.2. The sources differ on whether the link is an equivalence or an approximation, and none of them proves it.
• Suitability for large datasets. The core-set paper calls DBAL unsuitable for large datasets because of batch sampling 2009.00236 §3.1.3.
• Reported numbers across papers. Random-sampling baselines and some method results differ between papers at the same setting 2009.00236 §5. Compare dropout-based and other scores under one training recipe.
• Survey summaries of ensemble results. The uncertainty survey runs no experiments of its own. Its ensemble-versus-MC-dropout claims are summaries of cited work 2107.03342 §3.3. Check the original papers before relying on them.

Papers

• A Survey of Deep Active Learning: taxonomy of deep active learning with a section on Bayesian acquisition, where MC dropout is discussed; no experiments of its own.
• A Survey of Uncertainty in Deep Neural Networks: survey of uncertainty estimation that covers MC dropout as one Bayesian-style method and summarises ensemble comparisons; no experiments of its own.
• Nemani et al., UQ for engineering and prognostics: tutorial comparing MC dropout with Gaussian process regression, a Gaussian-head ensemble and SNGP on battery and turbofan regression tasks.
• Goan and Fookes, Bayesian Neural Networks: An Introduction and Survey: survey of Bayesian neural networks with the MC dropout mechanics and one toy regression comparing MC dropout with Bayes by Backprop and a Gaussian process.
• Davani et al., Dealing with Disagreements: multi-head BERT for subjective annotations, which uses MC dropout as a baseline for disagreement-based uncertainty.