Dropout

Overview

Dropout is a training-time regulariser. Each forward pass removes a random subset of activations (or, in DropConnect, of weights) so that the network cannot rely on any one of them. Seven cards use it this way, in six settings: as the alternative to an L2 penalty in a sparse, multi-field click-prediction network 1601.02376 §3.3; as a baseline and as a companion to a Lipschitz constraint on image and tabular benchmarks 1804.04368 §5; as DropConnect on a large fully connected layer of a medical-imaging ConvNet 1505.03046 §II-A; inside each per-feature network of an interpretable additive model on tabular data, together with whole-network dropout 2004.13912 app. A.3; as variational dropout on the recurrent units of BiLSTM sequence taggers 1707.09861 §4.2; and as fixed-rate dropout in small fully connected networks on sensor and biomechanics features 2411.04700 §3.2 2412.07823 §II.C.

The 2022 regularisation survey describes dropout and its variants in prose and gives no numbers of its own 2201.03299 §4.1. One card uses dropout for a different purpose: it randomises stand-in networks while poisoned training images are crafted, and the victims are trained without dropout 1905.05897 §3.6 1905.05897 §4. See the variant below.

Keeping dropout active at prediction time, and averaging the passes, is a separate use. Five further cards list this slug for that use 2006.12024 §2.4 2009.00236 §3.1.3 2107.03342 §3.2 2110.05719 §4.3.2 2305.04933 §3.2. They are covered on Monte Carlo dropout.

Variants

Unit dropout

Each unit's output is multiplied by a Bernoulli variable at training time. The click-prediction paper uses it in both its FNN and SNN models [arxiv:1601.02376 §4.6], and reports it as a better regulariser than an L2 penalty in every setting it compares 1601.02376 §3.3 1601.02376 §4.5. Its paper and code give different rates; see the caveats below. Card: Deep Learning over Multi-field Categorical Data.

The Lipschitz-constraint paper uses unit dropout as a baseline in every table and combines it with its own constraint. Its scripts set rates per layer type: VGG on CIFAR-10 uses 0.2 for convolutional and 0.5 for dense layers, and WRN uses 0.2 for convolutional layers cifar10-vgg.sh cifar10-wrn.sh. The paper does not state its rates in the text 1804.04368 §5. Card: Regularisation of Neural Networks by Enforcing Lipschitz Continuity.

Variational dropout for recurrent layers

Naive dropout draws a new mask at every time step of an LSTM. Variational dropout draws one mask per sequence and reuses it at all time steps, and it also applies dropout to the recurrent units 1707.09861 §4.2. The paper cites a prior work for this method and does not restate it. In its sweep over sequence-tagging configurations, variational dropout was the best option in 83.5% of 479 cases and gave the smallest spread. Its median gain over no dropout ranged from 0.31 points on POS to 1.98 on Entities, and over naive dropout from 0.19 to 1.32 1707.09861 §6.6. Dropout on both output and recurrent units was best in 62.6% of cases, with median gains of 0.05 to 0.82 points 1707.09861 §6.6. These results are for five tagging tasks with BiLSTM networks trained from scratch, with dropout rates from {0.05, 0.1, 0.25, 0.5} 1707.09861 §4.2. Card: score-distribution study of LSTM sequence taggers.

DropConnect

DropConnect drops individual weights instead of units. The detection paper names it as a regulariser its architecture builds on 1505.03046 §II-A. Its fully connected layer has 512 units with a DropConnect rate of 0.5, taken from the released configuration file and not stated in the paper layers-fc512-11pct-dc.cfg. The card cites no ablation of DropConnect, so its effect in that pipeline is not known from the card. Card: Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation.

Feature-net dropout

The additive model gives each input feature its own small network. Unit dropout is applied after each hidden layer of every feature network models.py. The paper also drops whole feature networks: during training, a feature's network is switched off for an individual example 2004.13912 app. A.3. Both rates are tuned per dataset by Bayesian optimisation 2004.13912 app. A.5. The unit rate search set is {0, 0.05, …, 0.9} and the feature-net search set is {0, 0.05, 0.1, 0.2} 2004.13912 app. A.6. The tuned unit rates are 0.1, 0.2, 0.8, 0.0 and 0.0 on COMPAS, MIMIC-II, Credit Fraud, FICO and Housing, and the feature-net rates are 0.05 on COMPAS and 0.0 elsewhere [arxiv:2004.13912 app. A.6]. The card cites no ablation of feature-net dropout. Its code applies TensorFlow dropout to the stacked feature outputs, which rescales surviving outputs by 1/(1 − p) during training models.py. The paper does not state that scaling. Card: Neural Additive Models: Interpretable Machine Learning with Neural Nets.

Fixed-rate dropout in small fully connected networks

Two cards use dropout as one fixed-rate ingredient of a small network, with no rate search. Neither card reports an ablation of its dropout.

The planetary-rover classifier places dropout at two points. The paper's text says 20% dropout between hidden layers 2411.04700 §3.2. Its code applies 10% dropout to the inputs and 20% after the last hidden layer, before the output model.py model.py. Card: Field Assessment of Force Torque Sensors for Planetary Rover Navigation.

The hip-moment estimator uses a three-layer network of 50 units. Each hidden layer runs linear, then batch normalization, then ReLU, then dropout with rate 0.2 2412.07823 §II.C. The card gives the order of these operations, so the dropout comes after the normalization and the activation. Card: Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications.

Dropout during crafting of poisons

The clean-label poisoning paper uses dropout in its stand-in networks while it crafts the poisoned images. It does not use dropout in the model that is attacked. At each crafting iteration, each stand-in is sampled as a thinned copy in which every unit is switched off with probability p and the kept units are scaled by 1/(1 − p) 1905.05897 §3.6. The crafting rate equals the rate the stand-in was trained with, and the values are 0.2, 0.25 and 0.3 1905.05897 §4. The victims are trained without dropout 1905.05897 §4. The paper's claim that this helps the poisons transfer rests on loss curves for one target, and no numeric ablation is printed 1905.05897 fig. 3. Card: Transferable Clean-Label Poisoning Attacks on Deep Neural Nets.

When to use which

The cards give conditions and comparisons, not a ranking that transfers to other tasks. The table follows what each card reports.

Implementation sketch

The loop below is the training-time core. It shows the two consistent ways to scale activations. The inverted form scales at training time; the plain form scales at inference. Pick one and keep it.

# Training-time dropout on one layer's activations
# Here `rate` is the drop share, unlike the keep rate in the click-prediction card
def dropout_train(h, rate, inverted):
    # Each unit is kept with probability 1 - rate
    mask = bernoulli(1 - rate, shape(h))
    if inverted:
        # Scale kept units up now, so inference needs no change
        return h * mask / (1 - rate)
    return h * mask

# Inference. The plain form scales at test time [arxiv:1804.04368 §4.3]
def dropout_infer(h, rate, inverted):
    if inverted:
        return h
    # Multiply by the kept share to match the unscaled training pass
    return h * (1 - rate)

# Variational mask for one recurrent layer [arxiv:1707.09861 §4.2]
def locked_mask(batch, units, rate):
    # One mask per sequence, shared by every time step
    keep = bernoulli(1 - rate, shape(batch, 1, units))
    return keep / (1 - rate)

mask = locked_mask(batch, units, rate)
for t in range(steps):
    # Same mask at each step; the mask is already scaled (inverted form)
    h[t] = h[t] * mask

Caveats that apply to every variant:

• The rate's meaning differs between cards. The click-prediction paper calls its value a keep rate: a value of 0.8 keeps 80% of units 1601.02376 §4.6. The Lipschitz paper gives its values as p and does not define them in the text 1804.04368 §5; its code uses the Keras Dropout layer maybe.py. The Lipschitz card reads Keras's documented argument as the drop share and does not verify it. The additive model's TensorFlow dropout also takes the drop share, scaling by 1/(1 − p) models.py. The sequence-tagging paper lists its rates without saying whether they are keep or drop shares 1707.09861 §4.2. Check the convention of your framework and of the paper before copying a number.
• Scaling must match between training and inference. The click-prediction code divides nothing at training time and uses undropped activations at inference 1601.02376 Paper vs code, pitfall "No inverted dropout" FNN.py. The Lipschitz paper describes weight scaling at test time 1804.04368 §4.3. Using the wrong pairing shifts the activation scale.
• Penalty terms may need dropout switched off. The additive model's output penalty is computed with dropout disabled graph_builder.py, so the penalty does not see a randomly thinned output. Check whether your framework's regulariser code does the same.
• Dropout changes what the constraint strength should be. The Lipschitz paper says a larger constraint value is needed with dropout, and that tuning without dropout and rescaling afterwards is not a valid shortcut 1804.04368 §4.3.
• Being in eval mode does not always turn dropout off. In one released implementation, a manual dropout call stays active whenever its rate is above zero resnet.py 1905.05897 pitfalls. Check that the switch you rely on actually disables it.

Contradictions

• Rate values, paper against code. The click-prediction paper gives a keep rate of about 0.8 for its FNN and about 0.99 for its SNN 1601.02376 §4.6. Its code uses 0.5 for the FNN on advertiser 2997, and 0.98 and 0.99 for the two SNN samplers FNN.py SNN_RBM.py SNN_DAE.py. The card records this as a difference, not a resolved value.
• Size of the gain. The click-prediction paper reports dropout better than L2 in every compared setting 1601.02376 §3.3. The Lipschitz paper says L2 penalties are weaker than dropout 1804.04368 §1, but its own CIFAR-10 table shows dropout alone 0.03 points above no regulariser on VGG19 and 0.33 points above on WRN-16-10 1804.04368 tab. 1. The two statements are about different comparisons, so both are kept here.
• Dropout placement, paper against code. The rover paper says 20% dropout between the hidden layers 2411.04700 §3.2. Its code applies one 20% dropout after the last hidden layer, before the output model.py. The card records this as a difference, not a resolved value.
• Default rate, paper against code. The sequence-tagging paper's grid is 0.05 to 0.5 and it names no single best rate 1707.09861 §4.2 1707.09861 §6.6. Its class default is (0.5, 0.5), and its example scripts use (0.25, 0.25) BiLSTM.py [code: Train_POS.py#L62]. Neither tuple is a value the paper reports.
• Best setting. The Lipschitz paper's text says a dropout combination is the best cell in several tables 1804.04368 §5. On CIFAR-10 WRN-16-10 the best cell is the constraint alone at 95.82 ± 0.19, above the combination at 95.68 ± 0.11 1804.04368 tab. 1.

Papers

• Deep Learning over Multi-field Categorical Data: dropout as a regulariser compared with L2, with its rates and scaling pitfall.
• Regularisation of Neural Networks by Enforcing Lipschitz Continuity: dropout as a baseline and as a companion to a Lipschitz constraint.
• Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation: DropConnect on a large fully connected layer.
• Neural Additive Models: Interpretable Machine Learning with Neural Nets: unit dropout and whole feature-net dropout, with per-dataset tuned rates.
• Score-distribution study of LSTM sequence taggers: variational dropout against naive dropout and no dropout across BiLSTM tagging configurations.
• Field Assessment of Force Torque Sensors for Planetary Rover Navigation: input and hidden dropout in a small terrain classifier, with a placement that differs between text and code.
• Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications: fixed 0.2 dropout after each hidden layer of a small estimator, with no ablation.
• A Survey on Regularization Methods for Convolutional Neural Networks: dropout and its variants described in prose; compiled results, not reproduced.
• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets: dropout in stand-in networks while crafting poisons; victims trained without it.
• Bayesian Neural Networks: An Introduction and Survey: MC dropout as variational inference; covered on the Monte Carlo dropout page.
• A Survey of Deep Active Learning: MC dropout as an uncertainty source for query selection; covered on the Monte Carlo dropout page.
• A Survey of Uncertainty in Deep Neural Networks: MC dropout as a test-time uncertainty method; covered on the Monte Carlo dropout page.
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations: MC dropout as a comparison baseline for annotator-disagreement uncertainty; covered on the Monte Carlo dropout page.
• Uncertainty tutorial for engineering prognostics: MC dropout compared with other uncertainty methods on prognostics data; covered on the Monte Carlo dropout page.