--- kind: concept slug: dropout title: Dropout summary: >- Randomly zeroing units, weights or whole sub-networks during training to regularise a network. Covers unit dropout, DropConnect, feature-net dropout and variational dropout for recurrent layers. Trade-offs: the rate is sensitive and the papers define it differently, the gain over no regulariser can be small, and the scaling used at inference must match the training form. tags: [regularization, training, dropout, recurrent-networks] papers: ["1505.03046", "1601.02376", "1707.09861", "1804.04368", "2004.13912", "2006.12024", "2009.00236", "2107.03342", "2110.05719", "2201.03299", "2305.04933", "2411.04700", "2412.07823", "1905.05897"] related: [monte-carlo-dropout, weight-decay, spectral-normalization, convolutional-neural-network, recurrent-neural-network, sequence-labeling, data-poisoning, uncertainty-estimation] status: draft updated_at: 2026-11-01 --- ## Overview Dropout is a training-time regulariser. Each forward pass removes a random subset of activations (or, in DropConnect, of weights) so that the network cannot rely on any one of them. Seven cards use it this way, in six settings: as the alternative to an L2 penalty in a sparse, multi-field click-prediction network [arxiv:1601.02376 §3.3]; as a baseline and as a companion to a Lipschitz constraint on image and tabular benchmarks [arxiv:1804.04368 §5]; as DropConnect on a large fully connected layer of a medical-imaging ConvNet [arxiv:1505.03046 §II-A]; inside each per-feature network of an interpretable additive model on tabular data, together with whole-network dropout [arxiv:2004.13912 app. A.3]; as variational dropout on the recurrent units of BiLSTM sequence taggers [arxiv:1707.09861 §4.2]; and as fixed-rate dropout in small fully connected networks on sensor and biomechanics features [arxiv:2411.04700 §3.2] [arxiv:2412.07823 §II.C]. The 2022 regularisation survey describes dropout and its variants in prose and gives no numbers of its own [arxiv:2201.03299 §4.1]. One card uses dropout for a different purpose: it randomises stand-in networks while poisoned training images are crafted, and the victims are trained without dropout [arxiv:1905.05897 §3.6] [arxiv:1905.05897 §4]. See the variant below. Keeping dropout active at prediction time, and averaging the passes, is a separate use. Five further cards list this slug for that use [arxiv:2006.12024 §2.4] [arxiv:2009.00236 §3.1.3] [arxiv:2107.03342 §3.2] [arxiv:2110.05719 §4.3.2] [arxiv:2305.04933 §3.2]. They are covered on [Monte Carlo dropout](/concepts/monte-carlo-dropout). ## Variants ### Unit dropout Each unit's output is multiplied by a Bernoulli variable at training time. The click-prediction paper uses it in both its FNN and SNN models [arxiv:1601.02376 §4.6], and reports it as a better regulariser than an L2 penalty in every setting it compares [arxiv:1601.02376 §3.3] [arxiv:1601.02376 §4.5]. Its paper and code give different rates; see the caveats below. Card: [Deep Learning over Multi-field Categorical Data](/papers/1601.02376). The Lipschitz-constraint paper uses unit dropout as a baseline in every table and combines it with its own constraint. Its scripts set rates per layer type: VGG on CIFAR-10 uses 0.2 for convolutional and 0.5 for dense layers, and WRN uses 0.2 for convolutional layers [code: experiments/cifar10-vgg.sh#L8] [code: experiments/cifar10-wrn.sh#L7]. The paper does not state its rates in the text [arxiv:1804.04368 §5]. Card: [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](/papers/1804.04368). ### Variational dropout for recurrent layers Naive dropout draws a new mask at every time step of an LSTM. Variational dropout draws one mask per sequence and reuses it at all time steps, and it also applies dropout to the recurrent units [arxiv:1707.09861 §4.2]. The paper cites a prior work for this method and does not restate it. In its sweep over sequence-tagging configurations, variational dropout was the best option in 83.5% of 479 cases and gave the smallest spread. Its median gain over no dropout ranged from 0.31 points on POS to 1.98 on Entities, and over naive dropout from 0.19 to 1.32 [arxiv:1707.09861 §6.6]. Dropout on both output and recurrent units was best in 62.6% of cases, with median gains of 0.05 to 0.82 points [arxiv:1707.09861 §6.6]. These results are for five tagging tasks with BiLSTM networks trained from scratch, with dropout rates from {0.05, 0.1, 0.25, 0.5} [arxiv:1707.09861 §4.2]. Card: [score-distribution study of LSTM sequence taggers](/papers/1707.09861). ### DropConnect DropConnect drops individual weights instead of units. The detection paper names it as a regulariser its architecture builds on [arxiv:1505.03046 §II-A]. Its fully connected layer has 512 units with a DropConnect rate of 0.5, taken from the released configuration file and not stated in the paper [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L96]. The card cites no ablation of DropConnect, so its effect in that pipeline is not known from the card. Card: [Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation](/papers/1505.03046). ### Feature-net dropout The additive model gives each input feature its own small network. Unit dropout is applied after each hidden layer of every feature network [code: neural_additive_models/models.py#L168-L176]. The paper also drops whole feature networks: during training, a feature's network is switched off for an individual example [arxiv:2004.13912 app. A.3]. Both rates are tuned per dataset by Bayesian optimisation [arxiv:2004.13912 app. A.5]. The unit rate search set is {0, 0.05, …, 0.9} and the feature-net search set is {0, 0.05, 0.1, 0.2} [arxiv:2004.13912 app. A.6]. The tuned unit rates are 0.1, 0.2, 0.8, 0.0 and 0.0 on COMPAS, MIMIC-II, Credit Fraud, FICO and Housing, and the feature-net rates are 0.05 on COMPAS and 0.0 elsewhere [arxiv:2004.13912 app. A.6]. The card cites no ablation of feature-net dropout. Its code applies TensorFlow dropout to the stacked feature outputs, which rescales surviving outputs by 1/(1 − p) during training [code: neural_additive_models/models.py#L243-L247]. The paper does not state that scaling. Card: [Neural Additive Models: Interpretable Machine Learning with Neural Nets](/papers/2004.13912). ### Fixed-rate dropout in small fully connected networks Two cards use dropout as one fixed-rate ingredient of a small network, with no rate search. Neither card reports an ablation of its dropout. The planetary-rover classifier places dropout at two points. The paper's text says 20% dropout between hidden layers [arxiv:2411.04700 §3.2]. Its code applies 10% dropout to the inputs and 20% after the last hidden layer, before the output [code: ml/model.py#L45] [code: ml/model.py#L66]. Card: [Field Assessment of Force Torque Sensors for Planetary Rover Navigation](/papers/2411.04700). The hip-moment estimator uses a three-layer network of 50 units. Each hidden layer runs linear, then batch normalization, then ReLU, then dropout with rate 0.2 [arxiv:2412.07823 §II.C]. The card gives the order of these operations, so the dropout comes after the normalization and the activation. Card: [Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications](/papers/2412.07823). ### Dropout during crafting of poisons The clean-label poisoning paper uses dropout in its stand-in networks while it crafts the poisoned images. It does not use dropout in the model that is attacked. At each crafting iteration, each stand-in is sampled as a thinned copy in which every unit is switched off with probability p and the kept units are scaled by 1/(1 − p) [arxiv:1905.05897 §3.6]. The crafting rate equals the rate the stand-in was trained with, and the values are 0.2, 0.25 and 0.3 [arxiv:1905.05897 §4]. The victims are trained without dropout [arxiv:1905.05897 §4]. The paper's claim that this helps the poisons transfer rests on loss curves for one target, and no numeric ablation is printed [arxiv:1905.05897 fig. 3]. Card: [Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](/papers/1905.05897). ## When to use which The cards give conditions and comparisons, not a ranking that transfers to other tasks. The table follows what each card reports. | Situation | Use | Why | |---|---|---| | Fully connected network on sparse, multi-field inputs and an L2 penalty is the alternative | Unit dropout, with the rate checked against the paper's own value | The click-prediction paper reports dropout better than L2 in its compared settings [arxiv:1601.02376 §3.3] [arxiv:1601.02376 §4.5] | | Few labelled examples and a norm or Lipschitz constraint is an option | Dropout combined with the constraint, then check the plain constraint too | The Lipschitz paper reports its gains are largest with few training examples [arxiv:1804.04368 §5.8]; on CIFAR-10 WRN-16-10, the constraint alone scores higher than the combination [arxiv:1804.04368 tab. 1] | | A strong baseline already exists and the budget for tuning is small | Measure dropout against no regulariser on your data before adopting it | On CIFAR-10 the paper's dropout gains over no regulariser are small: VGG19 90.46 vs 90.43 and WRN-16-10 95.46 vs 95.13 [arxiv:1804.04368 tab. 1] | | A large dense layer in a ConvNet that already uses per-view or per-patch training | DropConnect on that layer, if the framework supports it | The detection paper uses it on a 512-unit layer [arxiv:1505.03046 §II-A] [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L96]; the card has no ablation, so benefit is not shown | | Convolutional and dense layers in one network | Separate rates per layer type, starting from the values in the cards | The VGG script uses 0.2 for convolutional and 0.5 for dense layers [code: experiments/cifar10-vgg.sh#L8]; the WRN script uses 0.2 for convolutional layers [code: experiments/cifar10-wrn.sh#L7]. The paper does not justify these values | | A per-feature network model on tabular data, where the rate should be set per dataset | Unit dropout in each feature network, tuned on your data over the paper's search set | The tuned unit rate runs from 0.0 to 0.8 across five datasets [arxiv:2004.13912 app. A.6], so no single value transfers | | A small fully connected regressor or classifier on tabular or sensor features, with no sequence structure | Fixed dropout after the hidden layers at the rate the card uses (0.2 in both), then tune the rate on your data | Both cards use it, but neither reports an ablation, so the gain over no dropout is not shown [arxiv:2411.04700 §3.2] [arxiv:2412.07823 §II.C] | | A recurrent network (LSTM or BiLSTM) trained from scratch, such as a sequence tagger | Variational dropout, with the mask shared across time steps, on the recurrent units as well as the outputs; choose the rate on your data | Variational was the best option in 83.5% of 479 configurations and beat naive dropout by a median of 0.19 to 1.32 points [arxiv:1707.09861 §6.6]. The paper's comparison is on tagging tasks, so the gain on other tasks is not shown | | Choosing among CNN regularisers | Use the survey to pick a family, then the original paper for the recipe | Its results table is compiled from original papers and not rerun, and its rows differ in architecture, schedule and augmentation [arxiv:2201.03299 tab. 2] [arxiv:2201.03299 §7.2] | | Using the spread over dropout passes as an uncertainty estimate | Use the [Monte Carlo dropout](/concepts/monte-carlo-dropout) page; tune the rate and check calibration | The uncertainty tutorial warns the result depends strongly on the dropout rate, the number of dropout layers and training length, and reports overconfidence at low rates [arxiv:2305.04933 §3.2] [arxiv:2305.04933 §6.1] | ## Implementation sketch The loop below is the training-time core. It shows the two consistent ways to scale activations. The inverted form scales at training time; the plain form scales at inference. Pick one and keep it. ```pseudocode # Training-time dropout on one layer's activations # Here `rate` is the drop share, unlike the keep rate in the click-prediction card def dropout_train(h, rate, inverted): # Each unit is kept with probability 1 - rate mask = bernoulli(1 - rate, shape(h)) if inverted: # Scale kept units up now, so inference needs no change return h * mask / (1 - rate) return h * mask # Inference. The plain form scales at test time [arxiv:1804.04368 §4.3] def dropout_infer(h, rate, inverted): if inverted: return h # Multiply by the kept share to match the unscaled training pass return h * (1 - rate) # Variational mask for one recurrent layer [arxiv:1707.09861 §4.2] def locked_mask(batch, units, rate): # One mask per sequence, shared by every time step keep = bernoulli(1 - rate, shape(batch, 1, units)) return keep / (1 - rate) mask = locked_mask(batch, units, rate) for t in range(steps): # Same mask at each step; the mask is already scaled (inverted form) h[t] = h[t] * mask ``` Caveats that apply to every variant: - The rate's meaning differs between cards. The click-prediction paper calls its value a keep rate: a value of 0.8 keeps 80% of units [arxiv:1601.02376 §4.6]. The Lipschitz paper gives its values as `p` and does not define them in the text [arxiv:1804.04368 §5]; its code uses the Keras `Dropout` layer [code: arch/maybe.py#L6]. The Lipschitz card reads Keras's documented argument as the drop share and does not verify it. The additive model's TensorFlow dropout also takes the drop share, scaling by 1/(1 − p) [code: neural_additive_models/models.py#L243-L247]. The sequence-tagging paper lists its rates without saying whether they are keep or drop shares [arxiv:1707.09861 §4.2]. Check the convention of your framework and of the paper before copying a number. - Scaling must match between training and inference. The click-prediction code divides nothing at training time and uses undropped activations at inference [arxiv:1601.02376 Paper vs code, pitfall "No inverted dropout"] [code: python/FNN.py#L151-L152]. The Lipschitz paper describes weight scaling at test time [arxiv:1804.04368 §4.3]. Using the wrong pairing shifts the activation scale. - Penalty terms may need dropout switched off. The additive model's output penalty is computed with dropout disabled [code: neural_additive_models/graph_builder.py#L110-L117], so the penalty does not see a randomly thinned output. Check whether your framework's regulariser code does the same. - Dropout changes what the constraint strength should be. The Lipschitz paper says a larger constraint value is needed with dropout, and that tuning without dropout and rescaling afterwards is not a valid shortcut [arxiv:1804.04368 §4.3]. - Being in eval mode does not always turn dropout off. In one released implementation, a manual dropout call stays active whenever its rate is above zero [code: models/resnet.py#L43-L45] [arxiv:1905.05897 pitfalls]. Check that the switch you rely on actually disables it. ## Contradictions - **Rate values, paper against code.** The click-prediction paper gives a keep rate of about 0.8 for its FNN and about 0.99 for its SNN [arxiv:1601.02376 §4.6]. Its code uses 0.5 for the FNN on advertiser 2997, and 0.98 and 0.99 for the two SNN samplers [code: python/FNN.py#L42] [code: python/SNN_RBM.py#L55] [code: python/SNN_DAE.py#L48]. The card records this as a difference, not a resolved value. - **Size of the gain.** The click-prediction paper reports dropout better than L2 in every compared setting [arxiv:1601.02376 §3.3]. The Lipschitz paper says L2 penalties are weaker than dropout [arxiv:1804.04368 §1], but its own CIFAR-10 table shows dropout alone 0.03 points above no regulariser on VGG19 and 0.33 points above on WRN-16-10 [arxiv:1804.04368 tab. 1]. The two statements are about different comparisons, so both are kept here. - **Dropout placement, paper against code.** The rover paper says 20% dropout between the hidden layers [arxiv:2411.04700 §3.2]. Its code applies one 20% dropout after the last hidden layer, before the output [code: ml/model.py#L66]. The card records this as a difference, not a resolved value. - **Default rate, paper against code.** The sequence-tagging paper's grid is 0.05 to 0.5 and it names no single best rate [arxiv:1707.09861 §4.2] [arxiv:1707.09861 §6.6]. Its class default is (0.5, 0.5), and its example scripts use (0.25, 0.25) [code: neuralnets/BiLSTM.py#L38] [code: Train_POS.py#L62]. Neither tuple is a value the paper reports. - **Best setting.** The Lipschitz paper's text says a dropout combination is the best cell in several tables [arxiv:1804.04368 §5]. On CIFAR-10 WRN-16-10 the best cell is the constraint alone at 95.82 ± 0.19, above the combination at 95.68 ± 0.11 [arxiv:1804.04368 tab. 1]. ## Papers - [Deep Learning over Multi-field Categorical Data](/papers/1601.02376): dropout as a regulariser compared with L2, with its rates and scaling pitfall. - [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](/papers/1804.04368): dropout as a baseline and as a companion to a Lipschitz constraint. - [Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation](/papers/1505.03046): DropConnect on a large fully connected layer. - [Neural Additive Models: Interpretable Machine Learning with Neural Nets](/papers/2004.13912): unit dropout and whole feature-net dropout, with per-dataset tuned rates. - [Score-distribution study of LSTM sequence taggers](/papers/1707.09861): variational dropout against naive dropout and no dropout across BiLSTM tagging configurations. - [Field Assessment of Force Torque Sensors for Planetary Rover Navigation](/papers/2411.04700): input and hidden dropout in a small terrain classifier, with a placement that differs between text and code. - [Optimizing Locomotor Task Sets in Biological Joint Moment Estimation for Hip Exoskeleton Applications](/papers/2412.07823): fixed 0.2 dropout after each hidden layer of a small estimator, with no ablation. - [A Survey on Regularization Methods for Convolutional Neural Networks](/papers/2201.03299): dropout and its variants described in prose; compiled results, not reproduced. - [Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](/papers/1905.05897): dropout in stand-in networks while crafting poisons; victims trained without it. - [Bayesian Neural Networks: An Introduction and Survey](/papers/2006.12024): MC dropout as variational inference; covered on the Monte Carlo dropout page. - [A Survey of Deep Active Learning](/papers/2009.00236): MC dropout as an uncertainty source for query selection; covered on the Monte Carlo dropout page. - [A Survey of Uncertainty in Deep Neural Networks](/papers/2107.03342): MC dropout as a test-time uncertainty method; covered on the Monte Carlo dropout page. - [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](/papers/2110.05719): MC dropout as a comparison baseline for annotator-disagreement uncertainty; covered on the Monte Carlo dropout page. - [Uncertainty tutorial for engineering prognostics](/papers/2305.04933): MC dropout compared with other uncertainty methods on prognostics data; covered on the Monte Carlo dropout page.