Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

In plain words

Neural networks that learn images can memorise their training pictures instead of learning what matters. Regularization is the family of tricks that stops this. This survey sorts the recent tricks for convolutional networks into three places where they act.

The first place is the input: the training pictures are changed, for example by hiding a patch, blending two pictures, or flipping labels along with the picture. The second place is inside the network: some units, branches or whole blocks are randomly switched off or weakened during training. The third place is the target: the "correct answer" is softened so the network is less sure of itself.

The authors chose papers from the last few years and kept only those with public code. They then put the reported results in one table. They did not rerun any experiment, so the table shows what each paper reported, not what a reader would get.

TL;DR

A survey of CNN regularizers from the last few years, organised as input, internal and label methods §1.3. It is a map and a compiled results table tab. 2, not a new method §7. Use it to choose which family to try, then read the original paper for the exact recipe. The reported numbers are the original authors' and were not reproduced.

Problem

Training data is limited compared with what a learner sees, so a network can fit the training set without generalising to new images §1.1. The survey treats regularization as the set of methods that reduce test error on data outside the training set §1.2. It limits its scope to recent CNN methods, in its words "not older than five years" in the abstract and "no other study is older than four years" besides Dropout in §1.3 §1.3.

Its own count finds that about 44% of the surveyed works change the input, and about 22% change the labels §1.5 §1.6. The survey argues that label-level work is under-explored §1.6.

Method

The survey defines a regularized least-squares form. A regularizer Ω(A) is added to the data-fit term, and a scalar λ sets how strongly it acts eq. 3 §1.1. Its own notation there is not consistent: the first formulation uses factors U and V but the right-hand side uses W and Y eq. 1 §1.1. The paper does not use this form for any method; the methods below are described in prose.

The survey groups methods into three families §1.3:

1. Input (data augmentation) §3. These change the training images before the network sees them. Covered: Cutout §3.1, RandomErasing §3.2, AutoAugment §3.3, PBA §3.4, RandAugment §3.5, Mixup §3.6, CutMix §3.7, CutBlur §3.8, BatchAugment §3.9, FixRes §3.10 and Bag of Tricks §3.11.
2. Internal §4. These act on activations, branches or weights inside the network. Covered: Dropout and variants §4.1, MaxDropout §4.2, DropBlock §4.3, TargetDrop §4.4, AutoDrop §4.5, LocalDrop §4.6, residual-branch methods §4.7, Shake-Shake §4.8, ShakeDrop §4.9 and Manifold Mixup §4.10.
3. Label §5. These change the target vector. Covered: Label smoothing §5.1, TSLA §5.2, SLS §5.3 and JoCoR §5.4.

Two methods are short enough to restate as a procedure. The survey gives no formulas for either, so the steps below follow its prose.

# Cutout  [§3.1]: hide one square patch of each training image
# Input:  image (H, W, C); patch side s, set per dataset
# Output: image with a patch removed (H, W, C)

def cutout(image, side):
    # Choose the patch centre; the sampling rule is not stated  [§3.1]
    centre_row, centre_col = sample_centre(image)
    # Remove the pixels inside the square, clipped at the border  [§3.1]
    # The fill value is not stated.
    return remove_square(image, centre_row, centre_col, side)

# CutMix  [§3.7]: paste a box from image B into image A; mix labels by area
# Input:  image_a, image_b (H, W, C); label_a, label_b one-hot (K,)
# Output: mixed image (H, W, C); soft label (K,)

def cutmix(image_a, label_a, image_b, label_b):
    # Choose the box and the share of A that stays visible  [§3.7]
    # The box sampling rule is not stated.
    box, kept_share = sample_box(image_a)
    # Replace the box region of A with pixels of B  [§3.7]
    mixed = paste_region(image_a, image_b, box)
    # Weight each label by the area it still covers  [§3.7]
    # Example in the text: 30% airplane in a cat image gives 70/30 [§3.7]
    soft_label = kept_share * label_a + (1 - kept_share) * label_b
    return mixed, soft_label

Other methods described in prose, with the survey's own descriptions:

• Dropout drops each neuron with a Bernoulli draw at each training step; the survey notes it also applies to MLPs and RBMs §4.1.
• MaxDropout normalises a tensor, then zeroes values above a threshold p, so high activations are more likely to be removed §4.2.
• DropBlock drops contiguous regions of each feature map, starting with a small ratio and increasing it during training §4.3.
• Shake-Shake applies a random multiplier to each of the three residual branches in the forward pass and a different one in the backward pass; at inference each branch is multiplied by 0.5 §4.8. The random distribution is not stated.
• Label smoothing replaces one-hot targets with smoothed ones; the survey says it counters overfitting and overconfidence §5.1.
• TSLA uses smoothing only for part of training, then sets targets to 0 and 1 §5.2.

Concepts used: data-augmentation, dropout, label-smoothing, convolutional-neural-network.

Hyperparameters

Values as printed in the survey. The survey does not run experiments, so each value is the original paper's setting as reported by the survey.

Results

Author-reported: these are the original papers' numbers as compiled by the survey in its Table 2. Metric is classification error in %, lower is better. Setups differ across rows (architecture, training length, augmentation stack), so compare within a row only §7.2.

Survey's own ranking claims, not tested here: AutoAugment beats PBA on ResNeXt + Shake-Shake for CIFAR-10, and PBA beats AutoAugment on PyramidNet + ShakeDrop for CIFAR-10 §7.1 tab. 2. The survey also calls RandAugment the "likely best" input method and TSLA the best label method, and it gives its own guess for internal methods §7.1. These are the survey's opinions.

Paper vs code

No official code. The arXiv abstract and metadata link no repository, and the paper text has no project page. A GitHub repository search for the title words and for the author handle cfsantos returned nothing. The survey also lists code links for the methods it covers tab. 3 §7.6. These are third-party or framework links, not code from this survey, and the card does not use them.

Survey inclusion rule versus its own table §1.3 tab. 3: the survey says every included method has code "usually on Github", but the table gives "Available in the paper" for LocalDrop, TSLA and SLS, a PyTorch documentation page for Dropout, and a generic tensorflow/models link for label smoothing.

Pitfalls

• Survey, not method. Nothing here can be re-implemented as one method. Use the cited original papers for recipes.
• Numbers are not reproduced. Table 2 is a compilation. Rows differ in architecture, schedule and augmentation stack. The survey itself warns that small changes to the training protocol can shift results on their own §7.2.
• Table 2 has problems. The "+ Dropout + RE" row is printed three times with identical values, and one row is commented out. A stray "19.2" sits in the PyramidNet ImageNet cell. Several rows cite the wrong source (for example "+ ShaekDrop + Cutout" cites AutoAugment). Check any row against its original paper before use.
• Inconsistent counts in the text. The abstract says papers are "not older than five years"; §1.3 says "four years". AutoAugment is described as using "five different transformations" but a list of 14 follows §3.3.
• Residual-connection wording. §2 says the residual connection "fuses" outputs and §6.2.1 says it is a "concatenation". The card does not resolve which one each architecture uses.
• Equation 1. Variables U and V appear in the minimisation but W and Y on the right eq. 1. The text says a global minimum needs one factor fixed. Do not use eq. 1 as an implementation target.
• Mixup has no formula here. The survey describes Mixup in prose only §3.6. α is not stated; take it from the original paper.
• Hypothetical label examples. The Cutout label rule (a 25% crop lowering the active class from 1 to 0.75) is the survey's own suggestion, not a reported result §5.
• Subjective ranking. "Best" calls in §7.1 are based on the compiled table and on the survey's guesses. The survey says ShakeDrop's internal rank is "a guess" §7.1.
• Dataset count. The ImageNet training count printed in §6.1.2 was not checked against the dataset.

Lineage

• Surveys it compares with: two earlier regularization surveys, one general and one on Dropout only §1.4. Names are not given in the text.
• Covers the methods listed under Method. It does not name a successor.
• Superseded by: not stated.