Image Data Augmentation for Deep Learning: A Survey

In plain words

Deep image models learn from examples, and collecting labelled examples is expensive. Data augmentation makes more training examples out of the ones you already have, for instance by flipping, cropping or blanking out parts of a picture, or by blending two pictures together. The label has to stay valid after the change.

This paper is a review of those techniques. It sorts them into groups, from simple edits to methods that learn the edits from data or generate new images with neural networks. It then runs a set of common methods on three standard vision tasks and reports how much each task improved.

The authors' own measurements show small gains for most settings, and one detection model did slightly worse. The paper also lists open problems, such as the lack of theory about how much augmented data is enough.

TL;DR

A survey of image data augmentation that groups methods into basic manipulation, erasing, mixing, automated policy search, feature-space augmentation and GAN-based generation. Its own experiments apply the manipulation, erasing and mixing methods to semantic segmentation (PASCAL VOC), image classification (CIFAR-10, CIFAR-100, SVHN) and object detection (COCO2017). Author-reported: segmentation IoU rises by 0.53 to 2.61 points, classification accuracy by 0.07 to 2.80 points, and detection mAP by 0.4 points for Faster R-CNN and falls by 0.27 for CenterNet. No code is released and the training settings are not stated.

Problem

Deep networks need large training sets to avoid overfitting, and labelled image data is often limited §1. Augmentation is presented as a way to raise the quantity and diversity of training data §1. The authors name three gaps in the literature §1:

• Augmentation methods are usually task-specific, because the transform must be applied to the labels as well: a detector's boxes or a segmenter's masks must move with the image §1.
• There is no theory of how much augmented data is enough, and the size is usually set by experience §1 §5.
• Earlier surveys cover a single task, a single family of methods, or lack recent methods such as CutMix and AugMix §1.

The paper therefore sets out to review methods across tasks and to compare them empirically §1 fig. 1.

Method

This is a survey and an empirical comparison. It has no algorithm of its own, so the block below only shows the common shape of an augmentation step that the taxonomy covers. The paper does not specify how many operations are chosen or how often, so those details are left open. The taxonomy is given in fig. 1 and the reviewed methods are grouped as follows.

# Generic augmentation step, as the survey frames it  [§1] [§2]
# Input:  image (H, W, C), label, list of operations
# Output: augmented image and a label that still matches it

def augment(image, label, operations):
    # Apply each chosen operation; the count and order are not stated  [§4]
    for operation in operations:
        # Transform the label with the image, e.g. boxes or masks  [§1]
        image, label = operation(image, label)
    return image, label

1. Image manipulation §2.1 tab. 1. Flipping, rotation, scaling, noise injection, colour-space change, contrast change, sharpening, translation and cropping. The paper notes two problems: the transform is only sound if the data already follows something close to the true distribution, and translation and rotation leave empty borders that are usually filled with a constant (zero, black) value §2.1. It also notes that the object of interest must not leave the frame §2.1.
2. Image erasing §2.2. A sub-region of the input is removed and replaced by a constant or random value. Methods named are Cutout §2.2, Hide-and-Seek §2.2, Random Erasing §2.2, GridMask, a structured set of evenly spaced removed squares with controllable density and size §2.2, and FenceMask, which simulates object occlusion §2.2.
3. Image mixing §2.3. Two or more images, or sub-regions, are combined into one. Named methods are pairing samples (per-pixel average of two images), Mixup (convex combination of images and labels), CutMix (a patch from another image replaces the removed region), FMix (masks thresholded from low-frequency Fourier-space images), AugMix (several augmentation chains mixed by convex combination), and ManifoldMix (mixing hidden representations at several layers) §2.3.
4. Automated policy search §3.1. AutoAugment searches policies with reinforcement learning. Fast AutoAugment uses density matching to search faster. Population Based Augmentation produces non-stationary schedules. RandAugment removes the separate search step. KeepAugment uses a saliency map to keep informative regions. Augmentation-wise weight sharing makes search on large datasets affordable. OHL-Auto-Aug treats the policy as a probability distribution optimised jointly with the network §3.1.
5. Feature augmentation §3.2. The transform is applied to learned feature vectors instead of pixels. Named methods are noise addition and nearest-neighbour interpolation and extrapolation in feature space §3.2, FeatMatch, which refines features using within-class and across-class prototypes §3.2, and Moment Exchange, which swaps feature moments between two training images §3.2.
6. Deep generative models §3.3. GAN-based generation. Pix2Pix needs paired data §3.3. CycleGAN learns translation between two domains without pairs §3.3. StarGAN handles several domains with one generator and a domain label §3.3. StarGAN v2 adds style codes so that one input can produce diverse outputs within a target domain §3.3.

The paper also states a taxonomy-level position: the methods in groups 1 to 3 are the "basic" ones, and groups 4 to 6 the "advanced" ones §1.

Experimental protocol §4. The authors say they used the most common public datasets. The augmentation set in the experiments is the manipulation, erasing and mixing methods from groups 1 to 3 §4.1 §4.2 §4.3. The automated, feature-space and GAN methods are not evaluated in the experiments. Each model is trained with and without augmentation, and the paper reports the difference tab. 2 tab. 3 tab. 4.

Hyperparameters

The paper does not state training or augmentation hyperparameters for any experiment. The table lists what an implementer would need and where the paper looks for it.

Results

Author-reported. The paper ran each model with and without augmentation on three tasks. Setup: segmentation on PASCAL VOC with DeepLabV3+, PSPNet, GCNet and ISANet, metric mean IoU §4.1 tab. 2; classification on CIFAR-10, CIFAR-100 and SVHN with DenseNet, Wide-ResNet and Shake-ResNet, metric accuracy §4.2 tab. 3; detection on COCO2017 with Faster R-CNN and CenterNet, metrics mAP, AP50 and AP75 §4.3 tab. 4. Hardware, training length and the baseline recipe are not stated. "Reported change" is the printed column. The paper's column names are kept.

Segmentation, mean IoU (%) on PASCAL VOC tab. 2:

Classification, accuracy (%) tab. 3. "AAI" is the paper's average accuracy improvement column:

Detection on COCO2017 tab. 4. The "API" row is printed by the paper and is not defined in the text. The column header names the two detectors:

The text describes these as "promising" gains §4.3. The CenterNet rows are lower with augmentation on all three metrics tab. 4.

Paper vs code

No official code. The paper gives no repository or project link. A GitHub search by title words returned no repository described as this paper's code. A search by the first author's name returned user accounts, but the only repository on the account matching that name is a personal web page with no topic and no connection to augmentation. Surveys have no method code to compare, so nothing is checked here. The experiment scripts, configs and augmentation settings are not released, so the experiments cannot be reproduced from this card.

Pitfalls

• No code, no configs, no training recipe: the experimental results cannot be reproduced from the paper alone. Treat the numbers as the authors' single runs with unknown settings §4.
• Arithmetic inconsistencies in the printed tables. Recomputing from the printed "without" and "with" columns gives different values than the printed improvement column in several rows: DeepLabV3+ (0.43 vs 0.53) tab. 2; Shake-ResNet on CIFAR-10 (1.14 vs 1.11), Wide-ResNet on SVHN (0.08 vs 0.80) and Shake-ResNet on SVHN (0.03 vs 0.30) tab. 3; Faster R-CNN mAP (0.40 vs 2.40) tab. 4. Use the printed "without" and "with" columns, and recompute the differences yourself, rather than trusting the change column.
• The Table 4 "API" row is not defined in the text. The text says the metrics are "summarized for all methods in average sense" §4.3, which suggests averaging over augmentation methods, but the averaging is not described. The "with aug" value is therefore not a single configuration.
• Table 2's caption says "based on IoU and accuracy", but only IoU is reported tab. 2.
• The paper says the method set is "the same" across tasks §4.1 §4.2 §4.3. It names "mosaic" among the segmentation mixing methods §4.1, but mosaic is not described in the review sections. Implement mosaic from another source, not from this paper.
• The experiments cover only manipulation, erasing and mixing. Automated policy search (group 4), feature-space augmentation (group 5) and GAN-based generation (group 6) are described but not measured §3 §4. Do not use the tables to judge those groups.
• Gains are small and the spread between methods is not shown. With the reported ranges (for example +0.07 to +2.80 points in classification), the differences may fall within run-to-run variance. No variance is reported §4.
• CenterNet detection is lower with augmentation on all three metrics tab. 4, while the text describes the results as promising §4.3. Read the table, not the summary.
• The survey's selection criteria for the reviewed papers are not stated §1 §2 §3.
• The paper's own claims about why augmentation helps (e.g. about pairing samples and Mixup being "visually meaningless" to humans) are opinions, not measured results §5.
• The arXiv metadata lists version v2, and the LaTeX source used here is the one fetched for this card. Section and table numbers refer to that source.

Lineage

• Surveys discussed in the introduction, compared by scope rather than by method: a classification-only survey, a face-recognition survey, a survey on warping and oversampling, and an earlier general survey without task-level evaluation §1.
• The reviewed methods are the building blocks of the taxonomy: see §2 and §3 for the named papers.
• Superseded by: not applicable; no later version of this survey is described in this card.