SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization

In plain words

Image classifiers learn from training pictures, and a common trick is to make new training pictures by cutting a patch out of one picture and pasting it onto another, then mixing their labels in proportion to the patch size. The earlier version of this trick picked the patch at random. That patch could land on background, and the mixed label would then tell the model that the background belongs to the object.

SaliencyMix first finds the spot in the source picture that draws the eye, the most salient pixel, and cuts the patch around it. The patch is more likely to contain part of the object, so the label mix is more honest. The authors report better accuracy on CIFAR and ImageNet and some gain in robustness to adversarial noise. The cost is a saliency computation on every mixed batch, which adds a little training time.

TL;DR

SaliencyMix is a data augmentation for CNN training. It pastes a patch from one training image onto another at the same location, with the patch chosen around the most salient pixel of the source image, and mixes the two labels by the patch area. The paper reports top-1 error on CIFAR, ImageNet and object detection, all author-reported. The released code differs from the paper in the saliency box, the batch handling and the ImageNet dataset (see Paper vs code).

Problem

Regional dropout removes image regions and so discards informative pixels §1. CutMix pastes a random patch and mixes labels by patch size §1 §2.1. If the random patch falls on background, the mixed label still gives the source class weight, so the model can learn background features for that class §1 fig. 1. The paper assumes the classifier is sensitive to texture §1. Nothing is said about how often a random patch misses the object.

Method

SaliencyMix mixes one source patch into one target image. Both come from the training set §3. For a pair of images:

1. Compute a saliency map of the source image, I_vs = f(I_s), with f an OpenCV saliency detector eq. 1 §3.1 §3.3.
2. Take the pixel with the maximum saliency value eq. 2 §3.1.
3. Choose a patch whose size is set by the combination ratio λ, drawn from Uniform(0, 1) §3.1. The patch is centred on the peak pixel if that is possible, otherwise the peak is kept inside the patch §3.1.
4. Build a binary mask M that is 1 on the patch and 0 elsewhere. The augmented image is I_a = M ⊙ I_s + M' ⊙ I_t, with M' the complement of M eq. 3 §3.2.
5. Mix the labels by the ratio: y_a = λ y_t + (1 − λ) y_s eq. 4 §3.2.

The default scheme pastes the patch at the corresponding location in the target ("Salient to Corresponding"). The paper compares four other schemes in which the source is salient or non-salient and the target location is salient, non-salient or the same as the source §3.4 fig. 3(c–d). Among the five schemes, Salient to Corresponding gives the lowest classification error in the paper's experiments §3.4.

The paper compares four saliency detectors on CIFAR-10 with ResNet-18 and Tiny-ImageNet with ResNet-50. The OpenCV method cited as [OpenCV_VS] performs best, so it is used throughout §3.3 fig. 3(a–b). The paper calls it "fast self-tuning background subtraction" §3.3. The bibliography entry is Montabone and Soto (2010), a visual saliency method for human detection [bib: OpenCV_VS]. The released code uses OpenCV's StaticSaliencyFineGrained saliencymix.py.

Paper-level pseudocode, following §3.1 and §3.2. The paper gives no algorithm box and no formula for the patch side lengths, so those lines are marked.

# SaliencyMix for one (source, target) pair, reconstructed from §3.1-§3.2
# Input:  source (C, H, W) with label y_s; target (C, H, W) with label y_t
# Output: augmented image (C, H, W) and mixed label y_a

def saliency_mix(source, y_s, target, y_t):
    # Sample the combination ratio λ from Uniform(0, 1)  [§3.1]
    ratio = uniform(0, 1)

    # Saliency map of the source image  [§3.1] [eq. 1]
    saliency = detector(source)               # (H, W), gray-scale

    # Pixel with the maximum saliency value  [§3.1] [eq. 2]
    row, col = argmax(saliency)

    # Patch size set by λ; the exact side lengths are not stated  [§3.1]
    patch_h, patch_w = patch_size(ratio, H, W)

    # Centre the patch on the peak if possible; otherwise keep the
    # peak inside the patch  [§3.1]
    box = place_patch(row, col, patch_h, patch_w, H, W)

    # Binary mask M: 1 on the patch, 0 elsewhere  [§3.2]
    mask = box_mask(box, H, W)

    # Paste the source patch at the same location in the target
    # (default scheme)  [§3.2] [eq. 3] [§3.4]
    augmented = mask * source + (1 - mask) * target

    # Mix the labels by λ  [§3.2] [eq. 4]
    y_a = ratio * y_t + (1 - ratio) * y_s
    return augmented, y_a

Key equations, as written in the paper:

I_{vs} = f(I_s) eq. 1

i,j = \operatorname{argmax}(I_{vs}) eq. 2

I_a = M \odot I_s + M' \odot I_t eq. 3

y_a = \lambda\, y_t + (1-\lambda)\, y_s eq. 4

Hyperparameters

Values for the CIFAR and ImageNet experiments are from the text, with the code value next to them where the code sets one. "not stated" means the paper and the cited code say nothing. The code column is from the repository at code_commit.

Results

Author-reported. Setup: CIFAR-10 and CIFAR-100 (50,000 training and 10,000 test images each) with ResNet-18, ResNet-50 and WideResNet-28-10, 200 epochs, baselines as listed §4.1.1 tab. 1. ImageNet with ResNet-50 and ResNet-101, 300 epochs, best run per method §4.1.2 tab. 2. Detection: Faster R-CNN with ResNet-50 on Pascal VOC §4.2 tab. 3. Adversarial robustness: FGSM on ImageNet validation §4.4 tab. 4. Training time: ResNet-18 on CIFAR-10, 200 epochs §4.5 tab. 5. Hardware: four RTX 2080 Ti §4.

CIFAR, top-1 error (%), mean ± std over five runs tab. 1. "+" means traditional augmentation is also used.

Bold marks the best value in each column, as in the source table. The paper's text says it reduces CIFAR-10 error over CutMix by 1.85, 2.35 and 1.14 points for ResNet-18, ResNet-50 and WideResNet-28-10 without traditional augmentation, and by 5.69, 6.76 and 3.76 points on CIFAR-100 §4.1.1. These match the table.

ImageNet, error (%), best run per method tab. 2.

Transfer to Pascal VOC object detection, Faster R-CNN ResNet-50, mAP on VOC 2007 test (mean of three runs) tab. 3. Gain over baseline in brackets.

Adversarial robustness, top-1 accuracy (%) on FGSM-perturbed ImageNet validation, ResNet-50 tab. 4. The paper says this is +1.96 over CutMix §4.4.

Training time, ResNet-18 on CIFAR-10, 200 epochs, hours tab. 5. Paper says the extra time is due to saliency map generation and is negligible against the gain §4.5.

Qualitatively, the class activation maps of a ResNet-50 trained with SaliencyMix focus more on the target object than those trained with other augmentations §4.3 fig. 4. This is a visual comparison, with no metric.

Paper vs code

Code: SaliencyMix/SaliencyMix at commit 9ee02b87974efab1ee7a5229408a58a0e0bbf825, linked from the abstract. The repository has no LICENSE file, so no code is quoted here, only cited by path and line. The code contains two training scripts: SaliencyMix_CIFAR/ (PyTorch) and SaliencyMix-ImageNet/ (PyTorch). The CIFAR script is based on Cutout, and the ImageNet script on CutMix-PyTorch [README.md].

Differences between paper and code:

1. Saliency is computed once per batch, not per image. The paper describes a per-pair operation §3.1. The code computes one box from the first permuted partner image and pastes that same box from every partner image saliencymix.py; ImageNet is the same train.py.
2. Application is per batch. The paper does not say whether SaliencyMix is applied per sample or per batch. The code draws one uniform number per batch against salmix_prob saliencymix.py.
3. Box side and clipping. The paper says the patch size follows λ and the peak is centred "if possible", otherwise kept inside the patch §3.1. The code sets a box side of sqrt(1 − λ) of the image side [code: SaliencyMix_CIFAR/saliencymix.py#L195-L197], centres it on the peak and clips it to the image saliencymix.py. Clipping only truncates the box, so near the border the patch is smaller. It does not shift the box to keep the peak inside.
4. λ is recomputed from the clipped area. The label weight in the code is 1 minus the actual pasted area over the image area [code: SaliencyMix_CIFAR/saliencymix.py#L243], not the sampled λ as in eq. 4. The ImageNet code does the same train.py.
5. Loss form. The code computes the mixed loss as λ · CE(pred, y_t) + (1 − λ) · CE(pred, y_s) saliencymix.py. This is the same as cross-entropy against the mixed label of eq. 4, because cross-entropy is linear in the target.
6. Saliency input. The saliency detector receives the normalised image tensor, converted to a NumPy array saliencymix.py. The paper does not say what the detector receives. The output is multiplied by 255 and cast to uint8 saliencymix.py.
7. Batch size. The paper says 256 for CIFAR §4.1.1. The CIFAR code default and README use 128 saliencymix.py [code: SaliencyMix_CIFAR/README.md].
8. ImageNet data. The paper reports ImageNet-1k, 1000 classes §4.1.2. The ImageNet script reads ~/dataset/tiny-imagenet and sets 200 classes [code: SaliencyMix-ImageNet/train.py#L79-L80] train.py. The released script does not reproduce the ImageNet-1k setting without changes.
9. Probability. The paper gives no application probability. The CIFAR default is 0.5 saliencymix.py. The ImageNet README command uses 1.0 README.md.
10. Epochs and defaults. The ImageNet script defaults to 90 epochs and batch 128 train.py. The README command uses 300 epochs and batch 256 README.md. The LR schedule for 300 epochs matches the paper train.py.

No discrepancies were found in CIFAR optimiser, weight decay, LR schedule, epochs or WideResNet dropout; these match §4.1.1 saliencymix.py.

The paper names the saliency method in two ways. The text says "fast self-tuning background subtraction" §3.3. The bibliography cites Montabone and Soto (2010) for [OpenCV_VS] [bib]. The code uses OpenCV's StaticSaliencyFineGrained saliencymix.py. I could not confirm from the sources here that these are the same method.

Code-level pseudocode, as the released code runs it, not the paper's per-image rule:

# Released training step, CIFAR script
# [code: SaliencyMix_CIFAR/saliencymix.py#L234-L248]
# Input:  images (B, C, H, W), normalised; labels (B,)
# Output: loss for one optimisation step

def training_step(images, labels, beta, salmix_prob):
    # Apply SaliencyMix to the whole batch with probability salmix_prob
    # [code: SaliencyMix_CIFAR/saliencymix.py#L235]
    if uniform(0, 1) < salmix_prob:
        ratio = beta_sample(beta, beta)    # beta = 1.0 gives Uniform(0, 1)
        partners = random_permutation(B)
        # One box from the saliency peak of the first partner image only
        # [code: SaliencyMix_CIFAR/saliencymix.py#L241]
        box = saliency_box(images[partners[0]], ratio)
        # The same box is pasted from every partner image
        # [code: SaliencyMix_CIFAR/saliencymix.py#L242]
        images[:, :, box] = images[partners, :, box]
        # Recompute λ from the clipped box area
        # [code: SaliencyMix_CIFAR/saliencymix.py#L243]
        ratio = 1 - box_area(box) / (H * W)
        pred = model(images)
        loss = (ratio * cross_entropy(pred, labels)
                + (1 - ratio) * cross_entropy(pred, labels[partners]))
    else:
        loss = cross_entropy(model(images), labels)
    return loss

# Box for the batch-level step above
# [code: SaliencyMix_CIFAR/saliencymix.py#L191-L215]
# Input:  image (C, H, W), normalised; ratio in (0, 1)
# Output: clipped box (row range, column range)

def saliency_box(image, ratio):
    # Side of the box as a fraction of the image side
    # [code: SaliencyMix_CIFAR/saliencymix.py#L195-L197]
    scale = sqrt(1 - ratio)
    side_h = int(H * scale)
    side_w = int(W * scale)
    # Fine-grained static saliency map, scaled to 0-255 uint8
    # [code: SaliencyMix_CIFAR/saliencymix.py#L202-L204]
    saliency = static_saliency(image) * 255
    # Pixel with the largest saliency value
    # [code: SaliencyMix_CIFAR/saliencymix.py#L206-L208]
    row, col = argmax(saliency)
    # Centre the box on the peak and clip it to the image
    # [code: SaliencyMix_CIFAR/saliencymix.py#L210-L213]
    return clip_box(row, col, side_h, side_w, H, W)

Pitfalls

• Patch size is not formalised. The paper says the patch covers a share of the image set by λ, but gives no side-length formula §3.1. The code uses side sqrt(1 − λ), so area fraction 1 − λ before clipping [code: SaliencyMix_CIFAR/saliencymix.py#L195-L197]. An implementation should follow the code, and it should say so.
• Label weight drifts at borders. Because the code recomputes λ from the clipped area saliencymix.py, the label weight can differ from the sampled λ when the box is cut by the image edge. Eq. 4 does not show this.
• Peak placement differs. The paper's "keep the peak inside the patch" case is not in the code. The code only clips [code: SaliencyMix_CIFAR/saliencymix.py#L210-L213]. Near the image edge, the patch is smaller than λ suggests.
• The saliency pass runs on the CPU, once per batch. The code moves the image to the CPU and to NumPy, then runs OpenCV for one image per mixed batch saliencymix.py. The paper reports this as a small time cost §4.5 tab. 5. The time was measured on CIFAR-10 with ResNet-18 only. Each mixed step pays the copy and the detector call.
• OpenCV contrib is required. The README asks for opencv-contrib-python 4.2.0.32 README.md. The saliency module is in the contrib package. The paper does not name the OpenCV version.
• Saliency input and scale are unstated. The paper does not say whether the detector sees raw pixels or normalised values. The code gives the normalised tensor saliencymix.py. Results may depend on this, and the paper does not test it.
• Application probability is unstated. The paper gives no probability. The CIFAR code uses 0.5 and the ImageNet README 1.0 README.md. The two settings give different augmentation rates.
• Mixing is per batch in code, per pair in the paper. A batch either gets SaliencyMix on every sample or on none, and every sample in the batch shares one box and one saliency peak (code-level pseudocode above).
• Batch size for CIFAR is inconsistent. The paper says 256 §4.1.1, the code and README use 128 saliencymix.py.
• ImageNet numbers need the full ImageNet-1k pipeline. The released ImageNet script trains on a 200-class tiny-ImageNet path [code: SaliencyMix-ImageNet/train.py#L79-L80] train.py. The 21.26% and 20.09% results are on 1000 classes §4.1.2. Matching them needs more than this script.
• ImageNet numbers are best runs, not means. The paper reports "the best performance of each method" for ImageNet §4.1.2 tab. 2, while CIFAR numbers are means over five runs tab. 1. Do not compare them as if they were the same kind of number.
• Text claims do not match the tables. The paper says SaliencyMix gives "the best known top-1 error" of 2.76% on CIFAR-10 §1 §4.1.1. Table 1 shows AutoAugment at 2.60% on WideResNet-28-10 CIFAR-10+ tab. 1. The text says it "outperforms all other methods except PuzzleMix" §4.1.1.
• ImageNet ranking claim. The text says SaliencyMix "outperforms all other methods in comparison" §4.1.2. Table 2 shows PuzzleMix better on both top-1 (21.24 vs 21.26) and top-5 (5.71 vs 5.76) for ResNet-50 tab. 2.
• Rounded differences. The text gives top-1 reductions of 1.66, 1.31 and 0.14 for ResNet-50 over Cutout, Mixup and CutMix §4.1.2. From Table 2 the differences are 1.67, 1.32 and 0.14 tab. 2.
• Typo in §3.3. The text says the comparison is on "CIFAR10 and ImageNet", while the experiment uses Tiny-ImageNet §3.3 fig. 3.
• Attack strength not stated. The FGSM ε is not given, so the robustness numbers tab. 4 cannot be reproduced from the paper alone.
• Detection recipe is only partly stated. The detection section says to follow the original Faster R-CNN fine-tuning recipe §4.2. Optimiser, momentum, input size and anchor settings are not stated in the paper.

Lineage

• Builds on: CutMix (random patch cut and paste with label mixing), cited as [cutmix] in §2.1. Also Cutout and MixUp §2.1. No wiki card for these exists yet, so they are not linked.
• Compared against: CutMix, Cutout, MixUp, AutoAugment, PuzzleMix, Stochastic Depth, Manifold Mixup, DropBlock tab. 2 tab. 1.
• Related: Saliency detection, specifically the OpenCV method cited as [OpenCV_VS] §3.3. The text also mentions PuzzleMix as an optimisation-based mixing method §2.1.
• Superseded by: not known from content already in this wiki.