Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation

In plain words

Teaching a model to outline every object in a photo (instance segmentation) needs pixel-accurate masks drawn by people, which is slow and expensive. So you want to squeeze as much training signal as possible out of every labelled image.

The trick here is almost embarrassingly simple: take two labelled training photos, cut some of the outlined objects out of one, and stick them onto the other, like a collage. The labels come for free, because you already know exactly which pixels you pasted. Objects that end up completely hidden behind pasted ones are dropped from the labels, and partly hidden ones get their outlines trimmed.

Earlier versions of this idea tried hard to paste objects only where they would plausibly appear. This paper finds that random pasting works just as well, even if it produces odd scenes such as giraffes next to footballers at mismatched sizes. Combined with aggressive random zooming in and out of the images, it gives consistent gains across model sizes, and the biggest gains when labelled data is scarce or object classes are rare. It also costs nothing at inference time, since it only changes the training images.

TL;DR

Copy-Paste picks two training images, applies independent random scale jittering and horizontal flips to each, pastes a random subset of instances from one onto the other with a (optionally Gaussian-smoothed) alpha mask, and updates masks and boxes of occluded objects §3. Paste locations and choices are random; no visual context is modelled §3. It stacks with large scale jittering (resize range 0.1 to 2.0) and with self-training on pseudo-labelled images §4.2 §4.4. Author-reported: +0.5 to +1.1 mask AP over strong LSJ baselines on COCO across 7 models tab. 1, 57.3 box AP / 49.1 mask AP on COCO test-dev with self-training tab. 4, and large gains on LVIS rare classes tab. 7 tab. 8. Relevant if you train Mask R-CNN-style models and have instance masks; it needs masks, not only boxes.

Problem

Instance segmentation models are data-hungry while mask annotation is costly; the paper cites 22 worker hours per 1000 COCO instance masks §1. Standard augmentations (scale jitter, random resize) are generic and not object-aware §1. Prior Copy-Paste methods (Contextual Copy-Paste, InstaBoost) model the surrounding context to decide where to paste, adding complexity §1 §2. The paper asks whether purely random pasting is enough on strong, long-trained baselines §1.

Method

Copy-Paste builds each training example from two random training images. Both are augmented independently with scale jittering and horizontal flipping; then a random subset of ground-truth instances from one (the "pasted" image) is composited onto the other (the "main" image), and the main image's annotations are corrected for occlusion §3 fig. 2.

Algorithm: Simple Copy-Paste  [§3] [fig. 2] [app. A]
Input: labelled dataset D of images with instance masks, boxes, classes
1. Sample two images (I_main, I_paste) at random from D.
2. Apply random scale jittering + random horizontal flip to each image
   independently (with its masks). LSJ on both is the default [§4.1] [tab. 11].
3. Choose a random subset of the instances in I_paste
   (better than one object or all objects [tab. 10]).
4. Build a binary mask alpha = union of the chosen instances' GT masks.
   Optionally smooth alpha with a Gaussian filter (no measurable effect [tab. 10]).
5. Composite: I_new = I_paste * alpha + I_main * (1 - alpha).
6. Annotations:
   - add the chosen instances (masks, boxes, classes) from I_paste;
   - for each original instance of I_main, subtract alpha from its mask;
     drop it if the mask becomes empty (fully occluded), otherwise
     recompute its box from the remaining mask (partially occluded).
Output: I_new with updated annotations

Compositing rule, with \alpha the binary mask of pasted objects, I_1 the pasted image and I_2 the main image (unnumbered in the paper) §3:

I_{\text{new}} = I_1 \times \alpha + I_2 \times (1 - \alpha)

The paper states that the pasted objects' masks and boxes are taken from ground truth and that occluded objects' masks and boxes are updated §3; the exact mask subtraction and box recomputation in step 6 is our reading of that one sentence and is not spelled out further.

Scale jittering §3 fig. 3. Both variants resize and crop the image and use horizontal flips. Standard scale jittering (SSJ) resizes by a factor in 0.8 to 1.25 of the original size; large scale jittering (LSJ) uses 0.1 to 2.0. When the image is made smaller than the output size it is padded with gray pixels fig. 3. LSJ is used for all models unless stated otherwise §4.1.

Self-training Copy-Paste §3 §4.4:

1. Train a supervised teacher with Copy-Paste on labelled data.
2. Generate pseudo labels on unlabelled images.
3. Train a student on labelled + pseudo-labelled images, pasting ground-truth COCO instances into both labelled and pseudo-labelled images. Half of each batch is labelled COCO, half is pseudo-labelled data §4.4.

Pasting pseudo-labelled objects from the unlabelled set into COCO images gave no extra AP §4.4.

LVIS two-stage training §4.7. Stage 1 is the single-stage recipe. Stage 2 fine-tunes only the final classification layer of Mask R-CNN with the classification loss only, using Class-Balanced Loss with per-class weight

w_c = \frac{1-\beta}{1-\beta^{n_c}}, \quad \beta = 0.999

where n_c is the number of instances of class c; weights are divided by their mean and clipped to [0.01, 5] §4.7, footnote.

Hyperparameters

Results

Author-reported. Unless noted, COCO 2017 (118k train images) with LSJ baselines, random init, trained to convergence §4.1.

Copy-Paste across architectures and sizes on COCO val tab. 1:

Average over these 7 models: +1.3 box AP, +0.8 mask AP §4.2.

Per object size (COCO val) tab. 13:

Self-training, Eff-B7 FPN (640), COCO val tab. 2:

Where to paste during self-training tab. 3:

COCO state of the art tab. 4 († = test-time augmentation):

Self-training here uses unlabelled COCO and unlabelled Objects365 §4.5.

Data efficiency §4.3 fig. 5 fig. 1: on 10% of COCO (Eff-B7 FPN, 640), Copy-Paste adds 6.9 box AP over SSJ and 4.8 box AP over LSJ; with LSJ, 75% of COCO plus Copy-Paste roughly matches 100% of COCO without it. Copy-Paste plus LSJ versus SSJ alone is reported as up to 2x more data-efficient, with about +10 AP at 10% of data and +5 AP at full data fig. 1. mixup only helps in the low-data regime and not on top of LSJ §4.2 §4.3 fig. 5.

Training schedule and init §4.2 fig. 4: on Res-50 FPN (1024), Copy-Paste helps at 2x/3x schedules (24/36 epochs) and the gain grows with longer training; it helps with both ImageNet and random init, best with random init. The intro reports +1.0 AP at 48 epochs §1.

Transfer from COCO pre-training, Cascade / Eff-B7 NAS-FPN:

LVIS v1.0 val, single-stage, Eff-B7 FPN (640), mask AP tab. 8:

LVIS v1.0 val, two-stage, Eff-B7 FPN (640), mask AP tab. 9:

LVIS v1.0 val, two-stage models vs prior work tab. 7 († 2020 winner without TTA):

Ablations, Eff-B7 FPN (640), COCO tab. 10 tab. 11. Deltas in parentheses are relative to the full Copy-Paste row (50.0 / 43.7), not to the 48.5 / 42.7 baseline:

Other analyses: Copy-Paste raises per-class AP on all 80 COCO classes except hair drier, with the largest gains on classes with low baseline AP (average over 5 runs) app. B fig. 6. Using COCO-panoptic labels to split images into 42538 indoor and 71017 outdoor (4732 undetermined), random pairing pastes into a mismatched scene type in 46.8% of generated images app. C tab. 12:

Paper vs code

No official code. The paper does not link a repository.

Pitfalls

• Location is not sampled separately. The intro says objects are pasted "at random locations" §1, but the method only describes independent scale jitter and flip of both images followed by pasting §3; position randomness comes from the jitter/crop of the pasted image. No separate translation, rotation or other geometric transform is applied §2.
• Unstated knobs. The probability of applying Copy-Paste per example, the distribution of the random subset size, the Gaussian sigma for blending, the LR decay factor, and pseudo-label filtering thresholds are not stated.
• Occlusion handling is described in one sentence §3. You must subtract the pasted alpha from existing masks, drop instances whose mask becomes empty, and recompute boxes from the remaining mask; a box-only pipeline cannot do this correctly. Whether nearly-occluded (tiny remaining area) instances are dropped is not stated.
• Use a binary alpha for labels. If you Gaussian-blur alpha for blending, the paper does not say which mask is used for updating annotations; blending made no difference to AP, so simply skipping it is safe tab. 10.
• LSJ on the main image matters most. SSJ on the main image costs about 2 box AP regardless of the pasted image's jitter tab. 11. Copy-Paste on an SSJ pipeline is not the paper's headline setup.
• Long training hides in the headline numbers. LSJ + Copy-Paste from scratch needs 576 epochs to converge §4.1; at 2x/3x schedules gains are smaller fig. 4. Batch size 256 with sync BN on accelerators is assumed.
• Delta columns change reference. In the ablation tables, red deltas are relative to the full Copy-Paste model, not the baseline tab. 10 tab. 11. In tab. 4, the self-training Copy-Paste deltas (+2.5 / +2.2) are against the no-Copy-Paste baseline, although the text calls them a "further improvement" §4.5.
• Two LVIS headline numbers come from different tables. "+6.1 mask AP_r" §1 is the Eff-B7 NAS-FPN (1280) two-stage model tab. 7, while the single-stage gain is +8.7 AP_r with RFS tab. 8. RFS helps single-stage but hurts two-stage training §4.7. Cascade was not used for the LVIS SOTA model because it hurt AP_r §4.7, footnote.
• Dataset size inconsistencies. COCO train is given as 118k §4.1 but "120k" in the self-training section, and unlabelled COCO as 120k §4.1 but "110k" in the pseudo-labelled pool §4.4.
• Copying from pseudo-labelled images does not help. Paste ground-truth instances into pseudo-labelled backgrounds, not pseudo-labelled instances into labelled images §4.4.
• The largest model is not from scratch: its backbone is initialised from a self-training ImageNet checkpoint §4.1, so tab. 4 is not a pure random-init result.

Lineage

• Builds on: Cut, Paste and Learn (Dwibedi et al., ICCV 2017), keeping its paste-and-blend idea but pasting between images that already contain instances, without geometric transforms, on COCO/LVIS §2; Contextual Copy-Paste (Dvornik et al., ECCV 2018) and InstaBoost (Fang et al., ICCV 2019), dropping their context modelling §2. Detector: Mask R-CNN, Cascade R-CNN, EfficientNet, NAS-FPN §4.1.
• Compared against: mixup fig. 5; Repeat Factor Sampling and Class-Balanced Loss on LVIS tab. 8 tab. 9; SpineNet, DetectoRS, EfficientDet, YOLOv4 on COCO tab. 4; cRT and the LVIS 2020 winner tab. 7.
• Self-training recipe is similar to Noisy Student (Xie et al., CVPR 2020) and Rethinking Pre-training and Self-training (Zoph et al., NeurIPS 2020) §4.4.