Cut and Learn for Unsupervised Object Detection and Instance Segmentation

In plain words

Object detectors usually learn from boxes and masks that people drew by hand, and that labelling is slow and expensive. This paper asks whether a model that has learned from unlabelled pictures already knows where objects are.

The answer it proposes is yes, with a little help. A self-supervised image model groups the parts of a picture that belong together. The method uses those groups to cut each picture into a few rough object masks, then trains a detector on the masks. The detector is told not to be punished for finding objects the rough masks missed. After that it is retrained a few times on its own more confident guesses. All of this uses ImageNet pictures and no labels. The result is a detector that works on other kinds of images, such as video frames, paintings and sketches, without training on them.

TL;DR

• CutLER trains an unsupervised detector and instance segmenter on ImageNet images only, with no human labels §1 §3.5.
• Its three parts are MaskCut (rough multi-object masks), DropLoss (a loss that ignores predictions that do not overlap those masks), and multi-round self-training §3.
• Author-reported: zero-shot box AP50 averages 24.3 over 11 datasets, against 9.0 for FreeSOLO, which uses extra unlabelled in-domain data tab. 2.

Problem

Localisation (boxes, masks) needs annotations that are costly to collect. The introduction gives annotating about 164K COCO images with masks for 80 classes as more than 28K human hours §1. Earlier unsupervised methods either find one salient object per image (DINO, LOST, TokenCut), or they need in-domain unlabelled data or a specific detector architecture (FreeSOLO, DETReg, MaskDistill) §1 §2 tab. 1. The method assumes that self-supervised ViT features, such as DINO's, already group patches into objects §1 §3.

Method

The method has three stages §3.

1. MaskCut: several rough masks per image §3.2.

• Build a fully connected graph over the image's patches. The edge weight is the cosine similarity of the last-attention-layer key features of a DINO ViT-B/8 model §3.1 §3.5.
• Solve the generalised eigenproblem for the Normalized Cut and take the eigenvector of the second-smallest eigenvalue eq. 1.
• Threshold that eigenvector at its mean to get a binary mask eq. 2. Decide which side is foreground with two checks: the foreground should hold the patch with the largest absolute eigenvector value, and it should hold fewer than two of the four image corners §3.2.
• Zero out the foreground patches, recompute the similarity eq. 3, and repeat. The default is three stages §3.2 §3.5.
• Post-process the masks with a CRF and compute their boxes §3.5.

2. DropLoss: do not punish misses §3.3.

• Train a standard detector on the MaskCut masks as pseudo ground truth, but for each predicted region compute its maximum IoU with any pseudo ground-truth mask §3.3.
• Keep the detector's vanilla loss for that region only when that maximum IoU exceeds τ_IoU = 0.01 eq. 4 §3.5. Regions that overlap no pseudo mask get no loss, so the detector is not pushed to call them background §3.3.

3. Multi-round self-training §3.4.

• After a round, keep the predicted masks and boxes whose confidence is above a round-dependent score, and use them as new pseudo labels §3.4.
• Remove pseudo labels that overlap a kept prediction with IoU above 0.5 §3.4.
• Initialise the next round from the previous round's weights and train without DropLoss §3.5. Three rounds are the default, and the ablation shows gains that level off after three §3.4 tab. 9.

The detector is agnostic to the architecture. The paper uses Mask R-CNN and Cascade Mask R-CNN, with Cascade the default §3.5. It reports that ViTDet with a ViT-B backbone scores higher on COCO tab. 10. For the ResNet-50 backbone the weights are initialised from a self-supervised DINO model, and MoCo-v2, SwAV and CLD gave similar detection results §3.5.

Pseudocode for the three stages, in the paper's notation:

# MaskCut: up to n rough object masks per image  [§3.2] [eq. 1-3]
# Input:  key features of the DINO ViT, shape (N, d), N = patches
# Output: n binary masks, each (N,)

def maskcut(keys, n_masks, tau_ncut):
    # Nothing is masked out before the first stage  [§3.2]
    remaining = ones(N)
    masks = []
    for stage in range(n_masks):
        # Cosine similarity; keys of earlier foreground patches are zeroed,
        # norms come from the unmasked keys  [eq. 3] (reading; see Pitfalls)
        masked_keys = keys * remaining[:, None]
        W = (masked_keys @ masked_keys.T) / outer(norm(keys), norm(keys))
        # Sparsify: below tau_ncut -> 1e-5, otherwise 1  [§3.2]
        W = where(W < tau_ncut, 1e-5, 1.0)
        # Second-smallest eigenvector of the normalised Laplacian  [eq. 1]
        x = second_smallest_eigenvector(W)
        # Binary mask: patches at or above the mean of x  [eq. 2]
        mask = x >= mean(x)
        # Flip the foreground side if the two checks fail  [§3.2]
        # (the wording of the flip rule is ambiguous; see Pitfalls)
        if flip_needed(mask, x):
            mask = not mask
        masks.append(mask)
        remaining = remaining * (not mask)
    return masks

# DropLoss for one training image  [§3.3] [eq. 4]
# Input:  predicted regions (P), pseudo ground-truth masks (G)
# tau_iou = 0.01  [§3.5]

def drop_loss(regions, gt_masks, tau_iou):
    # Largest IoU of each region with any pseudo ground truth  [§3.3]
    iou_max = max_iou(regions, gt_masks)       # (P,)
    # Keep the vanilla loss only where the overlap exceeds tau  [eq. 4]
    keep = iou_max > tau_iou                   # (P,) boolean
    # Sum vs mean is not stated; which loss terms are masked is not stated
    return sum(keep * vanilla_loss(regions))

# Multi-round self-training  [§3.4] [§3.5]
# Input: unlabelled images, pseudo labels from MaskCut (round 0)

def self_train(images, pseudo_labels, n_rounds):
    detector = init_from_dino()                # [§3.5]
    for t in range(n_rounds):
        # Train with DropLoss only in round 0; no DropLoss after  [§3.5]
        detector = train(detector, images, pseudo_labels)
        # Score cut as printed: 0.75 - 0.5 t  [§3.4] (see Pitfalls)
        score_cut = 0.75 - 0.5 * t
        kept = predictions_above(detector, images, score_cut)
        # Drop pseudo labels that overlap a kept prediction, IoU > 0.5  [§3.4]
        pseudo_labels = concat(kept, drop_overlap(pseudo_labels, kept, 0.5))
    return detector

Hyperparameters

Results

Author-reported. Zero-shot means trained on ImageNet only and evaluated on the target dataset without finetuning §4.1. The metric is class-agnostic AP50 for boxes and masks, averaged over 11 datasets for Table 2 tab. 2. Benchmark coverage is listed in tab. 12: among the 11 datasets are UVO (video frames), Watercolor (paintings), Comic (sketches) and Clipart (clip arts) app. A.2 tab. 12. The baselines are FreeSOLO results reproduced with its official code and checkpoint, marked * in the tables tab. 3 tab. 4 tab. 5.

Ablation, one component at a time, on COCO and UVO (AP50 mask) tab. 6:

Other ablation numbers from the paper:

• TokenCut with one eigenvector reaches 4.9 AP50 mask on COCO, against 18.9 for the full method tab. 7.
• Self-training rounds on COCO AP50 mask: 17.7 (1 round), 18.5 (2), 18.9 (3), 18.9 (4) tab. 9.
• Detector architecture, COCO box AP50 without self-training: Mask R-CNN 20.3, Cascade 20.8, ViTDet 21.5 tab. 10.
• Pretraining data: with DINO and CutLER both on ImageNet, COCO box AP50 is 20.8; both on YFCC1M, 19.4; mixing the two datasets gives 14.9 or 14.8 tab. 11.
• The text states that CutLER finds instances that human annotators missed on COCO §4.1.

Paper vs code

Official code exists. The title block links https://github.com/facebookresearch/CutLER. Its top-level LICENSE is CC BY-NC-SA 4.0, a non-commercial licence. Under the wikillm licensing rules, no code was read for values, quoted or cited. This card therefore has no code, code_commit or code_license field, and it does not check paper-versus-code differences. Only the LICENSE file and the directory listing were checked. The discrepancies below are inside the paper.

Pitfalls

• Eq. 2 is typeset with the wrong index. The left side reads M^t_ij, but the condition compares the eigenvector. Implement it as: patch i is foreground when x^t_i ≥ mean(x^t) eq. 2.
• Foreground selection rule is ambiguous. Read literally, the flip condition is "criterion 1 fails while the set holds two corners, or criterion 2 fails". Criterion 2 is "fewer than two corners", so its failure already means two or more. That makes criterion 1 redundant, so the intended combination is not stated §3.2.
• Eq. 3 is indexed by pairs, not patches. I read it as: multiply each patch's key by the product of (1 − mask) from earlier stages, and take the norms from the unmasked keys. The text says "masking out these nodes", which supports this reading, but the normalisation is not spelled out eq. 3 §3.2.
• Stage count is ambiguous. The text says the process repeats t times and sets t = 3 §3.2. Section 3.5 says three stages, and Fig. 3 uses n = 3. Table 8 calls it N, masks per image. Whether this yields exactly three masks, or one plus three repeats, is not stated.
• τ_ncut default is not stated. The sweep in tab. 8 shows 0.15 scoring best on AP50 mask (17.7), with 0.2 close behind (17.6). The text calls the method robust to this value.
• DropLoss prose contradicts eq. 4. One sentence says regions with overlap above τ are dropped §3.3. Eq. 4 and the next sentence keep the loss only for overlap above τ, so low-overlap regions are dropped. The ablation text matches eq. 4: a higher threshold ignores more regions §5. The equation is the one to implement.
• DropLoss scope is not stated. The paper does not say whether the drop applies to classification, box and mask losses alike, or how the summed loss is normalised.
• Self-training score cut as printed is broken. The paper gives 0.75 − 0.5t §3.4. That is 0.25 for t = 1 and negative from t = 2, which would keep every prediction. The intended schedule and the round index are not stated.
• "Proposals" in self-training is undefined. The paper says it uses "predicted masks and proposals" above the score cut, without defining proposals or saying whether the cut applies to both §3.4.
• Self-training hyperparameters are incomplete. Only learning rate 0.01 and 80K iterations are given §3.5 app. A.1. Batch size, momentum and weight decay are not stated for those rounds.
• "Decreased by 5" is ambiguous. Read as division by 5 (0.005 to 0.001) after 80K iterations, which is a plausible but unstated reading §3.5 app. A.1.
• Text and table disagree on the LVIS-supervised comparison. The text says the method's AP50 mask on UVO is 4.8% higher than fully supervised SOLOv2 trained on LVIS §4.1. Table 5 gives 22.8 against 14.8, a difference of 8.0 tab. 5.
• Text and table disagree on one ablation gain. The text gives DropLoss a COCO AP50 mask gain of 0.9 §5. Table 6 gives 16.6 − 15.8 = 0.8 tab. 6.
• Fine-tuning details are incomplete. For 50% labels and above the paper says learning rates "decay by a factor of 2", with no starting rate or schedule. The std of the random head init is not stated app. A.5.
• Copy-paste is underspecified. The downscale range is given, but not how many objects are pasted or where §3.5.
• Dataset matching matters. Training DINO and CutLER on different datasets gives clearly worse COCO results (14.9 and 14.8 against 20.8 and 19.4) tab. 11. The paper therefore recommends the same distribution for both.
• Baselines are authors' reproductions. FreeSOLO numbers are rerun with its official code and checkpoint, not copied from its paper tab. 2 tab. 3 tab. 5.
• Comparisons across papers are weak. The datasets and metrics vary, and Table 2 compares a ResNet-50 model trained on ImageNet only against a ResNet-101 FreeSOLO fine-tuned on extra unlabelled data tab. 2.
• Costs the abstract hides. MaskCut runs per image, with three eigen-decompositions on a 480 × 480 input that yields 3,600 patches (derived). Then come 160K detector iterations at batch size 16, plus three self-training rounds of 80K iterations each. Hardware and wall-clock time are not stated.

Lineage

• Builds on: DINO for self-supervised ViT features §3.1 (2104.14294). TokenCut, which applies Normalized Cuts to DINO features and finds one object per image (arXiv:2209.00383) §2 §3.1. Normalized Cuts for graph bipartition §3.1. Copy-paste augmentation and a CRF post-process §3.5.
• Compared against: FreeSOLO (SOLOv2 detector, reproduced by the authors) tab. 2 tab. 3. LOST (arXiv:2109.14279) and MaskDistill (arXiv:2206.06363), named in the related work §2 tab. 1.
• Superseded by: not stated in this wiki.