Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework

In plain words

Training an object detector normally needs every object in every training photo to be boxed and named by a person, and that labeling is slow and expensive. Semi-supervised learning tries to get by with a few labeled photos plus many unlabeled ones.

Instant-Teaching lets the detector label the unlabeled photos for itself while it trains. At each step it guesses boxes on a lightly altered copy of an unlabeled photo, keeps only the confident guesses, and then trains on a heavily altered copy of the same photo to reproduce them. The guesses come from the model that is currently being trained, so they get better as training goes on, and no separate teacher model has to be trained first. An optional second model checks the first model's guesses, which the authors say reduces the effect of confident mistakes being reinforced.

The approach matters because it reports gains over the previous best semi-supervised detector on COCO and Pascal VOC, mainly when labels are scarce. These are the authors' own numbers.

TL;DR

• Semi-supervised object detection with a Faster R-CNN (ResNet-50 + FPN) §4.1.
• Every iteration: the current model pseudo-labels a weakly augmented unlabeled batch; the same images, strongly augmented, are trained against those labels §3.3 eq. 1.
• Strong augmentations are extended with Mixup and Mosaic on unlabeled images §3.3 §5.2.
• Optional co-rectify: two identically structured models with different initialisations refine each other's pseudo labels; only one model is used at inference §3.3 eq. 5.
• Author-reported: +4.2 mAP over STAC at 2% COCO labels (22.45 vs 18.25); about +1.0 mAP at 100% COCO; +5.36 mAP over STAC on VOC07 with VOC12 unlabeled tab. 1 tab. 2.

Problem

Supervised detectors need many accurately boxed and labeled images, which is costly §1. Semi-supervised object detection (SSOD) trains on a labeled set D_l and an unlabeled set D_u §3.1. The baseline the paper argues against is STAC, which first trains a teacher, then pre-generates pseudo labels that are never updated during student training. The authors identify two issues: the procedure is complicated and inefficient, and the fixed pseudo labels limit accuracy §1 §3.3.

Assumptions: the detector is Faster R-CNN with FPN, and the paper's experiments use COCO and Pascal VOC only §4.

Method

Instant-Teaching (Instant-Teaching*, when co-rectify is added) is a single training loop with two parts: instant pseudo labeling with weak-strong augmentation, and a co-rectify step §3.2.

Instant pseudo labeling §3.3. Each iteration has two steps:

1. The current model labels an unlabeled mini-batch on a weakly augmented view. The paper uses random flip as the only weak augmentation in all experiments unless stated otherwise §3.3. Non-maximum suppression and a confidence filter with threshold τ are always applied in this step; τ = 0.9 in all experiments §3.3.
2. The same unlabeled images get strong augmentations and the pseudo labels from step 1 are used as ground truth. The model is updated with a total loss: a supervised loss on labeled images and an unsupervised loss on strong views of unlabeled images §3.3 eq. 1. Labeled images keep the weak augmentation §3.3.

The total loss is ℓ = ℓ_s + λ_u ℓ_u eq. 1.

Supervised loss eq. 2: cross-entropy classification plus L1 box regression over anchors, summed over labeled images l. The regression term is weighted by λ eq. 2 §3.3.

Unsupervised loss eq. 3: the class probabilities and boxes are first predicted on the weakly augmented unlabeled image; the hard label ĉ = argmax of the class distribution is used as the target for classification on the strongly augmented image. The box regression term is only applied where the max class score is at least τ eq. 3 §3.3.

Weak-strong augmentation §3.3. Strong augmentations follow STAC: colour jitter, geometric transforms and Cutout (the "Color+Cutout" set in tab. 3). The paper adds Mixup eq. 4 and Mosaic fig. 2. Geometric transforms of pseudo boxes are not used; the authors leave that to future work §5.2. Mixup and Mosaic are applied only to unlabeled data §3.3 §5.2.

Mixup eq. 4, with α_m = 1.0 tab. 6: a λ_m is drawn from Beta(α_m, α_m). The unlabeled image is replaced by a weighted blend with a randomly chosen labeled image from the batch. Its class labels are blended with weight λ_m, and its boxes are the union of the two images' boxes eq. 4 §3.3.

Mosaic fig. 2: an unlabeled and a labeled image are combined in one of two random layouts, horizontal or vertical, and annotations are combined to match §3.3.

Co-rectify §3.3 eq. 5. Two models f_a (Model-a) and f_b (Model-b) with the same structure are trained at the same time. They differ in initial parameters, and in their data augmentations and pseudo labels within a batch §3.3. For Model-a's targets: Model-a predicts class probabilities c and boxes t on the weakly augmented image; Model-b's detection head refines them, taking t as proposals, giving c^r and t^r. The rectified class scores are the average of c and c^r, and the rectified boxes are a weighted average of t and t^r eq. 5 §3.3. Model-b's rectified targets are built in a similar way §3.3. At inference only Model-a is used, so inference time is unchanged §3.2.

# Instant-Teaching: one training iteration  [§3.3]
# Input:  labeled batch (images, boxes, classes); unlabeled batch
# Output: scalar loss  l = l_s + lambda_u * l_u  [eq. 1]

def training_step(labeled_batch, unlabeled_batch, model):
    # Pseudo-label unlabeled images with the current model on a
    # weakly augmented view (random flip)  [§3.3]
    weak_views = random_flip(unlabeled_batch)
    with no_gradient():
        class_probs, boxes = model.predict(weak_views)
    # Hard label = argmax class; keep boxes after NMS whose max
    # class score is at least tau  [§3.3] [eq. 3]
    pseudo_labels = filter_nms_by_score(class_probs, boxes, tau)
    # Strong views of the same images: Color+Cutout, then Mixup and
    # Mosaic on unlabeled images only  [§3.3] [eq. 4] [§5.2]
    strong_views, strong_labels = strong_augment(
        unlabeled_batch, pseudo_labels, labeled_batch)
    # Supervised loss on weakly augmented labeled images  [eq. 2]
    loss_sup = detection_loss(model, weak_aug(labeled_batch),
                              labeled_batch.targets, reg_weight=lam)
    # Unsupervised loss: classify strong views toward the pseudo
    # labels; regress only confident boxes  [eq. 3]
    loss_uns = detection_loss(model, strong_views, strong_labels,
                              reg_weight=lam, reg_mask=max_score >= tau)
    # Total loss, then one optimizer update  [eq. 1]
    loss = loss_sup + lambda_u * loss_uns
    optimizer.step(loss)
    return loss

# Co-rectify: targets for Model-a, refined by Model-b's head  [eq. 5]
# Input:  weakly augmented unlabeled images x_u
# Output: rectified class scores and boxes

def corrected_targets(model_a, model_b, x_u):
    # Model-a predicts class probabilities and boxes  [eq. 5]
    probs_a, boxes_a = model_a.predict(x_u)
    # Model-b's detection head refines them, using boxes_a as proposals  [eq. 5]
    probs_b, boxes_b = model_b.refine_head(x_u, proposals=boxes_a)
    # Class scores: plain average  [eq. 5]
    probs_star = 0.5 * (probs_a + probs_b)
    # Boxes: weighted average; weights not fully specified  [eq. 5]
    boxes_star = weighted_average(boxes_a, boxes_b, probs_a, probs_b)
    return probs_star, boxes_star

Hyperparameters

Values are as stated in the paper. "Not stated" means the paper does not give the value.

Full hyperparameter table: tab. 6 in app. A.2.

Results

Author-reported. Setup: Faster R-CNN with FPN and ResNet-50, MMDetection, 8 GPUs, ImageNet init §4.1. Baselines: supervised training on the same labeled data, STAC, and CSD (re-implemented by the authors, †) tab. 1. Values are mean ± std over runs where given.

MS-COCO, mAP tab. 1 (brackets are the paper's own gains over the supervised model)

Gains over supervised, as printed: Instant-Teaching* +9.00, +9.75, +8.28, +6.54, +2.57; STAC +4.92, +5.55, +5.91, +4.78, +1.58 tab. 1. The text gives the gains over STAC: +4.08 (1%), +4.2 (2%), 24.38 → 26.75 (5%), 28.64 → 30.40 (10%), and about +1.0 mAP at 100% §4.2.

Pascal VOC, AP (IoU) on VOC07 test tab. 2. Labeled: VOC07 trainval.

The paper reports gains of +5.10 / +6.40 over supervised for Instant-Teaching and Instant-Teaching* (AP 0.5:0.95, VOC12) tab. 2. The text gives 44.64 → 50.00 (+5.36) against STAC, and 46.01 → 50.80 for the COCO-subset setting §4.2.

Ablation: augmentation, 5% COCO labels, 8× unlabeled tab. 3. Strong augmentation sets; mAP.

* Color+Cutout is also applied to unlabeled images in the pseudo-labeling step tab. 3.

Unlabeled-set size, MS-COCO tab. 4. The "Full" column is not defined in the paper.

Confidence threshold τ, 10% COCO, λ_u = 1.0 tab. 5: τ = 0.3 → 26.30; 0.5 → 27.70; 0.7 → 28.70; 0.9 → 29.80 mAP.

λ_u sweep fig. 5: described only in a figure (best at 1.0; small drop at 1/4 to 4). No table.

Backbones, 2% COCO tab. 7 (appendix A.4): R50 supervised 12.70, Instant-Teaching 20.70, Instant-Teaching* 22.45; R101 15.80 / 22.10 / 23.50; X101-32x4d 16.60 / 22.40 / 24.20; R2N-101 17.4 / 23.5 / 25.9. Brackets give the paper's gains.

Pseudo-label quality fig. 3, fig. 6, fig. 7: qualitative and curve-based; the paper reports that the count of model-generated boxes rises over training §5.1, and that co-rectify gives higher mAP on confident pseudo labels §5.5. Figure values are not tabulated.

Convergence fig. 8 (app. A.3): loss curves for both models; no numbers.

Paper vs code

No official code.

Searched: the paper, its arXiv metadata and its bibliography give no code or project link. A GitHub repository search (instant-teaching, and instant-teaching with org:alibaba) gave one title match, mightyzau/Instant-Teaching (Apache-2.0). Its owner is an individual who is not an author and not an Alibaba account, so it was not used. A web search was not run, because that permission was denied.

Pitfalls

• LR schedule conflict. Section 4.1 says the rate drops 10× at 120k and 165k §4.1. Appendix A.1 and Table 6 give 120k, 160k and 180k app. A.1 tab. 6. The 0.0001 stage is said to be "≤ 180k" in A.1 but Table 6 gives only the total step count. Implement one of these and note which.
• Table 4 vs Table 1. At 5% COCO, Table 4 gives 25.60 "Full" and Table 1 gives 25.50 at 5%. At 10%, Table 4 gives 29.53 "Full" and Table 1 gives 29.45. Table 5 gives 29.80 at τ = 0.9, 10%. These are different settings or runs, and the paper does not say which is the main result.
• "Full" unlabeled set. Table 4 gives no definition for "Full". Read it as the full unlabeled split only with care; the 8× column has the same number as the ablation table tab. 3.
• Which schedule for ablations. Section 4.1 uses the quick schedule for 1–10% and standard for 100% §4.1. Ablations at 5% and 10% do not say which schedule they use; assume quick only if you need to match a table exactly.
• Pseudo-label filtering in eq. 3. Eq. 3 applies the τ mask to the regression term only, while the classification sum runs over all anchors. Section 3.3 says NMS and confidence filtering are applied with τ in the pseudo-labeling step §3.3. The paper does not say whether low-confidence anchors are excluded from the classification term eq. 3. Check the code before you implement it.
• Mixup of boxes and labels. Eq. 4 uses "∪" for both soft labels and boxes. Text calls the class labels "soft" and the boxes "pseudo annotations" §3.3. The paper does not say how boxes from the two images are clipped or dropped after blending, nor how the weights combine when a box is only in one image.
• Co-rectify weights. Eq. 5 writes the box weight as 1/(c_i + c_i^r) × (t_i c_i + t_i^r c_i^r). It does not say whether c_i is the max class score, the per-class vector, or the probability of the target class. The denominator reads as a scalar, but c is a vector. Not stated.
• Co-rectify targets. The text says Model-b's rectified targets are built "in a similar way" §3.3, but it does not say which target is used to train which model. Not stated.
• Co-rectify cost. Training runs two detectors and two heads on each batch. The paper reports inference cost only for the single model used §3.2. Training time and memory are not given.
• Extra forward pass. Each iteration runs the current model on every unlabeled image to make pseudo labels, before the training pass §3.3. The paper does not report the time or memory cost of this pass.
• Dataset numbers. The VOC unlabeled sets (~11k VOC12; ~95k COCO subset) are from the text §4.1. The exact image lists are not stated.
• Reported gains are over the paper's own reimplementations. CSD is the authors' implementation on Faster R-CNN tab. 1. Table 2 cites CSD with an R101-R-FCN backbone, a different detector, so its numbers are not directly comparable. Whether they were copied from the CSD paper is not stated.
• Schedules come from STAC. The quick and standard schedules are STAC's; the paper points to STAC's appendix for their full details §4.1 app. A.1.
• Confidence threshold at inference. Figure 6 uses a score threshold of 0.9 for evaluation of pseudo-label quality §5.5. The COCO evaluation uses score threshold 0.001 app. A.1. The two are not the same thing; do not mix them.
• Labeled and unlabeled counts per batch. A batch of 16 with a 1:1 ratio implies 8 and 8. The paper does not state it in those words §4.1.

Lineage

• Builds on: STAC, "A Simple Semi-Supervised Learning Framework for Object Detection" (Sohn et al.), arXiv:2005.04757 §1. Instant-Teaching keeps STAC's weak-strong augmentation scheme and its strong augmentations, and extends them with Mixup and Mosaic. It replaces STAC's pre-generated pseudo labels with labels produced at each iteration §3.3. There is no card for arXiv:2005.04757 in this wiki, so nothing from it is used here.
• Compared against: CSD (Jeong et al., NeurIPS 2019) tab. 1 tab. 2; the supervised baseline tab. 1; STAC tab. 1 tab. 2.
• Related techniques named in the paper: FixMatch and weak-strong augmentation in semi-supervised classification §3.3; confirmation bias §3.3; Mixup [zhang2017mixup] and Mosaic from YOLOv4 §3.3.
• Superseded by: not stated; no later work is cited in the paper.