Self-training

Overview

Self-training uses a model's own predictions as labels. A teacher produces labels for inputs that lack them, and a student is trained on the labels it keeps 2103.00550 §6.2. The shared assumption is that high-confidence predictions tend to be correct, for instance when classes form well-separated clusters. If the data break that assumption, the method can fail to help or even reduce accuracy 2103.00550 §2.1.

The unlabelled inputs need not be fresh images from the same source. Two variants below label what the model has not labelled before: unlabelled target-domain pixels in domain adaptation 1909.13589 §3.4, and background pixels that may hide old classes in continual segmentation 2011.11390 §3.3. A stream of unlabelled test images, with no source data at adaptation time, is another such input 2204.10377 §1.

Med-Gemini uses the same word for a different loop. It writes reasoning chains for labelled questions and keeps those that reach the gold answer, so its labels are not pseudo-labels on unlabelled inputs 2404.18416 §2.1. Its variant is listed below, separately.

Pseudo-labelling is one branch of semi-supervised learning. The survey files self-training under pseudo-labelling and separates it from disagreement-based methods such as co-training, which take pseudo-labels from two views that agree 2103.00550 §6.1.

Variants

Confidence-thresholded pseudo-labels

The teacher's most probable class becomes the target for each unlabelled sample, and the loss on those targets is added to the supervised loss with a weight that changes over training 2103.00550 §6.2. The survey gives no schedule for that weight and no confidence threshold. The fixed-threshold baselines in the FlexMatch comparison are reported only through that paper's table, not through a card here 2110.08263 tab. 1.

Class-wise adaptive thresholds (FlexMatch)

The fixed cut-off τ is replaced by one threshold per class. Each class's threshold is τ scaled by how many unlabelled samples are currently marked with that class, so classes the model still struggles with admit more pseudo-labels 2110.08263 §3.1. The scale is β(c) = σ(c) / max σ, where σ(c) counts the marked samples, and the mapping M(β) = β / (2 − β) gives the threshold T(c) = M(β(c)) · τ 2110.08263 eq. 11–12. The method adds no parameters and no forward passes; the marks are stored state, updated from the weak-view predictions 2110.08263 §3.1 alg. 1. A warm-up rule keeps the denominator at least the number of unused samples early in training 2110.08263 §3.2. The paper uses τ = 0.95 on the CIFAR and STL-10 experiments 2110.08263 §4 tab. 3.

The author-reported error on CIFAR-100 with 400 labels is 39.94%, against 46.42% for FixMatch with the fixed cut-off. On STL-10 with 40 labels it is 29.15% against 35.97% 2110.08263 tab. 1. The same tables show FixMatch as equal or better in some cells: 4.86% against 4.98% on CIFAR-10 with 250 labels, and 3.81% against 8.19% on SVHN with 40 labels. The authors attribute the SVHN result to class imbalance, where rare classes do not reach β near 1 and keep low thresholds 2110.08263 §4.1. Card: FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling.

Self-adaptive threshold with a fairness term (FreeMatch)

The global threshold is an exponential moving average (EMA) of the mean top weak-view confidence on unlabelled samples, starting at 1/C [arxiv:2205.07246 eq. 5]. Each class's threshold is that global value times the class's own EMA of predicted probability, divided by the largest such EMA, so the most-predicted class gets the full global threshold 2205.07246 eq. 6–7. A sample is kept if its top probability clears the threshold of its predicted class 2205.07246 eq. 8. Where FlexMatch scales by counts of marked samples, this variant scales by a running average of probabilities 2110.08263 §3.1 2205.07246 eq. 6–7. A fairness loss, with weight w_f, pushes the normalised distribution of kept strong-view predictions towards the normalised weak-view class statistics 2205.07246 eq. 9–12. The paper sets the unlabelled loss weight to 1 in all experiments and gives w_f per setting 2205.07246 app. D, tab. 5. The authors describe the extra cost over FixMatch as very small 2205.07246 app. C. The released configs use a fairness weight ten times below the appendix values for the same settings, so check the value against the card before reproducing Card.

The author-reported error on CIFAR-10 with 10 labels is 8.07%, against 13.85% for FlexMatch. On STL-10 with 40 labels it is 15.56% against 29.15% 2205.07246 tab. 1. The gap is small at 40 labels on CIFAR-10, 4.90% against 4.97% tab. 1. Removing the fairness term raises the CIFAR-10 10-label error to 10.37% 2205.07246 tab. 4. The paper reports the best error over all checkpoints, not the final one 2205.07246 §5.1. Card: FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning.

Noisy Student

An equal-or-larger student is trained on labelled data and on pseudo-labels from the teacher, with noise added to the student through RandAugment, dropout and stochastic depth. The student then becomes the next teacher, and the cycle repeats for a few rounds 2103.00550 §6.2. The survey describes the recipe but reports no accuracy numbers for it.

Noisy student for speech recognition (wav2vec-style encoder)

A wav2vec 2.0-style encoder is pre-trained on unlabelled audio. It is then fine-tuned with CTC as the first teacher, on 12.5 h of cleaned labelled speech 2609.24410 §8.2 §9. Each generation pseudo-labels one batch of unlabelled audio with the current teacher. The audio is augmented with Gaussian noise, time stretch and pitch shift, and a larger student is trained on the pseudo-labelled, clean and augmented data. The student becomes the next teacher 2609.24410 §6.2 §8.2 fig. 8. The paper runs five generations tab. 4. It states no pseudo-label filter, no augmentation magnitudes and no rule for accepting a student 2609.24410 §7.3 §6.2 §9. It is not stated whether each student starts from scratch §8.2 fig. 7.

The author-reported WER on their own evaluation set falls from 0.568 for the generation-zero teacher to 0.515 at generation five 2609.24410 tab. 4. The paper runs once, with no seeds or intervals. The gap between generations 2 and 3 is 0.003 WER, and the evaluation set size is not stated. Each generation is selected on the score it is reported on, so the gains may be optimistic 2609.24410 §9. The text's "8%" relative gain does not match the table, which gives about 9.3% 2609.24410 §9 tab. 4. CER returns to 0.279 at generation four, after 0.264 at five tab. 4. The paper gives no optimiser, learning rate, batch size or epoch count 2609.24410 §8.2. Card: End-to-end Jordanian dialect speech-to-text self-supervised learning framework.

Entropy-gated teacher-student for segmentation (U2PL)

Pixels whose teacher entropy falls in the top share are left out of the pseudo-label loss. The share starts at 20% and shrinks linearly to zero over training 2203.03884 §3.2 eq. 7. The excluded pixels are not thrown away: they are reused as negative examples in per-class queues for a pixel-level contrastive loss 2203.03884 §3.3. The teacher is an exponential moving average of the student 2203.03884 §3.1. On PASCAL VOC 2012 with ResNet-101 and DeepLabv3+, the author-reported mIoU at the 1/16 classic split is 67.98, against 45.77 for supervised-only training and 57.60 for the strongest listed baseline 2203.03884 tab. 1. Card: Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels.

Copy-Paste with self-training for detection and instance segmentation

The teacher is trained with Copy-Paste augmentation on labelled data. It then pseudo-labels unlabelled images, and a student trains on batches that are half labelled COCO and half pseudo-labelled data 2012.07177 §4.4. Ground-truth instances are pasted into the pseudo-labelled images. Pasting pseudo-labelled instances into labelled images gave no extra AP 2012.07177 §4.4. On EfficientNet-B7 FPN at 640 pixels, self-training alone gives +1.5 box AP and +1.3 mask AP; self-training combined with Copy-Paste gives +2.9 box AP and +2.3 mask AP over the baseline 2012.07177 tab. 2. Card: Ghiasi et al., Copy-Paste.

EMA teacher with focal classification for detection (Unbiased Teacher)

A Faster R-CNN with FPN and ResNet-50 is first trained on labelled boxes only, the burn-in stage 2102.09480 §3.1. The teacher is then an exponential moving average of the student: at each step it keeps 0.9996 of its own weights and takes 0.0004 from the student 2102.09480 eq. 3, tab. 5. The teacher sees a weakly augmented image (horizontal flip). Its detections go through class-wise NMS and a confidence cut-off of 0.7 2102.09480 §3.2, tab. 5. The student sees a strongly augmented copy (colour jitter, grayscale, blur, cutout) 2102.09480 tab. 6. Only the RPN and ROI classification losses use pseudo-labels; box-regression losses do not, because confidence does not measure localisation quality 2102.09480 §3.2. The ROI classifier uses focal loss in place of cross-entropy, which the authors argue limits the pull towards common classes and background 2102.09480 §3.3.

On COCO with 1% of the boxes labelled, the author-reported mAP is 20.75, against 13.97 for STAC and 9.05 for supervised-only training 2102.09480 tab. 1. In the ablation at the same label fraction, cross-entropy without the EMA teacher reaches 13.42 and focal loss with the EMA teacher 21.19 2102.09480 fig. 5. That ablation setting is 0.44 mAP above the headline 20.75 for the same label fraction, and the paper does not explain the gap. Card: Unbiased Teacher for Semi-Supervised Object Detection.

Self-produced guidance from a second head (MaxSquare, domain adaptation)

For unsupervised domain adaptation from synthetic GTA5 images to real Cityscapes images, a second low-level head on conv4 features is added 1909.13589 §3.4. For each target pixel, the guidance label is the argmax of the averaged output of the two heads. The label is kept only when either head gives that class a probability above 0.95 [arxiv:1909.13589 §3.4 eq. 15]. The second head is trained with cross-entropy on those labels, weighted by 0.1 1909.13589 §3.4 eq. 14, eq. 16.

The main change in the paper is a different target loss, the maximum squares loss 1909.13589 §3.2; the guidance head is an add-on. On top of the image-weighted loss, the head raises mIoU from 45.2 to 46.4 [arxiv:1909.13589 tab. 4]. The prose calls the gain significant, and the table shows that number. The authors report 46.4 against 42.2 for their entropy-minimisation rerun on the same split 1909.13589 tab. 2. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Old-model pseudo-labels for background pixels (PLOP, continual segmentation)

When a new step adds classes and old images are not stored, the new training labels mark old-class pixels as background. The frozen model from the previous step labels those pixels 2011.11390 §3.3. A background pixel gets the old model's argmax class only if its entropy is below a per-class threshold. The threshold is the median entropy of that class over training pixels. Other background pixels are ignored 2011.11390 §3.3 eq. 6–7. The loss on these labels is weighted by ν, the share of background pixels that kept a label 2011.11390 eq. 8.

The authors report 54.64 mIoU on VOC 2012 in the 15-1 Overlapped setting, against 29.29 for their MiB re-implementation [arxiv:2011.11390 tab. voc_sota_full]. On a 20% validation split, pseudo-labels alone raise 15-1 mIoU from 3.99 (cross-entropy only) to 19.74. Adding Local POD distillation gives 50.41, and ν weighting gives 52.31 2011.11390 tab. ablations. A pseudo-oracle that removes wrong pseudo-labels reaches 54.09 on the same split 2011.11390 tab. ablation_classif_distillation, so the headroom from better pseudo-labels is small there. Validation-split numbers are not comparable with the main tables. Card: PLOP: Learning without Forgetting for Continual Semantic Segmentation.

Soft-label self-training with masked inputs (building footprints)

The supervised model is the teacher. The student trains on unlabelled satellite tiles with the teacher's soft confidences as targets and a KL loss with focal weighting, γ = 0.25 2107.12283 §8. Random black masks are added to the student's inputs, so that it learns to handle no-data pixels 2107.12283 §8. The pool started at 100M tiles of 640 × 640 px. The supervised model kept 8.7M of them, mostly tiles without buildings 2107.12283 §3.3. The paper does not state the filter threshold or the number of iterations. It reports that fine-tuning on labels helped only in the first iteration 2107.12283 §8.

The author-reported gain is +0.057 mAP on average over the supervised teacher. The authors describe the mixup and edge-weighting effects as larger 2107.12283 §11 fig. 8. The abstract rounds the gain to +0.06, and the body value is used here. Card: Continental-Scale Building Detection from High Resolution Satellite Imagery.

Test-time adaptation with refined pseudo-labels (AdaContrast)

The unlabelled inputs are a target stream seen at adaptation time, and the source-trained classifier is updated on it without source images 2204.10377 §1. For each batch, one weak view and two strong views of each image are used 2204.10377 §3. The weak view is passed through a momentum copy of the model, and its feature is compared with the N nearest entries of a memory queue. The probabilities of those entries are averaged, and the argmax of the average is the pseudo-label [arxiv:2204.10377 §3.1 eq. 3–4]. No confidence threshold is applied: every pseudo-label supervises the strong view with cross-entropy 2204.10377 §3.3 eq. 7. Two more terms are added. An InfoNCE loss treats the momentum copy's key as positive and drops queue entries that share the current pseudo-label from the negatives 2204.10377 §3.2 eq. 5–6. A diversity term rewards a spread-out mini-batch mean prediction 2204.10377 §3.3 eq. 8–9. The three losses have weight 1 in every experiment 2204.10377 §3.3 eq. 10.

The author-reported per-class average on VisDA-C is 86.8%, against 83.0% for SHOT 2204.10377 tab. 1. On DomainNet-126 over seven shifts it is 67.8%, against 67.1% tab. 2. In the ablation, refinement without the contrastive term gives 86.5% at the base learning rate but 9.9% at ten times that rate. Adding the contrastive term gives 84.3% at the high rate, and the regularisation term gives 86.6% 2204.10377 tab. 4. Card: Contrastive Test-Time Adaptation.

Self-generated rationales filtered by gold answers (Med-Gemini)

Each labelled training question gets two reasoning chains, one written without search results and one with them. The search queries are written by the model itself 2404.18416 §2.1. Chains that end in a wrong answer are dropped before fine-tuning, so the filter needs the correct answer 2404.18416 §2.1. The fine-tuned model writes new chains, and the loop repeats until results stop improving 2404.18416 §2.1. The paper does not state the stopping rule. The author-reported gain from this self-training on MedQA is 3.2 points 2404.18416 §4.1 fig. 4. Card: Capabilities of Gemini Models in Medicine.

Re-clustering web labels in rounds (CAST, face data)

Here the "unlabelled" data are noisy labels. A web search gives each name a folder of face crops, and many folders hold faces of other people. CAST treats cleaning as self-training. A teacher embeds every face. Each folder is clustered, and only its largest cluster is kept. Folders are then merged or dropped by comparing their feature centres. A new student is trained on the cleaned data and becomes the next teacher 2103.04098 §3. The first teacher is an ArcFace ResNet-100 trained on MS1MV2. The text and the counts in the table indicate that each round starts from the same pre-processed set, not from the previous round's output 2103.04098 §3 tab. 2. No confidence threshold is used. The cut-offs are cosine thresholds on the centres: merge above 0.7, and drop the smaller folder between 0.5 and 0.7 2103.04098 §3. The DBSCAN parameters are not stated. The paper uses three rounds, and reports that a fourth gave saturated results §5.4.

The author-reported IJB-C TAR@FAR=1e-4 for ArcFace ResNet-100 trained on the cleaned WebFace42M is 97.70, against 96.03 for MS1MV2 [arxiv:2103.04098 tab. 5]. Over rounds on the web set, IJB-C goes from 97.28 after round 1 to 97.51, 97.70 and 97.66 tab. 7. On MS1M, the same procedure takes IJB-C from 93.45 to 94.31 after round 1 and 96.55 after round 3 tab. 7. That MS1M teacher was trained on IMDB-Face, not MS1MV2 §5.4. Within the same pipeline on MS1M, DBSCAN gives 96.55 on IJB-C, against 96.03 for K-Means and 96.42 to 96.48 for the two graph-based cleaners 2103.04098 tab. 8. The paper's dataset is released under a signed licence for research use, and its Table 1 calls it public §6 tab. 1. Check the licence before any reuse. Card: WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition.

Other described variants

The survey also describes Meta Pseudo Labels, where the teacher is updated from the student's loss on held-out data, and self-training by distillation as in SimCLRv2 2103.00550 §6.2. It gives no numbers for them, so treat them as unvalidated here.

In NLP data augmentation, self-training means labelling unlabelled text with a fine-tuned model. SentAugment retrieves web sentences close to task queries built from the labelled data 2110.01852 §2.3.4. This survey of text augmentation reports no experiments of its own, so the method is listed here without numbers 2110.01852 tab. 2.

When to use which

Training is slower than for supervised methods, and the paper gives no wall-clock figures 2203.03884 §5. The pseudo-labelling variants change only the weights, so inference cost is unchanged. The Med-Gemini loop also runs a search at inference for text tasks 2404.18416 §2.1.

Implementation sketch

The core loop shared by the variants. Where they differ, the comment says how.

# Generic self-training round  [arxiv:2103.00550 §6.2]
# Input: labelled (x, y), unlabelled u, teacher T trained on labelled data

def self_train_round(T, labelled, unlabelled, rounds):
    for r in range(rounds):
        # Teacher predicts class probabilities for unlabelled inputs
        probs = T.predict(unlabelled)               # (N, K)
        # Keep only the confident ones; threshold not stated in survey
        keep = probs.max(axis=1) > tau
        # AdaContrast: no threshold, every sample is kept; the label
        # comes from a vote over a feature queue  [arxiv:2204.10377 §3.1]
        # FlexMatch variant: tau per class, scaled by the class's
        # marked-sample count  [arxiv:2110.08263 eq. 11-12]
        # FreeMatch variant: EMA of confidence, scaled per class by an
        # EMA of predicted probability  [arxiv:2205.07246 eq. 5-7]
        # Segmentation variant: entropy quantile that shrinks to 0
        # over training instead of a fixed tau  [arxiv:2203.03884 eq. 7]
        targets = probs.argmax(axis=1)[keep]
        # Soft-label variant: targets = probs, KL loss with focal
        # gamma = 0.25  [arxiv:2107.12283 §8]
        # PLOP variant: only background pixels, entropy below the
        # per-class median  [arxiv:2011.11390 §3.3]
        # Student sees a stronger augmentation than the teacher
        # Noisy Student: RandAugment, dropout, stochastic depth
        # [arxiv:2103.00550 §6.2]
        # Copy-Paste variant: paste GT instances into pseudo images
        # [arxiv:2012.07177 §4.4]
        # CAST variant: no threshold; cluster each name's folder, keep the
        # largest cluster, then merge or drop folders by centre cosine
        # [arxiv:2103.04098 §3]
        S = train(new_model(), labelled + (unlabelled[keep], targets))
        # Student becomes the next teacher (Noisy Student)
        # EMA variant: T.w = a * T.w + (1 - a) * S.w after every step
        # [arxiv:2102.09480 eq. 3]
        T = S
    return T

Contradictions

• Discard or keep unsure predictions. The U2PL paper argues that dropping unsure pixels starves some classes of supervision and throws away which classes a pixel is not. It keeps those pixels as negatives instead 2203.03884 §1–§2. Its ablation finds reliable-only negatives give 77.30 mIoU at 1/8 against 79.01 with unreliable negatives [arxiv:2203.03884 tab. 4]. AdaContrast takes a third position: it applies no threshold and keeps every pseudo-label, with the queue vote doing the filtering 2204.10377 §3.3. Its headline is 86.8% on VisDA-C tab. 1. The survey presents confidence-based selection as the standard assumption behind self-training and does not weigh the alternative 2103.00550 §2.1, §6.2. The sides are stated here; the choice is for the implementer to test.
• Gains versus harm from unlabelled data. The copy-paste, U2PL and Unbiased Teacher papers report gains from unlabelled data on their own benchmarks 2012.07177 tab. 2 2203.03884 tab. 1 2102.09480 tab. 1. The survey warns that unlabelled data can make results worse when its distribution differs from the labelled set 2103.00550 §2.1, §8. The survey reports no numbers, so the two claims do not test each other.
• Imbalance as the cause of failure. The FlexMatch authors attribute its weak SVHN result to class imbalance, where rare classes keep low thresholds 2110.08263 §4.1. The FreeMatch authors add a fairness term for the same problem and report their best results in imbalanced settings 2205.07246 app. E, tab. 14. The two papers use different methods and splits, so neither result confirms the other's explanation. The page does not resolve which cause dominates.

Papers

• Domain Adaptation for Semantic Segmentation with Maximum Squares Loss: maximum squares loss for target pixels, plus an optional second head whose guidance labels come from both heads.
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling: per-class confidence thresholds that move with a class's learning status, added to FixMatch with no extra passes.
• FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning: global and per-class thresholds from running averages of the model's own confidence, plus a fairness loss on predicted class frequencies.
• Ghiasi et al., Copy-Paste: Copy-Paste augmentation and a self-training recipe for instance segmentation, with ablations on where pseudo-labelled data is pasted.
• A Survey on Deep Semi-supervised Learning: taxonomy of deep semi-supervised methods, with self-training and pseudo-labelling described in one shared notation; no experiments.
• PLOP: Learning without Forgetting for Continual Semantic Segmentation: old-model pseudo-labels for background pixels, gated by per-class entropy thresholds, in a class-incremental setting.
• Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels: U2PL, an entropy-gated teacher-student method for segmentation that reuses unreliable pixels as negatives.
• Unbiased Teacher for Semi-Supervised Object Detection: EMA teacher with weak and strong views for a Faster R-CNN detector, with focal loss on the ROI classifier.
• Continental-Scale Building Detection from High Resolution Satellite Imagery: soft-label self-training on 8.7M filtered satellite tiles, with black masks on the student's inputs; one ablation among several.
• Data Augmentation Approaches in Natural Language Processing: A Survey: survey of text augmentation, where self-training is one sampling method; no experiments.
• End-to-end Jordanian dialect speech-to-text self-supervised learning framework: wav2vec-style pre-training, then five noisy-student generations on the authors' own Jordanian Arabic speech; single run.
• Contrastive Test-Time Adaptation: source-free adaptation with queue-refined pseudo-labels, a contrastive term and a diversity term, on an unlabelled target stream.
• WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition: CAST, a self-training cleaning loop for web face folders, with the cleaned WebFace42M set and a time-limited test.
• Capabilities of Gemini Models in Medicine: Med-Gemini's loop of self-written reasoning chains filtered by the gold answer; one part of a larger medical model report.