--- kind: concept slug: self-training title: Self-training summary: >- Teacher-student loop: a model pseudo-labels unlabelled inputs and a student, often trained on stronger augmentation, learns from the kept labels. Variants differ in which labels they keep (confidence or entropy cut-offs, or soft labels) and in the teacher. Trade-off: extra passes and a second model versus gains from unlabelled data, which can also hurt. tags: [semi-supervised-learning, pseudo-labeling, teacher-student, data-augmentation, object-detection, continual-learning] papers: ["2012.07177", "2103.00550", "2203.03884", "1909.13589", "2011.11390", "2102.09480", "2107.12283", "2110.08263", "2205.07246", "2204.10377", "2404.18416", "2103.04098", "2110.01852", "2609.24410"] related: [consistency-regularization, data-augmentation, noisy-labels, contrastive-learning, semantic-segmentation, instance-segmentation, semi-supervised-learning, object-detection, domain-adaptation, entropy-minimization, continual-learning, curriculum-learning, test-time-training, face-recognition, dataset-curation] status: draft updated_at: 2026-11-01 --- ## Overview Self-training uses a model's own predictions as labels. A teacher produces labels for inputs that lack them, and a student is trained on the labels it keeps [arxiv:2103.00550 §6.2]. The shared assumption is that high-confidence predictions tend to be correct, for instance when classes form well-separated clusters. If the data break that assumption, the method can fail to help or even reduce accuracy [arxiv:2103.00550 §2.1]. The unlabelled inputs need not be fresh images from the same source. Two variants below label what the model has not labelled before: unlabelled target-domain pixels in domain adaptation [arxiv:1909.13589 §3.4], and background pixels that may hide old classes in continual segmentation [arxiv:2011.11390 §3.3]. A stream of unlabelled test images, with no source data at adaptation time, is another such input [arxiv:2204.10377 §1]. Med-Gemini uses the same word for a different loop. It writes reasoning chains for labelled questions and keeps those that reach the gold answer, so its labels are not pseudo-labels on unlabelled inputs [arxiv:2404.18416 §2.1]. Its variant is listed below, separately. Pseudo-labelling is one branch of semi-supervised learning. The survey files self-training under pseudo-labelling and separates it from disagreement-based methods such as co-training, which take pseudo-labels from two views that agree [arxiv:2103.00550 §6.1]. ## Variants ### Confidence-thresholded pseudo-labels The teacher's most probable class becomes the target for each unlabelled sample, and the loss on those targets is added to the supervised loss with a weight that changes over training [arxiv:2103.00550 §6.2]. The survey gives no schedule for that weight and no confidence threshold. The fixed-threshold baselines in the FlexMatch comparison are reported only through that paper's table, not through a card here [arxiv:2110.08263 tab. 1]. ### Class-wise adaptive thresholds (FlexMatch) The fixed cut-off τ is replaced by one threshold per class. Each class's threshold is τ scaled by how many unlabelled samples are currently marked with that class, so classes the model still struggles with admit more pseudo-labels [arxiv:2110.08263 §3.1]. The scale is β(c) = σ(c) / max σ, where σ(c) counts the marked samples, and the mapping M(β) = β / (2 − β) gives the threshold T(c) = M(β(c)) · τ [arxiv:2110.08263 eq. 11–12]. The method adds no parameters and no forward passes; the marks are stored state, updated from the weak-view predictions [arxiv:2110.08263 §3.1] [alg. 1]. A warm-up rule keeps the denominator at least the number of unused samples early in training [arxiv:2110.08263 §3.2]. The paper uses τ = 0.95 on the CIFAR and STL-10 experiments [arxiv:2110.08263 §4] [tab. 3]. The author-reported error on CIFAR-100 with 400 labels is 39.94%, against 46.42% for FixMatch with the fixed cut-off. On STL-10 with 40 labels it is 29.15% against 35.97% [arxiv:2110.08263 tab. 1]. The same tables show FixMatch as equal or better in some cells: 4.86% against 4.98% on CIFAR-10 with 250 labels, and 3.81% against 8.19% on SVHN with 40 labels. The authors attribute the SVHN result to class imbalance, where rare classes do not reach β near 1 and keep low thresholds [arxiv:2110.08263 §4.1]. Card: [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](/papers/2110.08263). ### Self-adaptive threshold with a fairness term (FreeMatch) The global threshold is an exponential moving average (EMA) of the mean top weak-view confidence on unlabelled samples, starting at 1/C [arxiv:2205.07246 eq. 5]. Each class's threshold is that global value times the class's own EMA of predicted probability, divided by the largest such EMA, so the most-predicted class gets the full global threshold [arxiv:2205.07246 eq. 6–7]. A sample is kept if its top probability clears the threshold of its predicted class [arxiv:2205.07246 eq. 8]. Where FlexMatch scales by counts of marked samples, this variant scales by a running average of probabilities [arxiv:2110.08263 §3.1] [arxiv:2205.07246 eq. 6–7]. A fairness loss, with weight w_f, pushes the normalised distribution of kept strong-view predictions towards the normalised weak-view class statistics [arxiv:2205.07246 eq. 9–12]. The paper sets the unlabelled loss weight to 1 in all experiments and gives w_f per setting [arxiv:2205.07246 app. D, tab. 5]. The authors describe the extra cost over FixMatch as very small [arxiv:2205.07246 app. C]. The released configs use a fairness weight ten times below the appendix values for the same settings, so check the value against the card before reproducing [Card](/papers/2205.07246). The author-reported error on CIFAR-10 with 10 labels is 8.07%, against 13.85% for FlexMatch. On STL-10 with 40 labels it is 15.56% against 29.15% [arxiv:2205.07246 tab. 1]. The gap is small at 40 labels on CIFAR-10, 4.90% against 4.97% [tab. 1]. Removing the fairness term raises the CIFAR-10 10-label error to 10.37% [arxiv:2205.07246 tab. 4]. The paper reports the best error over all checkpoints, not the final one [arxiv:2205.07246 §5.1]. Card: [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](/papers/2205.07246). ### Noisy Student An equal-or-larger student is trained on labelled data and on pseudo-labels from the teacher, with noise added to the student through RandAugment, dropout and stochastic depth. The student then becomes the next teacher, and the cycle repeats for a few rounds [arxiv:2103.00550 §6.2]. The survey describes the recipe but reports no accuracy numbers for it. ### Noisy student for speech recognition (wav2vec-style encoder) A wav2vec 2.0-style encoder is pre-trained on unlabelled audio. It is then fine-tuned with CTC as the first teacher, on 12.5 h of cleaned labelled speech [arxiv:2609.24410 §8.2] [§9]. Each generation pseudo-labels one batch of unlabelled audio with the current teacher. The audio is augmented with Gaussian noise, time stretch and pitch shift, and a larger student is trained on the pseudo-labelled, clean and augmented data. The student becomes the next teacher [arxiv:2609.24410 §6.2] [§8.2] [fig. 8]. The paper runs five generations [tab. 4]. It states no pseudo-label filter, no augmentation magnitudes and no rule for accepting a student [arxiv:2609.24410 §7.3] [§6.2] [§9]. It is not stated whether each student starts from scratch [§8.2] [fig. 7]. The author-reported WER on their own evaluation set falls from 0.568 for the generation-zero teacher to 0.515 at generation five [arxiv:2609.24410 tab. 4]. The paper runs once, with no seeds or intervals. The gap between generations 2 and 3 is 0.003 WER, and the evaluation set size is not stated. Each generation is selected on the score it is reported on, so the gains may be optimistic [arxiv:2609.24410 §9]. The text's "8%" relative gain does not match the table, which gives about 9.3% [arxiv:2609.24410 §9] [tab. 4]. CER returns to 0.279 at generation four, after 0.264 at five [tab. 4]. The paper gives no optimiser, learning rate, batch size or epoch count [arxiv:2609.24410 §8.2]. Card: [End-to-end Jordanian dialect speech-to-text self-supervised learning framework](/papers/2609.24410). ### Entropy-gated teacher-student for segmentation (U2PL) Pixels whose teacher entropy falls in the top share are left out of the pseudo-label loss. The share starts at 20% and shrinks linearly to zero over training [arxiv:2203.03884 §3.2 eq. 7]. The excluded pixels are not thrown away: they are reused as negative examples in per-class queues for a pixel-level contrastive loss [arxiv:2203.03884 §3.3]. The teacher is an exponential moving average of the student [arxiv:2203.03884 §3.1]. On PASCAL VOC 2012 with ResNet-101 and DeepLabv3+, the author-reported mIoU at the 1/16 classic split is 67.98, against 45.77 for supervised-only training and 57.60 for the strongest listed baseline [arxiv:2203.03884 tab. 1]. Card: [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](/papers/2203.03884). ### Copy-Paste with self-training for detection and instance segmentation The teacher is trained with Copy-Paste augmentation on labelled data. It then pseudo-labels unlabelled images, and a student trains on batches that are half labelled COCO and half pseudo-labelled data [arxiv:2012.07177 §4.4]. Ground-truth instances are pasted into the pseudo-labelled images. Pasting pseudo-labelled instances into labelled images gave no extra AP [arxiv:2012.07177 §4.4]. On EfficientNet-B7 FPN at 640 pixels, self-training alone gives +1.5 box AP and +1.3 mask AP; self-training combined with Copy-Paste gives +2.9 box AP and +2.3 mask AP over the baseline [arxiv:2012.07177 tab. 2]. Card: [Ghiasi et al., Copy-Paste](/papers/2012.07177). ### EMA teacher with focal classification for detection (Unbiased Teacher) A Faster R-CNN with FPN and ResNet-50 is first trained on labelled boxes only, the burn-in stage [arxiv:2102.09480 §3.1]. The teacher is then an exponential moving average of the student: at each step it keeps 0.9996 of its own weights and takes 0.0004 from the student [arxiv:2102.09480 eq. 3, tab. 5]. The teacher sees a weakly augmented image (horizontal flip). Its detections go through class-wise NMS and a confidence cut-off of 0.7 [arxiv:2102.09480 §3.2, tab. 5]. The student sees a strongly augmented copy (colour jitter, grayscale, blur, cutout) [arxiv:2102.09480 tab. 6]. Only the RPN and ROI classification losses use pseudo-labels; box-regression losses do not, because confidence does not measure localisation quality [arxiv:2102.09480 §3.2]. The ROI classifier uses focal loss in place of cross-entropy, which the authors argue limits the pull towards common classes and background [arxiv:2102.09480 §3.3]. On COCO with 1% of the boxes labelled, the author-reported mAP is 20.75, against 13.97 for STAC and 9.05 for supervised-only training [arxiv:2102.09480 tab. 1]. In the ablation at the same label fraction, cross-entropy without the EMA teacher reaches 13.42 and focal loss with the EMA teacher 21.19 [arxiv:2102.09480 fig. 5]. That ablation setting is 0.44 mAP above the headline 20.75 for the same label fraction, and the paper does not explain the gap. Card: [Unbiased Teacher for Semi-Supervised Object Detection](/papers/2102.09480). ### Self-produced guidance from a second head (MaxSquare, domain adaptation) For unsupervised domain adaptation from synthetic GTA5 images to real Cityscapes images, a second low-level head on conv4 features is added [arxiv:1909.13589 §3.4]. For each target pixel, the guidance label is the argmax of the averaged output of the two heads. The label is kept only when either head gives that class a probability above 0.95 [arxiv:1909.13589 §3.4 eq. 15]. The second head is trained with cross-entropy on those labels, weighted by 0.1 [arxiv:1909.13589 §3.4 eq. 14, eq. 16]. The main change in the paper is a different target loss, the maximum squares loss [arxiv:1909.13589 §3.2]; the guidance head is an add-on. On top of the image-weighted loss, the head raises mIoU from 45.2 to 46.4 [arxiv:1909.13589 tab. 4]. The prose calls the gain significant, and the table shows that number. The authors report 46.4 against 42.2 for their entropy-minimisation rerun on the same split [arxiv:1909.13589 tab. 2]. Card: [Domain Adaptation for Semantic Segmentation with Maximum Squares Loss](/papers/1909.13589). ### Old-model pseudo-labels for background pixels (PLOP, continual segmentation) When a new step adds classes and old images are not stored, the new training labels mark old-class pixels as background. The frozen model from the previous step labels those pixels [arxiv:2011.11390 §3.3]. A background pixel gets the old model's argmax class only if its entropy is below a per-class threshold. The threshold is the median entropy of that class over training pixels. Other background pixels are ignored [arxiv:2011.11390 §3.3 eq. 6–7]. The loss on these labels is weighted by ν, the share of background pixels that kept a label [arxiv:2011.11390 eq. 8]. The authors report 54.64 mIoU on VOC 2012 in the 15-1 Overlapped setting, against 29.29 for their MiB re-implementation [arxiv:2011.11390 tab. voc_sota_full]. On a 20% validation split, pseudo-labels alone raise 15-1 mIoU from 3.99 (cross-entropy only) to 19.74. Adding Local POD distillation gives 50.41, and ν weighting gives 52.31 [arxiv:2011.11390 tab. ablations]. A pseudo-oracle that removes wrong pseudo-labels reaches 54.09 on the same split [arxiv:2011.11390 tab. ablation_classif_distillation], so the headroom from better pseudo-labels is small there. Validation-split numbers are not comparable with the main tables. Card: [PLOP: Learning without Forgetting for Continual Semantic Segmentation](/papers/2011.11390). ### Soft-label self-training with masked inputs (building footprints) The supervised model is the teacher. The student trains on unlabelled satellite tiles with the teacher's soft confidences as targets and a KL loss with focal weighting, γ = 0.25 [arxiv:2107.12283 §8]. Random black masks are added to the student's inputs, so that it learns to handle no-data pixels [arxiv:2107.12283 §8]. The pool started at 100M tiles of 640 × 640 px. The supervised model kept 8.7M of them, mostly tiles without buildings [arxiv:2107.12283 §3.3]. The paper does not state the filter threshold or the number of iterations. It reports that fine-tuning on labels helped only in the first iteration [arxiv:2107.12283 §8]. The author-reported gain is +0.057 mAP on average over the supervised teacher. The authors describe the mixup and edge-weighting effects as larger [arxiv:2107.12283 §11] [fig. 8]. The abstract rounds the gain to +0.06, and the body value is used here. Card: [Continental-Scale Building Detection from High Resolution Satellite Imagery](/papers/2107.12283). ### Test-time adaptation with refined pseudo-labels (AdaContrast) The unlabelled inputs are a target stream seen at adaptation time, and the source-trained classifier is updated on it without source images [arxiv:2204.10377 §1]. For each batch, one weak view and two strong views of each image are used [arxiv:2204.10377 §3]. The weak view is passed through a momentum copy of the model, and its feature is compared with the N nearest entries of a memory queue. The probabilities of those entries are averaged, and the argmax of the average is the pseudo-label [arxiv:2204.10377 §3.1 eq. 3–4]. No confidence threshold is applied: every pseudo-label supervises the strong view with cross-entropy [arxiv:2204.10377 §3.3 eq. 7]. Two more terms are added. An InfoNCE loss treats the momentum copy's key as positive and drops queue entries that share the current pseudo-label from the negatives [arxiv:2204.10377 §3.2 eq. 5–6]. A diversity term rewards a spread-out mini-batch mean prediction [arxiv:2204.10377 §3.3 eq. 8–9]. The three losses have weight 1 in every experiment [arxiv:2204.10377 §3.3 eq. 10]. The author-reported per-class average on VisDA-C is 86.8%, against 83.0% for SHOT [arxiv:2204.10377 tab. 1]. On DomainNet-126 over seven shifts it is 67.8%, against 67.1% [tab. 2]. In the ablation, refinement without the contrastive term gives 86.5% at the base learning rate but 9.9% at ten times that rate. Adding the contrastive term gives 84.3% at the high rate, and the regularisation term gives 86.6% [arxiv:2204.10377 tab. 4]. Card: [Contrastive Test-Time Adaptation](/papers/2204.10377). ### Self-generated rationales filtered by gold answers (Med-Gemini) Each labelled training question gets two reasoning chains, one written without search results and one with them. The search queries are written by the model itself [arxiv:2404.18416 §2.1]. Chains that end in a wrong answer are dropped before fine-tuning, so the filter needs the correct answer [arxiv:2404.18416 §2.1]. The fine-tuned model writes new chains, and the loop repeats until results stop improving [arxiv:2404.18416 §2.1]. The paper does not state the stopping rule. The author-reported gain from this self-training on MedQA is 3.2 points [arxiv:2404.18416 §4.1] [fig. 4]. Card: [Capabilities of Gemini Models in Medicine](/papers/2404.18416). ### Re-clustering web labels in rounds (CAST, face data) Here the "unlabelled" data are noisy labels. A web search gives each name a folder of face crops, and many folders hold faces of other people. CAST treats cleaning as self-training. A teacher embeds every face. Each folder is clustered, and only its largest cluster is kept. Folders are then merged or dropped by comparing their feature centres. A new student is trained on the cleaned data and becomes the next teacher [arxiv:2103.04098 §3]. The first teacher is an ArcFace ResNet-100 trained on MS1MV2. The text and the counts in the table indicate that each round starts from the same pre-processed set, not from the previous round's output [arxiv:2103.04098 §3] [tab. 2]. No confidence threshold is used. The cut-offs are cosine thresholds on the centres: merge above 0.7, and drop the smaller folder between 0.5 and 0.7 [arxiv:2103.04098 §3]. The DBSCAN parameters are not stated. The paper uses three rounds, and reports that a fourth gave saturated results [§5.4]. The author-reported IJB-C TAR@FAR=1e-4 for ArcFace ResNet-100 trained on the cleaned WebFace42M is 97.70, against 96.03 for MS1MV2 [arxiv:2103.04098 tab. 5]. Over rounds on the web set, IJB-C goes from 97.28 after round 1 to 97.51, 97.70 and 97.66 [tab. 7]. On MS1M, the same procedure takes IJB-C from 93.45 to 94.31 after round 1 and 96.55 after round 3 [tab. 7]. That MS1M teacher was trained on IMDB-Face, not MS1MV2 [§5.4]. Within the same pipeline on MS1M, DBSCAN gives 96.55 on IJB-C, against 96.03 for K-Means and 96.42 to 96.48 for the two graph-based cleaners [arxiv:2103.04098 tab. 8]. The paper's dataset is released under a signed licence for research use, and its Table 1 calls it public [§6] [tab. 1]. Check the licence before any reuse. Card: [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](/papers/2103.04098). ### Other described variants The survey also describes Meta Pseudo Labels, where the teacher is updated from the student's loss on held-out data, and self-training by distillation as in SimCLRv2 [arxiv:2103.00550 §6.2]. It gives no numbers for them, so treat them as unvalidated here. In NLP data augmentation, self-training means labelling unlabelled text with a fine-tuned model. SentAugment retrieves web sentences close to task queries built from the labelled data [arxiv:2110.01852 §2.3.4]. This survey of text augmentation reports no experiments of its own, so the method is listed here without numbers [arxiv:2110.01852 tab. 2]. ## When to use which | Situation | Use | Why | |---|---|---| | Classification with very few labels, balanced classes, and a cheap retrain | Self-adaptive threshold with a fairness term (FreeMatch), or class-wise thresholds (FlexMatch) | FreeMatch reports 8.07% against 13.85% error for FlexMatch on CIFAR-10 with 10 labels [arxiv:2205.07246 tab. 1]. FlexMatch reports 39.94% against 46.42% on CIFAR-100 with 400 labels, and adds no model or forward pass [arxiv:2110.08263 tab. 1] [§3.1]. The gap narrows with more labels, so the choice matters most at the fewest labels [arxiv:2205.07246 tab. 1]. The survey explains Noisy Student but reports no accuracy, so it cannot rank it [arxiv:2103.00550 §6.2]. Check the confidence assumption first [arxiv:2103.00550 §2.1]. | | Classification with few labels and strongly imbalanced classes | Measure a fixed-threshold baseline first, then compare FreeMatch and FlexMatch on the same split | FlexMatch is worse than FixMatch on SVHN with 40 labels, 8.19% against 3.81% error, which the authors link to imbalance [arxiv:2110.08263 tab. 1] [§4.1]. FreeMatch reports 1.97% on SVHN with 40 labels [arxiv:2205.07246 tab. 1] and claims the best results in its imbalanced setting [arxiv:2205.07246 app. E, tab. 14]. Both are author-reported. | | Dense prediction (segmentation), few labelled images, many unlabelled | Entropy-gated pseudo-labels with an EMA teacher and contrastive negatives | Author-reported mIoU is 10.4 above the strongest listed baseline at 1/16 on VOC [arxiv:2203.03884 tab. 1]. Costs a teacher pass, an EMA copy and per-class queues of 30,000 to 50,000 entries [arxiv:2203.03884 app. A]. | | Instance masks available, large unlabelled pool, detection or instance segmentation | Pseudo-labels plus Copy-Paste on the student, batches half labelled and half pseudo-labelled | Author-reported +2.9 box AP over baseline [arxiv:2012.07177 tab. 2]. Needs masks, and the best reported result also uses unlabelled Objects365 [arxiv:2012.07177 §4.5]. | | Boxes labelled for 0.5–2% of images, unlabelled photos, Faster R-CNN-style detector | EMA teacher, weak and strong views, focal ROI loss (Unbiased Teacher) | Author-reported 20.75 mAP at 1% against 13.97 for STAC and 9.05 supervised [arxiv:2102.09480 tab. 1]. Costs a teacher forward pass per unlabelled batch; COCO runs are 180k iterations at 32 labelled and 32 unlabelled images [arxiv:2102.09480 app. "Training"]. | | Labelled synthetic source, unlabelled real target, segmentation | Change the target loss first; add guidance from a second head if the budget allows (MaxSquare) | Authors report 46.4 against 42.2 for their entropy-minimisation rerun [arxiv:1909.13589 tab. 2]. The guidance head adds 1.2 mIoU on top of the image-weighted loss [arxiv:1909.13589 tab. 4] and needs an extra ASPP head [arxiv:1909.13589 §3.4]. | | Classes added in steps, old images not stored | Old-model pseudo-labels on background pixels, entropy-gated and ν-weighted (PLOP) | Author-reported 54.64 against 29.29 mIoU on VOC 15-1 Overlapped [arxiv:2011.11390 tab. voc_sota_full]. Costs a frozen old-model pass per step and stored feature maps for distillation [arxiv:2011.11390 §3.2]. | | Large pool of unlabelled satellite or aerial tiles, segmentation model, budget for a full retrain on the pool | Soft-label self-training on filtered tiles, with black masks on the student's inputs | Author-reported +0.057 mAP over the supervised teacher, smaller than the mixup and edge-weighting effects [arxiv:2107.12283 §11]. The teacher pass filters 100M tiles down to 8.7M [arxiv:2107.12283 §3.3]. Measure against the supervised model first. | | Low-resource speech with a few hours of transcripts, a large untranscribed audio pool, and budget for several full retrains | Pre-train on the audio, then noisy-student generations with pseudo-labels and augmentation (Jordanian Arabic card) | Author-reported WER 0.568 to 0.515 over five generations on the authors' own set, from one run [arxiv:2609.24410 tab. 4]. Fine-tuning the first teacher took 83 h on two GPUs [§8.1]. The pseudo-label filter and augmentation strengths are not stated [§7.3] [§6.2], so the loop needs choices the authors did not report. | | Unlabelled test-time stream, no source data at adaptation time, classifier to adapt | Pseudo-labels refined by a feature queue, with a contrastive term and a diversity term (AdaContrast) | Author-reported 86.8% against 83.0% for SHOT on VisDA-C [arxiv:2204.10377 tab. 1]. Needs a momentum copy and two queues. Without the contrastive term, the ablation collapses at the higher learning rate [arxiv:2204.10377 tab. 4]. | | Labelled questions with a verifiable answer, a model that writes its own reasoning | Keep self-written chains that reach the gold answer, fine-tune, and repeat (Med-Gemini) | Author-reported 3.2 points on MedQA from self-training [arxiv:2404.18416 §4.1] [fig. 4]. The stopping rule and the search setup are not stated [§2.1], so the loop is hard to reproduce. | | Large web-scraped set grouped by name or identity, with many wrong members, and budget for several full retrains | Re-cluster each group with the latest teacher and merge or drop groups by centre similarity, for three rounds (CAST) | Author-reported IJB-C 97.70 for the cleaned WebFace42M against 96.03 for MS1MV2 [arxiv:2103.04098 tab. 5]. Round 4 scores 97.66 against 97.70 at round 3 [tab. 7], so the paper stops at three [§5.4]. The paper rules out clustering the whole set at once as impractical [§3]. The released data need a signed research licence [§6]. | | Unlabelled data may come from a different distribution | Keep a supervised-only baseline and measure against it | The survey warns that extra unlabelled data can reduce accuracy when assumptions fail [arxiv:2103.00550 §2.1, §8]. | | Labels are noisy as well as scarce | Read the noisy-label concept first | The survey lists noisy labels as an open problem for these methods [arxiv:2103.00550 §8]. | Training is slower than for supervised methods, and the paper gives no wall-clock figures [arxiv:2203.03884 §5]. The pseudo-labelling variants change only the weights, so inference cost is unchanged. The Med-Gemini loop also runs a search at inference for text tasks [arxiv:2404.18416 §2.1]. ## Implementation sketch The core loop shared by the variants. Where they differ, the comment says how. ```pseudocode # Generic self-training round [arxiv:2103.00550 §6.2] # Input: labelled (x, y), unlabelled u, teacher T trained on labelled data def self_train_round(T, labelled, unlabelled, rounds): for r in range(rounds): # Teacher predicts class probabilities for unlabelled inputs probs = T.predict(unlabelled) # (N, K) # Keep only the confident ones; threshold not stated in survey keep = probs.max(axis=1) > tau # AdaContrast: no threshold, every sample is kept; the label # comes from a vote over a feature queue [arxiv:2204.10377 §3.1] # FlexMatch variant: tau per class, scaled by the class's # marked-sample count [arxiv:2110.08263 eq. 11-12] # FreeMatch variant: EMA of confidence, scaled per class by an # EMA of predicted probability [arxiv:2205.07246 eq. 5-7] # Segmentation variant: entropy quantile that shrinks to 0 # over training instead of a fixed tau [arxiv:2203.03884 eq. 7] targets = probs.argmax(axis=1)[keep] # Soft-label variant: targets = probs, KL loss with focal # gamma = 0.25 [arxiv:2107.12283 §8] # PLOP variant: only background pixels, entropy below the # per-class median [arxiv:2011.11390 §3.3] # Student sees a stronger augmentation than the teacher # Noisy Student: RandAugment, dropout, stochastic depth # [arxiv:2103.00550 §6.2] # Copy-Paste variant: paste GT instances into pseudo images # [arxiv:2012.07177 §4.4] # CAST variant: no threshold; cluster each name's folder, keep the # largest cluster, then merge or drop folders by centre cosine # [arxiv:2103.04098 §3] S = train(new_model(), labelled + (unlabelled[keep], targets)) # Student becomes the next teacher (Noisy Student) # EMA variant: T.w = a * T.w + (1 - a) * S.w after every step # [arxiv:2102.09480 eq. 3] T = S return T ``` ## Contradictions - **Discard or keep unsure predictions.** The U2PL paper argues that dropping unsure pixels starves some classes of supervision and throws away which classes a pixel is not. It keeps those pixels as negatives instead [arxiv:2203.03884 §1–§2]. Its ablation finds reliable-only negatives give 77.30 mIoU at 1/8 against 79.01 with unreliable negatives [arxiv:2203.03884 tab. 4]. AdaContrast takes a third position: it applies no threshold and keeps every pseudo-label, with the queue vote doing the filtering [arxiv:2204.10377 §3.3]. Its headline is 86.8% on VisDA-C [tab. 1]. The survey presents confidence-based selection as the standard assumption behind self-training and does not weigh the alternative [arxiv:2103.00550 §2.1, §6.2]. The sides are stated here; the choice is for the implementer to test. - **Gains versus harm from unlabelled data.** The copy-paste, U2PL and Unbiased Teacher papers report gains from unlabelled data on their own benchmarks [arxiv:2012.07177 tab. 2] [arxiv:2203.03884 tab. 1] [arxiv:2102.09480 tab. 1]. The survey warns that unlabelled data can make results worse when its distribution differs from the labelled set [arxiv:2103.00550 §2.1, §8]. The survey reports no numbers, so the two claims do not test each other. - **Imbalance as the cause of failure.** The FlexMatch authors attribute its weak SVHN result to class imbalance, where rare classes keep low thresholds [arxiv:2110.08263 §4.1]. The FreeMatch authors add a fairness term for the same problem and report their best results in imbalanced settings [arxiv:2205.07246 app. E, tab. 14]. The two papers use different methods and splits, so neither result confirms the other's explanation. The page does not resolve which cause dominates. ## Papers - [Domain Adaptation for Semantic Segmentation with Maximum Squares Loss](/papers/1909.13589): maximum squares loss for target pixels, plus an optional second head whose guidance labels come from both heads. - [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](/papers/2110.08263): per-class confidence thresholds that move with a class's learning status, added to FixMatch with no extra passes. - [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](/papers/2205.07246): global and per-class thresholds from running averages of the model's own confidence, plus a fairness loss on predicted class frequencies. - [Ghiasi et al., Copy-Paste](/papers/2012.07177): Copy-Paste augmentation and a self-training recipe for instance segmentation, with ablations on where pseudo-labelled data is pasted. - [A Survey on Deep Semi-supervised Learning](/papers/2103.00550): taxonomy of deep semi-supervised methods, with self-training and pseudo-labelling described in one shared notation; no experiments. - [PLOP: Learning without Forgetting for Continual Semantic Segmentation](/papers/2011.11390): old-model pseudo-labels for background pixels, gated by per-class entropy thresholds, in a class-incremental setting. - [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](/papers/2203.03884): U2PL, an entropy-gated teacher-student method for segmentation that reuses unreliable pixels as negatives. - [Unbiased Teacher for Semi-Supervised Object Detection](/papers/2102.09480): EMA teacher with weak and strong views for a Faster R-CNN detector, with focal loss on the ROI classifier. - [Continental-Scale Building Detection from High Resolution Satellite Imagery](/papers/2107.12283): soft-label self-training on 8.7M filtered satellite tiles, with black masks on the student's inputs; one ablation among several. - [Data Augmentation Approaches in Natural Language Processing: A Survey](/papers/2110.01852): survey of text augmentation, where self-training is one sampling method; no experiments. - [End-to-end Jordanian dialect speech-to-text self-supervised learning framework](/papers/2609.24410): wav2vec-style pre-training, then five noisy-student generations on the authors' own Jordanian Arabic speech; single run. - [Contrastive Test-Time Adaptation](/papers/2204.10377): source-free adaptation with queue-refined pseudo-labels, a contrastive term and a diversity term, on an unlabelled target stream. - [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](/papers/2103.04098): CAST, a self-training cleaning loop for web face folders, with the cleaned WebFace42M set and a time-limited test. - [Capabilities of Gemini Models in Medicine](/papers/2404.18416): Med-Gemini's loop of self-written reasoning chains filtered by the gold answer; one part of a larger medical model report.