---
kind: concept
slug: semi-supervised-learning
title: Semi-supervised learning
summary: >-
  Training on a few labels plus many unlabelled inputs. Families: generative, consistency, graph, pseudo-label and hybrid methods; variants for dense prediction (images, LiDAR), detection, clustering, text, tabular and sensor data. Trade-off: extra passes and stored state; unlabelled data can hurt.
tags: [semi-supervised-learning, consistency-regularization, pseudo-labeling, teacher-student]
papers: ["2102.09480", "2103.00550", "2201.02609", "2203.03884", "2010.08240", "2105.05727", "2208.11296", "1905.00067", "2006.07988", "2012.06678", "2110.08263", "2205.07246", "2604.26818", "1709.00786", "2110.14048", "2012.13635", "1907.11879", "2405.05258"]
related: [consistency-regularization, self-training, contrastive-learning, semantic-segmentation, self-supervised-learning, noisy-labels, object-detection, novel-category-discovery, data-augmentation, domain-adaptation, graph-neural-network, generalized-pagerank, language-model-pretraining, masked-language-modeling, self-attention, multi-layer-perceptron, curriculum-learning, multi-task-learning, gradient-conflict-resolution, active-learning, multiple-instance-learning, transfer-learning, entropy-minimization, support-vector-machine, neural-symbolic-integration, differentiable-fuzzy-logic, time-series-classification]
status: draft
updated_at: 2026-11-01
---

## Overview

Labels are expensive, unlabelled inputs are cheap. Semi-supervised learning
(SSL) trains on a small labelled set L and a much larger unlabelled set U,
usually with |L| much smaller than |U| and K classes [arxiv:2103.00550 §2]. The
training objective adds three terms: a supervised loss on labelled data, an
unsupervised loss on unlabelled data, and a regularizer, each with a trade-off
weight [arxiv:2103.00550 §2 eq. 1]. Most deep methods differ in how they build
the unsupervised part.

The survey's central caveat is that the gain depends on assumptions about the
data: self-training needs confident predictions to be correct, co-training needs
two views, and the cluster, manifold and low-density-separation assumptions say
where the decision boundary may lie. When these fail, unlabelled data can leave
accuracy unchanged or lower it [arxiv:2103.00550 §2.1, §8]. Keep a
supervised-only baseline in every experiment. The survey itself covers image
classification [arxiv:2103.00550 §1]; the later variants below apply the same
ideas to segmentation, detection, clustering and text.

A later visual survey groups the same methods differently. It names five SSL
families (consistency regularization, self-training, graph regularization,
generative models and self-supervision) and argues that most of them rest on one
of three smoothness rationales: local smoothness across views, local smoothness
across neighbours, or global smoothness from the decision boundary
[arxiv:2208.11296 §2.2, §4.1, tab. 1, tab. 3]. That survey reports no
experiments of its own, so its taxonomy is a map, not a ranking
[arxiv:2208.11296 §1]. Card:
[Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](/papers/2208.11296).

## Variants

### Consistency regularization

The unlabelled input is perturbed in some way, and the prediction for the
perturbed view is pulled towards a target from the model itself. The survey
tabulates ten such methods by what they perturb and where the target comes from:
Ladder Network, Π Model, Temporal Ensembling, Mean Teacher, VAT, Dual Student,
SWA, VAdD, UDA and WCP [arxiv:2103.00550 §4 tab. 3]. Details live on the
[consistency regularization](/concepts/consistency-regularization) page. Cost
differs by method: the Π Model runs each sample forward twice per step, Temporal
Ensembling runs once but keeps a per-sample running target that is refreshed once
per epoch, and Mean Teacher keeps an exponential moving average (EMA) of the
weights as its teacher [arxiv:2103.00550 §4 eq. 17–19]. The visual survey
splits the same idea by where the perturbation happens: on the input (random
augmentation, VAT, MixUp-style mixing, automated augmentation) or in the model
(stochastic weights or structure, and weight or prediction ensembles)
[arxiv:2208.11296 §2.2.1]. Cards:
[A Survey on Deep Semi-supervised Learning](/papers/2103.00550) and
[Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](/papers/2208.11296).

### Generative models

Semi-supervised GANs reuse discriminator features, regularise a classifier with
generated samples, learn an inference model, or add generated samples to the
training data [arxiv:2103.00550 §3.1]. Semi-supervised VAEs extend the
variational bound with a label variable for unlabelled data, in models named M1,
M2, M1+M2, ADGM and others [arxiv:2103.00550 §3.2]. Card:
[A Survey on Deep Semi-supervised Learning](/papers/2103.00550).

### Graph-based methods

Nodes are samples, edges are similarities, and each node gets an embedding that
is trained with a reconstruction or a classification term. The survey covers
autoencoder embeddings (GAE, VGAE, SDNE, DNGR) and GNN embeddings (GCN, GAT,
GraphSAGE, GGNN) [arxiv:2103.00550 §5]. Most graph methods are transductive: they
predict only the nodes seen during training [arxiv:2103.00550 §2]. Card:
[A Survey on Deep Semi-supervised Learning](/papers/2103.00550).

Two node-classification papers change what a propagation layer computes, and
both are evaluated in the semi-supervised setting. MixHop replaces the single
neighbourhood average of a GCN layer with a concatenation over adjacency powers
`j` in a set `P`, each block with its own weights [arxiv:1905.00067 §3.1 eq. 4].
With `P = {1}` the layer reduces to the GCN layer [arxiv:1905.00067 §3.1]. The
authors prove that a vanilla GCN cannot represent a two-hop difference operator,
while MixHop with `P = {0, 1, 2}` can [arxiv:1905.00067 §3.3]. That is a
result about what the layer can express, not about accuracy. On the Planetoid
split (20 labelled nodes per class), the learned architecture reaches 71.4,
81.9 and 80.8 on Citeseer, Cora and Pubmed, against 70.3, 81.5 and 79.0 for the
GCN numbers the paper copies from Kipf and Welling. The GCN rows carry no
standard deviation, and the MixHop rows were rerun by the authors
[arxiv:1905.00067 tab. 1 §6.2]. On random splits the gains are 0.3, 1.1 and 1.6
points [arxiv:1905.00067 tab. 3]. Each power adds a sparse product per layer, so
cost grows with the largest power `j_max` [arxiv:1905.00067 §3.2]. Card:
[MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing](/papers/1905.00067).

Max-margin graph cuts keep a graph step but end in a kernel SVM. A regularized
harmonic function on a 5-nearest-neighbour graph gives each point a signed score:
the sign is the inferred label and the size is the confidence. A hinge-loss
kernel classifier is then trained only on points whose confidence is at least ε
[arxiv:2604.26818 §2 §3 eq. 5–8]. The graph step is convex, which the authors
contrast with the non-convex hat loss of semi-supervised SVMs [arxiv:2604.26818
§4.1]. The baseline is manifold regularization of SVMs (MR). On three UCI
datasets with 1 to 10% labels, with linear, cubic and RBF kernels, the authors
report lower error than MR in 29 of 36 settings [arxiv:2604.26818 fig. 4 §6.2].
For example, with a linear kernel and 1% labels on letter recognition, the error
is 15.79 against 30.94 for MR and 18.90 for a supervised SVM [arxiv:2604.26818
fig. 4]. The table reports no spread, and the stability bound is not used to set
the graph parameter [arxiv:2604.26818 §5.1]. The card records that no official
code was found. Card:
[Semi-supervised learning with max-margin graph cuts](/papers/2604.26818).

GPR-GNN keeps the neighbour averaging but learns one weight per propagation
step. A two-layer MLP produces a first representation, and the output is a
weighted sum of K + 1 propagated copies, with the weights trained end to end
[arxiv:2006.07988 §3 eq. 1]. The weights are not restricted to be positive. The
paper's filter view says that non-negative weights summing to one give a
low-pass filter, and negative weights let the model keep high-frequency content,
which the paper links to heterophilic graphs [arxiv:2006.07988 §4]. The paper
uses K = 10 [arxiv:2006.07988 §5] and initialises the weights as PPR-like
decays, one-hot vectors or at random [arxiv:2006.07988 §5]. Author-reported,
GPR-GNN has the best mean on 7 of 10 benchmarks. On Texas it reaches 92.92 against
92.26 for the MLP, and on Chameleon 67.48 against 60.96 for GCN
[arxiv:2006.07988 tab. 2]. On Citeseer, APPNP is higher (68.59 against 67.63),
and on Texas and Cornell the MLP is close to the GNNs, which the paper reads as
weak topology information [arxiv:2006.07988 §5 tab. 2]. The homophilic
benchmarks use 2.5% of nodes for training, and the heterophilic ones use 60%
[arxiv:2006.07988 §5 app. A.9]. The paper's over-smoothing argument assumes
an optimiser with a decreasing learning rate, such as SGD, while its experiments
use Adam [arxiv:2006.07988 §4 app. A.4 app. A.9]. Card:
[Adaptive Universal Generalized PageRank Graph Neural Network](/papers/2006.07988).

BertGCN applies the graph idea to text classification. It builds one graph over
the whole corpus, with document and word nodes, and starts each document node
from a BERT-style embedding. BERT and a two-layer GCN are trained jointly, and
unlabelled test documents take part in message passing, so the method is
transductive [arxiv:2105.05727 §1 §3.1]. The whole graph does not fit through
BERT in one pass, so a memory bank holds one embedding per document and is
refreshed each epoch [arxiv:2105.05727 §3.3]. With BERT-base, the reported test
accuracy is 89.3 on 20NG and 72.8 on Ohsumed, against 85.3 and 70.5 for BERT
alone. The gains over BERT on R8, R52 and MR are 0.3 points or less, computed
from the table [arxiv:2105.05727 tab. 1]. Card:
[BertGCN: Transductive Text Classification by Combining GCN and BERT](/papers/2105.05727).

### Pseudo-labelling and self-training

The model labels unlabelled inputs and trains on its own labels. Disagreement
methods such as deep co-training and Tri-Net use two or more heads or views and
add a sample when the heads agree [arxiv:2103.00550 §6.1]. Self-training methods
take the most probable class as the target and weight the unlabelled loss with a
schedule α(t) [arxiv:2103.00550 §6.2]. The survey also lists entropy
minimisation, Noisy Student, Meta Pseudo Labels, S4L, EnAET and SimCLRv2 in this
family [arxiv:2103.00550 §6.2]. Card:
[A Survey on Deep Semi-supervised Learning](/papers/2103.00550).

### Hybrid methods

Hybrids combine consistency, pseudo-labels and Mixup-style interpolation.
The survey covers ICT, MixMatch, ReMixMatch, DivideMix and FixMatch. In
FixMatch, a pseudo-label from a weakly augmented view becomes the target for a
strongly augmented view, and it is used only when the model is confident
[arxiv:2103.00550 §7]. Card:
[A Survey on Deep Semi-supervised Learning](/papers/2103.00550).

### Class-wise thresholds (FlexMatch)

FlexMatch keeps the FixMatch loop and replaces its single fixed cut-off with one
threshold per class. Each class's threshold moves with the number of unlabelled
samples the model currently assigns to that class [arxiv:2110.08263 §3.1]. A
class with few such samples gets a lower threshold and admits more pseudo-labels.
The threshold is T(c) = M(β(c)) · τ, where β(c) is the class count divided by
the largest count and M(x) = x / (2 − x) maps β back onto [0, 1]
[arxiv:2110.08263 §3.3 eq. 12]. The paper uses τ = 0.95 [arxiv:2110.08263 tab. 3].
Early in training most unlabelled samples are unassigned, so the denominator is
the larger of the largest count and the unassigned count [arxiv:2110.08263 §3.2
eq. 11]. Each unlabelled sample stores its last class above τ, so the counts
come from state the loop already keeps and no extra forward pass is needed
[arxiv:2110.08263 §3.1 alg. 1].

Author-reported error, on CIFAR-100 with 400 labels: 39.94 against 46.42 for
FixMatch. On STL-10 with 40 labels: 29.15 against 35.97 [arxiv:2110.08263 tab. 1].
The gain is not uniform. At 250 labels on CIFAR-10, FixMatch is slightly better
(4.86 against 4.98). On SVHN with 40 labels, FlexMatch is worse (8.19 against
3.81), and the authors attribute this to class imbalance: rare classes never
reach β close to 1, so their thresholds stay low [arxiv:2110.08263 tab. 1 §4.1]. The same per-class rule applied to
Pseudo-Labeling and UDA lowers error in most cells of the table
[arxiv:2110.08263 §4.1 tab. 1]. On ImageNet with 100
labels per class, top-1 error is 41.85 against 43.66 for FixMatch after 2^20
steps, and the authors say the models had not fully converged
[arxiv:2110.08263 §4.2 tab. 2]. Card:
[FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](/papers/2110.08263).

### Self-adaptive thresholds (FreeMatch)

FreeMatch keeps the pseudo-labelling loop and sets the threshold from the
model's own statistics. A global threshold follows an exponential moving average
(EMA) of the mean top confidence on weakly augmented unlabelled images. Each
class threshold is that global value scaled by the max-normalised EMA of the
class probabilities [arxiv:2205.07246 §4.1 eq. 5–7]. A fairness term, added
with its own weight, compares the normalised class-prediction statistics with a
target, to keep predictions spread across classes [arxiv:2205.07246 §4.2 eq. 11–12].
The paper's ablation on CIFAR-10 with 10 labels gives 8.07 with this term and
10.37 without it [arxiv:2205.07246 tab. 4]. The fairness weight is 0.01 or 0.05
depending on the setting [arxiv:2205.07246 app. D tab. 5].

Author-reported error, with 3 seeds and the best checkpoint of each run
[arxiv:2205.07246 §5.1]. On CIFAR-10 with 10 labels, 8.07 against 13.85 for
FlexMatch. On STL-10 with 40 labels, 15.56 against 29.15 [arxiv:2205.07246
tab. 1]. The gap is small with more labels: on CIFAR-10 with 4000 labels it is
4.10 against 4.19 [arxiv:2205.07246 tab. 1]. Several FlexMatch cells in this table
match the FlexMatch card's (for example 39.94, 29.15 and 8.19), but not all were
checked. The 10-label
standard deviations are wide (4.24 for FreeMatch, 12.04 for FlexMatch), so read
the ranking with care [arxiv:2205.07246 tab. 1]. The paper gives the extra cost
over FixMatch as very small, and its ImageNet run has the same per-iteration time
as FixMatch [arxiv:2205.07246 app. C tab. 2]. Card:
[FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](/papers/2205.07246).

### Entropy-filtered pseudo-labels with negative mining (dense prediction)

For semantic segmentation, the teacher's per-pixel entropy splits unlabelled
pixels into reliable and unreliable. Reliable pixels get their argmax class as
the pseudo-label. The unreliable share starts at 20% and shrinks linearly to 0
over training [arxiv:2203.03884 §3.2 eq. 5–7]. Unreliable pixels are not
discarded. They are kept as negatives for classes they are unlikely to belong to,
in one first-in-first-out queue per class, and a pixel-level InfoNCE loss with
temperature 0.5 uses them [arxiv:2203.03884 §3.3 eq. 4, 14–15]. The method is
called U2PL in the paper. Card:
[Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](/papers/2203.03884).

Reported numbers are author-reported, on PASCAL VOC 2012 with DeepLabv3+ and a
ResNet-101 encoder. With 1/4 of the blender-split labels, removing the contrastive
loss drops mIoU from 79.30 to 73.02 [arxiv:2203.03884 tab. 6]. On the classic
split with 1/16 labels, the paper reports 67.98 mIoU against 45.77 for the
supervised-only baseline [arxiv:2203.03884 tab. 1]. Before implementing, read the
card's "Paper vs code" section: the official code differs from the paper in the
contrastive weight, the anchor and negative counts, the anchor definition and the
Cityscapes warm-up [arxiv:2203.03884 §3.3, app. A].

### Scan mixing with a mean teacher (LiDAR segmentation)

LaserMix works on LiDAR scans. It splits each scan into bands by beam
inclination, then gives alternate bands to two mixed scans, so each band's
neighbours come from the other scan [arxiv:2405.05258 §3.2 eq. 6–7]. Labels
are mixed the same way. A student and an exponential-moving-average (EMA)
teacher train on the mixed scans, and the teacher's confident predictions on
unlabelled scans supply targets [arxiv:2405.05258 §3.3]. The paper's stated
objective is motivated by entropy minimisation [arxiv:2405.05258 §3.1 eq. 5],
but the released losses are cross-entropy, a mean-teacher term and a mixing
term, with no entropy term [arxiv:2405.05258 §3.3 eq. 8–9]. LaserMix++ adds
camera parts: pixels matched to points, mixing on (point, pixel) pairs,
distillation of image features onto LiDAR features, and a text-guided term
that uses CLIP class embeddings [arxiv:2405.05258 §4.1–4.5 eq. 10–16]. The
camera parts are specified only by the equations. The released code has the
LaserMix part and no camera branch [arxiv:2405.05258 §5.2] [code:
README.md#L177].

Author-reported, with Cylinder3D and mIoU. At 1% labels, LaserMix++ reaches
56.2 on SemanticKITTI, against 45.4 for supervised-only and 50.6 for LaserMix.
At the same label fraction it reaches 58.5 on nuScenes (55.3 LaserMix) and
47.3 on ScribbleKITTI (44.2 LaserMix) [arxiv:2405.05258 tab. 2]. The gain of
LaserMix++ over LaserMix at 1% is 3.1 to 5.6 points across these three
datasets, computed from the table. The paper's gain over LaserMix is 2% to
3% on average over its settings [arxiv:2405.05258 §5.3]. The paper claims
accuracy comparable to full supervision at 2× to 5× fewer annotations
[arxiv:2405.05258 §5.3]. No seeds or variance are reported
[arxiv:2405.05258 Pitfalls §5.4].

The released configs differ from the text. The SemanticKITTI config uses a
mixing weight of 1 where the paper uses 2.0, and it samples 4 to 8 areas
where the paper samples 2 to 6 [arxiv:2405.05258 §5.2]
[code: configs/lasermix/lasermix_cy3d_semi_semantickitti_10.py#L133–L134].
Pseudo-labels below the threshold get an ignore index in the code, and the
text does not say what happens to them [arxiv:2405.05258 §3.3]
[code: mmdet3d/models/segmentors/semi_base.py#L139–L141]. Decide which values
to use before comparing with the reported numbers. Card: [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](/papers/2405.05258).

### Teacher-student detection (Unbiased Teacher)

The teacher-student idea applied to Faster R-CNN with FPN and a ResNet-50
backbone. A burn-in stage trains on labelled images only. After that, an EMA
teacher labels weakly augmented unlabelled images, and the student learns from
strongly augmented views of the same images [arxiv:2102.09480 §3.1–3.2].
Pseudo-boxes are cleaned with class-wise NMS and kept only above a confidence
of 0.7 [arxiv:2102.09480 §3.2 tab. 5]. Box-regression losses are not applied to
pseudo-boxes; only the RPN and ROI classification losses use them [arxiv:2102.09480
§3.2]. The ROI classifier uses focal loss instead of cross-entropy, because the
authors argue that pseudo-labels are biased towards background and frequent
classes. The paper does not state the focal exponent [arxiv:2102.09480 §3.3].
Card: [Unbiased Teacher for Semi-Supervised Object Detection](/papers/2102.09480).

Author-reported, on COCO with Faster R-CNN. With 1% of the labels, AP50:95 is
20.75 against 9.05 supervised and 13.97 for STAC [arxiv:2102.09480 tab. 1]. The
gain holds across label fractions, from 16.94 against 6.83 at 0.5% to 31.50
against 23.86 at 10% [arxiv:2102.09480 tab. 1]. The ablation on the 1% split
shows that both the focal loss and the EMA teacher add to the result, from 13.42
with neither to 21.19 with both [arxiv:2102.09480 fig. 5]. The ablation's 21.19
and the headline 20.75 differ for the same setting, and the paper does not
explain the gap [arxiv:2102.09480 tab. 1 fig. 5]. Training is long: 180k
iterations with 32 labelled and 32 unlabelled images per step
[arxiv:2102.09480 app. "Training"].

### Open-world clustering (generalized category discovery)

The unlabelled set may mix images of labelled classes with images of classes
that have no labels, and the number of classes is not given
[arxiv:2201.02609 §1 §3]. The method avoids a classification head. It starts from
a ViT-B/16 with self-supervised DINO weights, fine-tunes with a noise-contrastive
loss on all images and a supervised contrastive loss on labelled images, mixed
with λ = 0.35 [arxiv:2201.02609 §3.1.1 eq. 1–3 §4.1]. Labels are then assigned by
semi-supervised k-means on the [CLS] features: labelled images keep the cluster
of their class, and unlabelled images go to the nearest centroid
[arxiv:2201.02609 §3.1.2]. The number of clusters k is found by searching for the
value that maximises accuracy on the labelled images, using Brent's algorithm on
[|Y_L|, 1000] [arxiv:2201.02609 §3.2 §4.1]. Card:
[Generalized Category Discovery](/papers/2201.02609).

Author-reported clustering accuracy over all unlabelled images: 91.5 on CIFAR10,
73.0 on CIFAR100 and 74.1 on ImageNet-100 [arxiv:2201.02609 tab. 2]; 51.3 on CUB,
39.0 on Stanford Cars and 35.4 on Herbarium19 [arxiv:2201.02609 tab. 3]. Plain
k-means on the DINO features reaches 83.6 on CIFAR10 and 52.0 on CIFAR100
[arxiv:2201.02609 tab. 2]. On CIFAR100, the ablation attributes the rise from
52.0 to 73.0 to the contrastive fine-tuning and the semi-supervised k-means
[arxiv:2201.02609 tab. 5]. Read the numbers with three cautions. The baselines
are adapted by the authors [arxiv:2201.02609 §3.3]. The Hungarian matching is
computed over all unlabelled images, so the numbers are not directly comparable
with novel-category-discovery papers [arxiv:2201.02609 app. E]. The paper's
headline gain on the generic datasets does not follow from its own table
[arxiv:2201.02609 §4.2 tab. 2]. No variance is reported.

### Silver pairs from a cross-encoder (sentence similarity)

A slow, accurate teacher labels unlabelled sentence pairs, and a fast bi-encoder
trains on those labels as soft targets. The cross-encoder is trained on the
labelled pairs, scores new pairs built from the unlabelled sentences, and the
bi-encoder trains on the labelled plus the scored pairs [arxiv:2010.08240 §3.1].
Which pairs are scored decides the result. The paper compares random sampling,
kernel-density matching, BM25 lexical neighbours, semantic-search neighbours and
their union [arxiv:2010.08240 §3.1]. Random pairs are mostly negatives. On
in-domain Spanish-STS, random sampling gives 62.05 against 72.07 for the
bi-encoder trained with no silver pairs [arxiv:2010.08240 tab. 5]. BM25 reaches
75.08 on the same task with 3,964 silver pairs, against 230,364 for KDE at 74.67
[arxiv:2010.08240 tab. 11]. The paper reports BM25 as the best strategy on
average at the smallest silver-set size [arxiv:2010.08240 §6.1].

For domain adaptation, the cross-encoder is trained on a labelled source domain
and labels unlabelled target pairs; no target labels are used
[arxiv:2010.08240 §3.2]. On Quora → Sprint, AUC(0.05) rises from 0.505 with a
source-only bi-encoder to 0.875. When Quora is the target, the same method is
level with the source-only baseline, 0.495 to 0.496 [arxiv:2010.08240 tab. 6].
The per-pair teacher pass runs during training only. The paper states that the
trained bi-encoder runs at the same speed as SBERT [arxiv:2010.08240 §5]. Card:
[Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](/papers/2010.08240).

### Pretraining on unlabelled tabular rows (TabTransformer)

Rows with categorical columns can also supply unlabelled data. TabTransformer
pretrains its attention layers on unlabelled rows and then fine-tunes them with
the supervised loss on the labelled rows [arxiv:2012.06678 §2]. The masked
objective (MLM) sets 30% of a row's categorical values to a missing-value
embedding and predicts each original class [arxiv:2012.06678 §2]. The
replaced-value objective (RTD) swaps those values for a random value from the
same column and asks, one head per column, whether each value was replaced
[arxiv:2012.06678 §2 app. A]. The text does not say how the random value is
drawn. The ablation finds that a fresh draw each epoch beats a fixed one: on
three datasets with 256 labels, relative AUC is +0.81 against −0.49
[arxiv:2012.06678 app. A]. The paper's ablation also finds 15% and 50% replacement
rates similar to the 30% used [arxiv:2012.06678 §2 app. A]. Pretraining epochs,
batch size and learning rate are not stated [arxiv:2012.06678 app. B.1].

Author-reported, as mean AUC. On the eight datasets with more than 30K rows, the
better of RTD and MLM beats the best other baseline (MLP (ER) in the table) at
50, 200 and 500 labels, by at least 1.2, 2.0 and 2.1 points
[arxiv:2012.06678 tab. 3 §3]. On the twelve smaller datasets, the best other
baseline is ahead at 50 labels (79.4 against 78.6 for RTD). RTD leads at 200 and
500 labels (81.6 against 81.2, and 83.4 against 82.3) [arxiv:2012.06678 tab. 4].
The paper gives as the reason for the 50-label loss that the fine-tuned head sees
only labelled rows [arxiv:2012.06678 §3.4]. Card:
[TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678).

### Auxiliary tasks with conflicting gradients (CAGrad)

The semi-supervised setting here is a multi-task one: the base method, ARML,
trains with auxiliary tasks alongside its main loss [arxiv:2110.14048 §5.4].
The task gradients can conflict, and CAGrad combines
them into one update. It starts from the average gradient and moves towards the
direction that most improves the worst task, within a ball of radius c around
the average [arxiv:2110.14048 §3.1]. The paper's semi-supervised runs fix c at
0.1 [arxiv:2110.14048 app. B.4]. On CIFAR-10, with 500, 1000 and 2000 labels,
ARML with CAGrad reaches 68.25, 74.37 and 82.81 average test accuracy. ARML alone
reaches 67.05, 73.22 and 81.35, and ARML with PCGrad reaches 67.49, 73.23 and
81.91 [arxiv:2110.14048 tab. 4]. These are author-reported means over three runs,
with gaps of about 1.2 to 1.5 points. The paper does not give the auxiliary
weights and the repository has no code for this experiment
[arxiv:2110.14048 §5.4 app. B.4]. Card:
[Conflict-Averse Gradient Descent for Multi-task Learning](/papers/2110.14048).

### Pretext tasks on unlabelled sensor signals (wearable HAR)

A network is pre-trained on unlabelled accelerometer windows to predict which
signal transform was applied to each window. The pretext task is multi-task:
one head per transform, all on a shared 1-D CNN trunk [arxiv:1907.11879 §3.1
§3.3]. The eight transforms are noise, scaling, sign inversion, negation within a
time window, time reversal, segment permutation, time warp and channel
shuffle. Their parameters are not stated [arxiv:1907.11879 §3.2]. The conv trunk is then copied into an
activity classifier, either frozen or with its last conv block fine-tuned
[arxiv:1907.11879 §3.1]. The semi-supervised variant pre-trains on the full
unlabelled target set and trains the classifier on 2 to 100 labelled windows
per class [arxiv:1907.11879 §4.3.4]. The paper compares this with an
autoencoder trained on the same windows and with end-to-end supervised
training [arxiv:1907.11879 §4.3.2].

Author-reported, as Cohen's kappa over 10 runs. With the trunk frozen, the
pretext features give 0.7310 on HHAR, 0.7642 on UniMiB and 0.8803 on UCI HAR,
against 0.6816, 0.7815 and 0.8789 for supervised training
[arxiv:1907.11879 tab. 2]. On UniMiB, WISDM and MotionSense the frozen
features trail supervised training, and under 5-fold user-split
cross-validation they trail on five of six datasets [arxiv:1907.11879 tab. 2
tab. 4]. The per-layer analysis on HHAR favours the second conv block (0.774)
over the last (0.679), but the main table uses the last block
[arxiv:1907.11879 §4.3.1]. The paper also reports the cost as equal to
supervised learning without giving runtime figures [arxiv:1907.11879 §3.1].
Card: [Multi-task Self-Supervised Learning for Human Activity Detection](/papers/1907.11879).

### Pathology: few labels in whole-slide images

A mini-review of machine learning for histopathology lists semi-supervised
learning as one answer to scarce labels, next to active learning, multiple
instance learning and transfer learning. It describes the method as using
unlabelled data to estimate the label distribution [arxiv:1709.00786 §4.2.2].
The review reports no experiments of its own [arxiv:1709.00786 §1], so use it
as a map of options rather than as evidence for any one method. Card:
[Machine learning methods for histopathological image analysis](/papers/1709.00786).

### Logic rules as the unlabelled signal (Logic Tensor Networks)

Logic Tensor Networks (LTN) score a knowledge base of first-order formulas
with real-valued truth values, and training maximises that satisfiability score
[arxiv:2012.13635 §3.2]. The paper does not describe its experiments as
semi-supervised learning. The reading here is ours, based on the card's concept
tags: several examples train with labels for only part of the data, and the
rules cover the rest. In the smokes example, all 14 individuals have known
smoking facts, but cancer facts are known only for the first group. Rules on
friendship and on smoking spread information to the other group
[arxiv:2012.13635 §4.7]. In the multi-label example, the only labels are
positive ones, with no negative examples. Mutual exclusion of blue and orange,
and of male and female, is written as rules [arxiv:2012.13635 §4.3]. The
clustering example uses no class labels. Its axioms ask every point to belong
to a cluster and nearby points to share one [arxiv:2012.13635 §4.6]. The card
records a final constraint satisfaction of 0.857 for that run, with no baseline
[arxiv:2012.13635 §4.6]. The smokes and multi-label results are qualitative
[arxiv:2012.13635 §4.3 §4.7]. Card:
[Logic Tensor Networks](/papers/2012.13635).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Classification with no GAN or VAE training budget | Consistency regularization (Π Model or Mean Teacher) | Only extra forward passes or an EMA copy are needed. Π Model doubles forward passes per step; Mean Teacher stores a second set of weights [arxiv:2103.00550 §4 eq. 17, 19] |
| Memory is tight and targets may lag a little | Temporal Ensembling | One forward pass per sample. Targets are refreshed once per epoch, so they trail the model [arxiv:2103.00550 §4 eq. 18] |
| Input has explicit edges, such as a citation or social graph | Graph-based embeddings (GCN, GAT, GraphSAGE) | Labels spread along edges. Most methods are transductive [arxiv:2103.00550 §2, §5] |
| Node classification where the graph may be heterophilic, or you do not know how far labels should spread | GPR-GNN (learned propagation weights, K = 10) | Weights may be negative, so the filter can keep high-frequency signal. Best mean on 7 of 10 benchmarks, author-reported. The MLP is close on Texas and Cornell, and APPNP is higher on Citeseer [arxiv:2006.07988 §4 §5 tab. 2]. The over-smoothing argument assumes SGD, but the experiments use Adam [arxiv:2006.07988 app. A.4 app. A.9] |
| Node classification where the useful signal may sit two hops out, and wider layers are affordable | MixHop (adjacency powers 0, 1, 2) | Each power has its own weights, so the layer can learn differences between neighbourhood distances [arxiv:1905.00067 §3.1 §3.3]. Gains over GCN on the Planetoid split are 0.4 to 1.8 points, and the GCN baseline has no spread reported [arxiv:1905.00067 tab. 1] |
| Dense per-pixel labels, few labelled images | Entropy-filtered pseudo-labels with negative mining (U2PL) | Uses the unreliable pixels that a confidence-only filter discards. Costs extra memory queues and a contrastive pass [arxiv:2203.03884 §3.3, app. A] |
| LiDAR semantic segmentation with few labelled scans and a LiDAR-only pipeline | Scan mixing with a mean teacher (LaserMix) | The released code implements this part. Author-reported 50.6 against 45.4 supervised on SemanticKITTI at 1% labels [arxiv:2405.05258 tab. 2]. Band edges and loss weights differ between paper and configs [arxiv:2405.05258 §5.2 Paper vs code] |
| LiDAR segmentation with paired camera images and a camera-to-LiDAR feature loss | LaserMix++ | Author-reported gains of 3.1 to 5.6 points over LaserMix at 1% labels. The camera parts have no released code and must be written from the equations [arxiv:2405.05258 tab. 2 §4.3–4.4] |
| Wearable accelerometer data, many unlabelled windows, a few labels per class | Multi-task pretext pre-training on transformed windows (TPN) | Author-reported kappa close to supervised training on some datasets, with a frozen trunk. Trails supervised on UniMiB, WISDM and MotionSense [arxiv:1907.11879 tab. 2]. Transform parameters and task weights are not stated [arxiv:1907.11879 §3.2 alg. 1] |
| Object detection with a few labelled boxes and many unlabelled photos from the same domain | Teacher-student detection (Unbiased Teacher) | Reported gains from 0.5% to 10% COCO labels. Needs a labelled burn-in, an EMA teacher and a long schedule [arxiv:2102.09480 tab. 1, app. "Training"] |
| Strong data augmentation is available and the labelled set is small | Hybrid methods (FixMatch) | FixMatch's unlabelled branch pairs a confident weak-view pseudo-label with a strong-view target [arxiv:2103.00550 §7] |
| Same setting as FixMatch, with classes of unequal difficulty and roughly balanced sizes | Class-wise thresholds (FlexMatch) | Adds one stored class per unlabelled sample and no forward pass [arxiv:2110.08263 §3.1 alg. 1]. CIFAR-100 with 400 labels: 39.94 against 46.42 error for FixMatch [arxiv:2110.08263 tab. 1]. Avoid when classes are imbalanced: on SVHN with 40 labels it is worse, 8.19 against 3.81 [arxiv:2110.08263 tab. 1 §4.1] |
| Sentence-pair scoring where the pairwise model is too slow for all pairs, and unlabelled sentences exist | Silver pairs from a cross-encoder with BM25 sampling (Augmented SBERT) | The teacher runs only on sampled pairs, and the bi-encoder keeps its serving cost. Random pair sampling hurts [arxiv:2010.08240 §3.1 §5 tab. 5] |
| Target domain has unlabelled text, source domain has labels | Cross-encoder labels on target pairs (AugSBERT domain adaptation) | Gains depend on the pair of domains: large on Quora → Sprint, none on Quora as target [arxiv:2010.08240 tab. 6] |
| Text classification over a corpus that includes unlabelled test documents, with a pretrained encoder | Document graph with BERT nodes (BertGCN) | Labels spread through word and document edges. Needs the whole corpus at training time and a memory bank refreshed each epoch [arxiv:2105.05727 §3.3] |
| Tabular data with categorical columns, many unlabelled rows and 50 to 500 labels | TabTransformer pretraining (RTD or MLM), then fine-tuning | The better objective beats the best other baseline on the eight larger datasets by at least 1.2, 2.0 and 2.1 points at 50, 200 and 500 labels [arxiv:2012.06678 tab. 3]. On smaller datasets an MLP (ER) is ahead at 50 labels [arxiv:2012.06678 tab. 4]. Compare with GBDT too, since the supervised model is level with it, 82.8 against 82.9 [arxiv:2012.06678 tab. 2] |
| Number of classes unknown, and unlabelled data may contain classes with no labels | Semi-supervised k-means on contrastive features (generalized category discovery) | Labels every unlabelled image without a head. The class count is estimated from labelled accuracy, so some ground-truth labels are needed [arxiv:2201.02609 §3.2 §4.1] |
| Want a map of visual SSL methods before choosing one, or a view of segmentation, detection and domain adaptation | Read the visual survey first | It sorts methods by smoothness rationale and summarises applications [arxiv:2208.11296 §4.1, §4.2]. It gives no numbers, so take results from the original papers [arxiv:2208.11296 §1] |
| Semi-supervised training already uses auxiliary tasks, and their gradients may conflict | Gradient combination with CAGrad (c = 0.1 in the paper's runs), on top of the auxiliary-task method | Author-reported gains of about 1.2 to 1.5 points over ARML on CIFAR-10 at 500 to 2000 labels, over three runs. Costs one backward pass per task per step [arxiv:2110.14048 tab. 4 §3.3] |
| Pseudo-labelling with very few labels (10 to 40), where a hand-picked fixed threshold is hard to tune | Self-adaptive thresholds (FreeMatch) | Threshold comes from the model's EMA confidence and class statistics, plus a fairness term. Author-reported error 8.07 against 13.85 for FlexMatch on CIFAR-10 with 10 labels, with wide spread [arxiv:2205.07246 tab. 1]. Advantage shrinks with more labels [arxiv:2205.07246 tab. 1] |
| Node or point labelling with a graph, and the final classifier should be a kernel SVM with a convex inference step | Max-margin graph cuts | Lower error than manifold regularization of SVMs in 29 of 36 UCI settings, author-reported [arxiv:2604.26818 fig. 4]. The SVM scale is not stated, so check one table row before using the γ value [arxiv:2604.26818 §3 §6.2] |
| Some facts about the data are known as rules, and labels cover only part of the data or lack negatives | Logic Tensor Networks (rules as a differentiable satisfiability loss) | Rules spread information to unlabelled items and supply exclusions. The paper's only baseline comparison is MNIST addition, where it reports 98.04 against 97.20 for DeepProbLog, author-reported [arxiv:2012.13635 tab. 2]. The rules must be written as fuzzy formulas, and the operators, the aggregation exponent and the stability constant need choosing [arxiv:2012.13635 §2.4 §4] |
| Unlabelled data may not match the labelled data, or classes are imbalanced | Any of the above, with a supervised-only baseline | Unlabelled data can hurt, and most methods assume balanced classes [arxiv:2103.00550 §2.1, §8] |

The survey does not rank methods, because it reports no accuracy numbers
[arxiv:2103.00550 §8]. Choose a variant by constraints, then check it against a
supervised baseline on your own data.

## Implementation sketch

A generic step that the variants share. Lines marked with a citation name the
choice that differs between them. The dense variant adds a contrastive term over
the unreliable pixels [arxiv:2203.03884 §3.3].

```pseudocode
# One semi-supervised training step, shared core of the variants
# Input: labelled batch (x_l, y_l), unlabelled batch x_u, step t of T
# Output: total loss

def ssl_step(student, teacher, x_l, y_l, x_u, t, T):
    # Supervised term on labelled data  [arxiv:2103.00550 §2 eq. 1]
    loss_s = cross_entropy(student(x_l), y_l)

    # Targets come from the model itself, with no gradient  [arxiv:2103.00550 §4]
    # Teacher is an EMA of the student, a past average, or the same net on
    # a perturbed view; the choice is the variant  [arxiv:2103.00550 tab. 3]
    with no_grad():
        probs_u = softmax(teacher(x_u))

    # Keep only the confident share of targets  [arxiv:2203.03884 eq. 5-7]
    # alpha_t shrinks linearly; U2PL starts at 0.2
    # A per-class cut-off on the confidence (FlexMatch) replaces this rule
    # [arxiv:2110.08263 §3.3]
    alpha = 0.2 * (1 - t / T)
    entropy = -sum(probs_u * log(probs_u), over_classes)
    gamma = percentile(entropy, 100 * (1 - alpha))
    reliable = entropy < gamma

    # Argmax pseudo-labels on reliable entries, ignore the rest  [eq. 6]
    pseudo_u = where(reliable, argmax(probs_u), IGNORE)

    # Pseudo-label loss over kept entries  [arxiv:2203.03884 eq. 3]
    # Guard against a batch with no kept entries
    loss_u = cross_entropy(student(x_u), pseudo_u, ignore=IGNORE)

    # Weight of the unsupervised term  [arxiv:2103.00550 eq. 1]
    # Not stated in the survey; U2PL normalises by the kept count [eq. 8]
    return loss_s + lambda_u * loss_u
```

## Contradictions

- **Unlabelled data helping or hurting.** The survey warns that unlabelled data
  can reduce accuracy below the supervised baseline when the data assumptions
  fail [arxiv:2103.00550 §2.1, §8]. U2PL reports gains over its supervised-only
  baseline in every row of its three result tables [arxiv:2203.03884 tab. 1–3].
  These are different tasks and the U2PL numbers are author-reported against
  baselines the authors re-implemented [arxiv:2203.03884 §4.2]. Both statements
  stand; neither settles the other.
- **Discarding or keeping unreliable predictions.** Unbiased Teacher drops
  detections below confidence 0.7 and does not reuse them [arxiv:2102.09480
  §3.2]. U2PL drops unreliable pixels from the pseudo-label loss but reuses them
  as negatives [arxiv:2203.03884 §3.3]. The two papers work on different tasks
  and neither compares against the other, so the cards do not show which choice
  is better.
- **Taxonomy placement.** The two surveys cut the field differently. The
  deep-SSL survey puts MixMatch and FixMatch under hybrid methods
  [arxiv:2103.00550 §7]. The visual survey lists MixMatch and ReMixMatch under two
  families and FixMatch under consistency and self-training, and says the grouping
  overlaps [arxiv:2208.11296 tab. 1, §2.2.1–2.2.2]. Neither places the methods
  wrongly; they answer different questions. The visual survey also notes that
  its names for the smoothness assumptions overlap [arxiv:2208.11296 §2.1, §2.2.2].
- **Cityscapes text and table.** The U2PL text describes notable margins on
  Cityscapes [arxiv:2203.03884 §4.2]. Its table shows the CutMix variant below
  AEL at 1/16, 1/8 and 1/4 of the labels. Only the AEL variant beats AEL at every
  fraction [arxiv:2203.03884 tab. 3].
- **Class-wise thresholds on SVHN with 40 labels.** The FlexMatch paper finds its
  per-class rule worse than FixMatch on SVHN with 40 labels, and attributes this
  to class imbalance [arxiv:2110.08263 tab. 1 §4.1]. FreeMatch also scales its
  threshold per class, and reports 1.97 on SVHN with 40 labels, against 3.81 for
  FixMatch and 8.19 for FlexMatch [arxiv:2205.07246 tab. 1]. The FreeMatch paper
  does not show which part of its design accounts for the gap, so the rule
  "avoid per-class thresholds under imbalance" does not hold without checking.
- **Fixed or class-wise cut-offs.** FlexMatch argues that one fixed cut-off wastes
  unlabelled data early in training and treats all classes alike
  [arxiv:2110.08263 §1]. Unbiased Teacher keeps pseudo-boxes above a fixed 0.7
  [arxiv:2102.09480 §3.2]. The two papers work on different tasks and do not
  compare with each other, so the cards do not show which rule suits detection.
- **GCN baselines across splits.** GCN on Cora scores 81.5 in MixHop's Planetoid
  table [arxiv:1905.00067 tab. 1] and 75.21 in GPR-GNN's table, where 2.5% of the
  nodes are labelled [arxiv:2006.07988 tab. 2 app. A.9]. The splits differ, so
  the two numbers should not be compared.
- **Optimiser behind the over-smoothing argument.** GPR-GNN's argument assumes
  SGD with a decreasing learning rate [arxiv:2006.07988 §4 app. A.4], while its
  experiments use Adam [arxiv:2006.07988 app. A.9]. The cards do not show that
  the argument carries over to Adam.
- **TabTransformer's dataset counts.** The abstract and §3 say 15 datasets, but
  the semi-supervised tables cover 20, and the text and table captions give
  different counts for the groups [arxiv:2012.06678 §3 tab. 3 tab. 4 app. C].
  Treat the 15-dataset supervised set and the 20-dataset semi-supervised set as
  separate benchmarks.

## Papers

- [A Survey on Deep Semi-supervised Learning](/papers/2103.00550): five-family taxonomy of deep SSL, with one loss notation and a table of consistency methods.
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](/papers/2203.03884): U2PL, which keeps unreliable pixels as contrastive negatives in dense prediction.
- [Unbiased Teacher for Semi-Supervised Object Detection](/papers/2102.09480): EMA teacher pseudo-boxes for Faster R-CNN, with focal loss on the ROI classifier.
- [Generalized Category Discovery](/papers/2201.02609): clustering with an unknown class count over mixed known and new classes, with labelled images pinned to their classes.
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](/papers/2010.08240): cross-encoder soft labels on sampled sentence pairs, with the sampling strategy as the key choice.
- [BertGCN: Transductive Text Classification by Combining GCN and BERT](/papers/2105.05727): BERT-initialised document nodes in a word-document graph, trained jointly with a memory bank.
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](/papers/2208.11296): visual SSL and UL taxonomy around three smoothness rationales, with applications to segmentation, detection and domain adaptation; no experiments.
- [MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing](/papers/1905.00067): concatenated propagation over adjacency powers, with per-power weights.
- [Adaptive Universal Generalized PageRank Graph Neural Network](/papers/2006.07988): learned propagation weights that can be negative, for homophilic and heterophilic graphs.
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](/papers/2012.06678): masked and replaced-value pretraining of column attention on unlabelled tabular rows.
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](/papers/2110.08263): per-class confidence thresholds on FixMatch, moved by how many samples each class has.
- [Conflict-Averse Gradient Descent for Multi-task Learning](/papers/2110.14048): CAGrad, a gradient-combination step that protects the worst task, with one semi-supervised CIFAR-10 experiment on ARML.
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](/papers/2205.07246): global and per-class thresholds from the model's own EMA statistics, with a fairness term on class predictions.
- [Semi-supervised learning with max-margin graph cuts](/papers/2604.26818): regularized harmonic labels on a k-NN graph, then a kernel hinge-loss SVM trained on confident points only.
- [Machine learning methods for histopathological image analysis](/papers/1709.00786): mini-review of histopathology ML, listing semi-supervised learning among remedies for scarce labels; no experiments.
- [Logic Tensor Networks](/papers/2012.13635): first-order rules as a differentiable satisfiability objective, with partial labels in several examples.
- [Multi-task Self-Supervised Learning for Human Activity Detection](/papers/1907.11879): transform-prediction pretext tasks on unlabelled accelerometer windows, then a classifier on the frozen or partly tuned trunk.
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](/papers/2405.05258): LaserMix band swapping with a mean teacher for LiDAR, and LaserMix++ camera and language terms, whose camera parts have no released code.