Semi-supervised learning

Overview

Labels are expensive, unlabelled inputs are cheap. Semi-supervised learning (SSL) trains on a small labelled set L and a much larger unlabelled set U, usually with |L| much smaller than |U| and K classes 2103.00550 §2. The training objective adds three terms: a supervised loss on labelled data, an unsupervised loss on unlabelled data, and a regularizer, each with a trade-off weight 2103.00550 §2 eq. 1. Most deep methods differ in how they build the unsupervised part.

The survey's central caveat is that the gain depends on assumptions about the data: self-training needs confident predictions to be correct, co-training needs two views, and the cluster, manifold and low-density-separation assumptions say where the decision boundary may lie. When these fail, unlabelled data can leave accuracy unchanged or lower it 2103.00550 §2.1, §8. Keep a supervised-only baseline in every experiment. The survey itself covers image classification 2103.00550 §1; the later variants below apply the same ideas to segmentation, detection, clustering and text.

A later visual survey groups the same methods differently. It names five SSL families (consistency regularization, self-training, graph regularization, generative models and self-supervision) and argues that most of them rest on one of three smoothness rationales: local smoothness across views, local smoothness across neighbours, or global smoothness from the decision boundary 2208.11296 §2.2, §4.1, tab. 1, tab. 3. That survey reports no experiments of its own, so its taxonomy is a map, not a ranking 2208.11296 §1. Card: Semi-Supervised and Unsupervised Deep Visual Learning: A Survey.

Variants

Consistency regularization

The unlabelled input is perturbed in some way, and the prediction for the perturbed view is pulled towards a target from the model itself. The survey tabulates ten such methods by what they perturb and where the target comes from: Ladder Network, Π Model, Temporal Ensembling, Mean Teacher, VAT, Dual Student, SWA, VAdD, UDA and WCP 2103.00550 §4 tab. 3. Details live on the consistency regularization page. Cost differs by method: the Π Model runs each sample forward twice per step, Temporal Ensembling runs once but keeps a per-sample running target that is refreshed once per epoch, and Mean Teacher keeps an exponential moving average (EMA) of the weights as its teacher 2103.00550 §4 eq. 17–19. The visual survey splits the same idea by where the perturbation happens: on the input (random augmentation, VAT, MixUp-style mixing, automated augmentation) or in the model (stochastic weights or structure, and weight or prediction ensembles) 2208.11296 §2.2.1. Cards: A Survey on Deep Semi-supervised Learning and Semi-Supervised and Unsupervised Deep Visual Learning: A Survey.

Generative models

Semi-supervised GANs reuse discriminator features, regularise a classifier with generated samples, learn an inference model, or add generated samples to the training data 2103.00550 §3.1. Semi-supervised VAEs extend the variational bound with a label variable for unlabelled data, in models named M1, M2, M1+M2, ADGM and others 2103.00550 §3.2. Card: A Survey on Deep Semi-supervised Learning.

Graph-based methods

Nodes are samples, edges are similarities, and each node gets an embedding that is trained with a reconstruction or a classification term. The survey covers autoencoder embeddings (GAE, VGAE, SDNE, DNGR) and GNN embeddings (GCN, GAT, GraphSAGE, GGNN) 2103.00550 §5. Most graph methods are transductive: they predict only the nodes seen during training 2103.00550 §2. Card: A Survey on Deep Semi-supervised Learning.

Two node-classification papers change what a propagation layer computes, and both are evaluated in the semi-supervised setting. MixHop replaces the single neighbourhood average of a GCN layer with a concatenation over adjacency powers j in a set P, each block with its own weights 1905.00067 §3.1 eq. 4. With P = {1} the layer reduces to the GCN layer 1905.00067 §3.1. The authors prove that a vanilla GCN cannot represent a two-hop difference operator, while MixHop with P = {0, 1, 2} can 1905.00067 §3.3. That is a result about what the layer can express, not about accuracy. On the Planetoid split (20 labelled nodes per class), the learned architecture reaches 71.4, 81.9 and 80.8 on Citeseer, Cora and Pubmed, against 70.3, 81.5 and 79.0 for the GCN numbers the paper copies from Kipf and Welling. The GCN rows carry no standard deviation, and the MixHop rows were rerun by the authors 1905.00067 tab. 1 §6.2. On random splits the gains are 0.3, 1.1 and 1.6 points 1905.00067 tab. 3. Each power adds a sparse product per layer, so cost grows with the largest power j_max 1905.00067 §3.2. Card: MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing.

Max-margin graph cuts keep a graph step but end in a kernel SVM. A regularized harmonic function on a 5-nearest-neighbour graph gives each point a signed score: the sign is the inferred label and the size is the confidence. A hinge-loss kernel classifier is then trained only on points whose confidence is at least ε 2604.26818 §2 §3 eq. 5–8. The graph step is convex, which the authors contrast with the non-convex hat loss of semi-supervised SVMs [arxiv:2604.26818 §4.1]. The baseline is manifold regularization of SVMs (MR). On three UCI datasets with 1 to 10% labels, with linear, cubic and RBF kernels, the authors report lower error than MR in 29 of 36 settings 2604.26818 fig. 4 §6.2. For example, with a linear kernel and 1% labels on letter recognition, the error is 15.79 against 30.94 for MR and 18.90 for a supervised SVM [arxiv:2604.26818 fig. 4]. The table reports no spread, and the stability bound is not used to set the graph parameter 2604.26818 §5.1. The card records that no official code was found. Card: Semi-supervised learning with max-margin graph cuts.

GPR-GNN keeps the neighbour averaging but learns one weight per propagation step. A two-layer MLP produces a first representation, and the output is a weighted sum of K + 1 propagated copies, with the weights trained end to end 2006.07988 §3 eq. 1. The weights are not restricted to be positive. The paper's filter view says that non-negative weights summing to one give a low-pass filter, and negative weights let the model keep high-frequency content, which the paper links to heterophilic graphs 2006.07988 §4. The paper uses K = 10 2006.07988 §5 and initialises the weights as PPR-like decays, one-hot vectors or at random 2006.07988 §5. Author-reported, GPR-GNN has the best mean on 7 of 10 benchmarks. On Texas it reaches 92.92 against 92.26 for the MLP, and on Chameleon 67.48 against 60.96 for GCN 2006.07988 tab. 2. On Citeseer, APPNP is higher (68.59 against 67.63), and on Texas and Cornell the MLP is close to the GNNs, which the paper reads as weak topology information 2006.07988 §5 tab. 2. The homophilic benchmarks use 2.5% of nodes for training, and the heterophilic ones use 60% 2006.07988 §5 app. A.9. The paper's over-smoothing argument assumes an optimiser with a decreasing learning rate, such as SGD, while its experiments use Adam 2006.07988 §4 app. A.4 app. A.9. Card: Adaptive Universal Generalized PageRank Graph Neural Network.

BertGCN applies the graph idea to text classification. It builds one graph over the whole corpus, with document and word nodes, and starts each document node from a BERT-style embedding. BERT and a two-layer GCN are trained jointly, and unlabelled test documents take part in message passing, so the method is transductive 2105.05727 §1 §3.1. The whole graph does not fit through BERT in one pass, so a memory bank holds one embedding per document and is refreshed each epoch 2105.05727 §3.3. With BERT-base, the reported test accuracy is 89.3 on 20NG and 72.8 on Ohsumed, against 85.3 and 70.5 for BERT alone. The gains over BERT on R8, R52 and MR are 0.3 points or less, computed from the table 2105.05727 tab. 1. Card: BertGCN: Transductive Text Classification by Combining GCN and BERT.

Pseudo-labelling and self-training

The model labels unlabelled inputs and trains on its own labels. Disagreement methods such as deep co-training and Tri-Net use two or more heads or views and add a sample when the heads agree 2103.00550 §6.1. Self-training methods take the most probable class as the target and weight the unlabelled loss with a schedule α(t) 2103.00550 §6.2. The survey also lists entropy minimisation, Noisy Student, Meta Pseudo Labels, S4L, EnAET and SimCLRv2 in this family 2103.00550 §6.2. Card: A Survey on Deep Semi-supervised Learning.

Hybrid methods

Hybrids combine consistency, pseudo-labels and Mixup-style interpolation. The survey covers ICT, MixMatch, ReMixMatch, DivideMix and FixMatch. In FixMatch, a pseudo-label from a weakly augmented view becomes the target for a strongly augmented view, and it is used only when the model is confident 2103.00550 §7. Card: A Survey on Deep Semi-supervised Learning.

Class-wise thresholds (FlexMatch)

FlexMatch keeps the FixMatch loop and replaces its single fixed cut-off with one threshold per class. Each class's threshold moves with the number of unlabelled samples the model currently assigns to that class 2110.08263 §3.1. A class with few such samples gets a lower threshold and admits more pseudo-labels. The threshold is T(c) = M(β(c)) · τ, where β(c) is the class count divided by the largest count and M(x) = x / (2 − x) maps β back onto [0, 1] 2110.08263 §3.3 eq. 12. The paper uses τ = 0.95 2110.08263 tab. 3. Early in training most unlabelled samples are unassigned, so the denominator is the larger of the largest count and the unassigned count [arxiv:2110.08263 §3.2 eq. 11]. Each unlabelled sample stores its last class above τ, so the counts come from state the loop already keeps and no extra forward pass is needed 2110.08263 §3.1 alg. 1.

Author-reported error, on CIFAR-100 with 400 labels: 39.94 against 46.42 for FixMatch. On STL-10 with 40 labels: 29.15 against 35.97 2110.08263 tab. 1. The gain is not uniform. At 250 labels on CIFAR-10, FixMatch is slightly better (4.86 against 4.98). On SVHN with 40 labels, FlexMatch is worse (8.19 against 3.81), and the authors attribute this to class imbalance: rare classes never reach β close to 1, so their thresholds stay low 2110.08263 tab. 1 §4.1. The same per-class rule applied to Pseudo-Labeling and UDA lowers error in most cells of the table 2110.08263 §4.1 tab. 1. On ImageNet with 100 labels per class, top-1 error is 41.85 against 43.66 for FixMatch after 2^20 steps, and the authors say the models had not fully converged 2110.08263 §4.2 tab. 2. Card: FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling.

Self-adaptive thresholds (FreeMatch)

FreeMatch keeps the pseudo-labelling loop and sets the threshold from the model's own statistics. A global threshold follows an exponential moving average (EMA) of the mean top confidence on weakly augmented unlabelled images. Each class threshold is that global value scaled by the max-normalised EMA of the class probabilities 2205.07246 §4.1 eq. 5–7. A fairness term, added with its own weight, compares the normalised class-prediction statistics with a target, to keep predictions spread across classes 2205.07246 §4.2 eq. 11–12. The paper's ablation on CIFAR-10 with 10 labels gives 8.07 with this term and 10.37 without it 2205.07246 tab. 4. The fairness weight is 0.01 or 0.05 depending on the setting 2205.07246 app. D tab. 5.

Author-reported error, with 3 seeds and the best checkpoint of each run 2205.07246 §5.1. On CIFAR-10 with 10 labels, 8.07 against 13.85 for FlexMatch. On STL-10 with 40 labels, 15.56 against 29.15 [arxiv:2205.07246 tab. 1]. The gap is small with more labels: on CIFAR-10 with 4000 labels it is 4.10 against 4.19 2205.07246 tab. 1. Several FlexMatch cells in this table match the FlexMatch card's (for example 39.94, 29.15 and 8.19), but not all were checked. The 10-label standard deviations are wide (4.24 for FreeMatch, 12.04 for FlexMatch), so read the ranking with care 2205.07246 tab. 1. The paper gives the extra cost over FixMatch as very small, and its ImageNet run has the same per-iteration time as FixMatch 2205.07246 app. C tab. 2. Card: FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning.

Entropy-filtered pseudo-labels with negative mining (dense prediction)

For semantic segmentation, the teacher's per-pixel entropy splits unlabelled pixels into reliable and unreliable. Reliable pixels get their argmax class as the pseudo-label. The unreliable share starts at 20% and shrinks linearly to 0 over training 2203.03884 §3.2 eq. 5–7. Unreliable pixels are not discarded. They are kept as negatives for classes they are unlikely to belong to, in one first-in-first-out queue per class, and a pixel-level InfoNCE loss with temperature 0.5 uses them 2203.03884 §3.3 eq. 4, 14–15. The method is called U2PL in the paper. Card: Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels.

Reported numbers are author-reported, on PASCAL VOC 2012 with DeepLabv3+ and a ResNet-101 encoder. With 1/4 of the blender-split labels, removing the contrastive loss drops mIoU from 79.30 to 73.02 2203.03884 tab. 6. On the classic split with 1/16 labels, the paper reports 67.98 mIoU against 45.77 for the supervised-only baseline 2203.03884 tab. 1. Before implementing, read the card's "Paper vs code" section: the official code differs from the paper in the contrastive weight, the anchor and negative counts, the anchor definition and the Cityscapes warm-up 2203.03884 §3.3, app. A.

Scan mixing with a mean teacher (LiDAR segmentation)

LaserMix works on LiDAR scans. It splits each scan into bands by beam inclination, then gives alternate bands to two mixed scans, so each band's neighbours come from the other scan 2405.05258 §3.2 eq. 6–7. Labels are mixed the same way. A student and an exponential-moving-average (EMA) teacher train on the mixed scans, and the teacher's confident predictions on unlabelled scans supply targets 2405.05258 §3.3. The paper's stated objective is motivated by entropy minimisation 2405.05258 §3.1 eq. 5, but the released losses are cross-entropy, a mean-teacher term and a mixing term, with no entropy term 2405.05258 §3.3 eq. 8–9. LaserMix++ adds camera parts: pixels matched to points, mixing on (point, pixel) pairs, distillation of image features onto LiDAR features, and a text-guided term that uses CLIP class embeddings 2405.05258 §4.1–4.5 eq. 10–16. The camera parts are specified only by the equations. The released code has the LaserMix part and no camera branch 2405.05258 §5.2 [code: README.md#L177].

Author-reported, with Cylinder3D and mIoU. At 1% labels, LaserMix++ reaches 56.2 on SemanticKITTI, against 45.4 for supervised-only and 50.6 for LaserMix. At the same label fraction it reaches 58.5 on nuScenes (55.3 LaserMix) and 47.3 on ScribbleKITTI (44.2 LaserMix) 2405.05258 tab. 2. The gain of LaserMix++ over LaserMix at 1% is 3.1 to 5.6 points across these three datasets, computed from the table. The paper's gain over LaserMix is 2% to 3% on average over its settings 2405.05258 §5.3. The paper claims accuracy comparable to full supervision at 2× to 5× fewer annotations 2405.05258 §5.3. No seeds or variance are reported 2405.05258 Pitfalls §5.4.

The released configs differ from the text. The SemanticKITTI config uses a mixing weight of 1 where the paper uses 2.0, and it samples 4 to 8 areas where the paper samples 2 to 6 2405.05258 §5.2 lasermix_cy3d_semi_semantickitti_10.py. Pseudo-labels below the threshold get an ignore index in the code, and the text does not say what happens to them 2405.05258 §3.3 semi_base.py. Decide which values to use before comparing with the reported numbers. Card: Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving.

Teacher-student detection (Unbiased Teacher)

The teacher-student idea applied to Faster R-CNN with FPN and a ResNet-50 backbone. A burn-in stage trains on labelled images only. After that, an EMA teacher labels weakly augmented unlabelled images, and the student learns from strongly augmented views of the same images 2102.09480 §3.1–3.2. Pseudo-boxes are cleaned with class-wise NMS and kept only above a confidence of 0.7 2102.09480 §3.2 tab. 5. Box-regression losses are not applied to pseudo-boxes; only the RPN and ROI classification losses use them [arxiv:2102.09480 §3.2]. The ROI classifier uses focal loss instead of cross-entropy, because the authors argue that pseudo-labels are biased towards background and frequent classes. The paper does not state the focal exponent 2102.09480 §3.3. Card: Unbiased Teacher for Semi-Supervised Object Detection.

Author-reported, on COCO with Faster R-CNN. With 1% of the labels, AP50:95 is 20.75 against 9.05 supervised and 13.97 for STAC 2102.09480 tab. 1. The gain holds across label fractions, from 16.94 against 6.83 at 0.5% to 31.50 against 23.86 at 10% 2102.09480 tab. 1. The ablation on the 1% split shows that both the focal loss and the EMA teacher add to the result, from 13.42 with neither to 21.19 with both 2102.09480 fig. 5. The ablation's 21.19 and the headline 20.75 differ for the same setting, and the paper does not explain the gap 2102.09480 tab. 1 fig. 5. Training is long: 180k iterations with 32 labelled and 32 unlabelled images per step 2102.09480 app. "Training".

Open-world clustering (generalized category discovery)

The unlabelled set may mix images of labelled classes with images of classes that have no labels, and the number of classes is not given 2201.02609 §1 §3. The method avoids a classification head. It starts from a ViT-B/16 with self-supervised DINO weights, fine-tunes with a noise-contrastive loss on all images and a supervised contrastive loss on labelled images, mixed with λ = 0.35 2201.02609 §3.1.1 eq. 1–3 §4.1. Labels are then assigned by semi-supervised k-means on the [CLS] features: labelled images keep the cluster of their class, and unlabelled images go to the nearest centroid 2201.02609 §3.1.2. The number of clusters k is found by searching for the value that maximises accuracy on the labelled images, using Brent's algorithm on [|Y_L|, 1000] 2201.02609 §3.2 §4.1. Card: Generalized Category Discovery.

Author-reported clustering accuracy over all unlabelled images: 91.5 on CIFAR10, 73.0 on CIFAR100 and 74.1 on ImageNet-100 2201.02609 tab. 2; 51.3 on CUB, 39.0 on Stanford Cars and 35.4 on Herbarium19 2201.02609 tab. 3. Plain k-means on the DINO features reaches 83.6 on CIFAR10 and 52.0 on CIFAR100 2201.02609 tab. 2. On CIFAR100, the ablation attributes the rise from 52.0 to 73.0 to the contrastive fine-tuning and the semi-supervised k-means 2201.02609 tab. 5. Read the numbers with three cautions. The baselines are adapted by the authors 2201.02609 §3.3. The Hungarian matching is computed over all unlabelled images, so the numbers are not directly comparable with novel-category-discovery papers 2201.02609 app. E. The paper's headline gain on the generic datasets does not follow from its own table 2201.02609 §4.2 tab. 2. No variance is reported.

Silver pairs from a cross-encoder (sentence similarity)

A slow, accurate teacher labels unlabelled sentence pairs, and a fast bi-encoder trains on those labels as soft targets. The cross-encoder is trained on the labelled pairs, scores new pairs built from the unlabelled sentences, and the bi-encoder trains on the labelled plus the scored pairs 2010.08240 §3.1. Which pairs are scored decides the result. The paper compares random sampling, kernel-density matching, BM25 lexical neighbours, semantic-search neighbours and their union 2010.08240 §3.1. Random pairs are mostly negatives. On in-domain Spanish-STS, random sampling gives 62.05 against 72.07 for the bi-encoder trained with no silver pairs 2010.08240 tab. 5. BM25 reaches 75.08 on the same task with 3,964 silver pairs, against 230,364 for KDE at 74.67 2010.08240 tab. 11. The paper reports BM25 as the best strategy on average at the smallest silver-set size 2010.08240 §6.1.

For domain adaptation, the cross-encoder is trained on a labelled source domain and labels unlabelled target pairs; no target labels are used 2010.08240 §3.2. On Quora → Sprint, AUC(0.05) rises from 0.505 with a source-only bi-encoder to 0.875. When Quora is the target, the same method is level with the source-only baseline, 0.495 to 0.496 2010.08240 tab. 6. The per-pair teacher pass runs during training only. The paper states that the trained bi-encoder runs at the same speed as SBERT 2010.08240 §5. Card: Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks.

Pretraining on unlabelled tabular rows (TabTransformer)

Rows with categorical columns can also supply unlabelled data. TabTransformer pretrains its attention layers on unlabelled rows and then fine-tunes them with the supervised loss on the labelled rows 2012.06678 §2. The masked objective (MLM) sets 30% of a row's categorical values to a missing-value embedding and predicts each original class 2012.06678 §2. The replaced-value objective (RTD) swaps those values for a random value from the same column and asks, one head per column, whether each value was replaced 2012.06678 §2 app. A. The text does not say how the random value is drawn. The ablation finds that a fresh draw each epoch beats a fixed one: on three datasets with 256 labels, relative AUC is +0.81 against −0.49 2012.06678 app. A. The paper's ablation also finds 15% and 50% replacement rates similar to the 30% used 2012.06678 §2 app. A. Pretraining epochs, batch size and learning rate are not stated 2012.06678 app. B.1.

Author-reported, as mean AUC. On the eight datasets with more than 30K rows, the better of RTD and MLM beats the best other baseline (MLP (ER) in the table) at 50, 200 and 500 labels, by at least 1.2, 2.0 and 2.1 points 2012.06678 tab. 3 §3. On the twelve smaller datasets, the best other baseline is ahead at 50 labels (79.4 against 78.6 for RTD). RTD leads at 200 and 500 labels (81.6 against 81.2, and 83.4 against 82.3) 2012.06678 tab. 4. The paper gives as the reason for the 50-label loss that the fine-tuned head sees only labelled rows 2012.06678 §3.4. Card: TabTransformer: Tabular Data Modeling Using Contextual Embeddings.

Auxiliary tasks with conflicting gradients (CAGrad)

The semi-supervised setting here is a multi-task one: the base method, ARML, trains with auxiliary tasks alongside its main loss 2110.14048 §5.4. The task gradients can conflict, and CAGrad combines them into one update. It starts from the average gradient and moves towards the direction that most improves the worst task, within a ball of radius c around the average 2110.14048 §3.1. The paper's semi-supervised runs fix c at 0.1 2110.14048 app. B.4. On CIFAR-10, with 500, 1000 and 2000 labels, ARML with CAGrad reaches 68.25, 74.37 and 82.81 average test accuracy. ARML alone reaches 67.05, 73.22 and 81.35, and ARML with PCGrad reaches 67.49, 73.23 and 81.91 2110.14048 tab. 4. These are author-reported means over three runs, with gaps of about 1.2 to 1.5 points. The paper does not give the auxiliary weights and the repository has no code for this experiment 2110.14048 §5.4 app. B.4. Card: Conflict-Averse Gradient Descent for Multi-task Learning.

Pretext tasks on unlabelled sensor signals (wearable HAR)

A network is pre-trained on unlabelled accelerometer windows to predict which signal transform was applied to each window. The pretext task is multi-task: one head per transform, all on a shared 1-D CNN trunk [arxiv:1907.11879 §3.1 §3.3]. The eight transforms are noise, scaling, sign inversion, negation within a time window, time reversal, segment permutation, time warp and channel shuffle. Their parameters are not stated 1907.11879 §3.2. The conv trunk is then copied into an activity classifier, either frozen or with its last conv block fine-tuned 1907.11879 §3.1. The semi-supervised variant pre-trains on the full unlabelled target set and trains the classifier on 2 to 100 labelled windows per class 1907.11879 §4.3.4. The paper compares this with an autoencoder trained on the same windows and with end-to-end supervised training 1907.11879 §4.3.2.

Author-reported, as Cohen's kappa over 10 runs. With the trunk frozen, the pretext features give 0.7310 on HHAR, 0.7642 on UniMiB and 0.8803 on UCI HAR, against 0.6816, 0.7815 and 0.8789 for supervised training 1907.11879 tab. 2. On UniMiB, WISDM and MotionSense the frozen features trail supervised training, and under 5-fold user-split cross-validation they trail on five of six datasets [arxiv:1907.11879 tab. 2 tab. 4]. The per-layer analysis on HHAR favours the second conv block (0.774) over the last (0.679), but the main table uses the last block 1907.11879 §4.3.1. The paper also reports the cost as equal to supervised learning without giving runtime figures 1907.11879 §3.1. Card: Multi-task Self-Supervised Learning for Human Activity Detection.

Pathology: few labels in whole-slide images

A mini-review of machine learning for histopathology lists semi-supervised learning as one answer to scarce labels, next to active learning, multiple instance learning and transfer learning. It describes the method as using unlabelled data to estimate the label distribution 1709.00786 §4.2.2. The review reports no experiments of its own 1709.00786 §1, so use it as a map of options rather than as evidence for any one method. Card: Machine learning methods for histopathological image analysis.

Logic rules as the unlabelled signal (Logic Tensor Networks)

Logic Tensor Networks (LTN) score a knowledge base of first-order formulas with real-valued truth values, and training maximises that satisfiability score 2012.13635 §3.2. The paper does not describe its experiments as semi-supervised learning. The reading here is ours, based on the card's concept tags: several examples train with labels for only part of the data, and the rules cover the rest. In the smokes example, all 14 individuals have known smoking facts, but cancer facts are known only for the first group. Rules on friendship and on smoking spread information to the other group 2012.13635 §4.7. In the multi-label example, the only labels are positive ones, with no negative examples. Mutual exclusion of blue and orange, and of male and female, is written as rules 2012.13635 §4.3. The clustering example uses no class labels. Its axioms ask every point to belong to a cluster and nearby points to share one 2012.13635 §4.6. The card records a final constraint satisfaction of 0.857 for that run, with no baseline 2012.13635 §4.6. The smokes and multi-label results are qualitative 2012.13635 §4.3 §4.7. Card: Logic Tensor Networks.

When to use which

The survey does not rank methods, because it reports no accuracy numbers 2103.00550 §8. Choose a variant by constraints, then check it against a supervised baseline on your own data.

Implementation sketch

A generic step that the variants share. Lines marked with a citation name the choice that differs between them. The dense variant adds a contrastive term over the unreliable pixels 2203.03884 §3.3.

# One semi-supervised training step, shared core of the variants
# Input: labelled batch (x_l, y_l), unlabelled batch x_u, step t of T
# Output: total loss

def ssl_step(student, teacher, x_l, y_l, x_u, t, T):
    # Supervised term on labelled data  [arxiv:2103.00550 §2 eq. 1]
    loss_s = cross_entropy(student(x_l), y_l)

    # Targets come from the model itself, with no gradient  [arxiv:2103.00550 §4]
    # Teacher is an EMA of the student, a past average, or the same net on
    # a perturbed view; the choice is the variant  [arxiv:2103.00550 tab. 3]
    with no_grad():
        probs_u = softmax(teacher(x_u))

    # Keep only the confident share of targets  [arxiv:2203.03884 eq. 5-7]
    # alpha_t shrinks linearly; U2PL starts at 0.2
    # A per-class cut-off on the confidence (FlexMatch) replaces this rule
    # [arxiv:2110.08263 §3.3]
    alpha = 0.2 * (1 - t / T)
    entropy = -sum(probs_u * log(probs_u), over_classes)
    gamma = percentile(entropy, 100 * (1 - alpha))
    reliable = entropy < gamma

    # Argmax pseudo-labels on reliable entries, ignore the rest  [eq. 6]
    pseudo_u = where(reliable, argmax(probs_u), IGNORE)

    # Pseudo-label loss over kept entries  [arxiv:2203.03884 eq. 3]
    # Guard against a batch with no kept entries
    loss_u = cross_entropy(student(x_u), pseudo_u, ignore=IGNORE)

    # Weight of the unsupervised term  [arxiv:2103.00550 eq. 1]
    # Not stated in the survey; U2PL normalises by the kept count [eq. 8]
    return loss_s + lambda_u * loss_u

Contradictions

• Unlabelled data helping or hurting. The survey warns that unlabelled data can reduce accuracy below the supervised baseline when the data assumptions fail 2103.00550 §2.1, §8. U2PL reports gains over its supervised-only baseline in every row of its three result tables 2203.03884 tab. 1–3. These are different tasks and the U2PL numbers are author-reported against baselines the authors re-implemented 2203.03884 §4.2. Both statements stand; neither settles the other.
• Discarding or keeping unreliable predictions. Unbiased Teacher drops detections below confidence 0.7 and does not reuse them [arxiv:2102.09480 §3.2]. U2PL drops unreliable pixels from the pseudo-label loss but reuses them as negatives 2203.03884 §3.3. The two papers work on different tasks and neither compares against the other, so the cards do not show which choice is better.
• Taxonomy placement. The two surveys cut the field differently. The deep-SSL survey puts MixMatch and FixMatch under hybrid methods 2103.00550 §7. The visual survey lists MixMatch and ReMixMatch under two families and FixMatch under consistency and self-training, and says the grouping overlaps 2208.11296 tab. 1, §2.2.1–2.2.2. Neither places the methods wrongly; they answer different questions. The visual survey also notes that its names for the smoothness assumptions overlap 2208.11296 §2.1, §2.2.2.
• Cityscapes text and table. The U2PL text describes notable margins on Cityscapes 2203.03884 §4.2. Its table shows the CutMix variant below AEL at 1/16, 1/8 and 1/4 of the labels. Only the AEL variant beats AEL at every fraction 2203.03884 tab. 3.
• Class-wise thresholds on SVHN with 40 labels. The FlexMatch paper finds its per-class rule worse than FixMatch on SVHN with 40 labels, and attributes this to class imbalance 2110.08263 tab. 1 §4.1. FreeMatch also scales its threshold per class, and reports 1.97 on SVHN with 40 labels, against 3.81 for FixMatch and 8.19 for FlexMatch 2205.07246 tab. 1. The FreeMatch paper does not show which part of its design accounts for the gap, so the rule "avoid per-class thresholds under imbalance" does not hold without checking.
• Fixed or class-wise cut-offs. FlexMatch argues that one fixed cut-off wastes unlabelled data early in training and treats all classes alike 2110.08263 §1. Unbiased Teacher keeps pseudo-boxes above a fixed 0.7 2102.09480 §3.2. The two papers work on different tasks and do not compare with each other, so the cards do not show which rule suits detection.
• GCN baselines across splits. GCN on Cora scores 81.5 in MixHop's Planetoid table 1905.00067 tab. 1 and 75.21 in GPR-GNN's table, where 2.5% of the nodes are labelled 2006.07988 tab. 2 app. A.9. The splits differ, so the two numbers should not be compared.
• Optimiser behind the over-smoothing argument. GPR-GNN's argument assumes SGD with a decreasing learning rate 2006.07988 §4 app. A.4, while its experiments use Adam 2006.07988 app. A.9. The cards do not show that the argument carries over to Adam.
• TabTransformer's dataset counts. The abstract and §3 say 15 datasets, but the semi-supervised tables cover 20, and the text and table captions give different counts for the groups 2012.06678 §3 tab. 3 tab. 4 app. C. Treat the 15-dataset supervised set and the 20-dataset semi-supervised set as separate benchmarks.

Papers

• A Survey on Deep Semi-supervised Learning: five-family taxonomy of deep SSL, with one loss notation and a table of consistency methods.
• Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels: U2PL, which keeps unreliable pixels as contrastive negatives in dense prediction.
• Unbiased Teacher for Semi-Supervised Object Detection: EMA teacher pseudo-boxes for Faster R-CNN, with focal loss on the ROI classifier.
• Generalized Category Discovery: clustering with an unknown class count over mixed known and new classes, with labelled images pinned to their classes.
• Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks: cross-encoder soft labels on sampled sentence pairs, with the sampling strategy as the key choice.
• BertGCN: Transductive Text Classification by Combining GCN and BERT: BERT-initialised document nodes in a word-document graph, trained jointly with a memory bank.
• Semi-Supervised and Unsupervised Deep Visual Learning: A Survey: visual SSL and UL taxonomy around three smoothness rationales, with applications to segmentation, detection and domain adaptation; no experiments.
• MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing: concatenated propagation over adjacency powers, with per-power weights.
• Adaptive Universal Generalized PageRank Graph Neural Network: learned propagation weights that can be negative, for homophilic and heterophilic graphs.
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings: masked and replaced-value pretraining of column attention on unlabelled tabular rows.
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling: per-class confidence thresholds on FixMatch, moved by how many samples each class has.
• Conflict-Averse Gradient Descent for Multi-task Learning: CAGrad, a gradient-combination step that protects the worst task, with one semi-supervised CIFAR-10 experiment on ARML.
• FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning: global and per-class thresholds from the model's own EMA statistics, with a fairness term on class predictions.
• Semi-supervised learning with max-margin graph cuts: regularized harmonic labels on a k-NN graph, then a kernel hinge-loss SVM trained on confident points only.
• Machine learning methods for histopathological image analysis: mini-review of histopathology ML, listing semi-supervised learning among remedies for scarce labels; no experiments.
• Logic Tensor Networks: first-order rules as a differentiable satisfiability objective, with partial labels in several examples.
• Multi-task Self-Supervised Learning for Human Activity Detection: transform-prediction pretext tasks on unlabelled accelerometer windows, then a classifier on the frozen or partly tuned trunk.
• Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving: LaserMix band swapping with a mean teacher for LiDAR, and LaserMix++ camera and language terms, whose camera parts have no released code.