A Survey on Deep Semi-supervised Learning

In plain words

Labels are expensive. A doctor has to look at every scan, a linguist has to tag every sentence. Unlabelled data, on the other hand, is cheap and plentiful. Semi-supervised learning trains on a small labelled set together with a large unlabelled one, so that the unlabelled data improves the model beyond what the labels alone allow.

This survey sorts the deep-learning versions of that idea into five families. Generative methods learn what the data looks like, using GANs or variational autoencoders, and use that knowledge to help the classifier. Consistency regularization asks the model to give the same answer when its input or its weights are perturbed a little, for example a slightly shifted image or a different dropout mask. Graph-based methods connect similar examples in a graph and let label information flow along the edges. Pseudo-labeling lets the model, or a second model, label the unlabelled data itself and then train on its most confident guesses. Hybrid methods such as MixMatch and FixMatch combine several of these tricks.

For each family the survey writes the training losses in one shared notation, draws the architectures side by side, and tabulates which perturbations each method uses. It runs no experiments, so it tells you how the methods differ, not which one scores best. It closes with open problems: unlabelled data can make a model worse when it does not match the labelled data, class imbalance, noisy labels, and a lack of theory explaining why these methods work.

TL;DR

A taxonomy survey, not a method paper (arXiv v2, August 2021). It sorts deep semi-supervised learning (DSSL) into five families by loss and model design §1 fig. 1: generative (semi-supervised GANs §3.1, semi-supervised VAEs §3.2), consistency regularization §4, graph-based (autoencoder and GNN embeddings) §5, pseudo-labeling (disagreement-based §6.1 and self-training §6.2), and hybrid methods built on Mixup §7. Useful for implementers: one notation for all losses (eq. 1 and the per-method consistency constraints), the consistency-method summary tab. 3, the graph-autoencoder summary tab. 4 and the augmentation and network-perturbation map tab. 5. Coverage is image classification first §1. No code, no experiments, and several equations carry typos or notation clashes (see Pitfalls).

Problem

Supervised deep learning needs a lot of high-quality labels, which are often expensive or slow to produce, for example medical labels that need expensive machines and several experts §1. SSL uses abundant unlabelled data alongside a few labels. The authors define semi-supervised classification, clustering (constrained clustering) and regression, then restrict the survey to image classification and claim the ideas carry over to detection, segmentation, clustering and regression §1. Their stated gap: earlier surveys predate deep methods such as adversarial training, cover only parts of the field (evaluation, or only generative and teacher-student models), or use a different taxonomy without comparisons or future directions §1.

Method

The "method" of a survey is its taxonomy. Section numbering of the compiled paper: §1 Introduction, §2 Background (§2.1 assumptions, §2.2 classical methods, §2.3 related paradigms, §2.4 datasets and applications), §3 Generative methods (§3.1 GANs, §3.2 VAEs), §4 Consistency regularization, §5 Graph-based (§5.1 autoencoder, §5.2 GNN), §6 Pseudo-labeling (§6.1 disagreement-based, §6.2 self-training), §7 Hybrid, §8 Challenges and future directions, §9 Conclusion. There is no appendix. Figures: fig. 1 taxonomy, fig. 2 semi-supervised GAN architectures, fig. 3 semi-supervised VAE graphical models, fig. 4 consistency architectures, fig. 5 graph-based architectures, fig. 6 pseudo-label architectures, fig. 7 hybrid architectures. Equation numbers below were counted from the LaTeX source up to eq. 49. Equations after that (from Tri-Net's labelling rule in §6.1 on) are cited by section, because the source suppresses some numbers and the count is uncertain.

Common objective §2

Labelled set X_L=\{x_i\}_{i=1}^L with labels Y_L, unlabelled set X_U of size U, usually L \ll U, K classes §2:

\min_{\theta} \sum_{x\in X_L,y\in Y_L}\mathcal{L}_s(x,y,\theta)+\alpha \sum_{x\in X_U}\mathcal{L}_u(x,\theta)+\beta \sum_{x\in X}\mathcal{R}(x,\theta) eq. 1

\mathcal{L}_s is a per-example supervised loss (e.g. cross-entropy), \mathcal{L}_u an unsupervised loss, \mathcal{R} a regularizer (e.g. a consistency loss), and \alpha,\beta>0 trade-off weights. The authors say they often do not separate \mathcal{L}_u from \mathcal{R} §2. Transductive SSL predicts only the training-time unlabelled points; inductive SSL must generalise to unseen data. Most graph-based methods are transductive, most others inductive §2.

Assumptions §2.1: self-training (high-confidence predictions tend to be correct, e.g. when classes form well-separated clusters), co-training (two conditionally independent, individually sufficient views), generative model (data from a mixture with correct p(y), p(x\mid y)), cluster (same cluster implies same class), low-density separation (decision boundary in low-density regions), manifold (neighbours on a low-dimensional manifold share labels). If the data violate them, SSL can fail to help or reduce accuracy §2.1.

Classical methods §2.2: self-learning, co-training, EM on generative models, transductive SVMs (low-density separation; non-convex), and graph label propagation (labels of labelled nodes stay fixed and spread to neighbours by similarity).

Related paradigms §2.3: transfer learning (domains may differ, whereas SSL assumes i.i.d. train and test), weakly-supervised learning (SSL is its "incomplete supervision" case), PU learning (a special case of SSL), meta-learning, and self-supervised learning (cites the self-supervised survey).

Datasets tab. 2: image classification (MNIST, SVHN, STL-10, CIFAR-10/100, ImageNet), object detection (PASCAL VOC, MS COCO, ILSVRC), 3D detection (SUN RGB-D, ScanNet, KITTI), video salient object detection (VOS, DAVIS, FBMS), semantic segmentation (PASCAL VOC, PASCAL Context, MS COCO, Cityscapes, CamVid, SiftFlow, StanfordBG), text classification (AG News, DBpedia (printed as "BPpedia"), Yahoo! Answers, IMDB, Yelp, Snippets, Ohsumed, TagMyNews, MR, Elec, Rotten Tomatoes, RCV1), dialogue policy (MultiWOZ), dialogue generation (Cornell Movie Dialogs, Ubuntu Dialogue Corpus), sequence learning (IMDB, Rotten Tomatoes, 20 Newsgroups, DBpedia, CoNLL 2003 NER, CoNLL 2000 chunking, Twitter POS, Universal Dependencies, CCG supertagging), semantic role labelling (CoNLL-2009, CoNLL-2013), QA (SQuAD, TriviaQA).

Generative methods §3

Semi-supervised GANs §3.1 fig. 2. Base GAN game:

\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p(x)}[\log D(x)] + \mathbb{E}_{z \sim p_{z}}[\log(1 - D(G(z)))] eq. 2

Four ways to use GANs for SSL: reuse discriminator features, regularise a classifier with generated samples, learn an inference model, or add generated samples as training data §3.1. The simplest recipe adds a supervised term such as \mathbb{E}_{(x,y)\in X_l}[\log D(y\mid x)] to the GAN value function §3.1.

• CatGAN: K-way discriminator; objective uses the entropy H[p(y\mid x,D)] (confident on real), H[p(y\mid G(z),D)] (uncertain on generated), marginal class entropy H[p(y\mid D)] (use all classes), plus cross-entropy on labelled data §3.1.
• CCGAN: in-painting. Mask m, generator output x_G=G(m\odot x,z), in-painted image x_I=(1-m)\odot x_G+m\odot x. The discriminator separates real from in-painted images, and its penultimate layer is shared with the classifier §3.1.
• SGAN / Improved GAN: (K{+}1)-class discriminator, the extra class meaning "generated". The generator is trained by feature matching, \|\mathbb{E}_{x\in X} D(x)-\mathbb{E}_{z\sim p(z)}D(G(z))\|_2^2. Other listed tricks: minibatch discrimination, historical averaging, one-sided label smoothing, virtual batch norm §3.1. Classifier objective:

\max_D \mathbb{E}_{(x,y)\sim p(x,y)}\log p_D(y\mid x,y\le K) + \mathbb{E}_{x\sim p(x)}\log p_D(y\le K\mid x)+\mathbb{E}_{x\sim p_G}\log p_D(y=K+1\mid x) eq. 3

The first term is supervised, the last two are unsupervised (unlabelled and generated data) §3.1.

• GoodBadGAN: a deliberately "bad" generator. Adds the penalty \mathbb{E}_{x\sim p_G}\log p(x)\,\mathbb{I}[p(x)>\epsilon] (only high-density generated samples are penalised) and the conditional-entropy term \mathbb{E}_{x \sim p_x} \sum_{k=1}^{K} \log p_D(k\mid x) to eq. 3 §3.1.
• Localized GAN: local generator G(x,z); |D(G(x,z+\sigma z))-D(G(x,z))|^2\approx \|\nabla_x^G D(x)\|^2 \sigma z eq. 4, so the regulariser \sum_{k=1}^{K}\mathbb{E}_{x\sim p_x}\|\nabla_x^G \log p(y=k\mid x)\|^2 makes the classifier smooth along the manifold §3.1.
• CT-GAN: WGAN, \min_G\max_D \mathbb{E}_{x\sim p_x}D(x)-\mathbb{E}_{z\sim p_z}D(G(z)) eq. 5, plus gradient penalty at \hat{x}=\epsilon x+(1-\epsilon)G(z), plus a consistency term on two perturbations x',x'' of each real x. For SSL the discriminator uses eq. 3 instead of eq. 5, plus the consistency term §3.1.
• BiGAN / ALI: encoder E maps x\to z'; the discriminator tells (x,E(x)) from (G(z),z) eq. 6. ALI uses the Improved-GAN discriminator for SSL §3.1.
• Augmented BiGAN: feature matching \|\mathbb{E}_{x\in X}D(E(x),x)-\mathbb{E}_{z\sim p(z)}D(z,G(z))\|^2_2 for G and E, plus a third pair term \mathbb{E}_{x}[\log(1-D(E(x),G_x(E(x))))] against class switching §3.1.
• Triple GAN: separate classifier C; three-player game eq. 7:

\min_{C,G}\max_{D} \mathbb{E}_{(x,y)\sim p(x,y)}[\log D(x,y)] + \alpha\,\mathbb{E}_{(x,y)\sim p_c(x,y)}[\log(1-D(x,y))] + (1-\alpha)\,\mathbb{E}_{(x,y)\sim p_g(x,y)}[\log(1-D(G(y,z),y))] eq. 7

• Enhanced TGAN (class-wise mean feature matching, two collaborating classifiers, semantic matching), MarginGAN (generator maximises margin of generated samples, classifier decreases it), Triangle GAN (two generators E,G, two discriminators D_1,D_2), Structured GAN (two independent latents: y for designated semantics, z for the rest, two adversarial games for p(x,y) and p(x,z)), R^3-CGAN (CutMix-built cross-category and real-fake instances regularise C and D_1) §3.1.

Semi-supervised VAEs §3.2 fig. 3. ELBO:

\log p(x)=\log \mathbb{E}_{q(z\mid x)}\Big[\frac{p(z)p(x\mid z)}{q(z\mid x)}\Big] \ge \mathbb{E}_{q(z\mid x)}[\log p(z)p(x\mid z)-\log q(z\mid x)] eq. 8–9

• SSVAE M1: latent-feature model; samples from q_\phi(z\mid x) are classifier features. M2: p_\theta(x\mid z,y)p(z)p(y) with y latent for unlabelled data; q_\phi predicts missing labels. M1+M2: p_\theta(x,y,z_1,z_2)=p(y)p(z_2)p_\theta(z_1\mid y,z_2)p_\theta(x\mid z_1) eq. 10. M2 is the base of the family §3.2.
• ADGM: auxiliary variable a, q_\phi(a,y,z\mid x)=q_\phi(z\mid a,y,x)q_\phi(y\mid a,x)q_\phi(a\mid x) eq. 11; SDGM reverses the x–a dependence; the original paper reports the z\to x skip connection is needed for convergence §3.2.
• Infinite VAE: Dirichlet-process mixture of VAEs, p(c,\pi,x,z)=p(c\mid\pi)p_{\alpha}(\pi)p_{\theta}(x\mid c,z)p(z) eq. 12, trained with Gibbs sampling plus variational inference §3.2.
• Disentangled VAE (factorisation q_\phi(y\mid x,z)q_\phi(z\mid x), importance sampling) eq. 13; SDVAE (disentangled v and non-interpretable u independent given x) eq. 14; ReVAE (labels captured by auxiliary variables, conditional prior p_\psi(z\mid y) for test-time inference) §3.2.

Consistency regularization §4 fig. 4

Teacher-student framing. The student has weights \theta, the target (teacher) \theta'. Generic constraint §4:

\mathbb{E}_{x\in X } \mathcal{R}(f(\theta, x), \mathcal{T}_x) eq. 15

\mathcal{R} is usually MSE or KL divergence; methods differ in how the target \mathcal{T}_x is built. Because targets come from the model itself, these methods suffer confirmation bias, which better targets reduce §4. Summary tab. 3:

Per-method details §4:

• Ladder Network: noise \zeta in every hidden layer of a corrupted encoder that shares weights with a clean encoder; a decoder g(\cdot,\cdot) denoises each layer, and the unsupervised loss is the per-layer MSE between clean z_i and reconstructed \hat z_i §4. Written as \mathbb{E}_{x\in X}\mathcal{R}(f(\theta,x),g(f(\theta,x+\zeta))) eq. 16.
• \Pi Model: each sample, labelled and unlabelled, goes forward twice with different augmentation and dropout eq. 17.
• Temporal Ensembling: one forward pass per sample; target is the per-sample ensemble Z_i \leftarrow \alpha Z_i + (1-\alpha) z_i, updated after each epoch with momentum \alpha eq. 18.
• Mean Teacher: teacher weights are the EMA of student weights over training steps eq. 19.
• VAT: \mathbb{E}_{x\in X}\mathcal{R}(f(\theta,x),g(\theta,x+\gamma^{adv})) with \gamma^{adv}=\arg\max_{\gamma;\|\gamma\|\le\delta}\mathcal{R}(f(\theta,x),g(\theta,x+\gamma)) eq. 20.
• Dual Student: two students with different initialisations. x is a stable sample for student i if \mathcal{C}_{x}^{i}=\{p_{x}^{i}=p_{\bar{x}}^{i}\}_1 \,\&\, (\{\mathcal{M}_{x}^{i}>\xi\}_1 \,\|\, \{\mathcal{M}_{\bar{x}}^{i}>\xi\}_1) with \mathcal{M}_{x}^{i}=\|f(\theta^i,x)\|_\infty eq. 21. Stabilization loss eq. 22: if both are stable, the student with the larger \varepsilon is pulled towards the other (\{\varepsilon^i>\varepsilon^j\}_1\,\mathcal{R}(f(\theta^i,x),f(\theta^j,x))); otherwise \mathcal{C}^i\,\mathcal{R}(f(\theta^i,x),f(\theta^j,x)).
• SWA / fast-SWA: SGD on the consistency loss keeps exploring distant solutions rather than converging; averaging weights along a cyclical-LR trajectory (fast-SWA averages several points per cycle with a longer cycle) gives flatter solutions eq. 23.
• VAdD: \epsilon^{adv}=\arg\max_{\epsilon;\|\epsilon^s-\epsilon\|_2\le \delta H}\mathcal{R}(f(\theta,x,\epsilon^s),f(\theta,x,\epsilon)), with \epsilon^s a random dropout mask, \delta the noise intensity and H the dropout-layer dimension eq. 24.
• WCP: worst-case additive weight noise, \mathbb{E}\,\mathcal{R}(f(\theta,x),g(\theta+\zeta,x)) eq. 25 with a norm constraint on \zeta, and worst-case DropConnect, \mathbb{E}\,\mathcal{R}(f(\theta,x),f((1-\alpha)\theta,x)) with \alpha=1 for a dropped connection eq. 26.
• UDA: replace simple noise with strong augmentation (AutoAugment, RandAugment, back-translation) eq. 27.

Graph-based methods §5 fig. 5

The survey covers graph-embedding SSL only: nodes are samples, edges are similarities, f_{\mathbf z}: v\mapsto \mathbf z_v\in\mathbb R^d, d\ll|\mathcal V| §5. Unified loss: a supervised term over X_l plus \alpha\sum_{x\in X}\mathcal R(x,f_{\mathbf z}) eq. 28. Shallow embeddings (DeepWalk, LINE, node2vec) are lookup tables by node ID; deep embeddings split into autoencoder-based and GNN-based §5.

Autoencoder-based §5.1: each node i has a neighbourhood vector \mathbf s_i\in\mathbb R^{|\mathcal V|}; \text{Dec}(\text{Enc}(\mathbf s_i))\approx \mathbf s_i eq. 29, loss \sum_i\|\text{Dec}(\mathbf z_i)-\mathbf s_i\|_2^2 eq. 30. GAE: \text{Enc}(A,X)=\text{GraphConv}(\sigma(\text{GraphConv}(A,X))) eq. 31, \text{Dec}(\mathbf z_u,\mathbf z_v)=\mathbf z_u^\top\mathbf z_v eq. 32. VGAE maximises \mathbb{E}_{q(\mathbf Z\mid X,A)}[\log p(A\mid\mathbf Z)]-\text{KL}[q(\mathbf Z\mid X,A)\,\|\,p(\mathbf Z)] eq. 33 with a diagonal-Gaussian q per node eq. 34 and an inner-product-sigmoid edge likelihood eq. 35. Summary tab. 4:

SDNE preserves first- and second-order proximity (autoencoder plus a Laplacian-eigenmaps penalty); DNGR uses random surfing, a PPMI matrix and stacked denoising autoencoders §5.1.

GNN-based §5.2: message passing \mathbf h_u^{(k+1)}=\text{Update}^{(k)}(\mathbf h_u^{(k)},\mathbf m^{(k)}_{\mathcal N(u)}) with \mathbf m^{(k)}_{\mathcal N(u)}=\text{Aggregate}^{(k)}(\{\mathbf h_v^{(k)}\}) eq. 36–37; node embedding \mathbf z_u=\mathbf h_u^{(K)}; a classifier on labelled nodes is then applied to unlabelled nodes §5.2.

• Basic GNN: \mathbf h_u^{(k)}=\sigma(\mathbf W_{\text{self}}^{(k)}\mathbf h_u^{(k-1)}+\mathbf W_{\text{neigh}}^{(k)}\sum_{v\in\mathcal N(u)}\mathbf h_v^{(k-1)}+\mathbf b^{(k)}) eq. 38.
• GCN: symmetric normalisation \mathbf m_{\mathcal N(u)}=\sum_{v\in\mathcal N(u)}\mathbf h_v/\sqrt{|\mathcal N(u)||\mathcal N(v)|} eq. 39; update \mathbf h_u^{(k)}=\sigma(\mathbf W^{(k)}\sum_{v\in\mathcal N(u)\cup\{u\}}\mathbf h_v/\sqrt{|\mathcal N(u)||\mathcal N(v)|}) eq. 40.
• GAT: \mathbf m_{\mathcal N(u)}=\sum_v\alpha_{u,v}\mathbf h_v eq. 41, \alpha_{u,v}=\text{softmax}_v(\mathbf a^\top[\mathbf W\mathbf h_u\oplus\mathbf W\mathbf h_v]) eq. 42.
• GraphSAGE: concatenate the base update with \mathbf h_u against over-smoothing eq. 43.
• GGNN: \mathbf h_u^{(k)}=\text{GRU}(\mathbf h_u^{(k-1)},\mathbf m^{(k)}_{\mathcal N(u)}) eq. 44; LSTM variants also mentioned §5.2.

Pseudo-labeling §6 fig. 6

Unlike consistency methods, these rely on confident pseudo-labels added to the training set §6.

Disagreement-based §6.1.

• Deep co-training: views x=(v_1,v_2), classifiers \mathcal C_1,\mathcal C_2. Agreement loss \mathcal{L}_{ct}=H(\tfrac12(\mathcal C_1(v_1)+\mathcal C_2(v_2)))-\tfrac12(H(\mathcal C_1(v_1))+H(\mathcal C_2(v_2))) eq. 45; supervised \mathcal{L}_s=H(y,\mathcal C_1(v_1))+H(y,\mathcal C_2(v_2)) eq. 46; View Difference Constraint: the classifiers must disagree on adversarial examples \mathcal X' eq. 47, implemented as a cross-entropy between one classifier on x and the other on adversarial examples from generators g_1,g_2 eq. 48. Other uses: RGB and depth as two views, original and "anonymous" (opposite) reviews in sentiment classification §6.1.
• Tri-Net: output smearing builds three training sets from X_L; a shared module M_S feeds three heads M_1,M_2,M_3, initialised with the summed softmax cross-entropy eq. 49. When two heads agree on an unlabelled sample, that pseudo-label is added to the third head's set and the third head is fine-tuned; heads are periodically fine-tuned on their own sets to keep diversity. The pseudo-label is the argmax of the summed head probabilities §6.1.

Self-training §6.2.

• EntMin: push unlabelled predictions to low entropy so the boundary avoids dense regions.
• Pseudo-label: the max-probability class of each unlabelled sample is its target; loss = mean labelled loss + \alpha(t) × mean unlabelled loss over mini-batches §6.2.
• Noisy Student: EfficientNet teacher trained on labels pseudo-labels unlabelled data; an equal-or-larger student trains on both with RandAugment, dropout and stochastic depth; the student becomes the next teacher, repeated for a few iterations §6.2.
• S^4L: adds self-supervised losses: rotation prediction over \{0^\circ,90^\circ,180^\circ,270^\circ\} (cross-entropy), or Exemplar (8 copies per image via inception crop, random horizontal flip, HSV colour randomisation; batch-hard triplet loss with soft margin) §6.2.
• Meta Pseudo Labels: teacher g_\phi produces target distributions; the student f_\theta updates on them; the student's loss on a held-out validation set after the update gives the teacher's gradient, alternating §6.2.
• EnAET: four spatial transforms (projective, affine, similarity, Euclidean) and a combined non-spatial transform (colour, contrast, brightness, sharpness, four strength parameters). AET loss \mathbb{E}_{x,t(x)}\|D[E(x),E(t(x))]-t(x)\|^2 is added as a regulariser, plus KL between P(y\mid x) and P(y\mid t(x)) §6.2.
• SimCLRv2: (1) contrastive pre-training with \ell_{i,j}=-\log\frac{\exp(\text{sim}(z_i,z_j)/\tau)}{\sum_{k=1}^{2N}\mathbb{I}_{[k\ne i]}\exp(\text{sim}(z_i,z_k)/\tau)}, cosine similarity, temperature \tau; (2) supervised fine-tuning on 1% or 10% of labels; (3) self-training by distillation, \mathcal L^{\text{distill}}=-\sum_{x_i}\sum_y P^T(y\mid x_i;\tau)\log P^S(y\mid x_i;\tau) §6.2.

Hybrid methods §7 fig. 7

Built on Mixup: \tilde x=\lambda x_i+(1-\lambda)x_j, \tilde y=\lambda y_i+(1-\lambda)y_j, \lambda\in[0,1] §7.

• ICT: \mathbb{E}\,\mathcal{R}(f(\theta,\text{Mix}_\lambda(x_i,x_j)),\text{Mix}_\lambda(f(\theta',x_i),f(\theta',x_j))) on unlabelled pairs, with \theta' an EMA of \theta as in Mean Teacher §7.
• MixMatch, as the survey describes it §7:

Algorithm: MixMatch (survey's description)  [§7]
Input: labelled batch, unlabelled batch, K augmentations, Beta(α, α)
1. Augment each labelled (x_i, y_i) once.
2. Make K weak augmentations of each unlabelled x_j.
3. Pseudo-label ȳ_j = mean prediction over the K augmentations.
4. Sharpen ȳ_j with temperature scaling -> ỹ_j  (temperature not stated).
5. Concatenate augmented labelled and pseudo-labelled unlabelled examples; shuffle -> W.
6. W_L = first L items of W, W_U = the rest.
7. Mixup(augmented labelled batch, W_L) and Mixup(augmented unlabelled batch, W_U),
   x' = λ x_1 + (1-λ) x_2, λ ~ Beta(α, α).
8. Loss: cross-entropy on mixed labelled + mean squared error on mixed unlabelled.

• ReMixMatch: adds distribution alignment (marginal of unlabelled predictions matched to the label marginal) and augmentation anchoring (several CTAugment strong views pulled towards the prediction on one weak "anchor" view), replacing MixMatch's consistency term §7.
• DivideMix: noisy-label learning. Two networks; per network a GMM on per-sample losses splits data into labelled and unlabelled for the other network's next epoch (co-divide); MixMatch with co-refinement and co-guessing §7.
• FixMatch: pseudo-label from the weakly augmented view (flip-and-shift) is used only if confident, and is the target for a strongly augmented view (RandAugment or CTAugment, followed by Cutout) §7.

Augmentation and network-perturbation map tab. 5:

Open problems §8

Theory: what augmentations, training procedures and losses actually contribute is unclear, and a single weight treats all unlabelled examples equally (per-example weighting is one fix). Domain knowledge: when labelled and unlabelled distributions differ or assumptions fail, more unlabelled data can make results worse. Noisy labels. Class imbalance (most methods assume uniformly distributed classes). Robustness via adversarial training and strong augmentation (Cutout, RandomErasing, HideAndSeek, GridMask, AutoAugment, RandAugment, CTAugment, Mixup). Safe SSL: methods that never significantly degrade performance when unlabelled data is added §8.

Hyperparameters

The survey has no experiments and no hyperparameters of its own. Values it quotes from surveyed methods (check each against the original paper before relying on it):

Results

No experiments of its own, and no accuracy numbers are reported for the surveyed methods, so the survey cannot be used to rank methods by performance. The only quantitative comparison is the time-complexity column of tab. 4 (above).

Paper vs code

No official code.

Pitfalls

• Notation clashes. \alpha means the eq. 1 trade-off weight, the Triple GAN mixing weight eq. 7, the Temporal Ensembling momentum §4, the Dirichlet parameter eq. 12, the WCP DropConnect mask eq. 26, the GAT attention weight eq. 41 and the Mixup Beta parameter §7. K is both the number of classes §2 and MixMatch's number of augmentations §7; L is both the labelled-set size §2 and MixMatch's labelled batch split §7. D is the discriminator, the degree matrix and the EnAET decoder tab. 1 §6.2. Read each equation in its own subsection.
• Ladder Network formula differs between text and table. eq. 16 includes the denoising decoder g, tab. 3 drops it. The text says the loss is per layer, but both formulas show only one output-level term. Use the per-layer description §4.
• Mean Teacher noise. eq. 19 uses different noise \zeta,\zeta' for student and teacher; tab. 3 uses the same \zeta for both. Which one is meant is not stated.
• VAT is written with g(\theta,\cdot) in eq. 20 and f in tab. 3, and the text writes the constraint as \gamma\le\delta rather than \|\gamma\|\le\delta §4. How the arg max for \gamma^{adv} is computed is not described.
• Disentangled VAE bound eq. 13 is missing the \log in front of q_\phi(y,z\mid x), and takes the expectation over q_\phi(z\mid x,y) although the text has just said that factorisation is not used. Do not implement it as printed.
• VGAE likelihood eq. 35 has index j unbound (the product runs over i only) and sums the two Bernoulli branches without the per-edge structure. Use eq. 33 as the objective and take the edge likelihood from the original VGAE paper.
• Unified graph loss eq. 28 prints the supervised term without a loss symbol, \sum(x,y,f_{\mathbf z}).
• GCN degrees. eq. 40 sums over \mathcal N(u)\cup\{u\} but normalises with |\mathcal N(u)|, |\mathcal N(v)|, so it is not stated whether the degree counts the self-loop. eq. 39 also has misplaced absolute-value bars in the source.
• Deep co-training loss eq. 48 contains an undefined p_1, and the sentence before it pairs the classifiers and generators differently (\mathcal C_2(x) vs \mathcal C_1(g_2(x))) from the formula. Check the original paper.
• Citation slips. Enhanced TGAN and MarginGAN are said to build on Triple GAN, but the reference attached is the one used for Triangle GAN §3.1. GAT is attributed to "Cucurull et al." with no reference §5.2. Verify attributions before linking.
• Missing training details. FixMatch's confidence threshold and unlabelled loss form, MixMatch's sharpening temperature and K, Pseudo-label's \alpha(t) schedule and the Mean Teacher EMA decay are not given §6.2 §7. The survey is a map to the original papers, not a substitute for them.
• Temporal Ensembling targets are stale. They are refreshed once per epoch §4, so they lag the current model; the survey gives no bias correction or start-up handling.
• Unlabelled data can hurt. If labelled and unlabelled distributions differ, or the cluster or manifold assumptions fail, adding unlabelled data can reduce accuracy below the supervised baseline §2.1 §8. Keep a supervised-only baseline.
• Class balance assumed. SSL methods commonly assume classes are uniformly distributed, and on highly imbalanced data most learners lean towards the majority class §8.
• Scope. Image classification is the focus §1; graph methods are mostly transductive §2; the graph section covers embedding methods only, not classical Laplacian regularisation §5.

Lineage

• Builds on: classical SSL surveys and books (Chapelle et al., Zhu) and the Oliver et al. evaluation study §1; this survey adds deep methods and a five-family taxonomy.
• Related survey in this wiki: Self-supervised Learning: Generative or Contrastive, cited in §2.3.
• Covers, among others: Improved GAN, Triple GAN, BiGAN/ALI, SSVAE (M1/M2), ADGM, Ladder Network, Π Model, Temporal Ensembling, Mean Teacher, VAT, UDA, GAE/VGAE, GCN, GAT, GraphSAGE, Pseudo-label, Noisy Student, Meta Pseudo Labels, SimCLRv2, ICT, MixMatch, ReMixMatch, DivideMix, FixMatch.
• Compared against: not applicable (survey).