Semi-Supervised and Unsupervised Deep Visual Learning: A Survey

In plain words

Deep vision models usually learn from large sets of labelled photos. Labelling is slow and expensive, and a model trained on a narrow set of labels may fail on new lighting, viewpoints or places. Unlabelled images are cheap, so two families of methods try to use them. Semi-supervised learning trains on a few labels and many unlabelled images in the same run. Unsupervised learning first learns general image features from unlabelled images alone, and those features are then fine-tuned on a task that has labels.

This survey sorts the main methods in both families by the idea they rely on. One idea is to make the model give the same answer to an image and to a changed copy of it. Another is to let the model label its own confident guesses. A third is to make neighbouring images get the same label through a graph. A fourth is to invent a task from the image itself, such as predicting a rotation or filling in a hidden patch. The survey also shows how these ideas are used for segmentation, object detection and adapting a model to a new domain, and it lists open problems.

TL;DR

A survey, not a method paper. It groups semi-supervised learning (SSL) into five families and unsupervised learning (UL) into three, with one table each tab. 1 tab. 2. It argues that most SSL and UL methods rest on one of three smoothness rationales: consistency across transformations, smoothness between neighbours, or pseudo-targets from global decision boundaries §4.1 tab. 3. It covers segmentation, detection and domain adaptation §4.2, and lists open problems in open-set, incremental and multi-modal learning §5. It reports no experiments of its own §1.

Problem

Supervised deep models need large labelled training sets §1. Collecting them is expensive, and may be restricted by privacy, security or ethics rules §1. Supervised models can also memorise their labelled data, pick up annotator bias, and generalise poorly when the test distribution differs from the training one §1. The survey cites two annotation costs from other work: about 1.5 hours for one pixel-wise annotated Cityscapes image §4.2.1, and up to 42 seconds per bounding box for detection §4.2.2. Its question is how unlabelled images can improve generalisation or give a better starting point for a model §1.

Method

The paper has no method of its own. Its contribution is a set of taxonomies, a shared formal objective, and a discussion of how the families relate §1. Each method is described in one sentence to one paragraph, with formulas only where the families share a form. For implementation details, read the cited original paper.

Scope and setups

• SSL uses a small labelled set \mathcal{D}_l with K classes and a large unlabelled set \mathcal{D}_u, and trains both in one stage. It is evaluated by whether it beats its own supervised baseline, and whether the gain grows as the unlabelled share grows §2.1.
• UL uses only unlabelled data for the main training. It is two-stage: pre-train on unlabelled images, then fine-tune on a labelled downstream task. It is evaluated by linear probing (frozen features with a linear or k-NN classifier) or by fine-tuning for detection or segmentation §3.1.
• Unified objective eq. 1:

\min_{\theta} \; \lambda_l \sum_{(\mathbf{x},y) \in \mathcal{D}_L} \mathcal{L}_{\text{sup}}(\mathbf{x}, y, \theta) + \lambda_u \sum_{\mathbf{x} \in \mathcal{D}_U} \mathcal{L}_{\text{unsup}}(\mathbf{x}, \theta) eq. 1

Here \theta are the parameters of a deep network and \lambda_l, \lambda_u balance the two terms. For UL pre-training, \lambda_l = 0 §1. Many SSL methods make \lambda_u a ramp-up function of the training iteration t §2.1.

# Unified SSL/UL loss for one update  [eq. 1] [§1]
# Inputs: a labeled batch (x, y) and an unlabeled batch x
# Output: scalar loss; UL pre-training uses lambda_l = 0  [§1]

def ssl_ul_loss(model, labeled_batch, unlabeled_batch, lambda_l, lambda_u):
    # Supervised term on the labeled pairs  [eq. 1]
    supervised = sup_loss(model, labeled_batch)
    # Unsupervised term; its form depends on the family  [eq. 1] [§2.2]
    unsupervised = unsup_loss(model, unlabeled_batch)
    # Weighted sum; many SSL methods ramp lambda_u up over t  [§2.1]
    return lambda_l * supervised + lambda_u * unsupervised

SSL families (Table 1 tab. 1)

Consistency regularization §2.2.1. The model must give consistent outputs for an input and a variant of it. Input variants come from a transformation \hat{x} = q_x(x; \epsilon), and the loss is eq. 2:

\min_{\theta} \; \sum_{x \in \mathcal{D}} d\big(p(y|x;\theta),\, \hat{p}(y|\hat{x};\theta)\big) eq. 2

Model variants come from a changed model \hat{\theta} = f_\theta(\theta; \eta), and the loss is eq. 3:

\min_{\theta} \; \sum_{x \in \mathcal{D}} d\big(p(y|x;\theta),\, \hat{p}(y|x;\hat{\theta})\big) eq. 3

The discrepancy d is a divergence or distance such as KL divergence, cross-entropy or mean squared error §2.2.1.

• Input side: random augmentation (Π-model, ensemble transformations); adversarial perturbation (Virtual Adversarial Training); MixUp-style mixing (ICT, MixMatch); automated augmentation (ReMixMatch, UDA, FixMatch) §2.2.1.
• Model side: stochastic perturbation of weights or structure (Pseudo-Ensemble, Ladder Network, Virtual Adversarial Dropout, Worst-Case Perturbations); ensembling in weight or prediction space through averages (Temporal Ensembling, Mean Teacher, SWA, UASD) §2.2.1.

Self-training §2.2.2. The model labels unlabelled samples it is confident about, then learns from those labels. Three forms:

• Entropy minimisation eq. 4 pushes each unlabelled prediction towards low entropy:

\min_{\theta} \; \sum_{x \in \mathcal{D}} \Big( -\sum_{j=1}^{K} p(y_j|x;\theta) \log p(y_j|x;\theta) \Big) eq. 4

It can act in prediction space (Pseudo-Label; MixMatch sharpens averaged predictions with a temperature; FixMatch keeps one-hot labels only above a confidence threshold) or in feature space (a memory module keeps one centre per class and assigns samples to the nearest one) §2.2.2.

• Co-training trains two or more classifiers on different views and lets each one label data for the others eq. 5:

\min_{\theta} \; \sum_{x \in \mathcal{D}} d\big(p_1(y|x;\theta_1), z_2\big) + d\big(p_2(y|x;\theta_2), z_1\big) eq. 5

Tri-training labels a sample when two of three classifiers agree and their confidence is above a threshold §2.2.2.

• Distillation trains a student on a teacher's targets eq. 6:

\min_{\theta} \; \sum_{x \in \mathcal{D}} d\big(p_s(y|x;\theta_s),\, z_t\big) eq. 6

Model distillation uses the teacher's soft or one-hot labels (Noisy Student, UASD). Data distillation averages a teacher's outputs over several transformations of the same input (Radosavovic et al.), and uses two stages §2.2.2.

Graph-based regularization §2.2.3. A nearest-neighbour graph encodes similarity between samples. Feature-space methods add a contrastive-style loss that pulls graph neighbours together and pushes non-neighbours apart by a margin eq. 7:

\min_{\theta} \sum_{x_i, x_j \in \mathcal{D}} \begin{cases} \|h(x_i) - h(x_j)\|^2 & \text{if } W_{ij} = 1 \\ \max\big(0,\, m - \|h(x_i) - h(x_j)\|\big)^2 & \text{if } W_{ij} = 0 \end{cases} eq. 7

Methods named: EmbedNN and Teacher Graph (feature space), Graph Convolutional Networks, and Label Propagation (prediction space, alternating label propagation and network updates) §2.2.3 tab. 1.

Deep generative models §2.2.4. Two streams:

• VAEs add a class variable to the latent space and an explicit classification loss on labelled data (class-conditional VAE, ADGM) §2.2.4.
• GANs use a (K{+}1)-class discriminator: real labelled samples get one of the K classes, generated samples get class K{+}1. The objective has a supervised term eq. 8 and an unsupervised term eq. 9:

\max_{\theta} \; \sum_{x \in \mathcal{D}} \log p(y \mid x_l,\, y < K{+}1) eq. 8

\max_{\theta} \; \sum_{x \in \mathcal{D}} \log\big(1 - p(y{=}K{+}1 \mid x_l)\big) - \log p(y{=}K{+}1 \mid x_G) eq. 9

CatGAN, feature-matching GAN, ALI, BadGAN and Localized GAN follow this scheme. Localized GAN also regularises the generator to keep samples near their source on the manifold §2.2.4.

Self-supervised learning for SSL §2.2.5. Self-supervision is task-agnostic, so it can be pre-training or an auxiliary loss. Examples: S4L (rotation prediction and exemplar invariance as auxiliary tasks), SimCLR and SimCLRv2 (contrastive pre-training, then supervised or semi-supervised fine-tuning) §2.2.5.

UL families (Table 2 tab. 2)

Pretext tasks §3.2.1. Hand-made proxy tasks need no labels.

• Pixel-level tasks reconstruct pixel values eq. 10:

\min_{\theta} \; \sum_{x \in \mathcal{D}} \lVert G_\theta(x) - \hat{x} \rVert^2 eq. 10

G_\theta is usually an encoder–decoder, and the encoder is reused downstream. The named tasks are auto-encoding, inpainting (including masked auto-encoders), denoising and colorization §3.2.1 fig. 8.

• Instance-level tasks predict an applied transformation z from the transformed input \Phi_z(x) eq. 11:

\min_{\theta} \; \sum_{x \in \mathcal{D}} \mathcal{L}_{\text{unsup}}\big(\Phi_z(x),\, z,\, \theta\big) eq. 11

Examples: rotation, scaling and tiling, patch ordering, and jigsaw puzzles §3.2.1 fig. 9.

Discriminative models §3.2.2. They enforce invariance to views, neighbours or groups.

• Instance discrimination, mostly contrastive, uses an InfoNCE loss eq. 12:

\min_{\theta} \; \sum_{x_i \in \mathcal{D}} -\log \frac{\exp\big(f_\theta(x_i) \cdot f_\theta(x_i^+)/\tau\big)}{\sum_{j=1}^{M} \exp\big(f_\theta(x_i) \cdot f_\theta(x_j)/\tau\big)} eq. 12

x_i^+ is a second view of the same instance, and the M samples in the denominator are one positive and M{-}1 negatives. Negatives come from large batches (SimCLR uses 4096), a memory bank (InstDis, MoCo, PIRL, CMC), or a queue (MoCo) §3.2.2. Positive pairs come from augmentation or other transformations §3.2.2. Negative-free methods (BYOL, SimSiam, DirectPred) match two views with a stop-gradient to avoid collapse; Barlow Twins drives a cross-correlation matrix to the identity §3.2.2 fig. 10.

• Deep clustering uses cluster memberships as pseudo-labels eq. 13:

\min_{\theta} \; \sum_{x \in \mathcal{D}} \mathcal{L}_{\text{unsup}}(x, \hat{y}, \theta) eq. 13

Offline methods alternate clustering and network training: DeepCluster (k-means), JULE (agglomerative clustering), SeLa (Sinkhorn–Knopp optimal transport). Online methods couple both in one loop: IIC maximises mutual information between an image's predictions and those of a perturbed copy; SwAV assigns samples to learned prototypes §3.2.2 fig. 11.

Deep generative models §3.2.3. A GAN is trained with the standard minimax game eq. 14:

\min_{G} \max_{D} \; \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))] eq. 14

Representations come from the discriminator (DCGAN; self-supervised GAN and transformation GAN, which add a transformation-prediction task) or from the generator-side encoder (BiGAN; BigBiGAN, with a stronger architecture and an extra unary discriminator loss) §3.2.3 fig. 12.

Shared rationales (Discussion §4.1)

The survey argues that most SSL and UL methods share one of three smoothness assumptions fig. 13 tab. 3:

• Local smoothness across views eq. 15: a sample and its transformed copy get the same label. This covers consistency regularization in SSL and instance discrimination in UL §4.1.1.

\min_{\theta} \; \sum_{x_i \in \mathcal{D}} \mathcal{L}_{\text{unsup}}\big(f(x_i),\, f(\hat{x}_i)\big) eq. 15

• Local smoothness across neighbours eq. 16: a sample and a nearby sample get the same label. This covers graph-based regularization in SSL and neighbourhood consistency in UL §4.1.1.

\min_{\theta} \; \sum_{x_i \in \mathcal{D}} \mathcal{L}_{\text{unsup}}\big(f(x_i),\, f(x_j)\big) eq. 16

• Global smoothness eq. 17: a sample gets a target from class boundaries the model has found, which are assumed to lie in low-density regions. This covers self-training in SSL and deep clustering in UL §4.1.2.

\min_{\theta} \; \sum_{x_i \in \mathcal{D}} \mathcal{L}_{\text{unsup}}\big(f(x_i),\, z_i\big) eq. 17

The setups also differ. SSL trains on labelled and unlabelled data together. UL pre-trains on unlabelled data and then fine-tunes, and its unlabelled and labelled data may differ in distribution, which makes transfer harder to predict §4.1.3.

tab. 3

Applications (§4.2)

• Semantic segmentation §4.2.1. Consistency is applied per pixel. The first method perturbs inputs with CutOut and CutMix and enforces pixel-level agreement; cross-consistency training adds noise to features instead. Self-training repeats teacher-based re-labelling of unlabelled images, or co-trains two models. Contrastive pixel losses choose positives at the same location in two crops, or features before and after a colour change, with negatives sampled from other images §4.2.1.
• Object detection §4.2.2. Consistency methods flip boxes or recompute overlapping boxes under MixUp. Teacher–student methods derive pseudo-boxes from an EMA teacher or from NMS on the model's own outputs. Self-training annotates unlabelled images with a teacher, then refines the labels interactively or selects them by uncertainty §4.2.2.
• Unsupervised domain adaptation §4.2.3. Consistency uses dual MixUp across categories and domains, and self-ensembling with a Mean Teacher. Self-training uses entropy minimisation, pseudo-labels or co-training on target data, combined with distribution alignment. Generative models translate images between domains at pixel level (PixelDA, CyCADA and others). Self-supervised auxiliary tasks such as jigsaw puzzles or rotation are trained alongside the supervised source objective §4.2.3.

Emerging trends (§5)

• Open-set learning §5.1: unlabelled data may contain unseen classes. UASD and DS³L down-weight the unsupervised loss in proportion to how likely a sample is to be from an unseen class. Later methods use curriculum learning, an OOD classifier with iterative self-training, or one-vs-all inlier/outlier classifiers (OpenMatch). Open challenges: OOD detection, long-tailed unlabelled data, and discovering unseen classes §5.1.
• Incremental learning §5.2: unlabelled data arrives as a stream, and old data may be unavailable. Methods limit forgetting with importance weights or distillation, and adapt to new classes with online clustering or mixture-of-Gaussian updates. Open challenges: catastrophic forgetting, new concepts, and confirmation bias of pseudo-labels on non-i.i.d. streams §5.2.
• Multi-modal learning §5.3: vision–language BERT-style models use a masked-word (Cloze) objective and linguistic–visual alignment. Contrastive language supervision (CLIP, ALIGN) aligns images and text. Audio–visual methods use the natural correspondence in videos. Open challenge: the shared-semantics assumption fails when modalities carry different content §5.3.

Hyperparameters

The survey is not a training recipe, so it gives almost no training values. Everything that is stated is listed below.

Results

Not applicable. The paper reports no experiments and no benchmark table of its own §1. The only concrete quantities in the text, apart from the formulas, are two annotation costs and the SimCLR batch size, all cited to other work §3.2.2 §4.2.1 §4.2.2. Results of the cited methods must be taken from their original papers.

Paper vs code

No official code. The paper gives no code or project link (checked the LaTeX source for URLs and the arXiv metadata, which has no comment field). I searched GitHub for "semi-supervised unsupervised visual survey" and "semi-supervised survey" via the repositories API. The results were third-party repositories, none owned by an author or an author's lab, so none was used.

Pitfalls

• Survey, not a method. Each method gets one sentence or a short paragraph. Schedules, architectures, thresholds and most loss weights are absent. Implement from the cited originals, not from this card.
• Sign in Eq. 9. As printed, the generated-sample term is -\log p(y{=}K{+}1 \mid x_G) inside a maximisation. That pushes the generated-class probability down, while the text says generated samples belong to class K{+}1 §2.2.4 eq. 9. Check the original GAN-based SSL paper before implementing. This may be a typesetting or sign error.
• Eq. 8 index. The sum runs over \mathcal{D}, but the term uses only the labelled sample x_l. The paper does not say whether unlabelled samples enter this sum §2.2.4 eq. 8.
• Undefined targets in Eq. 5. z_1 and z_2 are never defined. The choice of hard or soft targets appears only in prose §2.2.2 eq. 5.
• Eq. 7 does not match the prose. The text says the regulariser minimises pairwise similarities. The equation pulls W = 1 pairs together and pushes W = 0 pairs apart by margin m, using squared norms in one branch and not the other. The text calls W_{ij} an affinity matrix with soft weights, but Eq. 7 treats it as a hard 0/1 neighbour indicator. The margin is not given a value §2.2.3 eq. 7.
• Eq. 12 has unstated details. The paper does not say whether features are normalised, and gives no temperature value. The denominator includes the positive, so M counts the positive plus negatives §3.2.2 eq. 12.
• Notation drift in Eq. 1. The equation writes \mathcal{D}_L and \mathcal{D}_U, while the text defines \mathcal{D}_l and \mathcal{D}_u §1 eq. 1.
• Wrong citation in Table 2. BiGAN is cited to the BigGAN paper, while the text cites the BiGAN paper for BiGAN and the BigGAN paper for BigGAN tab. 2 §3.2.3.
• Naming mismatches. Table 1 writes "EmbedNN" and the text writes "EmbedCNN" tab. 1 §2.2.3. UASD is expanded once as "USAD" §2.2.2.
• Families overlap. MixMatch and ReMixMatch appear under two SSL families, and UASD under two others tab. 1. The taxonomy is a grouping, not a partition.
• Overlapping names for assumptions. §2.1 lists cluster, structure and low-density separation as aliases of the manifold assumption. §2.2.2 later treats low-density separation as its own basis for entropy minimisation. Read these as related, not identical.
• Figures and claims cited from other work. Cityscapes annotation time (1.5 h per image) §4.2.1, box annotation time (42 s) §4.2.2 and SimCLR's batch size §3.2.2 come from other papers. They were not checked here.
• Dated snapshot. The arXiv v1 dates from August 2022, and the open problems in §5 describe the state of the field then. Recent work may have addressed them.

Lineage

• Builds on: nothing stated as a method. The paper positions itself against earlier SSL, self-supervised and combined surveys cited in §1. None of them has a card here.
• Compared against: the same surveys, which the paper says focus on one setup, on non-deep methods, or lack a full taxonomy §1.
• Superseded by: not known from content already in this wiki.