---
kind: concept
slug: consistency-regularization
title: Consistency regularization
summary: >-
  A loss asking a model to give the same prediction for an input and a perturbed copy, or for an input and a frozen reference model. Variants differ in what they perturb (noise, adversarial direction, augmentation, dropout, weights) and where the target comes from (same pass, past epochs, weight average, pseudo-labels, frozen model). Trade-off: extra passes or stored state versus target freshness.
tags: [semi-supervised-learning, consistency-regularization, teacher-student, augmentation, prompt-learning, lidar, semantic-segmentation]
papers: ["2103.00550", "2102.09480", "2208.11296", "2110.08263", "2205.07246", "2307.06948", "2405.05258"]
related: [noisy-labels, semi-supervised-learning, self-training, object-detection, semantic-segmentation, data-augmentation, curriculum-learning, catastrophic-forgetting, soft-prompt-tuning]
status: draft
updated_at: 2026-11-01
---

## Overview

Consistency regularization is a family of semi-supervised losses. The model is
trained on a small labelled set as usual. On unlabelled inputs it is not given
a label. Instead, the prediction for one view of an input must match a target
built from another view of the same input [arxiv:2103.00550 §4]. The shared
form is a distance R between the student's output and a target T, averaged over
unlabelled inputs, and it is added to the supervised loss with a trade-off
weight [arxiv:2103.00550 §2, §4].

The survey's framing: targets come from the model itself, so these methods are
teacher-student methods. The target is the prediction of a teacher that is the
same network, an averaged copy, or a stored average of past predictions. Because
the target is made by the model, the model can reinforce its own early errors.
This is called confirmation bias, and better target construction is the main
way the variants try to reduce it [arxiv:2103.00550 §4].

The variants differ in two choices: which perturbation makes the second view
(see the table in [Variants](#variants)), and how the target is kept. The
survey groups perturbations into input augmentations (noise, stochastic
augmentation, adversarial perturbation, AutoAugment, RandAugment, CTAugment,
Mixup) and network transformations (dropout, EMA, SWA, stochastic depth,
DropConnect) [arxiv:2103.00550 tab. 5].

Consistency is one of the five families of deep semi-supervised learning in the
survey's taxonomy, next to generative, graph-based, pseudo-labeling and hybrid
methods [arxiv:2103.00550 §1]. The assumptions behind it are that the decision
boundary lies in low-density regions and that neighbours on the data manifold
share a label [arxiv:2103.00550 §2.1]. If those assumptions fail, unlabelled
data can lower accuracy [arxiv:2103.00550 §2.1].

A second survey, which covers visual recognition, counts consistency regularization
as one of five semi-supervised families [arxiv:2208.11296 tab. 1]. It places the
same idea under local smoothness across views: a sample and its transformed copy
should get the same label, which is also the basis of instance discrimination in
unsupervised learning [arxiv:2208.11296 §4.1.1] [eq. 15]. The two surveys name
different families (see [Contradictions](#contradictions)).

The same loss is also used without unlabelled data. Here the reference is the
unmodified pretrained model, and the goal is to stop a small fine-tune from
drifting away from what that model already knows. The adapted model is pulled
toward the frozen model's features and predictions on the same training inputs
[arxiv:2307.06948 §3.2.1]. This is the anchor form of the idea, and it is
distinct from the teacher-student form above, where the teacher is the model
being trained or a copy of it.

## Variants

Variants without a card of their own are described in the survey card,
[A Survey on Deep Semi-supervised Learning](/papers/2103.00550). Variants with a
card link to it in their section.

### Input noise with a decoder: Ladder Network

Noise is added in every hidden layer of a corrupted encoder that shares weights
with a clean encoder. A decoder denoises each layer, and the unsupervised loss is
the per-layer reconstruction error between clean and reconstructed activations
[arxiv:2103.00550 §4] [eq. 16]. The survey notes that its text and its summary
table disagree on the exact formula, so use the per-layer description
[arxiv:2103.00550 §4] (see [Contradictions](#contradictions)).

### Two stochastic passes: Π Model

Each sample, labelled or not, goes through the network twice with different
random augmentation and dropout. The loss is the distance between the two
outputs [arxiv:2103.00550 §4] [eq. 17]. This needs no extra model state, but it
costs two forward passes per sample [arxiv:2103.00550 §4] [eq. 17].

### Per-sample ensemble targets: Temporal Ensembling

Each sample gets one forward pass per step. Its target is an exponential moving
average of its own past predictions, stored per sample and updated after each
epoch with a momentum [arxiv:2103.00550 §4] [eq. 18]. The target therefore lags
the current model, and the survey does not give a bias correction or a start-up
rule [arxiv:2103.00550 §4] (see [When to use which](#when-to-use-which)).

### Weight-averaged teacher: Mean Teacher

The teacher is an exponential moving average of the student's weights over
training steps. The student is pulled toward the teacher's prediction on a
differently augmented copy of the input [arxiv:2103.00550 §4] [eq. 19]. The
survey's summary table and eq. 19 disagree on which noise is applied to the
teacher, and the EMA decay is not stated [arxiv:2103.00550 §4]. Mean Teacher
recurs in the hybrid ICT method, which uses the same teacher on Mixup pairs
[arxiv:2103.00550 §7].

### Adversarial direction: VAT

The perturbation is not random. It is the small input change, within a fixed
radius, that moves the prediction the most. The loss compares the prediction on
the clean input with the prediction on that adversarial input [arxiv:2103.00550
§4] [eq. 20]. The survey does not describe how the maximising perturbation is
computed, and it does not give the radius [arxiv:2103.00550 §4].

### Adversarial direction plus dropout: VAdD

Adversarial perturbation is combined with random dropout masks. The adversarial
dropout mask is searched within a bounded distance of a random mask. The bound
scales with the dropout layer's dimension and a noise intensity [arxiv:2103.00550
§4] [eq. 24]. Neither value is given in the survey [arxiv:2103.00550 §4].

### Two students with stability filtering: Dual Student

Two students start from different initial weights. A sample is stable for a
student when that student's prediction agrees across the sample's two versions
and its confidence is above a threshold ξ [arxiv:2103.00550 §4] [eq. 21]. If both
students find the sample stable, the one with the larger ε is pulled toward the
other. Otherwise the pull is gated by each student's own stability flag
[arxiv:2103.00550 §4] [eq. 22]. The threshold ξ is not stated in the survey
[arxiv:2103.00550 §4].

### Weight averaging along a trajectory: SWA

Stochastic gradient descent on the consistency loss keeps moving between
distant solutions instead of settling. Weights are averaged along a cyclical
learning-rate path, and fast-SWA averages several points per cycle over a
longer cycle. The survey reports this as giving flatter solutions
[arxiv:2103.00550 §4] [eq. 23].

### Worst-case weight and structure noise: WCP

Worst-case additive noise is applied to the weights, under a norm constraint.
Worst-case DropConnect drops individual connections. The consistency target is
the output of the perturbed network [arxiv:2103.00550 §4] [eq. 25–26].

### Strong augmentation: UDA

Simple noise is replaced by strong, domain-specific augmentation. For images
these are AutoAugment and RandAugment, and for text back-translation
[arxiv:2103.00550 §4] [eq. 27]. The survey's augmentation map lists UDA under
AutoAugment and RandAugment [arxiv:2103.00550 tab. 5].

### Weak and strong views for detection: Unbiased Teacher

[Unbiased Teacher for Semi-Supervised Object Detection](/papers/2102.09480)
applies the teacher-student pattern to object detection. A Faster R-CNN detector
is first trained on labelled images only (burn-in), and the teacher and student
are then started from those weights [arxiv:2102.09480 §3.1]. The teacher is an
exponential moving average of the student [arxiv:2102.09480 §3.2] [eq. 3]. It
sees a weakly augmented image (horizontal flip) and proposes boxes. The student
sees a strongly augmented copy (colour jitter, grayscale, blur, cutout) and
learns from those boxes [arxiv:2102.09480 §3.2] [tab. 6].

The target is a hard pseudo-label, not a distance between outputs. Duplicate
boxes are removed per class, and boxes below a confidence threshold of 0.7 are
dropped [arxiv:2102.09480 §3.2] [tab. 5]. Only the classification losses of
the region-proposal and box-classifier heads take pseudo-labels. Box
regression is not trained on them, because confidence does not measure how
well a box is placed [arxiv:2102.09480 §3.2].

The paper argues that class imbalance biases pseudo-labels toward background
and common classes. It counters this with a focal loss on the box classifier
and the slow teacher [arxiv:2102.09480 §3.3] [fig. 5]. The EMA rate multiplies
the teacher: with 0.9996, the teacher keeps 99.96% of its weights at each step,
and using the value the other way round inverts the method [arxiv:2102.09480
eq. 3].

The paper reports results for its own setup only. These are author-reported
gains at 0.5% to 10% of the COCO train2017 images labelled, and it is compared
only with STAC and CSD [arxiv:2102.09480 tab. 1] (see
[Contradictions](#contradictions)).

### Mixed LiDAR scans with a mean teacher: LaserMix and LaserMix++

[LaserMix++](/papers/2405.05258) applies the teacher-student pattern to
semi-supervised LiDAR semantic segmentation. Two scans are cut into bands by
laser-beam inclination angle, and the bands are swapped between them, so each
band keeps its neighbours from the other scan [arxiv:2405.05258 §3.2] [eq. 7].
The student predicts the mixed scan, and the mixed labels come from ground truth
and from the teacher's pseudo-labels on the unlabelled part [arxiv:2405.05258
§3.3]. A mean-teacher term asks the student's prediction on the unlabelled scan
to match the teacher's, as a squared ℓ2 distance [arxiv:2405.05258 §3.3] [eq. 9].
The teacher's pseudo-labels are kept only above a confidence threshold, which
the paper sets at about 0.9 [arxiv:2405.05258 §5.4].

The camera extension adds three more terms on points that project to a valid
pixel. One distils image features onto the LiDAR features [arxiv:2405.05258
§4.3] [eq. 13]. Another aligns them with text-aligned image features
[arxiv:2405.05258 §4.4] [eq. 15]. The overall loss is a weighted sum of these
terms with the supervised and mixing losses [arxiv:2405.05258 §4.5] [eq. 16].
The paper reports these camera terms as part of its results, but the repository
it links releases only the LaserMix part (see [Contradictions](#contradictions)).

The paper's reported gain, for Cylinder3D on SemanticKITTI with 1% of the
labels, is 56.2 mIoU, against 45.4 for supervised-only training and 50.6 for
LaserMix [arxiv:2405.05258 tab. 2]. These are author-reported. The text reports
no seeds or variance for them [arxiv:2405.05258]. The paper compares
against LiM3D, which reports higher numbers on ScribbleKITTI at 10% to 50%, and
attributes the difference to that method's test-time ensembles [arxiv:2405.05258
§5.3] [tab. 2].

### Per-class thresholds on confident pseudo-labels: FlexMatch

[FlexMatch](/papers/2110.08263) builds on FixMatch, which uses a weak view to
produce a pseudo-label and trains a strong view toward it with a masked
cross-entropy. FlexMatch changes only the confidence cut-off. A fixed threshold
τ is replaced by a threshold per class, T(c) = M(β(c)) · τ, with M(x) = x / (2 − x)
[arxiv:2110.08263 §3.1] [eq. 12]. The learning effect β(c) is the number of
unlabelled samples currently marked as class c, divided by the largest such
count. During warm-up the denominator also counts the unused samples
[arxiv:2110.08263 §3.2] [eq. 11]. A class the model still struggles with has
few marks, so its threshold falls and more of its samples are used. The marks
are stored per unlabelled sample, so the method needs no extra model and no
extra forward pass [arxiv:2110.08263 §3.1] [alg. 1].

The gains are author-reported. On CIFAR-100 with 400 labels, error falls from
46.42% (FixMatch) to 39.94%. On STL-10 with 40 labels it falls from 35.97% to
29.15%, and on ImageNet-1K with 100 labels per class, top-1 error falls from
43.66% to 41.85% [arxiv:2110.08263 tab. 1] [tab. 2]. The paper also reports
cases where FixMatch is as good or better: 250 labels on CIFAR-10, and SVHN with
40 labels, where FlexMatch is worse (8.19% against 3.81%). The authors attribute
the SVHN result to class imbalance [arxiv:2110.08263 tab. 1] [§4.1].

### Self-adaptive thresholds from running averages: FreeMatch

[FreeMatch](/papers/2205.07246) keeps the FixMatch pipeline and replaces the
fixed τ with state the model maintains itself. The global threshold is an
exponential moving average (EMA) of the mean top weak-view probability over each
unlabelled batch [arxiv:2205.07246 §4.1] [eq. 5]. A second EMA tracks the mean
predicted probability per class. Dividing it by its largest entry and multiplying
by the global threshold gives each class its own cut-off [arxiv:2205.07246 §4.1]
[eq. 6–7]. Unlike the mark counts in FlexMatch, the state here is averaged
probabilities, not per-sample labels [arxiv:2110.08263 §3.1]. The paper's
motivation is a two-Gaussian analysis in which the best threshold rises as the
model grows more confident. That analysis is a simplified binary model, not a
proof for deep networks [arxiv:2205.07246 §2] [eq. 2].

The paper adds a self-adaptive fairness loss. It is a cross-entropy between a
target distribution from the EMA statistics and a guess from the strong-view
predictions over the kept samples, which keeps predictions spread across classes
[arxiv:2205.07246 §4.2] [eq. 11]. It is added to the supervised and masked
unsupervised losses with weight w_f [arxiv:2205.07246 eq. 12]. The paper gives
w_f as 0.01 or 0.05 by setting [arxiv:2205.07246 app. D, tab. 5]. The released
configs use a tenth of that, which the [FreeMatch card](/papers/2205.07246) lists
under Paper vs code.

Author-reported results, against FlexMatch: CIFAR-10 with 10 labels, 13.85%
error falls to 8.07%; STL-10 with 40 labels, 29.15% falls to 15.56%; CIFAR-100
with 400 labels, 39.94% falls to 37.98% [arxiv:2205.07246 tab. 1]. On CIFAR-10
at 40 labels the gap is small (4.90% against 4.97%). The spread across seeds at
10 labels is wide, ±12.04 for FlexMatch and ±4.24 for FreeMatch [arxiv:2205.07246
tab. 1]. Removing the fairness term at 10 labels gives 10.37% error, against 8.07%
with it [arxiv:2205.07246 tab. 4]. The extra cost is the EMA and histogram
bookkeeping, which the paper calls very small [arxiv:2205.07246 app. C] [tab. 2].

### Frozen pretrained model as the anchor: PromptSRC

[PromptSRC](/papers/2307.06948) trains deep vision-language prompts for CLIP
with the frozen encoders. The task loss is cross-entropy on the prompted
image-text similarities [arxiv:2307.06948 §3.2] [eq. 1]. Three agreement terms
are added, all computed on the same mini-batch. An L1 term keeps the prompted
image features close to the frozen CLIP image features, and a second L1 term
does the same for the text features, each with its own weight [arxiv:2307.06948
§3.2.1] [eq. 2]. A KL term pulls the prompted class distribution toward the
frozen CLIP distribution [arxiv:2307.06948 §3.2.1] [eq. 3]. The frozen text
target for each class is an average over a set of prompt templates, and that
average is used only during training [arxiv:2307.06948 §3.2.3].

The fourth ingredient, epoch-level prompt averaging, is not a consistency term.
It averages the prompts over epochs with Gaussian weights, and the averaged
prompts are used at inference [arxiv:2307.06948 §3.2.2] [eq. 6] [eq. 7]
[fig. 3]. The paper cites weight-space ensembling work as the motivation for
this step [arxiv:2307.06948 §3.2.2].

The paper reports author-reported base-to-novel results on 11 datasets with a
harmonic mean of 79.97, against 78.55 for MaPLe [arxiv:2307.06948 tab. 1]. It
uses 46K tunable parameters, against 3.55M for MaPLe, and it beats MaPLe on 8
of the 11 datasets [arxiv:2307.06948 §4.3]. Its ablation on the same split adds
about 2 points of harmonic mean over the IVLP baseline from the consistency
loss, about 0.15 more from the epoch averaging, and about 0.27 more from the
textual-diversity term [arxiv:2307.06948 tab. 2]. The costs are one extra
forward pass through the frozen image encoder per step [arxiv:2307.06948 §4.7],
and about 10% more training compute than IVLP on the reported setup [tab. 7].
The reported setup uses CLIP ViT-B/16 only [arxiv:2307.06948 §4.1].

### Consistency inside other methods

Two methods in the survey's hybrid family also use a consistency term. ICT
trains the student on Mixup-mixed unlabelled pairs and asks the mixed prediction
to match the mixed teacher predictions [arxiv:2103.00550 §7]. FixMatch uses a
pseudo-label from a weakly augmented view as the target for a strongly augmented
view, applied only above a confidence threshold [arxiv:2103.00550 §7]. The survey
classes both as hybrid methods, which combine several families
[arxiv:2103.00550 §7].

### Consistency in dense and domain-shift tasks

The visual survey reports the same idea used beyond classification. In
segmentation, consistency is applied per pixel. One method perturbs inputs with
CutOut and CutMix and enforces agreement at each pixel, and cross-consistency
training adds noise to features instead [arxiv:2208.11296 §4.2.1]. In object
detection, consistency methods flip boxes or recompute overlapping boxes under
MixUp, and teacher–student methods derive pseudo-boxes from an EMA teacher or
from non-maximum suppression on the model's own outputs [arxiv:2208.11296 §4.2.2].
For unsupervised domain adaptation, consistency uses dual MixUp across categories
and domains, and self-ensembling with a Mean Teacher [arxiv:2208.11296 §4.2.3].
The survey gives no numbers for these methods [arxiv:2208.11296 §1]. For the
detection method with reported results, see the Unbiased Teacher variant above.
For LiDAR segmentation with reported results, see the LaserMix and LaserMix++
variant above.

## When to use which

The surveys report no accuracy numbers for these variants, so this table is based
on what each variant needs and where its target comes from, not on which one
performs best [arxiv:2103.00550 §4, §8] [arxiv:2208.11296 §1]. The rows for
Unbiased Teacher and FlexMatch use the authors' own reported results.

| Situation | Use | Why |
|---|---|---|
| Simplest code, no extra model state, two forward passes are affordable | Π Model | Same weights, two augmented passes [arxiv:2103.00550 §4] [eq. 17] |
| Cannot afford a second pass per step, storage per sample is acceptable | Temporal Ensembling | One pass per step, target from past predictions [arxiv:2103.00550 §4] [eq. 18]. Targets lag by up to an epoch |
| Targets should track the current model during training | Mean Teacher | Teacher is an EMA of the student's weights, updated each step [arxiv:2103.00550 §4] [eq. 19] |
| A small input change has a clear worst direction, and an adversarial search per step is acceptable | VAT | Loss uses the adversarial input; the search method is not described [arxiv:2103.00550 §4] [eq. 20] |
| Dropout-heavy network, want adversarial and dropout noise together | VAdD | Adversarial mask searched near a random dropout mask [arxiv:2103.00550 §4] [eq. 24] |
| Strong domain augmentation exists (images: RandAugment or AutoAugment; text: back-translation) | UDA | Strong augmentation replaces simple noise [arxiv:2103.00550 §4] [eq. 27] |
| Two model copies are affordable and noisy targets are a concern | Dual Student | Filters unstable samples before the pull [arxiv:2103.00550 §4] [eq. 21–22] |
| Want solutions that generalise through flatter minima | SWA | Averages weights along a cyclical path [arxiv:2103.00550 §4] [eq. 23] |
| Network can carry a decoder per layer | Ladder Network | Per-layer denoising loss [arxiv:2103.00550 §4] [eq. 16] |
| Object detection with few labelled images and many unlabelled images | Unbiased Teacher | EMA teacher on weak views, student on strong views, pseudo-boxes filtered by a 0.7 threshold; author-reported gains over supervised-only at 0.5 to 2% COCO labels [arxiv:2102.09480 §3.2] [tab. 1] |
| LiDAR semantic segmentation with few labelled scans, and labelled and unlabelled scans from the same sensor setup | LaserMix++ (or LaserMix alone) | Band swap between scans and a mean-teacher term on the mixed scans [arxiv:2405.05258 §3.2] [§3.3] [eq. 7] [eq. 9]. Author-reported 56.2 mIoU at 1% labels on SemanticKITTI, against 45.4 supervised-only [arxiv:2405.05258 tab. 2]. The same-setup assumption is the paper's [arxiv:2405.05258 §4.1]. The camera terms have no released code, so start with LaserMix |
| Image classification with a FixMatch-style pipeline, where the fixed threshold leaves much unlabelled data unused or hard classes rarely pass it | FlexMatch | Per-class threshold from mark counts, no extra passes [arxiv:2110.08263 §3.1] [eq. 12]. Its result on imbalanced SVHN was worse than FixMatch [tab. 1] |
| Very few labels, and you do not want per-sample label state or a hand-set threshold | FreeMatch | Global and per-class thresholds from EMAs of confidence and class probabilities, plus a fairness term [arxiv:2205.07246 §4.1] [eq. 11]. Its largest author-reported gains are at 10 to 40 labels [tab. 1]. Its fairness weight in the released configs differs from the paper's table, see its card |
| Adapting a large frozen pretrained model with few labels, and the unseen classes must not get worse | PromptSRC | Prompts pulled toward the frozen model's features and logits, plus epoch averaging [arxiv:2307.06948 §3.2] [eq. 4]. Author-reported base-to-novel gain over MaPLe on 11 datasets, with far fewer tunable parameters [tab. 1] [§4.3]. Needs the frozen model's forward pass each step [§4.7] |

The Unbiased Teacher row is the only detection method in the cited papers with
reported numbers, and those numbers are the authors' own. The 2208.11296 survey
covers detection methods without results [arxiv:2208.11296 §4.2.2], and the
2103.00550 survey is restricted to classification [arxiv:2103.00550 §1].

If the unlabelled data may not match the labelled data, or the cluster and
manifold assumptions may fail, the survey warns that more unlabelled data can
reduce accuracy. Keep a supervised-only baseline when testing any variant
[arxiv:2103.00550 §2.1, §8]. The survey also notes that most methods assume
balanced classes [arxiv:2103.00550 §8], so check the class counts of the
unlabelled data before choosing a variant. For the separate problem of label
noise in the labelled set, see [noisy labels](/concepts/noisy-labels).

## Implementation sketch

Common core of the teacher-student variants. The variant decides how the two
views are built and how the teacher is updated.

```pseudocode
# One training step, shared by the consistency variants [arxiv:2103.00550 §2, §4]
def consistency_step(student, teacher, x_l, y_l, x_u, weight):
    # Supervised loss on the labelled batch, e.g. cross-entropy [arxiv:2103.00550 eq. 1]
    loss_sup = cross_entropy(student(x_l), y_l)
    # Build two views of each unlabelled input: noise, dropout, augmentation or
    # an adversarial step, depending on the variant [arxiv:2103.00550 tab. 3]
    view_a = perturb(x_u)
    view_b = perturb(x_u)
    # Target from the teacher, with no gradient [arxiv:2103.00550 eq. 15]
    target = stop_gradient(teacher(view_b))
    # Distance R between student and target, usually MSE or KL [arxiv:2103.00550 §4].
    # Pseudo-label variants use a classification loss on hard labels instead
    # [arxiv:2102.09480 §3.2]
    loss_cons = distance(student(view_a), target)
    # Trade-off weight on the consistency term [arxiv:2103.00550 eq. 1]
    loss = loss_sup + weight * loss_cons
    update(student, loss)
    # Teacher by EMA of the student's weights [arxiv:2103.00550 eq. 19]. Here
    # decay is the share the teacher keeps: 0.9996 in [arxiv:2102.09480 eq. 3]
    teacher = ema(teacher, student, decay)
    return loss
```

FlexMatch replaces the distance with a masked cross-entropy to a pseudo-label
from the weak view, and keeps the per-class threshold state across steps
[arxiv:2110.08263 eq. 8] [alg. 1]. The paper does not say whether that
pseudo-label is hard or soft [arxiv:2110.08263 eq. 8].

FreeMatch uses the same masked pseudo-label loss, with thresholds updated from
EMA state each step, and adds a fairness term on the strong-view predictions
[arxiv:2205.07246 alg. 1] [eq. 8] [eq. 11].

The anchor form in PromptSRC has no EMA teacher. The target is the frozen
pretrained model's feature and logit output for the same input, and only the
prompts are updated [arxiv:2307.06948 alg. 1]. The sketch's `teacher` becomes
`frozen_clip`, and the `ema` line is dropped.

## Contradictions

Within the sources, these points matter when implementing the variants. Two
of them are numeric disagreements inside one paper.

- Ladder Network: the text gives a per-layer loss with a denoising decoder, and
  the summary table omits the decoder [arxiv:2103.00550 §4] [eq. 16].
- Mean Teacher: eq. 19 uses different noise for the student and the teacher,
  while the summary table uses the same noise for both [arxiv:2103.00550 §4].
- VAT: the formula uses a different network symbol from the summary table, and
  the text states the radius constraint in a different form
  [arxiv:2103.00550 §4].
- Unbiased Teacher: the main table reports 20.75 mAP at 1% COCO labels, while
  the focal-plus-EMA ablation reports 21.19 for the same setting
  [arxiv:2102.09480 tab. 1] [fig. 5]. Its eq. 2 has a plus sign in front of
  the gradient step, and the code uses a descent step [arxiv:2102.09480 eq. 2].
- PromptSRC: the text says the consistency loss raises novel accuracy by 3.95
  points, while table 2 gives 75.38 minus 71.79, which is 3.59
  [arxiv:2307.06948 §4.2] [tab. 2]. The text also says the training compute is
  0.13 times higher than IVLP, while table 7 gives about 10% more
  [arxiv:2307.06948 §4.7] [tab. 7]. The table values are used above.
- The two surveys group methods differently. 2103.00550 names five families:
  generative, consistency, graph-based, pseudo-labeling and hybrid
  [arxiv:2103.00550 §1]. 2208.11296 also names five, but its families are
  consistency, self-training, graph regularization, generative models and
  self-supervision, with hybrid methods placed under several of them
  [arxiv:2208.11296 tab. 1].
- LaserMix and LaserMix++: the paper sets the mixing weight λ_mix to 2.0 and
  the mean-teacher weight λ_mt to 250 [arxiv:2405.05258 §5.2]. The released
  LaserMix configs set λ_mix to 1, and λ_mt to 250, 500 or 200 depending on
  the config. The paper's pseudo-label threshold is about 0.9
  [arxiv:2405.05258 §5.4], and the released configs use 0.9 for SemanticKITTI
  and 0.95 for nuScenes. These values come from the
  [LaserMix++ card, Paper vs code](/papers/2405.05258), which read the released
  code.
- EMA direction in LaserMix: the paper's ablation speaks of decay between 0.9
  and 0.99 [arxiv:2405.05258 §5.4]. The released config passes momentum 0.01,
  and its update keeps 0.99 of the teacher each step, per the same
  [card](/papers/2405.05258). Read the convention in each source before
  copying a value.
- LaserMix++ camera terms: the paper gives no weight for the text-alignment
  step and describes its features as cosine distances, while eqs. 13 and 15 are
  written as inner products [arxiv:2405.05258 §4.3] [§4.4] [eq. 13] [eq. 15].

Check each against the original paper before implementing. The sources do not
resolve them.

## Papers

- [A Survey on Deep Semi-supervised Learning](/papers/2103.00550): taxonomy of consistency methods, their losses in one notation, and a map of the perturbations each one uses.
- [Unbiased Teacher for Semi-Supervised Object Detection](/papers/2102.09480): EMA teacher with weak and strong views for detection, with pseudo-box filtering and a focal loss on the box classifier.
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](/papers/2110.08263): per-class confidence thresholds for FixMatch that follow how well each class is learned, with no extra passes.
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](/papers/2208.11296): five semi-supervised families and their use in segmentation, detection and domain adaptation, with no experiments.
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](/papers/2205.07246): global and per-class thresholds from running averages, plus a fairness term that keeps predictions spread across classes.
- [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](/papers/2307.06948): consistency to a frozen CLIP model for prompt tuning, with epoch-level prompt averaging to limit forgetting of unseen classes.
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](/papers/2405.05258): LiDAR segmentation with band-swapped scans and a mean teacher, plus camera distillation and text guidance in LaserMix++.