Consistency regularization

Overview

Consistency regularization is a family of semi-supervised losses. The model is trained on a small labelled set as usual. On unlabelled inputs it is not given a label. Instead, the prediction for one view of an input must match a target built from another view of the same input 2103.00550 §4. The shared form is a distance R between the student's output and a target T, averaged over unlabelled inputs, and it is added to the supervised loss with a trade-off weight 2103.00550 §2, §4.

The survey's framing: targets come from the model itself, so these methods are teacher-student methods. The target is the prediction of a teacher that is the same network, an averaged copy, or a stored average of past predictions. Because the target is made by the model, the model can reinforce its own early errors. This is called confirmation bias, and better target construction is the main way the variants try to reduce it 2103.00550 §4.

The variants differ in two choices: which perturbation makes the second view (see the table in Variants), and how the target is kept. The survey groups perturbations into input augmentations (noise, stochastic augmentation, adversarial perturbation, AutoAugment, RandAugment, CTAugment, Mixup) and network transformations (dropout, EMA, SWA, stochastic depth, DropConnect) 2103.00550 tab. 5.

Consistency is one of the five families of deep semi-supervised learning in the survey's taxonomy, next to generative, graph-based, pseudo-labeling and hybrid methods 2103.00550 §1. The assumptions behind it are that the decision boundary lies in low-density regions and that neighbours on the data manifold share a label 2103.00550 §2.1. If those assumptions fail, unlabelled data can lower accuracy 2103.00550 §2.1.

A second survey, which covers visual recognition, counts consistency regularization as one of five semi-supervised families 2208.11296 tab. 1. It places the same idea under local smoothness across views: a sample and its transformed copy should get the same label, which is also the basis of instance discrimination in unsupervised learning 2208.11296 §4.1.1 eq. 15. The two surveys name different families (see Contradictions).

The same loss is also used without unlabelled data. Here the reference is the unmodified pretrained model, and the goal is to stop a small fine-tune from drifting away from what that model already knows. The adapted model is pulled toward the frozen model's features and predictions on the same training inputs 2307.06948 §3.2.1. This is the anchor form of the idea, and it is distinct from the teacher-student form above, where the teacher is the model being trained or a copy of it.

Variants

Variants without a card of their own are described in the survey card, A Survey on Deep Semi-supervised Learning. Variants with a card link to it in their section.

Input noise with a decoder: Ladder Network

Noise is added in every hidden layer of a corrupted encoder that shares weights with a clean encoder. A decoder denoises each layer, and the unsupervised loss is the per-layer reconstruction error between clean and reconstructed activations 2103.00550 §4 eq. 16. The survey notes that its text and its summary table disagree on the exact formula, so use the per-layer description 2103.00550 §4 (see Contradictions).

Two stochastic passes: Π Model

Each sample, labelled or not, goes through the network twice with different random augmentation and dropout. The loss is the distance between the two outputs 2103.00550 §4 eq. 17. This needs no extra model state, but it costs two forward passes per sample 2103.00550 §4 eq. 17.

Per-sample ensemble targets: Temporal Ensembling

Each sample gets one forward pass per step. Its target is an exponential moving average of its own past predictions, stored per sample and updated after each epoch with a momentum 2103.00550 §4 eq. 18. The target therefore lags the current model, and the survey does not give a bias correction or a start-up rule 2103.00550 §4 (see When to use which).

Weight-averaged teacher: Mean Teacher

The teacher is an exponential moving average of the student's weights over training steps. The student is pulled toward the teacher's prediction on a differently augmented copy of the input 2103.00550 §4 eq. 19. The survey's summary table and eq. 19 disagree on which noise is applied to the teacher, and the EMA decay is not stated 2103.00550 §4. Mean Teacher recurs in the hybrid ICT method, which uses the same teacher on Mixup pairs 2103.00550 §7.

Adversarial direction: VAT

The perturbation is not random. It is the small input change, within a fixed radius, that moves the prediction the most. The loss compares the prediction on the clean input with the prediction on that adversarial input [arxiv:2103.00550 §4] eq. 20. The survey does not describe how the maximising perturbation is computed, and it does not give the radius 2103.00550 §4.

Adversarial direction plus dropout: VAdD

Adversarial perturbation is combined with random dropout masks. The adversarial dropout mask is searched within a bounded distance of a random mask. The bound scales with the dropout layer's dimension and a noise intensity [arxiv:2103.00550 §4] eq. 24. Neither value is given in the survey 2103.00550 §4.

Two students with stability filtering: Dual Student

Two students start from different initial weights. A sample is stable for a student when that student's prediction agrees across the sample's two versions and its confidence is above a threshold ξ 2103.00550 §4 eq. 21. If both students find the sample stable, the one with the larger ε is pulled toward the other. Otherwise the pull is gated by each student's own stability flag 2103.00550 §4 eq. 22. The threshold ξ is not stated in the survey 2103.00550 §4.

Weight averaging along a trajectory: SWA

Stochastic gradient descent on the consistency loss keeps moving between distant solutions instead of settling. Weights are averaged along a cyclical learning-rate path, and fast-SWA averages several points per cycle over a longer cycle. The survey reports this as giving flatter solutions 2103.00550 §4 eq. 23.

Worst-case weight and structure noise: WCP

Worst-case additive noise is applied to the weights, under a norm constraint. Worst-case DropConnect drops individual connections. The consistency target is the output of the perturbed network 2103.00550 §4 eq. 25–26.

Strong augmentation: UDA

Simple noise is replaced by strong, domain-specific augmentation. For images these are AutoAugment and RandAugment, and for text back-translation 2103.00550 §4 eq. 27. The survey's augmentation map lists UDA under AutoAugment and RandAugment 2103.00550 tab. 5.

Weak and strong views for detection: Unbiased Teacher

Unbiased Teacher for Semi-Supervised Object Detection applies the teacher-student pattern to object detection. A Faster R-CNN detector is first trained on labelled images only (burn-in), and the teacher and student are then started from those weights 2102.09480 §3.1. The teacher is an exponential moving average of the student 2102.09480 §3.2 eq. 3. It sees a weakly augmented image (horizontal flip) and proposes boxes. The student sees a strongly augmented copy (colour jitter, grayscale, blur, cutout) and learns from those boxes 2102.09480 §3.2 tab. 6.

The target is a hard pseudo-label, not a distance between outputs. Duplicate boxes are removed per class, and boxes below a confidence threshold of 0.7 are dropped 2102.09480 §3.2 tab. 5. Only the classification losses of the region-proposal and box-classifier heads take pseudo-labels. Box regression is not trained on them, because confidence does not measure how well a box is placed 2102.09480 §3.2.

The paper argues that class imbalance biases pseudo-labels toward background and common classes. It counters this with a focal loss on the box classifier and the slow teacher 2102.09480 §3.3 fig. 5. The EMA rate multiplies the teacher: with 0.9996, the teacher keeps 99.96% of its weights at each step, and using the value the other way round inverts the method [arxiv:2102.09480 eq. 3].

The paper reports results for its own setup only. These are author-reported gains at 0.5% to 10% of the COCO train2017 images labelled, and it is compared only with STAC and CSD 2102.09480 tab. 1 (see Contradictions).

Mixed LiDAR scans with a mean teacher: LaserMix and LaserMix++

LaserMix++ applies the teacher-student pattern to semi-supervised LiDAR semantic segmentation. Two scans are cut into bands by laser-beam inclination angle, and the bands are swapped between them, so each band keeps its neighbours from the other scan 2405.05258 §3.2 eq. 7. The student predicts the mixed scan, and the mixed labels come from ground truth and from the teacher's pseudo-labels on the unlabelled part [arxiv:2405.05258 §3.3]. A mean-teacher term asks the student's prediction on the unlabelled scan to match the teacher's, as a squared ℓ2 distance 2405.05258 §3.3 eq. 9. The teacher's pseudo-labels are kept only above a confidence threshold, which the paper sets at about 0.9 2405.05258 §5.4.

The camera extension adds three more terms on points that project to a valid pixel. One distils image features onto the LiDAR features [arxiv:2405.05258 §4.3] eq. 13. Another aligns them with text-aligned image features 2405.05258 §4.4 eq. 15. The overall loss is a weighted sum of these terms with the supervised and mixing losses 2405.05258 §4.5 eq. 16. The paper reports these camera terms as part of its results, but the repository it links releases only the LaserMix part (see Contradictions).

The paper's reported gain, for Cylinder3D on SemanticKITTI with 1% of the labels, is 56.2 mIoU, against 45.4 for supervised-only training and 50.6 for LaserMix 2405.05258 tab. 2. These are author-reported. The text reports no seeds or variance for them 2405.05258. The paper compares against LiM3D, which reports higher numbers on ScribbleKITTI at 10% to 50%, and attributes the difference to that method's test-time ensembles [arxiv:2405.05258 §5.3] tab. 2.

Per-class thresholds on confident pseudo-labels: FlexMatch

FlexMatch builds on FixMatch, which uses a weak view to produce a pseudo-label and trains a strong view toward it with a masked cross-entropy. FlexMatch changes only the confidence cut-off. A fixed threshold τ is replaced by a threshold per class, T(c) = M(β(c)) · τ, with M(x) = x / (2 − x) 2110.08263 §3.1 eq. 12. The learning effect β(c) is the number of unlabelled samples currently marked as class c, divided by the largest such count. During warm-up the denominator also counts the unused samples 2110.08263 §3.2 eq. 11. A class the model still struggles with has few marks, so its threshold falls and more of its samples are used. The marks are stored per unlabelled sample, so the method needs no extra model and no extra forward pass 2110.08263 §3.1 alg. 1.

The gains are author-reported. On CIFAR-100 with 400 labels, error falls from 46.42% (FixMatch) to 39.94%. On STL-10 with 40 labels it falls from 35.97% to 29.15%, and on ImageNet-1K with 100 labels per class, top-1 error falls from 43.66% to 41.85% 2110.08263 tab. 1 tab. 2. The paper also reports cases where FixMatch is as good or better: 250 labels on CIFAR-10, and SVHN with 40 labels, where FlexMatch is worse (8.19% against 3.81%). The authors attribute the SVHN result to class imbalance 2110.08263 tab. 1 §4.1.

Self-adaptive thresholds from running averages: FreeMatch

FreeMatch keeps the FixMatch pipeline and replaces the fixed τ with state the model maintains itself. The global threshold is an exponential moving average (EMA) of the mean top weak-view probability over each unlabelled batch 2205.07246 §4.1 eq. 5. A second EMA tracks the mean predicted probability per class. Dividing it by its largest entry and multiplying by the global threshold gives each class its own cut-off 2205.07246 §4.1 eq. 6–7. Unlike the mark counts in FlexMatch, the state here is averaged probabilities, not per-sample labels 2110.08263 §3.1. The paper's motivation is a two-Gaussian analysis in which the best threshold rises as the model grows more confident. That analysis is a simplified binary model, not a proof for deep networks 2205.07246 §2 eq. 2.

The paper adds a self-adaptive fairness loss. It is a cross-entropy between a target distribution from the EMA statistics and a guess from the strong-view predictions over the kept samples, which keeps predictions spread across classes 2205.07246 §4.2 eq. 11. It is added to the supervised and masked unsupervised losses with weight w_f 2205.07246 eq. 12. The paper gives w_f as 0.01 or 0.05 by setting 2205.07246 app. D, tab. 5. The released configs use a tenth of that, which the FreeMatch card lists under Paper vs code.

Author-reported results, against FlexMatch: CIFAR-10 with 10 labels, 13.85% error falls to 8.07%; STL-10 with 40 labels, 29.15% falls to 15.56%; CIFAR-100 with 400 labels, 39.94% falls to 37.98% 2205.07246 tab. 1. On CIFAR-10 at 40 labels the gap is small (4.90% against 4.97%). The spread across seeds at 10 labels is wide, ±12.04 for FlexMatch and ±4.24 for FreeMatch [arxiv:2205.07246 tab. 1]. Removing the fairness term at 10 labels gives 10.37% error, against 8.07% with it 2205.07246 tab. 4. The extra cost is the EMA and histogram bookkeeping, which the paper calls very small 2205.07246 app. C tab. 2.

Frozen pretrained model as the anchor: PromptSRC

PromptSRC trains deep vision-language prompts for CLIP with the frozen encoders. The task loss is cross-entropy on the prompted image-text similarities 2307.06948 §3.2 eq. 1. Three agreement terms are added, all computed on the same mini-batch. An L1 term keeps the prompted image features close to the frozen CLIP image features, and a second L1 term does the same for the text features, each with its own weight [arxiv:2307.06948 §3.2.1] eq. 2. A KL term pulls the prompted class distribution toward the frozen CLIP distribution 2307.06948 §3.2.1 eq. 3. The frozen text target for each class is an average over a set of prompt templates, and that average is used only during training 2307.06948 §3.2.3.

The fourth ingredient, epoch-level prompt averaging, is not a consistency term. It averages the prompts over epochs with Gaussian weights, and the averaged prompts are used at inference 2307.06948 §3.2.2 eq. 6 eq. 7 fig. 3. The paper cites weight-space ensembling work as the motivation for this step 2307.06948 §3.2.2.

The paper reports author-reported base-to-novel results on 11 datasets with a harmonic mean of 79.97, against 78.55 for MaPLe 2307.06948 tab. 1. It uses 46K tunable parameters, against 3.55M for MaPLe, and it beats MaPLe on 8 of the 11 datasets 2307.06948 §4.3. Its ablation on the same split adds about 2 points of harmonic mean over the IVLP baseline from the consistency loss, about 0.15 more from the epoch averaging, and about 0.27 more from the textual-diversity term 2307.06948 tab. 2. The costs are one extra forward pass through the frozen image encoder per step 2307.06948 §4.7, and about 10% more training compute than IVLP on the reported setup tab. 7. The reported setup uses CLIP ViT-B/16 only 2307.06948 §4.1.

Consistency inside other methods

Two methods in the survey's hybrid family also use a consistency term. ICT trains the student on Mixup-mixed unlabelled pairs and asks the mixed prediction to match the mixed teacher predictions 2103.00550 §7. FixMatch uses a pseudo-label from a weakly augmented view as the target for a strongly augmented view, applied only above a confidence threshold 2103.00550 §7. The survey classes both as hybrid methods, which combine several families 2103.00550 §7.

Consistency in dense and domain-shift tasks

The visual survey reports the same idea used beyond classification. In segmentation, consistency is applied per pixel. One method perturbs inputs with CutOut and CutMix and enforces agreement at each pixel, and cross-consistency training adds noise to features instead 2208.11296 §4.2.1. In object detection, consistency methods flip boxes or recompute overlapping boxes under MixUp, and teacher–student methods derive pseudo-boxes from an EMA teacher or from non-maximum suppression on the model's own outputs 2208.11296 §4.2.2. For unsupervised domain adaptation, consistency uses dual MixUp across categories and domains, and self-ensembling with a Mean Teacher 2208.11296 §4.2.3. The survey gives no numbers for these methods 2208.11296 §1. For the detection method with reported results, see the Unbiased Teacher variant above. For LiDAR segmentation with reported results, see the LaserMix and LaserMix++ variant above.

When to use which

The surveys report no accuracy numbers for these variants, so this table is based on what each variant needs and where its target comes from, not on which one performs best 2103.00550 §4, §8 2208.11296 §1. The rows for Unbiased Teacher and FlexMatch use the authors' own reported results.

The Unbiased Teacher row is the only detection method in the cited papers with reported numbers, and those numbers are the authors' own. The 2208.11296 survey covers detection methods without results 2208.11296 §4.2.2, and the 2103.00550 survey is restricted to classification 2103.00550 §1.

If the unlabelled data may not match the labelled data, or the cluster and manifold assumptions may fail, the survey warns that more unlabelled data can reduce accuracy. Keep a supervised-only baseline when testing any variant 2103.00550 §2.1, §8. The survey also notes that most methods assume balanced classes 2103.00550 §8, so check the class counts of the unlabelled data before choosing a variant. For the separate problem of label noise in the labelled set, see noisy labels.

Implementation sketch

Common core of the teacher-student variants. The variant decides how the two views are built and how the teacher is updated.

# One training step, shared by the consistency variants [arxiv:2103.00550 §2, §4]
def consistency_step(student, teacher, x_l, y_l, x_u, weight):
    # Supervised loss on the labelled batch, e.g. cross-entropy [arxiv:2103.00550 eq. 1]
    loss_sup = cross_entropy(student(x_l), y_l)
    # Build two views of each unlabelled input: noise, dropout, augmentation or
    # an adversarial step, depending on the variant [arxiv:2103.00550 tab. 3]
    view_a = perturb(x_u)
    view_b = perturb(x_u)
    # Target from the teacher, with no gradient [arxiv:2103.00550 eq. 15]
    target = stop_gradient(teacher(view_b))
    # Distance R between student and target, usually MSE or KL [arxiv:2103.00550 §4].
    # Pseudo-label variants use a classification loss on hard labels instead
    # [arxiv:2102.09480 §3.2]
    loss_cons = distance(student(view_a), target)
    # Trade-off weight on the consistency term [arxiv:2103.00550 eq. 1]
    loss = loss_sup + weight * loss_cons
    update(student, loss)
    # Teacher by EMA of the student's weights [arxiv:2103.00550 eq. 19]. Here
    # decay is the share the teacher keeps: 0.9996 in [arxiv:2102.09480 eq. 3]
    teacher = ema(teacher, student, decay)
    return loss

FlexMatch replaces the distance with a masked cross-entropy to a pseudo-label from the weak view, and keeps the per-class threshold state across steps 2110.08263 eq. 8 alg. 1. The paper does not say whether that pseudo-label is hard or soft 2110.08263 eq. 8.

FreeMatch uses the same masked pseudo-label loss, with thresholds updated from EMA state each step, and adds a fairness term on the strong-view predictions 2205.07246 alg. 1 eq. 8 eq. 11.

The anchor form in PromptSRC has no EMA teacher. The target is the frozen pretrained model's feature and logit output for the same input, and only the prompts are updated 2307.06948 alg. 1. The sketch's teacher becomes frozen_clip, and the ema line is dropped.

Contradictions

Within the sources, these points matter when implementing the variants. Two of them are numeric disagreements inside one paper.

• Ladder Network: the text gives a per-layer loss with a denoising decoder, and the summary table omits the decoder 2103.00550 §4 eq. 16.
• Mean Teacher: eq. 19 uses different noise for the student and the teacher, while the summary table uses the same noise for both 2103.00550 §4.
• VAT: the formula uses a different network symbol from the summary table, and the text states the radius constraint in a different form 2103.00550 §4.
• Unbiased Teacher: the main table reports 20.75 mAP at 1% COCO labels, while the focal-plus-EMA ablation reports 21.19 for the same setting 2102.09480 tab. 1 fig. 5. Its eq. 2 has a plus sign in front of the gradient step, and the code uses a descent step 2102.09480 eq. 2.
• PromptSRC: the text says the consistency loss raises novel accuracy by 3.95 points, while table 2 gives 75.38 minus 71.79, which is 3.59 2307.06948 §4.2 tab. 2. The text also says the training compute is 0.13 times higher than IVLP, while table 7 gives about 10% more 2307.06948 §4.7 tab. 7. The table values are used above.
• The two surveys group methods differently. 2103.00550 names five families: generative, consistency, graph-based, pseudo-labeling and hybrid 2103.00550 §1. 2208.11296 also names five, but its families are consistency, self-training, graph regularization, generative models and self-supervision, with hybrid methods placed under several of them 2208.11296 tab. 1.
• LaserMix and LaserMix++: the paper sets the mixing weight λ_mix to 2.0 and the mean-teacher weight λ_mt to 250 2405.05258 §5.2. The released LaserMix configs set λ_mix to 1, and λ_mt to 250, 500 or 200 depending on the config. The paper's pseudo-label threshold is about 0.9 2405.05258 §5.4, and the released configs use 0.9 for SemanticKITTI and 0.95 for nuScenes. These values come from the LaserMix++ card, Paper vs code, which read the released code.
• EMA direction in LaserMix: the paper's ablation speaks of decay between 0.9 and 0.99 2405.05258 §5.4. The released config passes momentum 0.01, and its update keeps 0.99 of the teacher each step, per the same card. Read the convention in each source before copying a value.
• LaserMix++ camera terms: the paper gives no weight for the text-alignment step and describes its features as cosine distances, while eqs. 13 and 15 are written as inner products 2405.05258 §4.3 §4.4 eq. 13 eq. 15.

Check each against the original paper before implementing. The sources do not resolve them.

Papers

• A Survey on Deep Semi-supervised Learning: taxonomy of consistency methods, their losses in one notation, and a map of the perturbations each one uses.
• Unbiased Teacher for Semi-Supervised Object Detection: EMA teacher with weak and strong views for detection, with pseudo-box filtering and a focal loss on the box classifier.
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling: per-class confidence thresholds for FixMatch that follow how well each class is learned, with no extra passes.
• Semi-Supervised and Unsupervised Deep Visual Learning: A Survey: five semi-supervised families and their use in segmentation, detection and domain adaptation, with no experiments.
• FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning: global and per-class thresholds from running averages, plus a fairness term that keeps predictions spread across classes.
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting: consistency to a frozen CLIP model for prompt tuning, with epoch-level prompt averaging to limit forgetting of unseen classes.
• Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving: LiDAR segmentation with band-swapped scans and a mean teacher, plus camera distillation and text guidance in LaserMix++.