SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning

In plain words

Semi-supervised learning trains on a few labelled examples plus many unlabelled ones. A common trick is to let the model label the unlabelled examples itself and keep only the labels it is confident about. This paper argues that a hard confidence cut-off throws away many correct labels, while lowering the cut-off lets in wrong ones.

SoftMatch gives every unlabelled example a weight instead of a keep-or-drop decision. Confident examples get the full weight, somewhat unsure ones get a partial weight, and only very unsure ones fade out. The shape of this weighting is estimated from the model's own confidence during training, so it is not a fixed number.

It also nudges the predicted class frequencies toward even, so that classes the model rarely predicts are not ignored.

TL;DR

• Threshold-based pseudo-labelling trades quantity against quality: a high cut-off keeps correct labels only, a low one also keeps wrong labels §1.
• SoftMatch weights each unlabelled sample with a truncated Gaussian on its top class probability, with mean and variance estimated by EMA from the batch §3.1 eq. 5-7.
• It adds Uniform Alignment of the class-prediction distribution, used only for computing the weights, not for the pseudo-label §3.2 eq. 8-9.
• Author-reported: lowest error in most settings of the classic, long-tailed, ImageNet and text experiments. The exceptions are listed in Results tab. 2 tab. 3 tab. 4 tab. 5.
• Code is public (USB, MIT). The paper's footnoted repository only redirects to USB and TorchSSL.

Problem

Pseudo-labelling with a confidence threshold keeps unlabelled samples whose top class probability exceeds a fixed or adaptive cut-off and ignores the rest §2.2. The paper formalises two quantities. Quantity is the average weight given to unlabelled samples. Quality is the fraction of correct pseudo-labels among the weighted ones §2.1 eq. 3-4.

For a threshold tau, the paper shows that quantity and quality move in opposite directions as tau changes §2.2. In its two-moon example with 4 labels and a 3-layer MLP, the paper reports that about 71% of correct pseudo-labels are excluded by FixMatch's high threshold, and that about 16% of the pseudo-labels used by FlexMatch are incorrect §1 fig. 1. The assumption behind this is that the marginal distribution of confidences is uniform over the kept region, which the paper calls over-simplified §2.2.

Method

The unsupervised loss is written as a weighted cross-entropy with weights lambda(p) in [0, lambda_max] §2.1 eq. 2. Thresholding is a step function for lambda. SoftMatch instead sets lambda from a truncated Gaussian fitted to the top probability §3.1 eq. 5.

1. Supervised loss. Cross-entropy on the labelled batch, unchanged from standard pseudo-labelling §2.1 eq. 1.
2. Gaussian statistics. Compute the batch mean and unbiased variance of the top class probability on the weakly augmented unlabelled batch §3.1 eq. 6. Smooth them over steps with EMA momentum m §3.1 eq. 7.
3. Uniform alignment. Keep an EMA of the batch-mean class-probability vector. Rescale each prediction by the ratio of the uniform distribution to that EMA and renormalise. This aligned vector is used only for the weight §3.2 eq. 8.
4. Weight. Apply the truncated Gaussian to the top probability of the aligned vector: weight 1 at or above the mean, Gaussian decay below it §3.1 eq. 9. The 2-sigma range is realised by dividing the variance by 4 §4.1 app. A.5.
5. Pseudo-label. Take the arg-max of the unaligned weak prediction, as a hard label §3.2 alg. 1.
6. Loss. Weight the cross-entropy of the strongly augmented view against that hard label, and add it to the supervised loss §2.1 eq. 2 alg. 1.

Key equations, transcribed from the paper:

\mathcal{L}_{s} = \frac{1}{B_{L}} \sum_{i=1}^{B_{L}} \mathcal{H}\left(\mathbf{y}_i, \mathbf{p}(\mathbf{y}\mid\mathbf{x}_{i}^{l})\right) eq. 1

\mathcal{L}_{u} = \frac{1}{B_{U}} \sum_{i=1}^{B_{U}} \lambda(\mathbf{p}_i)\, \mathcal{H}\left(\hat{\mathbf{p}}_i, \mathbf{p}(\mathbf{y}\mid\Omega(\mathbf{x}_i^{u}))\right) eq. 2

Here p is the prediction on the weak view omega(x^u), p-hat is its one-hot arg-max, and Omega is the strong view. The total loss is L = L_s + L_u §2.1.

f(\mathbf{p}) = \mathbb{E}_{\mathcal{D}_U}[\lambda(\mathbf{p})] \in [0, \lambda_{\max}] eq. 3

g(\mathbf{p}) = \sum_{i}^{N_U} \mathbb{1}(\hat{\mathbf{p}}_i = \mathbf{y}_i^u) \frac{\lambda(\mathbf{p}_i)}{\sum_{j}^{N_U} \lambda(\mathbf{p}_j)} \in [0, 1] eq. 4

Equations 3 and 4 define quantity and quality. They are analysis tools, and the quality definition uses ground-truth labels that are unavailable in training §2.1.

\lambda(\mathbf{p}) = \begin{cases} \lambda_{\max} \exp\left(-\frac{(\max(\mathbf{p})-\mu_t)^{2}}{2 \sigma_t^2}\right), & \max(\mathbf{p}) < \mu_t, \\ \lambda_{\max}, & \text{otherwise.} \end{cases} eq. 5

\hat{\mu}_b = \frac{1}{B_U} \sum_{i=1}^{B_U} \max(\mathbf{p}_i), \qquad \hat{\sigma}_b^2 = \frac{1}{B_U} \sum_{i=1}^{B_U} \left(\max(\mathbf{p}_i) - \hat{\mu}_b\right)^2 eq. 6

\hat{\mu}_t = m\, \hat{\mu}_{t-1} + (1-m)\, \hat{\mu}_b, \qquad \hat{\sigma}_t^2 = m\, \hat{\sigma}_{t-1}^2 + (1-m)\, \frac{B_U}{B_U-1}\, \hat{\sigma}_b^2 eq. 7

The paper uses the unbiased variance in the EMA and initialises mu-hat at 1/C and sigma-hat-squared at 1.0 §3.1.

\operatorname{UA}(\mathbf{p}) = \operatorname{Normalize}\left(\mathbf{p} \cdot \frac{\mathbf{u}(C)}{\hat{\mathbb{E}}_{B_U}[\mathbf{p}]}\right), \quad \operatorname{Normalize}(\cdot) = \frac{(\cdot)}{\sum(\cdot)} eq. 8

Here u(C) is the uniform distribution over C classes, and the expectation is an EMA of batch predictions on unlabelled data §3.2.

\lambda(\mathbf{p}) = \begin{cases} \lambda_{\max} \exp\left(-\frac{(\max(\operatorname{UA}(\mathbf{p})) - \hat{\mu}_t)^{2}}{2 \hat{\sigma}_t^2}\right), & \max(\operatorname{UA}(\mathbf{p})) < \hat{\mu}_t, \\ \lambda_{\max}, & \text{otherwise.} \end{cases} eq. 9

Algorithm 1 gives the same steps in the paper's own pseudo-code alg. 1 app. A.2. The sketch below follows it, with the variance scaling from §4.1 and the detachment from the code.

# SoftMatch loss for one training step  [alg. 1] [eq. 1-2] [eq. 6-9]
# Inputs: labeled batch (x, y) of size B_L; unlabeled batch u of size B_U
#         EMA momentum m (0.999 in the paper's runs); C classes
# State:  mu, var (scalars), p_model (C,); initial values mu = 1/C, var = 1.0

def softmatch_loss(x, y, u, state):
    # Supervised cross-entropy on the labeled batch  [eq. 1]
    loss_sup = mean_cross_entropy(y, model(x))

    # Weak-view prediction on the unlabeled batch; no gradient  [alg. 1]
    p = softmax(model(weak(u))).detach()             # (B_U, C)

    # Batch mean and unbiased variance of the top probability  [eq. 6]
    mu_b = mean(max(p))
    var_b = unbiased_variance(max(p))

    # EMA of the Gaussian parameters  [eq. 7]
    state.mu = m * state.mu + (1 - m) * mu_b
    state.var = m * state.var + (1 - m) * var_b

    # EMA of the mean prediction, then uniform alignment  [§3.2] [eq. 8]
    state.p_model = m * state.p_model + (1 - m) * mean(p, axis=0)
    aligned = normalize(p * (uniform(C) / state.p_model))

    # Truncated Gaussian weight on the aligned top probability  [eq. 9]
    # Variance divided by 4 gives the 2-sigma range  [§4.1]
    # lambda_max is taken as 1.0 here  [alg. 1]
    # Weights carry no gradient
    # [code: semilearn/algorithms/softmatch/utils.py#L53-L54]
    top = max(aligned)
    weight = where(top < state.mu,
                   exp(-(top - state.mu) ** 2 / (2 * state.var / 4)),
                   1.0)                                # (B_U,)

    # Hard pseudo-label from the unaligned weak prediction  [§3.2] [alg. 1]
    pseudo = argmax(p, axis=1)                         # (B_U,)

    # Weighted cross-entropy on the strong view  [eq. 2] [alg. 1]
    loss_unsup = mean(weight * cross_entropy(pseudo, model(strong(u))))

    # Total loss L = L_s + L_u  [§2.1]
    return loss_sup + loss_unsup

Relation to thresholding: the weight is 1 for every sample whose aligned top probability is at or above the running mean, so the Gaussian acts as a soft, adaptive threshold §3.1. The paper's quantity and quality analysis is in app. A.1.

Hyperparameters

Values as stated in the paper, with code values where the paper is silent or the code differs. Each row is cited to the paper section or table and, where relevant, to the code.

SoftMatch-specific

Classic image classification (CIFAR-10/100, SVHN, STL-10, ImageNet) tab. 6 §4.1

Long-tailed image classification (CIFAR-10-LT, CIFAR-100-LT) tab. 7 §4.2

Text classification (AG News, DBpedia, IMDb, Amazon-5, Yelp-5) tab. 8 §4.3

Results

Author-reported. Each setting uses the paper's backbone and the protocol above; numbers are top-1 error (%), mean ± std over the stated seeds or splits. Values are copied from the paper's tables; best values are not marked here.

Classic image classification tab. 2. Setup: CIFAR-10 / CIFAR-100 / SVHN / STL-10, 3 seeds, 2^20 steps. The paper cites * values from the original papers.

The paper states that ReMixMatch is best on CIFAR-100 with 2,500 and 10,000 labels, and SoftMatch is comparable to FlexMatch there §4.1.

ImageNet tab. 3. Setup: ResNet-50, top-1 error (%), column headers as printed (see Pitfalls for the "10%" wording).

Long-tailed classification tab. 4. Setup: WRN-28-2, 5 splits, mean ± std over the 5 splits.

Text classification tab. 5. Setup: BERT-Base, 3 seeds, top-1 error (%). Label counts as printed.

Ablations app. A.4 app. A.5 tab. 9-12: on CIFAR-10 with 250 labels, error is 4.82 ± 0.09 with the 2-sigma range, 4.97 ± 0.13 with sigma, and 4.84 ± 0.15 with 3 sigma tab. 11. The uniform UA target gives 4.82 ± 0.09, the labelled-class ground-truth target 4.83 ± 0.12, and the estimated labelled target 4.90 ± 0.23 tab. 12. The paper says the choice of EMA momentum among 0.99, 0.999 and 0.9999 changes final error little (4.92, 4.82, 4.86) but changes convergence speed tab. 10 app. A.5. On CIFAR-10 at 40 labels, the learned truncated Gaussian gives 4.91 ± 0.12, and a fixed (non-learned) Gaussian with mu = 1.0 and sigma = 0.3 gives 7.73 ± 1.44 tab. 9 app. A.4. These are the paper's numbers.

Paper vs code

The paper's footnote links github.com/Hhhhhhao/SoftMatch. That repository has only a README (checked 2026-11-05, default branch) and no LICENSE. The README says the code is published in USB (microsoft/Semi-supervised-learning) and TorchSSL (TorchSSL/TorchSSL), and that the classic-setting results (its Table 1, this card's tab. 2) were run in TorchSSL. The paper's own footnote also puts the long-tailed experiments in TorchSSL and the text experiments in USB §4 [footnote 1]. The card uses USB as code at commit 1ef4cbeb, and both repositories were read. USB's LICENSE.txt is MIT. TorchSSL's LICENSE is MIT (commit f26e1d42).

1. Alignment target differs in TorchSSL. The paper's UA uses the uniform target u(C) §3.2 eq. 8. The TorchSSL code (used by the paper for Table 2) divides by an EMA of labelled-batch predictions, not uniform softmatch.py. The paper's ablation labels that variant "estimated labelled target" and reports 4.90 ± 0.23 against uniform 4.82 ± 0.09 on CIFAR-10 at 250 labels tab. 12. USB uses uniform by default softmatch.py dist_align.py.

2. Statistics are taken on different vectors. In USB, the alignment is applied before the Gaussian statistics are updated, so mu and var use aligned probabilities softmatch.py utils.py. In TorchSSL, the statistics are updated on the unaligned weak prediction before alignment softmatch.py. The paper does not say which is used, and eq. 6 is written in terms of max(p).

3. Labelled, weak and strong views share one forward pass. Both codes concatenate the labelled, weak and strong batches into a single forward pass softmatch.py softmatch_cifar10_250_0.yaml softmatch.py. The paper does not state this, so batch-normalisation statistics are computed over the joint batch.

4. Temperature. The code's pseudo-label temperature is T = 0.5 softmatch.py. With hard labels, argmax is unchanged, so it has no effect. The paper does not mention T.

5. Classic settings match. The USB classic configs for CIFAR-10 at 250 labels use lr 0.03, 2^20 steps, batch 64, uratio 7, weight decay 5e-4 (CIFAR-100 1e-3), EMA 0.999, cosine with 7/16 factor softmatch_cifar10_250_0.yaml softmatch_cifar100_400_0.yaml build.py. No discrepancy found for these values.

6. Text configs differ from the paper's table. The USB config for AG News 40 labels uses AdamW lr 5e-5 softmatch_ag_news_40_0.yaml, weight decay 5e-4 [L31], layer decay 0.65 [L32], batch 8 [L15], 102,400 steps [L10], warm-up 5,120 steps [L11], and max length 512 [L42]. The paper's Table 8 gives lr 1e-5, weight decay 1e-4, batch 16, 2^18 steps, and does not mention layer decay, warm-up or max length tab. 8 §4.3. The paper says text results are from USB §4.3 [footnote 1]; the config may be from a later run.

7. No code for ImageNet or long-tailed runs. Neither repository has a SoftMatch config for ImageNet or for the long-tailed setting. Those values are paper-only tab. 3 tab. 7.

Pitfalls

• Quantity lower bound assumes half the batch is below the mean. The derivation that f(p) >= lambda_max / 2 assumes exactly half of the unlabelled samples have max(p) below mu_t, justified by "mu_t is the mean" app. A.1. A mean does not split a skewed distribution in half, so the bound is only approximate. The USB code keeps a commented-out median alternative utils.py.
• Variance scaling wording. §4.1 says divide sigma-hat by 4 for 2 sigma, and the appendix says divide the estimated variance by 4 §4.1 app. A.5. Both codes divide the variance by 4 in the exponent utils.py softmatch.py. Implement the variance divided by 4, as in the pseudocode above.
• Initial behaviour. The paper initialises mu-hat at 1/C and sigma-hat-squared at 1.0 app. A.2. The top probability is always at least 1/C, so at the start every sample has weight 1 until mu-hat moves. With momentum 0.999, the EMA's memory is about 1/(1-m) = 1,000 steps; this is derived from the stated momentum, not stated in the paper.
• lambda_max unspecified. The paper never gives a value §2.1. Algorithm 1 uses 1.0 outside the Gaussian region alg. 1, which implies lambda_max = 1.
• UA EMA details not in paper. The momentum and first-batch initialisation of the mean-prediction EMA are not stated. The USB code initialises it from the first batch and uses ema_p = 0.999 dist_align.py.
• Stop-gradient on weights. The paper does not say whether gradients flow through lambda or the pseudo-labels. Both codes compute weights and targets without gradient softmatch.py softmatch.py.
• Text claims swap two margins. The paper says SoftMatch beats FlexMatch by 7.73% on CIFAR-100 with 400 labels and by 2.84% on STL-10 with 40 labels §4.1. From tab. 2, the CIFAR-100 gap is 39.94 − 37.10 = 2.84 and the STL-10 gap is 29.15 − 21.42 = 7.73. The two values are swapped in the text.
• Long-tailed margin. The paper says SoftMatch beats the second-best by 2.4% at gamma = 150 on CIFAR-10-LT §4.2. In tab. 4, the second-best is FixMatch at 29.62, a gap of 2.22; 2.40 is the gap to FlexMatch (29.80).
• ImageNet label budget. The text says ImageNet "with 10% labels" §4.1. The table columns say 100k and 400k labels tab. 3. The paper does not state the ImageNet training-set size used for the percentage.
• Batch sizes for ImageNet. The paper says B_U = 7 x B_L for all datasets §4.1, but its table gives ImageNet B_L = B_U = 128 tab. 6.
• Long-tailed training length. The paper does not state the number of training steps for long-tailed runs tab. 7 app. A.3.2. The training time is about one day on V100. Set steps and schedule K explicitly when reproducing.
• Evaluation weights. Classic and long-tailed runs evaluate an EMA of the weights (0.999), while text runs evaluate the fine-tuned weights directly tab. 6 tab. 7 tab. 8.
• Where SoftMatch is not best. On CIFAR-100 with 2,500 and 10,000 labels, ReMixMatch (26.03, 20.02) and FlexMatch (26.49, 21.90) are lower than SoftMatch (26.66, 22.03). On SVHN with 1,000 labels, UDA (1.89) and FixMatch (1.96) are lower than SoftMatch (2.01) tab. 2. On Yelp-5, FlexMatch is lower (39.01 against 39.31) tab. 5. The paper's text acknowledges the CIFAR-100 and SVHN cases §4.1. All results are author-reported, over 3 seeds or 5 splits, and were not reproduced here.
• Compute. The paper reports the SoftMatch overhead as trivial app. A.2. The classic runs take about 3 to 7 days per setting on a V100 app. A.3.1.

Lineage

• Builds on: confidence-thresholded pseudo-labelling as in FixMatch, with the class-wise and dynamic thresholds of FlexMatch and Dash, and distribution alignment from ReMixMatch §1 §2.2 §3.2. The paper does not give arXiv ids for these in the text.
• Compared against: PseudoLabel, MeanTeacher, MixMatch, ReMixMatch, UDA, FixMatch, FlexMatch, and Influence (the sample-weight method of Ren et al.) tab. 2 §5.
• Superseded by: not stated in the paper's content.