FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning

In plain words

Semi-supervised learning trains an image classifier on a few labeled images and many unlabeled ones. A common trick is to let the model label the unlabeled images itself and learn from the guesses it is sure about. "Sure enough" is decided by a confidence threshold, and that threshold is usually a fixed number someone picked by hand. A threshold that is too low lets wrong guesses into training. One that is too high throws away most of the unlabeled data, and training slows down.

FreeMatch removes the hand-picked number. It tracks how confident the model has been on unlabeled images over recent training, and it tracks how often the model predicts each class. The threshold rises as the model gets more confident, and it is lower for classes the model still rarely predicts. A second term nudges the model to spread its predictions across classes, which matters most when labels are very scarce. Think of it as a bar that lowers itself while the student is still learning and rises as the student gets better.

TL;DR

• Replaces the fixed confidence threshold of pseudo-labeling with a self-adaptive one: a global threshold from the model's average confidence, scaled per class by the model's class-wise prediction rate §4.1.
• Adds a self-adaptive fairness loss that compares normalised class-prediction statistics against a target §4.2 eq. 11.
• Author-reported gains are largest with very few labels (CIFAR-10 with 10 labels, STL-10 with 40 labels) §5.2 tab. 1.
• The official code (semilearn package) differs from the paper in several settings, notably the fairness weight (see Paper vs code).

Problem

Pseudo-labeling and consistency methods assign hard labels to unlabeled images whose top predicted probability exceeds a threshold, and they skip the rest §3 eq. 4. Earlier methods use one fixed threshold, and the paper cites a fixed high value of 0.95 as an example §1. The authors argue that a fixed or ad-hoc, hand-scheduled threshold ignores the model's learning status and does not account for differences between classes §1 §2.

A motivating two-Gaussian analysis gives the intuition §2: the fraction of unlabeled data that passes a threshold depends on the threshold and on how far apart the class distributions are, and the best threshold rises as the model becomes more confident §2 eq. 2. The analysis is a simplified binary model, not a proof for deep networks.

The paper's assumptions are a standard semi-supervised setup: a labeled set and an unlabeled set, weak and strong augmentations, and known number of classes C §3.

Method

Notation. Labeled batch size is B. The unlabeled batch has μB images, where μ is the unlabeled-to-labeled ratio. q_b denotes the weak-view class probabilities and Q_b the strong-view probabilities for unlabeled image b. Q̂_b and q̂_b are their hard one-hot labels. C is the number of classes §3.

1. Base objective. Supervised cross-entropy on labeled data eq. 3. For unlabeled data, the weak view gives a hard label, and the strong view is trained with cross-entropy against it, gated by a confidence mask eq. 4 §3. The paper uses weak augmentation (random crop and flip) and strong augmentation (RandAugment) §3.
2. Global threshold τ_t. An exponential moving average (EMA) of the mean maximum weak-view probability over the unlabeled batch. Initialised at 1/C eq. 5.
3. Local statistics p̃_t. An EMA of the mean weak-view probability vector, initialised at 1/C eq. 6. Its histogram counterpart h̃_t is an EMA of the histogram of hard predictions, which the algorithm updates the same way eq. 10 alg. 1.
4. Self-adaptive threshold per class. Each class threshold is τ_t times the max-normalised entry of p̃_t, so the most-predicted class gets the full τ_t eq. 7.
5. Masked unsupervised loss. A weak-view sample is kept if its maximum probability reaches the threshold of its own predicted class. The loss is the masked cross-entropy on the strong view, averaged over μB eq. 8 alg. 1. Note the strict inequality in eq. 8 versus the ≥ used in the algorithm (see Pitfalls).
6. Self-adaptive fairness (SAF). Over the kept samples, compute the strong-view mean probability p̄ and the histogram h̄ of strong hard labels eq. 9. Normalise p̃_t/h̃_t and p̄/h̄ to sum to one eq. 11. The fairness loss is the negative cross-entropy between these two distributions eq. 11.
7. Total loss. L = L_s + w_u L_u + w_f L_f eq. 12. The paper sets w_u = 1 for all experiments and w_f per setting (see Hyperparameters) app. D, tab. 5.

Full update rules as written in the paper:

\tau_t = \begin{cases} \frac{1}{C}, & t = 0 \\ \lambda \tau_{t-1} + (1-\lambda)\frac{1}{\mu B}\sum_{b=1}^{\mu B}\max(q_b), & \text{otherwise} \end{cases} eq. 5

\tilde{p}_t(c) = \begin{cases} \frac{1}{C}, & t = 0 \\ \lambda \tilde{p}_{t-1}(c) + (1-\lambda)\frac{1}{\mu B}\sum_{b=1}^{\mu B} q_b(c), & \text{otherwise} \end{cases} eq. 6

\tau_t(c) = \operatorname{MaxNorm}(\tilde{p}_t(c))\cdot\tau_t = \frac{\tilde{p}_t(c)}{\max\{\tilde{p}_t(c') : c' \in [C]\}}\cdot\tau_t eq. 7

\mathcal{L}_u = \frac{1}{\mu B}\sum_{b=1}^{\mu B}\mathbb{1}\big(\max(q_b) > \tau_t(\arg\max q_b)\big)\cdot\mathcal{H}(\hat{q}_b, Q_b) eq. 8

\bar{p} = \frac{1}{\mu B}\sum_{b=1}^{\mu B}\mathbb{1}\big(\max(q_b)\ge\tau_t(\arg\max q_b)\big)\,Q_b, \qquad \bar{h} = \operatorname{Hist}_{\mu B}\big(\mathbb{1}(\ldots)\,\hat{Q}_b\big) eq. 9

\tilde{h}_t = \lambda\tilde{h}_{t-1} + (1-\lambda)\operatorname{Hist}_{\mu B}(\hat{q}_b) eq. 10

\mathcal{L}_f = -\mathcal{H}\Big(\operatorname{SumNorm}\big(\tfrac{\tilde{p}_t}{\tilde{h}_t}\big),\ \operatorname{SumNorm}\big(\tfrac{\bar{p}}{\bar{h}}\big)\Big), \quad \operatorname{SumNorm}(x) = \frac{x}{\sum x} eq. 11

\mathcal{L} = \mathcal{L}_s + w_u\,\mathcal{L}_u + w_f\,\mathcal{L}_f eq. 12

Here H(·,·) is cross-entropy, Hist is a histogram over the μB samples, and λ is the EMA decay eq. 5–eq. 11. The paper's initial value for h̃_t is not stated in the text; the code uses 1/C utils.py.

Per training step, the algorithm updates τ_t, p̃_t and h̃_t from the weak view, then computes the per-class thresholds, the masked unsupervised loss, p̄, h̄ and the fairness loss alg. 1 app. C. The extra cost over FixMatch is the EMA and histogram bookkeeping, which the paper describes as very small app. C; its ImageNet table reports 0.4 s/iter for FixMatch and FreeMatch and 0.6 s/iter for FlexMatch tab. 2.

Pseudocode of one training step, following alg. 1 and eq. 3–eq. 12. sum_norm(x) = x / sum(x). Tensors are written with their shape:

# FreeMatch loss at iteration t  [alg. 1] [app. C]
# Inputs: labeled (x, y); unlabeled weak view u_w and strong view u_s
# State: tau (scalar), p_tilde (C,), h_tilde (C,); lam = EMA decay
# sum_norm(x) = x / sum(x); max-normalisation is x / max(x)

def freematch_loss(x, y, u_w, u_s, state, lam, w_u, w_f):
    # Supervised cross-entropy on labeled data  [eq. 3]
    loss_sup = cross_entropy(model(x), y)

    # Weak-view probabilities, treated as constants  [§4.1]
    probs_w = softmax(model(u_w)).detach()      # (muB, C)
    max_probs = max(probs_w, axis=1)            # (muB,)
    pseudo = argmax(probs_w, axis=1)            # (muB,)

    # Global threshold: EMA of mean top confidence  [eq. 5]
    state.tau = lam * state.tau + (1 - lam) * mean(max_probs)

    # Per-class EMA of mean predicted probability  [eq. 6]
    state.p_tilde = (
        lam * state.p_tilde + (1 - lam) * mean(probs_w, axis=0)
    )

    # Per-class EMA of the histogram of hard predictions  [eq. 10]
    hist = mean(one_hot(pseudo, C), axis=0)     # (C,)
    state.h_tilde = lam * state.h_tilde + (1 - lam) * hist

    # Class threshold: max-normalised p_tilde scales tau  [eq. 7]
    scale = state.p_tilde / max(state.p_tilde)  # (C,)
    thresholds = scale * state.tau              # (C,)

    # Keep a weak sample if it clears its predicted class threshold  [eq. 8]
    mask = max_probs >= thresholds[pseudo]      # (muB,)

    # Masked cross-entropy, strong view vs hard pseudo-labels  [eq. 8]
    logits_s = model(u_s)                       # (muB, C)
    loss_unsup = mean(mask * cross_entropy(logits_s, pseudo))

    # Strong-view mean probability and histogram over kept samples  [eq. 9]
    probs_s = softmax(logits_s)                 # (muB, C)
    hard_s = argmax(probs_s, axis=1)            # (muB,)
    p_bar = mean(mask[:, None] * probs_s, axis=0)              # (C,)
    h_bar = mean(mask[:, None] * one_hot(hard_s, C), axis=0)   # (C,)

    # Fairness: normalised target against normalised prediction  [eq. 11]
    target = sum_norm(state.p_tilde / state.h_tilde)           # (C,)
    guess = sum_norm(p_bar / h_bar)                            # (C,)
    # Equals -H(target, guess); minimised as written  [eq. 11]
    loss_fair = sum(target * log(guess + 1e-12))

    # Total objective: w_u = 1; w_f per setting  [eq. 12] [tab. 5]
    return loss_sup + w_u * loss_unsup + w_f * loss_fair

Initial values: τ_0 = 1/C, p̃_0 = 1/C eq. 5 eq. 6. With C = 10 the first threshold is 0.1 §4.1.

Hyperparameters

Values from the paper's appendix and experimental setup. Code values are in Paper vs code.

Results

Author-reported. Setup: TorchSSL codebase, identical backbones and hyperparameters for all methods, 3 seeds, best error over checkpoints, top-1 error in percent, mean ± std §5.1 tab. 1. Lower is better. The paper's bold and underline markings are not reproduced; read the numbers directly.

Table 1 tab. 1: error rates on CIFAR-10/100, SVHN and STL-10. Column headers give the number of labeled samples (CIFAR-10 has 10, 40, 250, 4000 labels; CIFAR-100 has 400, 2500, 10000; SVHN has 40, 250, 1000; STL-10 has 40, 1000). Fully supervised reference rows are in the last line.

FreeMatch against FlexMatch, the strongest baseline in most columns tab. 1: CIFAR-10 with 10 labels 13.85 → 8.07; STL-10 with 40 labels 29.15 → 15.56; CIFAR-100 with 400 labels 39.94 → 37.98. The abstract's "error rate reduction" values (5.78, 13.59 and, on ImageNet, 1.28) are differences in percentage points between these error rates, not relative reductions §1 §5.2 tab. 1 tab. 2.

Table 2 tab. 2: ImageNet with 100 labels per class (100k labels), top-1 / top-5 error and runtime per iteration.

Qualitative results (STL-10, 40 labels): the class-average threshold of FreeMatch is lower than FlexMatch and FixMatch early on, the sampling rate is higher, and the threshold rises as training proceeds fig. 3 §5.3. The paper also reports t-SNE plots and a confusion matrix app. E. Their numbers are not reproduced here.

Ablations §5.4:

Table 3 tab. 3: thresholding schemes, CIFAR-10 with 40 labels, error (%). Fixed τ (FixMatch) 7.47±0.28; FlexMatch scheme 4.97±0.06; max-normalised local threshold alone 5.13±0.03; global threshold alone 6.06±0.65; global × FlexMatch scaling 8.40±2.49; CBST 16.65±2.90; AdaMatch relative threshold 6.09±0.65; SAT (global and local) 4.92±0.04. The paper notes that L_f is excluded from this ablation.

Table 4 tab. 4: fairness terms, CIFAR-10 with 10 labels, error (%). No fairness 10.37±7.70; U log p̄ 9.57±6.67; U log SumNorm(p̄/h̄) 12.07±5.23; AdaMatch DA 32.94±1.83; ReMixMatch DA 11.06±8.21; SAF (this paper) 8.07±4.24.

Table 11 tab. 11 (EMA decay, CIFAR-10 with 40 labels): 0.9 → 4.94±0.06; 0.99 → 4.92±0.08; 0.999 → 4.90±0.04; 0.9999 → 5.03±0.07.

Imbalanced SSL app. E, tab. 14: CIFAR-10-LT and CIFAR-100-LT results are given in the appendix. In the FixMatch, FlexMatch and FreeMatch rows shown there, FreeMatch is best or tied at every imbalance setting.

Paper vs code

Code: official repository microsoft/Semi-supervised-learning (linked in the paper's abstract), cloned at commit 1ef4cbebcc0b368158315aeb425053858cf6c845. The FreeMatch implementation is semilearn/algorithms/freematch/freematch.py and the threshold hook is semilearn/algorithms/freematch/utils.py. Configs are under config/classic_cv/freematch/.

• Fairness weight w_f is 10× lower in the code. The paper uses 0.01 or 0.05 depending on the setting app. D, tab. 5. Every FreeMatch config in config/classic_cv/freematch/ sets ent_loss_ratio: 0.001, for example freematch_cifar10_40_0.yaml. The code's argument default is 0.01 freematch.py, so the configs override it. The paper's settings are not what the configs run.
• SVHN threshold range. The paper restricts the self-adaptive threshold to [0.9, 0.95] on SVHN §5.1. The code clips the global threshold to [0, 0.95] utils.py, and only when clip_thresh is set. The SVHN config sets it True freematch_svhn_40_0.yaml. The lower bound of 0.9 is not in the code.
• SVHN warm-up not found. The paper's 2-epoch labeled-only warm-up on SVHN §5.1 has no matching setting in the SVHN config or the FreeMatch class. num_warmup_iter exists in the config parser with default 0, but it only feeds the learning-rate schedule config.py.
• ImageNet setting not in this repo. The ImageNet (100 labels per class) setting from the paper tab. 2 has no FreeMatch config under config/classic_cv/. The usb_cv FreeMatch configs use other datasets and ViT backbones (for example config/usb_cv/freematch/freematch_stl10_100_0.yaml, which uses a ViT and learning rate 1e-4) and are not the paper's ImageNet run.
• Fairness normalisation differs but cancels. The paper divides p̄ and h̄ by μB eq. 9. The code averages p̄ over the kept samples and normalises the histogram by the kept count freematch.py freematch.py. Both differ by a constant factor that SumNorm removes, so the loss value is the same up to this factor.
• Mask operator. The paper writes strict > in eq. 8 and ≥ in the algorithm. The code uses ge (≥) utils.py.
• Pseudo-label temperature T. The config sets T = 0.5 freematch_cifar10_40_0.yaml for every run, but the paper uses hard pseudo-labels, and the hook returns the argmax before T is applied when hard_label is True pseudo_label.py. T has no effect in the paper's setting.
• Extra options not in the paper. use_quantile (replaces the mean with the 0.8 quantile of the confidences) defaults to False freematch.py utils.py. The paper does not mention it.
• Matches checked. Learning rate 0.03, momentum 0.9, EMA 0.999, batch 64, μ = 7, weight decay (5e-4 and 1e-3 for CIFAR-100), cosine schedule with 7/16 cycles build.py, and 2^20 steps all agree with §5.1 and app. D. The joint forward on labeled and unlabeled views (use_cat: True) is set in the configs freematch_cifar10_40_0.yaml and handled at freematch.py. The paper does not say whether labeled and unlabeled batches share one forward pass, so this is a code choice that affects BatchNorm statistics.

No discrepancies found in the threshold update itself: the global threshold utils.py, the p̃ and h̃ EMAs utils.py, the max-normalised per-class threshold and mask utils.py, and the masked unsupervised loss averaged over μB consistency.py match eq. 5–8.

Pitfalls

• Reported error is the best checkpoint, not the final one. The paper reports "the best error rates of all checkpoints" over 3 seeds §5.1. The table columns are therefore optimistic relative to the last checkpoint. The same protocol is used for the baselines, but a reimplementation should report both numbers.
• Initial threshold is 1/C. With C = 10 it starts at 0.1, so almost every weak sample is accepted in the first steps. The mask only tightens as the EMA moves eq. 5 utils.py.
• Slow EMA. λ = 0.999 means a time constant of roughly 1000 steps, so the thresholds lag the model. The paper's EMA ablation shows the result is flat across 0.9 to 0.999 but worse at 0.9999 tab. 11.
• Division by histogram and p̄/h̄. Both 1/h̃_t and 1/h̄ blow up when a class is never predicted. The code replaces the infinite scaler with 0, which removes that class from the fairness term freematch.py freematch.py. The paper does not state this. A small epsilon is added only in the log freematch.py.
• Detached targets. In the code the weak-view probabilities, the EMA state and the scalers 1/h are detached, so gradients flow only through the strong-view predictions utils.py freematch.py freematch.py. The paper does not state which terms take gradients; keep this detachment when reimplementing.
• Empty mask. If no unlabeled sample passes the threshold, the fairness loss is set to 0 freematch.py. The paper is silent on this case.
• Scarce labels dominate the claims. The largest gains are at 10 to 40 labels. Several cells have high standard deviation (for example CIFAR-10 with 10 labels: 8.07 ± 4.24 versus FlexMatch 13.85 ± 12.04 tab. 1), so the gap is within a few standard deviations for some settings. Read the significance tests in the paper's appendix before relying on the ranking app. E.
• Compute is large. The schedule is 2^20 steps. Two days on one V100 for CIFAR-10, about 10 days for CIFAR-100 and STL-10 app. D. The ImageNet run uses 4 V100 GPUs §5.2 footnote.
• Repository license text. The README and the source headers say MIT and Microsoft Corporation, but LICENSE.txt in the clone is MIT text with a 2021 copyright line for a different author. code_license: MIT follows the README and LICENSE.txt. Check this before copying any code, since the copyright line does not match.

Lineage

• Builds on: the pseudo-labeling and consistency-regularisation line of work, with the "weak and strong augmentation" strategy from UDA, and the fairness term from prior work on pseudo-label bias §3. These papers are named in the text, but they have no cards here, so nothing from them is used.
• Compared against: Π Model, Pseudo Label, VAT, Mean Teacher, MixMatch, ReMixMatch, UDA, FixMatch, Dash, MPL and FlexMatch tab. 1; CBST and the AdaMatch relative threshold in the threshold ablation tab. 3; AdaMatch DA and ReMixMatch DA in the fairness ablation tab. 4.
• Superseded by: not known from the content of this wiki.