Entropy minimization

Overview

Entropy minimization adds an unsupervised term to the training loss. For each unlabelled input, the term is the Shannon entropy of the model's predicted distribution, and minimising it pushes that distribution toward a confident answer 1909.13589 §3.1. The total loss is the usual supervised loss on labelled data plus a weighted entropy term on the unlabelled data 1909.13589 §3.1.

The gradient of the binary form grows without bound as the predicted probability approaches 0 or 1 1909.13589 §3.2. The paper argues that this makes inputs the model already predicts confidently dominate the update, while uncertain inputs get little signal 1909.13589 §1.

At test time the same idea is applied to one unlabelled test image at a time. The entropy is taken of the mean prediction over several views of that image, and only the views whose own entropy is low enter the mean 2308.06038 §3.1 eq. 3 eq. 4. Entropy therefore appears twice: once to choose the confident views, and once in the objective. The parameters tuned are four CLIP context tokens, not the whole model 2308.06038 §4.1.

The variants below change the target term, how it is weighted, which predictions it is computed on, or whether source data is available during adaptation. One method replaces the per-sample term with a term on the batch-average prediction 2204.10377 §3.3.

Variants

Plain entropy minimization

The Shannon entropy of the target prediction is added to the source cross-entropy, with weight λ_T 1909.13589 §3.1. The paper's unsupervised domain adaptation setup uses it on GTA5 to Cityscapes with DeepLabv2 and ResNet-101, and reports 42.2 mIoU for the authors' own run 1909.13589 tab. 2. It is also the baseline for the classification experiment on Office-31, where the paper reports 84.1 average accuracy with ResNet-50 1909.13589 tab. 1. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Maximum squares loss

Replaces the entropy with the negative sum of squared target probabilities 1909.13589 §3.2. The paper states that its gradient grows linearly in the probability instead of without bound 1909.13589 §3.2. On GTA5 to Cityscapes it reaches 44.3 mIoU with the same backbone, and 85.6 average accuracy on Office-31 1909.13589 tab. 2 1909.13589 tab. 1. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Image-wise class weighting

Weights each image's target term by the class counts of its own argmax prediction, with a hyperparameter α 1909.13589 §3.3. The paper sets α = 0.2 in its experiments 1909.13589 §4.2. In its ablation it lifts plain entropy from 42.2 to 43.5 and maximum squares from 44.3 to 45.2 1909.13589 tab. 4. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Entropy with a second guidance head

A second output head on intermediate features is trained on confident labels formed from both heads' predictions 1909.13589 §3.4. The paper only tests this on top of maximum squares, not plain entropy, and reports 46.4 mIoU with the full combination 1909.13589 tab. 4. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Mean-prediction entropy over confident views (test-time)

For one test image, the method makes several views of it, keeps the views whose own entropy is low, averages their class probabilities, and minimises the entropy of that average 2308.06038 §3.1 eq. 3 eq. 4. The paper describes this as the objective of test-time prompt tuning (TPT). The TPT baseline reaches 46.65 average accuracy for ResNet-50 and 61.59 for ViT-B/16 on the natural-shift table 2308.06038 tab. 1. The method needs no labels. Card: Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning.

Diffusion views with a cosine filter (test-time)

DiffTPT adds 63 Stable Diffusion image variations to the usual crop and flip views, 128 views in total 2308.06038 §4.1. It keeps the 80% of views most similar to the test image, then the 30% with lowest entropy 2308.06038 §4.1. The paper does not name the representation the cosine is taken in 2308.06038 §3.4. Against TPT, its printed averages are higher by 2.06 and 0.69 points on the natural-shift table, and by 2.87 and 1.48 points on the cross-dataset table, for ResNet-50 and ViT-B/16 respectively 2308.06038 tab. 1 tab. 2. These differences are author-reported and were computed from the printed averages, not taken from the paper's text.

Conditional entropy in a source-free clustering term

Here the source data is not used during adaptation. Only the source-trained classifier and unlabelled target images are available 2502.19316 §1. The classifier's clustering regularizer has two parts: the conditional entropy of its predictions on target images, and a KL term between each prediction and the prediction on an adversarially perturbed copy of the image 2502.19316 §3.3 eq. 7 eq. 8 eq. 9. A generator, trained with the classifier, supplies labelled samples, and a weight anchor keeps the classifier near its source weights 2502.19316 §3.1 §3.2. The paper does not say when the clustering term starts 2502.19316 §3.4 alg. 1. Its ablation removes the smoothness term but keeps the entropy term, so the entropy term's own effect is not isolated 2502.19316 tab. 5. Author-reported results are 99.4 on SVHN to MNIST against 76.4 for the source-only model 2502.19316 tab. 1, 89.6 Office-31 average against 76.1 with ResNet-50 tab. 2, and 81.6 on VisDA17 against 52.4 with ResNet-101 tab. 3. The paper says most compared methods use source data during adaptation 2502.19316 §4.2. Card: Model Adaptation: Unsupervised Domain Adaptation without Source Data.

Batch-average diversity term instead of per-sample entropy (source-free)

AdaContrast does not minimise per-sample entropy. Its authors argue that entropy minimisation ignores relations between samples and disturbs calibration on the target 2204.10377 §2 §4.3. Instead, for each mini-batch of target predictions, it adds the sum of p̄ log p̄ over classes, where p̄ is the mean prediction over the batch 2204.10377 §3.3 eq. 8 eq. 9. That quantity is the negative entropy of the batch mean, so minimising it spreads the batch's predictions over classes. It is the only term in the method that acts on class balance across the batch [arxiv:2204.10377 §3.3]. The per-sample terms are a cross-entropy against a refined pseudo label and a contrastive loss 2204.10377 §3.2 §3.3 eq. 5 eq. 7.

The ablation's regularisation column raises VisDA-C from 85.7 to 86.8 at the base learning rate, and from 84.3 to 86.6 at ten times that rate. On DomainNet-126 it changes the average from 67.9 to 67.8 2204.10377 tab. 4. The ablation has one row per component, so this is the effect of the term given the other components, not of entropy minimisation. The released code computes the mean over each mini-batch of 128 images, not over the whole target set the paper's equation describes 2204.10377 §3.3 target.py. Card: Contrastive Test-Time Adaptation.

Class-level fairness term in pseudo-labelling (semi-supervised)

FreeMatch does not minimise a per-sample entropy. It trains a classifier on labelled data and on unlabelled data whose hard pseudo-labels pass a confidence threshold 2205.07246 eq. 4 eq. 8. The threshold comes from the model's running confidence, scaled per class by a running prediction rate 2205.07246 eq. 5 eq. 7. The only term that works on class distributions is a fairness loss, the negative cross-entropy between two normalised class distributions: one from running statistics and one from the kept strong-view predictions 2205.07246 eq. 9 eq. 11. The paper gives its weight as 0.01 or 0.05 depending on the setting [arxiv:2205.07246 app. D, tab. 5]. Author-reported errors are 8.07 on CIFAR-10 with 10 labels against 13.85 for FlexMatch, and 15.56 against 29.15 on STL-10 with 40 labels 2205.07246 tab. 1. The CIFAR-10 gap is within the spread of the runs (±4.24 for FreeMatch, ±12.04 for FlexMatch), and the comparison table has no plain entropy-minimization row. Card: FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning.

When to use which

Evidence here comes from five papers, each on different settings. Segmentation numbers use DeepLabv2 with ResNet-101 1909.13589 §4.2. Semi-supervised classification numbers use WRN backbones on CIFAR-10 and STL-10 2205.07246 §5.1. Test-time numbers use CLIP with author-reported results on 1000 sampled test images per dataset 2308.06038 §4.1. Source-free numbers use a DIRT-T-style network for digits and signs and ResNet-50 or ResNet-101 for Office-31 and VisDA17 2502.19316 §4.1. The contrastive test-time adaptation numbers use ResNet-101 on VisDA-C and ResNet-50 on DomainNet-126 2204.10377 §4.1. No setting is shared across papers, so the numbers should not be compared across rows.

The DiffTPT paper does not state the license or terms of the Stable Diffusion checkpoint it uses for generation 2308.06038 §3.3. Check those terms before you redistribute generated views.

Implementation sketch

Plain entropy minimization, with the maximum squares swap noted below. The loss is written in the paper's notation and omits the λ_T schedule.

# Source batch is labelled, target batch is not  [arxiv:1909.13589 §3.1]
def entropy_min_loss(x_s, y_s, x_t, lambda_t):
    # Supervised pixel cross-entropy on the source  [arxiv:1909.13589 §3.1]
    log_p_s = log_softmax(model(x_s))
    loss_src = cross_entropy(log_p_s, y_s)

    # Predicted distribution on the unlabelled target  [arxiv:1909.13589 §3.1]
    log_p_t = log_softmax(model(x_t))
    p_t = exp(log_p_t)
    # Shannon entropy per pixel, averaged over pixels  [arxiv:1909.13589 §3.1]
    loss_ent = -mean(sum(p_t * log_p_t, axis=class))

    return loss_src + lambda_t * loss_ent


# Maximum squares swap for the target term  [arxiv:1909.13589 §3.2]
# loss_ms = -mean(sum(p_t ** 2, axis=class))

Test-time variant, for one test image and its views. The paper's own normaliser is ρ_H·N with N the number of views 2308.06038 eq. 4.

# One test image, N views of it (augmented or generated)  [arxiv:2308.06038 §3.1]
def test_time_entropy_loss(clip, views, ctx, rho):
    # Class probabilities for every view  # (N, K)
    probs = softmax(clip.logits(views, ctx))
    # Entropy of each view's own prediction  [arxiv:2308.06038 eq. 4]
    ent = -sum(probs * log(probs), axis=class)  # (N,)
    # Keep the k = floor(rho * N) views with lowest entropy  [arxiv:2308.06038 §4.1]
    k = floor(rho * N)
    keep = lowest_k_mask(ent, k)
    # Mean prediction over the kept views  [arxiv:2308.06038 eq. 4]
    p_mean = sum(probs[keep], axis=views) / k
    # Entropy of the mean prediction is the loss  [arxiv:2308.06038 eq. 3]
    return -sum(p_mean * log(p_mean))

The source-free entropy term is the same mean per-sample entropy as the plain form, computed on target images inside the clustering regularizer 2502.19316 eq. 7. Its weight is λ_clu, which is 0.1 for digits and 1 elsewhere according to the paper's text, with the sign-task value unstated 2502.19316 §4.1. The paper does not state the perturbation radius ξ or the number of power-iteration steps 2502.19316 eq. 8.

# Source-free: entropy inside the clustering regulariser  [arxiv:2502.19316 §3.3]
def clustering_entropy(C, x_t):
    # Predicted class probabilities on target images  # (B, K)
    probs = softmax(C(x_t))
    # Mean per-sample entropy, minimised  [arxiv:2502.19316 eq. 7]
    return mean(-sum(probs * log(probs), axis=1))

Batch-average diversity term from AdaContrast. It is added to the loss with weight 1 and acts on the mini-batch, not on single samples.

# Batch-average diversity term  [arxiv:2204.10377 §3.3]
def diversity_loss(logits):
    # Mean class probability over the mini-batch  [arxiv:2204.10377 eq. 8]
    p_bar = mean(softmax(logits), axis=0)           # (C,)
    # Sum of p log p over classes, i.e. the negative entropy of p_bar  [eq. 9]
    return sum(p_bar * log(p_bar))

Segmentation runs use λ_T = 0.1 1909.13589 §4.2. Classification uses 0.03 for entropy minimization and 0.3 for maximum squares [arxiv:1909.13589 §4.2]. The paper's sensitivity study for maximum squares covers λ_T from 0.02 to 0.5 and gives 43.0 to 44.3 mIoU 1909.13589 tab. 5. How the target term is normalised (per pixel, per image or per batch) changes the effective λ_T, so keep the normalisation fixed when you compare runs.

For the test-time variants, DiffTPT uses 4 optimisation steps at learning rate 5e-3 on 4 context tokens, starting from the prompt "a photo of a" 2308.06038 §4.1. Going from one to four steps lifts its cross-dataset result from 56.6 to 58.8, and further steps slightly reduce it 2308.06038 fig. 7.

Contradictions

No cross-paper numerical conflict is established, since the papers report different tasks and settings. Within the segmentation paper, the entropy baseline appears twice with close but different numbers: 42.2 as the authors' reproduction (MinEnt†) and 42.3 as the MinEnt result reported by ADVENT 1909.13589 tab. 2. The paper does not reconcile them.

Within the DiffTPT paper, the abstract and conclusion state an average gain of 5.13% over TPT. §4.2 gives 5.1% (ResNet-50) and 2.3% (ViT-B/16) for the cross-dataset scenario, and the printed averages give 2.87 and 1.48 points 2308.06038 §4.2 tab. 2. The paper does not say how the 5.13% figure is computed, so treat it as unverified.

Within the AdaContrast paper, the text says its regularisation term adds 0.9 points on VisDA-C at ten times the base learning rate. Its ablation table gives 84.3 to 86.6, which is 2.3 points 2204.10377 §4.4 tab. 4. This page uses the table.

Papers

• Domain Adaptation for Semantic Segmentation with Maximum Squares Loss: compares plain entropy minimization with a maximum squares term, image-wise class weights and a guidance head for unsupervised segmentation adaptation.
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning: applies an entropy objective over the mean of confident test-time views to CLIP prompt tokens, and adds diffusion views with a similarity filter.
• Model Adaptation: Unsupervised Domain Adaptation without Source Data: adapts a source-trained classifier to unlabelled target images with a class-conditional generator, a weight anchor and a clustering regularizer that includes conditional entropy.
• Contrastive Test-Time Adaptation: adapts a source-trained classifier to unlabelled target batches with a contrastive loss and a batch-average diversity term in place of per-sample entropy minimisation.
• FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning: sets the pseudo-label threshold from the model's running confidence, scaled per class, and adds a class-level fairness term built on a negative cross-entropy.