Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

In plain words

Segmentation means giving every pixel of a picture a label such as "road" or "person". Labelling every pixel by hand is expensive, so this paper trains with a small labeled set plus a large pile of unlabeled photos. A teacher copy of the model guesses labels for the unlabeled pixels, and a student copy learns from those guesses.

The usual trick is to trust only the guesses the teacher is sure about and throw away the rest. The authors point out that the unsure pixels are often unsure between two or three similar classes, yet quite sure they are not any of the others. So instead of discarding them, the method uses them as negative examples: "this pixel is not a car, not a train". Each class keeps a queue of such negative examples, so every class gets a steady supply. A contrastive loss then pulls confident pixels of a class towards that class's average and pushes them away from the queued negatives.

The cut-off between sure and unsure pixels is not fixed. The share of pixels treated as unsure starts at one fifth and shrinks as training goes on.

TL;DR

• Semi-supervised semantic segmentation: a few labeled images, many unlabeled ones, teacher-student self-training with DeepLabv3+ on ResNet-101 §4.1.
• Pixels whose teacher entropy is in the top fraction are excluded from the pseudo-label loss §3.2 but reused as negatives in a per-class memory queue for a pixel-level InfoNCE loss §3.3.
• The unreliable fraction starts at 20% and decays linearly to zero over training §3.2 eq. 7.
• Author-reported gains over supervised-only and prior semi-supervised methods on PASCAL VOC 2012 and Cityscapes §4.2, tabs. 1–3.
• Several details differ between the paper and its code (anchor set, positive centre, query and negative counts, λ_c, Cityscapes warm-up). See "Paper vs code".

Problem

Supervised segmentation needs dense, expensive annotation §1. Self-training with pseudo-labels usually keeps only confident predictions §1, §2. Pixels the model cannot classify confidently then never receive a learning signal, and some classes can be starved of supervision §1, fig. 1. Discarding unreliable pixels also loses information about which classes they are not §1, fig. 2. The method assumes a standard teacher-student setup with an exponential moving average (EMA) teacher §3.1.

Method

Overview §3.1. Labeled and unlabeled images are sampled in equal numbers per step §3.1. The total loss is L = L_s + λ_u L_u + λ_c L_c eq. 1. L_s is pixel cross-entropy on labeled images eq. 2. L_u is cross-entropy on pseudo-labels from unlabeled images eq. 3. L_c is a pixel-level InfoNCE loss over the pixels that L_u does not use eq. 4. The teacher's weights are an EMA of the student's §3.1. The full procedure is in alg. 1.

Step 1: entropy and reliability §3.2. The teacher predicts softmax probabilities p_ij for each unlabeled pixel. The per-pixel entropy H is computed from p_ij eq. 5. The entropy threshold γ_t is the quantile of H at 100·(1 − α_t), so the top α_t fraction of pixels is unreliable §3.2.

Step 2: shrinking unreliable share §3.2. α_t = α_0·(1 − t/T), where t is the epoch, T the total number of epochs and α_0 = 20% eq. 7.

Step 3: pseudo-labels §3.2. A reliable pixel (H < γ_t) gets its argmax class as pseudo-label. All other pixels are ignored eq. 6.

Step 4: unsupervised loss weight §3.2. λ_u = η · (B_u·H·W) / (number of non-ignored pixels in the batch), with η = 1 eq. 8. The pseudo-label cross-entropy is computed on the surviving pixels eq. 3.

Step 5: anchors §3.3. For each class c, anchor pixels are labeled pixels of class c with teacher probability p_c above δ_p = 0.3 eq. 9, plus unlabeled pixels with pseudo-label c and p_c above δ_p eq. 10. The anchor set is their union eq. 11.

Step 6: positive §3.3. The positive for class c is the mean of the teacher's representations over the anchors of class c eq. 12.

Step 7: negatives §3.3. A labeled pixel is a negative for class c if its ground truth is not c and c is among the top r_l = 3 probabilities eq. 14. An unlabeled pixel is a negative for c if it is unreliable (H > γ_t) and the rank of c lies in [r_l, r_h) with r_h = 20 eq. 15. Ranks are 0-based, rank 0 being the argmax §3.3. Negatives are pushed into a FIFO queue Q_c per class §3.3, alg. 1.

Step 8: InfoNCE §3.3. For each class, M anchors are sampled, and each one gets one positive and N negatives sampled from Q_c. Cosine similarities are divided by τ = 0.5. The loss is averaged over classes and anchors eq. 4. Anchor features carry gradients from the student; positive and negative features come from the teacher §3.3, alg. 1.

# One U2PL training iteration  [alg. 1] [eq. 1]
# Input:  labeled images_l and masks_l, unlabeled images_u, epoch t
# Output: total loss

def training_step(images_l, masks_l, images_u, epoch):
    # Teacher predicts class probabilities on unlabeled pixels  [§3.2]
    probs_u = softmax(teacher(images_u))           # (B_u, C, H, W)
    # Per-pixel entropy of those probabilities  [eq. 5]
    entropy = -sum(probs_u * log(probs_u), over_classes)  # (B_u, H, W)

    # Unreliable share shrinks linearly from 20% towards 0  [eq. 7]
    alpha = 0.2 * (1 - epoch / total_epochs)
    # Entropy threshold at the (1 - alpha) quantile  [§3.2]
    gamma = percentile(entropy, 100 * (1 - alpha))
    reliable = entropy < gamma
    # Argmax pseudo-labels for reliable pixels, ignore the rest  [eq. 6]
    pseudo_u = where(reliable, argmax(probs_u), IGNORE)

    # Supervised cross-entropy on labeled images  [eq. 2]
    loss_s = cross_entropy(student(images_l), masks_l)
    # Share of kept pixels rescales the unsupervised loss  [eq. 8]
    num_pixels = batch_u * height * width
    weight_u = num_pixels / count(pseudo_u != IGNORE)
    # Pseudo-label loss on the kept pixels, with eta = 1  [eq. 3]
    loss_u = weight_u * cross_entropy(student(images_u), pseudo_u)

    # Contrastive term over every class present in the batch  [§3.3]
    loss_c = 0
    for c in range(C):
        # Anchors: confident pixels of class c, p_c > 0.3  [eq. 9-11]
        anchors = anchor_pixels(c, labels_l, pseudo_u, probs, delta=0.3)
        # Positive: mean teacher representation of the anchors  [eq. 12]
        positive = mean(teacher_features[anchors])
        # Negatives: labeled non-c pixels in top 3, unlabeled unreliable
        # pixels with rank of c in [3, 20)  [eq. 14-15]
        queue[c].push(negative_pixels(c, labels_l, probs_u, entropy))
        # M anchors and N negatives per anchor  [§3.3] [eq. 4]
        queries = sample(student_features[anchors], M)
        keys = sample(queue[c], N)
        loss_c += infonce(queries, positive, keys, tau=0.5)  # [eq. 4]
    loss_c = loss_c / (C * M)                      # average  [eq. 4]

    # Total loss, lambda_u folded into loss_u, lambda_c = 0.1  [eq. 1]
    return loss_s + loss_u + lambda_c * loss_c

After each optimizer step, the teacher's parameters move towards the student's by an exponential moving average §3.1. The paper does not give the momentum value.

Strong augmentation §4.2. The main variant, "U2PL (w/ CutMix)", applies CutMix to the unlabeled images §4.2, tab. 3. The paper does not describe a second variant "U2PL (w/ AEL)" beyond its table row tab. 3.

Architecture §4.1. ResNet-101 pre-trained on ImageNet is the encoder and DeepLabv3+ the decoder. The segmentation head and the representation head are both built from two Conv-BN-ReLU blocks. The representation head outputs 256 dimensions §4.1.

Hyperparameters

Results

Author-reported, mIoU on the validation sets. Setup: ResNet-101 encoder and DeepLabv3+ decoder for all rows §4.2. Baselines are re-implemented by the authors (marked † in the paper) or taken from prior work §4.2. "Strongest listed" is the highest prior row in the same table.

PASCAL VOC 2012, classic split (1,464 labeled candidates) tab. 1

PASCAL VOC 2012, blender split (10,582 labeled candidates) tab. 2

Cityscapes val tab. 3

Ablations tab. 4–7, PASCAL blender split, 1/4 (2646) unless stated.

• Negatives from unreliable pixels (top 20% entropy) 79.01 at 1/8 and 79.30 at 1/4. Reliable-only negatives give 77.30 and 77.35; all pixels give 77.40 and 77.57 tab. 4.
• Rank thresholds (r_l, r_h) = (3, 20) give the best result of those tested; (1, 3) gives 79.03 at 1/4 and (10, 20) gives 78.94 tab. 5.
• Components at 1/4: no contrastive loss 73.02; L_c alone 77.08; with queue, PRT and high-entropy filtering 78.49; with DPA and high-entropy filtering 79.07; all components 79.30 tab. 6.
• α_0 = 20% is best: 40%, 30%, 20%, 10% give 76.77, 77.34, 79.01, 77.80 at 1/8 tab. 7.
• Binary cross-entropy in place of InfoNCE reaches 79.64 at 1/4 on blender app. tab. A4 and 75.91 on Cityscapes at 1/4 app. tab. A3.

Paper vs code

Code: official repository https://github.com/Haochen-Wang409/U2PL at commit 976c1169c4b168e5889383c78d3fc0fe4dc530ac, Apache-2.0. It belongs to the authors: the first author's GitHub user matches the author name "Haochen Wang", and the paper's footnote links the project page under the same user name §1 footnote. Checked: train_semi.py, u2pl/utils/loss_helper.py, u2pl/utils/utils.py, u2pl/dataset/augmentation.py, u2pl/models/decoder.py, and the configs under experiments/.

• Contrastive weight λ_c. Paper: λ_c = 0.1 app. tab. A1. Code: loss_weight is read with default 1 and neither config sets it, so λ_c = 1 train_semi.py config.yaml.
• Anchors and negatives per class. Paper: M = 50 anchors and N = 256 negatives per anchor §3.3 eq. 4 app. tab. A1. Code: num_queries = 256 is the number of anchors and num_negatives = 50 the negatives per anchor config.yaml loss_helper.py loss_helper.py. The two numbers are swapped relative to the paper.
• Reliable anchor set. Paper: anchors use reliable pixels, H ≤ γ_t, which is the lowest-entropy 1 − α_t share, about 80% at the start §3.3 eq. 10 §3.2. Code: the anchor mask uses entropy at or below the α_t-th percentile, i.e. the lowest-entropy α_t share, about 20% at the start and shrinking towards 0 by the end train_semi.py. The negative (unreliable) mask in the code, top α_t share train_semi.py, matches the paper.
• Positive centre. Paper: mean over A_c, which includes the p_c > δ_p filter eq. 12. Code: mean of the teacher's representations over all reliable pixels of class c, without the δ_p filter loss_helper.py.
• Warm-up for Cityscapes. Paper: warm start with supervised loss for 20 epochs on Cityscapes and 1 on VOC app. A. Code: Cityscapes config sets sup_only_epoch: 0 config.yaml. VOC has no such key, so the default 1 applies train_semi.py. VOC matches; Cityscapes does not.
• Teacher momentum. Paper: EMA update, no value given §3.1. Code: momentum 0.99 with a ramp-up min(1 − 1/(iter + 1), 0.99) config.yaml train_semi.py.
• Positive normalisation. Paper: 1/(C·M) eq. 4. Code: divides by the number of classes present in the batch loss_helper.py. Matches only when all classes are present.
• Negative gate δ_n. The code has current_class_negative_threshold: 1 config.yaml. The paper has no such term. It is applied as prob < 1 loss_helper.py, which is effectively always true.
• Augmentation probability. Paper: CutMix used, probability not stated §4.2. Code: applied to each unlabeled batch with probability 0.5 train_semi.py.
• Heads. Paper: two Conv-BN-ReLU blocks, 512-d input from ASPP §4.1. Code: the heads take the concatenation of ASPP and low-level features (512 channels), apply two 3×3 conv blocks with dropout 0.1, and a final 1×1 conv. The representation head ends in a 1×1 conv to 256 channels decoder.py decoder.py. Dropout and the 1×1 output are not in the paper.
• Batch size. Paper: batch 16 §4.1. Code: per-process batch 4 for VOC and 2 for Cityscapes, with comments for 4 and 8 GPUs config.yaml config.yaml. The total of 16 holds only with those GPU counts.
• Auxiliary loss (Cityscapes only). Code: auxiliary head with weight 0.4 and width 1024 config.yaml. Not in the paper.
• Anchor and negative sampling. Code samples with replacement loss_helper.py loss_helper.py. The paper does not say.

Pitfalls

• Shrinking contrastive sets. Because both the anchor set (lowest-entropy α_t share) and the negative set (top α_t share) use the decaying α_t, both shrink towards zero late in training train_semi.py. The paper's anchor definition does not do this (see "Paper vs code"). Pick one definition and measure it.
• Teacher in train mode for the contrastive pass. The teacher runs its contrastive forward pass in train mode under no-grad train_semi.py. The EMA copies parameters only train_semi.py, so the teacher's BatchNorm running statistics come from that pass. The teacher's pseudo-labels use those statistics in eval mode train_semi.py. The paper is silent on this.
• Division by zero. The loss weight divides by the count of kept pixels loss_helper.py. If a batch has no kept pixels the weight is infinite or NaN. The code does not guard this.
• Queue behaviour. Queues persist across iterations and epochs and store teacher features utils.py. Background uses 50,000 entries and foreground classes 30,000 train_semi.py. Early in training the queues are empty and the contrastive term for those classes is skipped loss_helper.py.
• Cutoffs depend on the batch. Thresholds are percentiles of the entropy of the current batch, not of the whole unlabeled set train_semi.py. Results will move with batch size and crop.
• Training cost. The paper notes that training is slow compared with supervised methods §5. It gives no wall-clock numbers.
• Cityscapes results versus claims. Only the "w/ AEL" variant beats AEL at every fraction in table 3. The CutMix variant is below AEL at 1/16, 1/8 and 1/4 tab. 3. The text describes "notable margins" §4.2 without that caveat.
• Figures. Figures 1 to 4 and A1 to A2 illustrate entropy, the pipeline and the qualitative results fig. 1–4 fig. A1–A2. They have no numbers that are not in the tables.

Lineage

• Builds on: CutMix for the strong augmentation variant §4.2; the contrastive pixel-level loss and its hyperparameters from Reco §3.3; self-training with confidence filtering in FixMatch and UPS, which the paper contrasts against §2.
• Compared against: Mean Teacher, CCT, GCT, PseudoSeg, CutMix, CPS, PC²Seg and AEL §4.2.
• Related: Negative learning, which uses only confident negatives §2. The paper argues for using unreliable pixels as negatives instead §2.