Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation

In plain words

A segmentation model learns to label every pixel of a street scene. Training it needs many pixel-level labels, which are slow and costly to draw. A common shortcut is to train on synthetic images from a video game, where the labels come for free. The catch is that synthetic images look different from real ones, and a model trained on them does poorly on real photos.

This paper tries to close that gap without labels for the real photos. First, a generator restyles the synthetic images so they look more like the real target domain. It is built to change only the look and keep the objects where they are, so each restyled image can keep the label of its synthetic original. Second, a "teacher" copy of the model keeps a smoothed average of the "student" model's weights. The student is nudged to agree with the teacher's predictions on real images, which are shown with random noise and dropout. The teacher's predictions act as guesses for the unlabeled real images.

The authors' claim is that the restyled images make the teacher-student step work for segmentation, where geometric augmentation misaligns pixel labels.

TL;DR

• Unsupervised domain adaptation from labeled synthetic street scenes (GTA5, SYNTHIA) to unlabeled Cityscapes images, for semantic segmentation §1 §4.1.
• Two parts: TGCF-DA, a GAN that restyles source images to the target look while keeping their semantics, with no cycle-consistency §3.1-§3.3; and self-ensembling (teacher as an EMA of the student) with a consistency loss on target images §3.4-§3.5.
• Author-reported mIoU: 42.5 (GTA5) and 38.5 (SYNTHIA) against 28.3 and 22.6 for source-only tab. 1.
• Relevant if you need synthetic-to-real segmentation with no target labels and cannot afford cycle-consistency training. Hyperparameters for optimization are largely not stated (see Pitfalls).

Problem

Models trained on labeled synthetic scenes lose accuracy on real scenes because of domain shift §1. Adversarial feature alignment is the usual remedy, but the authors argue that aligning global distributions can cause negative transfer, where target features are matched to the wrong class §1. Cycle-consistency GAN translation avoids some of that, but it needs an extra target-to-source generator and the authors say it can be too strong when target data are scarce §1. Standard geometric augmentation, which self-ensembling uses for classification, can misalign pixel labels in segmentation §1 §3.5. The paper aims to get a self-ensembling model working for segmentation by replacing the augmentation with GAN-generated images §1 §3.

Method

Overview. The method has two stages fig. 1 fig. 2. Stage one trains TGCF-DA. A target-guided generator G takes a source image x_s and a target image x_t and outputs an image with source content and target style §3.1. A frozen segmenter f_seg, pretrained on labeled source data, is used to keep the semantic content of the output §3.2. The generator and a discriminator D are trained jointly §3.3. Once trained, G produces augmented images that carry the labels of their source images §3.3. Stage two trains the segmentation network with self-ensembling. A student f_S and a teacher f_T share the architecture §3.4. Each mini-batch contains source, augmented and target samples §3.4.

Target-guided generator §3.1. The generator treats a source image as content and a target image as style. A source encoder produces a content representation, and a target encoder produces the style. The target encoder, built from multiple fully connected layers, predicts a per-channel scale and shift, gamma_t and beta_t, which are applied to the normalized source feature maps by adaptive instance normalization (AdaIN) eq. 1. The style is injected at intermediate residual blocks, while the spatial layout of the source image is kept §3.1.

\tilde{F}_s^i = \gamma_t^i \, \frac{F_s^i - \mu(F_s^i)}{\sigma(F_s^i)} + \beta_t^i eq. 1

Semantic constraint §3.2. The generated image must keep the pixel labels of the source image. The constraint is the pixel-wise cross-entropy between the pretrained segmenter's output on the generated image and the source label map, with f_seg fixed eq. 2. The authors argue this constraint also stabilizes adversarial training when there is no cycle-consistency §3.2.

L_{sem}(f_{seg}, G) = -\frac{1}{HW} \sum_{k=1}^{HW} \sum_{c=1}^{C} y_s^{(k,c)} \log f_{seg}\big(G(x_s, x_t)\big)^{(k,c)} eq. 2

TGCF-DA objective §3.3. The discriminator D is the multi-scale discriminator of §4.2. The adversarial objective is least-squares GAN (LSGAN), with spectral normalization applied for stability §3.3. The GAN term is printed as eq. 3. The total objective adds the semantic term with weight lambda_sem eq. 4. The paper sets lambda_sem to 10 in all experiments §4.2.

L_{GAN}(G, D) = E_{(x_s, x_t)}\big[D(G(x_s, x_t))^2\big] + E_{x_t}\big[(D(x_t) - 1)^2\big] eq. 3

L_{TGCF\text{-}DA} = L_{GAN} + \lambda_{sem} L_{sem} eq. 4

Self-ensembling §3.4. The teacher's weights are an exponential moving average (EMA) of the student's weights, with decay alpha eq. 5. The student is trained with a supervised cross-entropy on source images and augmented images §3.4. A consistency loss is the mean squared error between the softmax of the student's and the teacher's prediction maps, computed on target images only eq. 6. The total loss is the supervised loss plus a weight delta_con times the consistency loss eq. 7. The consistency is taken from the output of the final classifier, before upsampling §4.2.

t_i = \alpha\, t_{i-1} + (1 - \alpha)\, s_i eq. 5

L_{con}(f_S, f_T) = E_{x_t}\big[\,\|\sigma(f_S(x_t)) - \sigma(f_T(x_t))\|^2\,\big] eq. 6

L_{total} = L_{sup} + \delta_{con}\, L_{con} eq. 7

Ramp-up of the consistency weight §5.3. The weight is delta = 1 + delta_0 * exp(-5 (1 - x)^2), where x is the ratio of current epochs to total epochs, and delta_0 is a ramp-up coefficient. The "+1" differs from the usual sigmoid ramp-up and is there so the consistency term still matters at the start of training §5.3. The authors found ramp-up necessary, contrary to the self-ensembling classification work they build on §3.4.

Augmentation of target samples §3.5. Ordinary geometric augmentation is not used on target images because it misaligns pixels. Instead, Gaussian noise is added to the target inputs that go to the student and the teacher, and dropout is used to perturb the weights §3.5.

The pseudocode below is assembled from §3.1 to §3.5 and §5.3. The paper has no algorithm box.

# Stage 1: TGCF-DA generator and discriminator  [§3.1] [§3.2] [§3.3]
# Input:  source (x_s, y_s), target x_t; f_seg frozen
# Output: generator used to build augmented images with source labels

def tgcf_da_loss(x_s, y_s, x_t):
    # Source content with target style  [§3.1] [eq. 1]
    fake = generator(x_s, x_t)                 # (B, 3, H, W)
    # Adversarial term as printed in eq. 3; the generator-side form and
    # the update split between G and D are not stated  [eq. 3]
    adv = gan_loss(discriminator, fake, x_t)
    # Semantic constraint: frozen segmenter must read y_s on fake  [eq. 2]
    sem = cross_entropy(f_seg(fake), y_s)
    # Total objective with lambda_sem = 10  [eq. 4] [§4.2]
    return adv + lambda_sem * sem

# Stage 2: self-ensembling step  [§3.4] [§3.5] [§5.3]
# Input:  source batch, augmented batch (source labels), target batch
# Teacher f_T is an EMA of student f_S  [eq. 5]

def self_ensemble_step(x_source, y_source, x_aug, y_aug, x_target,
                       epoch_ratio, it):
    # Supervised cross-entropy on source and augmented  [§3.4]
    sup_source = cross_entropy(student(x_source), y_source)
    sup_aug = cross_entropy(student(x_aug), y_aug)
    # Relative weighting of the two terms is not stated
    loss_sup = sup_source + sup_aug
    # Target views: Gaussian noise on input, dropout on weights  [§3.5]
    # Noise std and dropout rate are not stated
    student_out = student(x_target + gaussian_noise(x_target))
    teacher_out = teacher(x_target + gaussian_noise(x_target))
    # Consistency on target samples only, MSE of softmax maps  [eq. 6]
    loss_con = mse(softmax(student_out), softmax(teacher_out))
    # Ramp-up coefficient delta_0 = 30  [§5.3] [tab. 2]
    ramp_up_coef = 30
    delta = 1 + ramp_up_coef * exp(-5 * (1 - epoch_ratio) ** 2)
    # Total loss  [eq. 7]
    loss = loss_sup + delta * loss_con
    # Student update by gradient descent; optimizer not stated  [§4.2]
    update(student, loss)
    # Teacher EMA; alpha = 0.99 for first 37K iterations, then 0.999
    # [§5.3] [eq. 5]
    alpha = 0.99 if it < 37000 else 0.999
    teacher = alpha * teacher + (1 - alpha) * student
    return loss

Hyperparameters

Results

Author-reported. Setup: synthetic-to-real, Cityscapes validation set as evaluation, VGG-16 segmenter, mIoU §4.1 §4.3 tab. 1. All numbers are mIoU in percent. The text says "14.2%" and "13.1%" gains; these are percentage points.

Ablation rows from the same table tab. 1:

Sensitivity of the two self-ensembling hyperparameters, GTA5 and SYNTHIA mIoU, as printed tab. 2. The metric is not named in the caption; the delta_0 = 30 and alpha = 0.999 cells match the main table's "Ours" rows, so they are presumably mIoU:

The paper reports that the segmentation with self-ensembling alone does not hold up across training, while the teacher and student of the full method both improve §4.4 fig. 4. Per-class results show the largest gains on majority classes and smaller gains on minority classes §5.2 fig. 6. The paper gives no runtime or memory measurements for TGCF-DA, although it calls the method computationally efficient §3.3 §5.1.

Paper vs code

No official code.

Searched: the paper text and the arXiv abstract page (no code link, no project page); GitHub repository search for "TGCF-DA", "tgcfda", "cycle-free gan augmentation", "self-ensembling gan domain adaptation", and "domain adaptation semantic segmentation gan augmentation"; the KAIST-VICLab organisation; and the GitHub profiles of author Taekyung Kim. No repository from the authors matched.

Pitfalls

• No code to check against. Every value in the Hyperparameters table comes from the paper text, so the optimizer, learning rates, batch sizes and iteration counts are not stated. The paper defers architecture details to a supplementary that is not in our source archive §4.2.
• Headline SYNTHIA gain does not match the table. The text says the method beats the baseline by 13.1 points on SYNTHIA §4.3. Table 1(b) gives 38.5 - 22.6 = 15.9 mIoU, or 46.6 - 24.1 = 22.5 mIoU*. The GTA5 gain of 14.2 matches (42.5 - 28.3). Use the table values.
• Which network is reported is not stated. The paper does not say whether Table 1 numbers come from the student or the teacher §4.3.
• Adversarial term of the generator. Eq. 3 is written in the discriminator's form (squared D(G) plus a squared (D(x_t) - 1) term). The generator's adversarial loss and the alternating update schedule are not written out eq. 3 §3.3. The code is the only place to check this, and none is available.
• Ramp-up "epochs" versus iterations. The ramp-up uses epoch ratio x §5.3, but the EMA switch is given in iterations §5.3. Total epochs and iterations are not stated, so the ramp-up cannot be pinned to steps.
• EMA choice is not clear-cut in Table 2. On SYNTHIA alpha = 0.99 (38.5) beats 0.999 (37.8), while on GTA5 0.999 is best. The paper uses the 0.99-then-0.999 schedule for all runs §5.3 tab. 2.
• Possible selection on the reported split. The text names the Cityscapes validation set as the evaluation set §4.1. The sensitivity tables do not name a split tab. 2, so the hyperparameters may have been chosen on the same set that is reported.
• Unclear noise and dropout. The text says Gaussian noise is fed to the student and teacher "respectively" §3.5. It is unclear whether each network gets its own noise draw. Noise std, dropout rate and dropout placement are not stated.
• Consistency loss normalization. Eq. 6 writes a squared norm, but whether it is averaged over pixels, classes or both is not stated.
• Relative weight of source versus augmented supervision. Both enter L_sup §3.4, with no weighting given.
• Augmentation size. The number of augmented images generated per source image is not stated §3.3.
• f_seg is not confirmed. The text says "such as FCN-8s" §3.2 while §4.2 describes the segmenter as VGG-16 with ASPP. Whether the pretrained f_seg and the trained segmenter share an architecture is not stated.
• Ramp-up is not a standard sigmoid. The "+1" and the Gaussian-shaped exponent §5.3 change the early-training contribution of the consistency term. The authors motivate the "+1", but the choice is not ablated against the sigmoid form.
• Cost is not measured. The "computationally efficient" claim §3.3 has no runtime or memory numbers in the text.
• Figure and table numbers were counted in LaTeX source order (Figures 1 to 8, Tables 1 and 2). Floats in the compiled PDF may be placed differently, so check numbering against the PDF.
• Ambiguous "%". Gains quoted with "%" are absolute mIoU points, not relative changes.

Lineage

• Builds on: self-ensembling for domain adaptation in classification §2 §3.4, which is the teacher-student EMA setup with the consistency loss; EMA weight update from Tarvainen and Valpola §2 §5.3; AdaIN §3.1; the MUNIT-style generator §4.2; LSGAN and spectral normalization §3.3; multi-scale discriminator from the cited high-resolution work §3.3; ASPP decoder from DeepLab §4.2. Changed here: no cycle-consistency §1 §3.3; target-style GAN augmentation replaces geometric augmentation §3.5; weight ramp-up is used §3.4.
• Compared against: Curriculum DA, CyCADA, MCD, LSD-seg, AdaptSegNet, ROAD, Conservative Loss, DCAN and CBST §4.3 tab. 1.
• Superseded by: not known from wiki content.