All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation

In plain words

Segmentation models learn to label every pixel of a photo, but labelling real street photos by hand is slow. A cheaper route is to train on computer-game renderings, where the labels come for free. The catch is that game images look different from real ones, and a model trained on them often stumbles on real photos. This is the "domain shift" problem.

The paper's idea is to split an image into two parts. One part describes the layout and shapes of the scene, which should look the same whether the picture came from a game or a camera. The other part describes the surface look of the picture: colours, textures and lighting. The segmentation model only uses the layout part, so the look of the game does not confuse it. The method also learns to swap the look of a game picture with the look of a real one while keeping its layout. A game picture given a real-world look keeps the game's layout, so the game's labels can be reused to train on real-looking images without any new hand labelling.

TL;DR

• Unsupervised domain adaptation for semantic segmentation: labels on a source domain (GTA5), none on the target domain (Cityscapes) §1 §4.1.
• DISE encodes each image into a domain-invariant structure code and a domain-specific texture code, reconstructs images, and swaps textures across domains §3.1.
• Segmentation uses only the structure code. Swapped images inherit source labels, which give extra supervision §3.2 fig. 2.
• Author-reported: 45.4 mIoU on GTA5 to Cityscapes with a DeepLab-v2 ResNet-101 base, against 42.4 for the output-space baseline tab. 2 tab. 4.
• Official PyTorch code exists, but it targets PyTorch 0.3.1 and has several differences from the paper (see Paper vs code) README.md.

Problem

Supervised segmentation needs pixel-level labels, which are costly to collect for real street scenes §1. Synthetic datasets such as GTA5 and SYNTHIA come with labels, but models trained on them often fail on real images because of domain shift in low-level appearance §1. Earlier adaptation methods align the whole feature or output distribution between domains. The paper argues that this assumes the two domains can be aligned as a whole, which is often impractical when textures differ strongly §1 §2.

The method assumes that the high-level structure of an image is the most informative part for segmentation and can be shared across domains, while texture is domain-specific §1 §3.1. The setting requires annotated source images and unannotated target images §3.1.

Method

DISE has five sub-networks §3.1:

• common encoder E_c, shared by both domains, producing the structure code z_c (the layer4 output of the ResNet-101 base) §3.3;
• two private encoders E_p^s and E_p^t, one per domain, producing texture codes z_p (8-dimensional) §3.3;
• a shared decoder D that rebuilds an image from (z_c, z_p, domain flag) §3.3;
• a pixel-wise classifier T on z_c, the DeepLab-v2 classifier §3.3;
• discriminators for the output space, the translated images and the domain classifiers used by the adversarial terms §3.2.

Given a source image x^s, E_c gives z_c^s and E_p^s gives z_p^s. Their combination reconstructs x^s. The same holds for target images. Then the two domains' texture codes are swapped: the decoder combines the structure of one image with the texture of the other, yielding a translated image §3.1 fig. 2. Translated source-to-target images carry the source labels as pseudo-labels, because the hypothesis is that segmentation depends only on structure §3.1 §3.2.

Training minimises a weighted sum of the losses below §3.2 eq. 1. The paper says the weights were chosen empirically but does not state their values; the code's values are in the Hyperparameters table.

• Segmentation loss on source labels, cross-entropy through E_c and T §3.2.
• Output-space adversarial loss: a patch discriminator on the softmax predictions tries to tell source from target, and E_c and T are updated to fool it §3.2 eq. 2.
• Reconstruction loss: a perceptual distance between each input and its self-reconstruction §3.2 eq. 3 eq. 4.
• Translation structure loss: perceptual distance between a translated image and the image that supplied its structure, using higher VGG layers §3.2 eq. 5.
• Translation texture loss: distance between channel-wise activation means of a translated image and the image that supplied its texture, using early VGG layers §3.2 eq. 6 eq. 7.
• Translation adversarial loss, LSGAN with patch discriminators, on translated images against the other domain §3.2.
• Label transfer loss: cross-entropy on translated source-to-target images with the source labels, updating E_c and T §3.2.

The target labels are never used in the loss §3.2. The perceptual distance uses a pre-trained VGG network, over the layers relu1_1, relu2_1, relu3_1, relu4_1 and relu5_1 §3.2 eq. 3. Perceptual and texture losses use an L1 distance §3.2 eq. 3 eq. 6.

Key equations, as in the paper:

\mathcal{L} = \lambda_{seg}^{s}\mathcal{L}_{seg}^{s} + \lambda_{seg\_adv}\mathcal{L}_{seg\_adv} + \lambda_{rec}\mathcal{L}_{rec} + \lambda_{trans\_str}\mathcal{L}_{trans\_str} + \lambda_{trans\_tex}\mathcal{L}_{trans\_tex} + \lambda_{trans\_adv}\mathcal{L}_{trans\_adv} + \lambda_{seg}^{s2t}\mathcal{L}_{seg}^{s2t} eq. 1

\mathcal{L}_{seg\_adv}(\theta_c, \theta_t) = -\frac{1}{H'W'} \sum_{h',w'} \log\big(D_{adv}^{seg}(\hat{y}^t)_{h',w'}\big), \quad H' = H/16,\; W' = W/16 eq. 2

\mathcal{L}_{perc}(x, y; w) = \sum_{l \in L} \frac{w^{(l)}}{N^{(l)}} \left\| \psi^{(l)}(x) - \psi^{(l)}(y) \right\|_1, \quad L = \{\text{relu1\_1}, \dots, \text{relu5\_1}\} eq. 3

\mathcal{L}_{tex}(x, y; w) = \sum_{l \in L} \frac{w^{(l)}}{C^{(l)}} \sum_c \left\| \mu_c(\psi^{(l)}(x)) - \mu_c(\psi^{(l)}(y)) \right\|_1 eq. 6

Equations 4, 5 and 7 are the paper's instantiations of the perceptual and texture distances on the reconstructed, structure-swapped and texture-swapped pairs eq. 4 eq. 5 eq. 7.

Algorithm (training step). The paper gives no algorithm box; this is the loss structure of §3.1 and §3.2, with weights left as named variables.

# DISE training loss  [§3.1] [§3.2] [eq. 1]
# Input:  source batch (x_s, y_s) with labels; target batch x_t, no labels
# Output: scalar loss; each sub-network is updated by its own optimiser

def dise_loss(x_s, y_s, x_t):
    # Structure codes from the shared encoder  [§3.1]
    z_c_s = common_encoder(x_s)              # (B, 2048, h, w)
    z_c_t = common_encoder(x_t)              # (B, 2048, h, w)

    # Texture codes from the private encoders  [§3.1]
    z_p_s = private_encoder_s(x_s)           # (B, 8)
    z_p_t = private_encoder_t(x_t)           # (B, 8)

    # Self-reconstructions, shared decoder, domain flag  [§3.1] [eq. 4]
    rec_s = decoder(z_c_s, z_p_s, domain=0)
    rec_t = decoder(z_c_t, z_p_t, domain=1)

    # Texture swap: structure of one image, texture of the other  [§3.1]
    trans_s2t = decoder(z_c_s, z_p_t, domain=1)
    trans_t2s = decoder(z_c_t, z_p_s, domain=0)

    # Segmentation from structure codes; source labels are supervised  [§3.2]
    pred_s = classifier(z_c_s)
    pred_t = classifier(z_c_t)
    loss_seg = cross_entropy(pred_s, y_s)

    # Output-space adversarial loss on target predictions  [§3.2] [eq. 2]
    loss_seg_adv = -mean(log(seg_disc(softmax(pred_t))))

    # Reconstruction and translation perceptual losses  [§3.2] [eq. 3-5]
    loss_rec = perc(rec_s, x_s, w_rec) + perc(rec_t, x_t, w_rec)
    loss_trans_str = perc(trans_s2t, x_s, w_str) \
        + perc(trans_t2s, x_t, w_str)

    # Texture loss on channel-wise activation means  [§3.2] [eq. 6] [eq. 7]
    loss_trans_tex = tex(trans_s2t, x_t, w_tex) \
        + tex(trans_t2s, x_s, w_tex)

    # LSGAN adversarial loss on translated images  [§3.2]
    loss_trans_adv = lsgan_gen(trans_s2t) + lsgan_gen(trans_t2s)

    # Label transfer: source labels on translated source-to-target images
    # [§3.2]
    pred_trans = classifier(common_encoder(trans_s2t))
    loss_label = cross_entropy(pred_trans, y_s)

    # Weighted sum; the paper does not state the weights  [§3.2] [eq. 1]
    return (
        w_seg * loss_seg
        + w_seg_adv * loss_seg_adv
        + w_rec * loss_rec
        + w_trans_str * loss_trans_str
        + w_trans_tex * loss_trans_tex
        + w_trans_adv * loss_trans_adv
        + w_label * loss_label
    )

Hyperparameters

Values from the paper unless marked "code". Where the paper and code differ, both are listed in Paper vs code.

Results

Author-reported. Setup: adaptation from GTA5 (source, labels) to Cityscapes (target, unlabelled train split), evaluated by per-class IoU on the Cityscapes validation split §4.1 §4.2. Base model is DeepLab-v2 ResNet-101 for DISE; baselines use the base shown tab. 2.

Headline: DISE reaches 45.4 mIoU, the highest in the GTA5 table. The best baseline in the table is Hong et al. at 44.5 with an FCN8s base tab. 2. The text reports these as state of the art §4.2.

Table 2 (GTA5 to Cityscapes, per-class IoU, 19 classes) tab. 2 §4.2. Bold marks in the source are not reproduced.

Table 3 (SYNTHIA to Cityscapes, per-class IoU, 16 classes) tab. 3 §4.2. Saleh et al. is not in this table.

Table 4 (ablation on GTA5 to Cityscapes, mIoU) tab. 4 §4.3. A = segmentation loss on source; B = output-space adversarial loss; C = reconstruction, translation structure, translation texture and translation adversarial losses; D = label transfer loss.

Qualitative results are in fig. 3 (Cityscapes predictions) and fig. 4 (translated samples in both directions) §4.2 §4.4.

Paper vs code

Code: https://github.com/a514514772/DISE-Domain-Invariant-Structure-Extraction at commit 97c3940d54c451f92e4d40a3d642caf9c6ceca58, linked from the paper's footnote in §4 §4 (the paper names the repository, so it is the authors' own link). The repository is GPL-3.0. We read it for facts and cite paths only; no code is quoted here.

• Source subset: the paper's text describes 24,996 GTA5 images and does not state a subset §4.1. The code trains on train_modified.txt, 18,111 entries train_modified.txt. The paper's LaTeX source has a commented-out note about about 18,000 images after manual removal, which is not part of the published text. Treat the subset as the code's choice.
• Training resolution: the paper resizes inputs to 512x1024 before cropping 256x512 §3.3. The code resizes GTA5 to 720x1280 first [code: train_dise_gta2city.py#L78] train_dise_gta2city.py.
• Private encoder input: the paper says E_p encodes the image §3.1. The code feeds E_p the 64-channel stem output of the shared encoder, not RGB train_dise_gta2city.py model.py.
• Private encoder depth: the paper says four convolution blocks §3.3. The code has five stride-2 convolutions, then average pooling and a 1x1 convolution model.py.
• Decoder input: the paper concatenates z_p, z_c and the flag §3.3. The code maps z_p through an MLP to 256 channels and broadcasts them spatially model.py model.py.
• Output-space adversarial loss: the paper applies one discriminator at the output of T §3.2. The code uses two (aux head and final head) with weights 2e-4 and 1e-3 train_dise_gta2city.py [code: train_dise_gta2city.py#L334-L340].
• Auxiliary segmentation head: the paper has one classifier on z_c §3.2. The code also supervises an auxiliary head on layer3 features, weight 0.1, for the source and label-transfer losses train_dise_gta2city.py train_dise_gta2city.py seg_model.py.
• Learning rates: the paper gives 1e-3 for the decoder and 1e-4 for "the others" §3.3. The code uses 1e-3 for both private encoders and the decoder, and 1e-4 for domain classifiers and image discriminators [code: train_dise_gta2city.py#L87-L88] train_dise_gta2city.py.
• Translation losses: the paper weights structure and texture with separate λ's §3.2 eq. 1. The code has one weight, 0.05, for both [code: train_dise_gta2city.py#L391]. The texture term uses unweighted channel means of layers 1 to 3, and the structure term uses layers 4 and 5 only [code: util/loss.py#L38-L43].
• Label transfer gradient: the code runs the classifier on rec_s2t.detach() for the translated-image segmentation loss train_dise_gta2city.py. So that term does not update the decoder. The paper says the label loss trains E_c and T §3.2, which matches.
• Translation discriminators: the paper does not describe an update interval. The code updates them every fifth iteration train_dise_gta2city.py train_dise_gta2city.py.
• Checkpoint selection: the code evaluates on Cityscapes validation every 1,000 iterations and keeps the best mIoU train_dise_gta2city.py train_dise_gta2city.py. The paper does not say how the reported checkpoint was chosen §4.2.
• Validation resolution: during training, validation labels are resized to 512x1024 train_dise_gta2city.py CityLoader.py, so the selection mIoU is computed at that resolution. evaluate.py upsamples predictions to 1024x2048 as the paper states evaluate.py evaluate.py.
• Target labels: the target label list is loaded train_dise_gta2city.py and used only for visual output train_dise_gta2city.py train_dise_gta2city.py. No loss reads it. This matches the paper.
• Pretrained init: the paper says VOC pre-training §3.3. The code loads a checkpoint from a URL with the layer5 head dropped for 19 classes [code: model/seg_model.py#L7] seg_model.py. The checkpoint's training data is not stated in the code.
• Software: the README targets PyTorch 0.3.1 README.md. The training script uses APIs of that era, such as Variable(..., volatile=True) and loss.data[0] train_dise_gta2city.py [code: train_dise_gta2city.py#L514]. It will not run unchanged on current PyTorch.

Other results-text mismatches, found only in the paper itself:

• Table 2 gives Tsai et al. 42.4 mIoU tab. 2. Table 4 gives 42.6 for "Seg-map adaptation", which the text says "corresponds to the method in" Tsai et al. tab. 4 §4.3. Two numbers for the same method.
• The SYNTHIA text says DISE is superior in Building and Sky §4.2. In Table 3, DISE is 77.1 against 77.5 for three baselines in Building and 81.2 against 82.5 for Tsai et al. in Sky tab. 3.

Pitfalls

• Loss weights are not stated in the paper. Eq. 1 names them eq. 1, and the code's values are in the Hyperparameters table. Using the paper alone, an implementer must pick them, and the choice changes the result.
• The code's total loss adds an auxiliary segmentation head and a second adversarial discriminator. The paper describes neither [code: train_dise_gta2city.py#L386-L391]. An implementation that follows the paper alone is a different model.
• The structure code is 2048 channels. The code's dilated layers (dilation 2 in layer3, 4 in layer4) imply output stride 8 seg_model.py. The paper does not state the output stride §3.3.
• The private codes are 8-dimensional §3.3. Their input is the stem output of the shared encoder in the code train_dise_gta2city.py, so an implementation that encodes RGB images directly will not match the code.
• The perceptual and texture terms are defined with normalisers w/N and w/C eq. 3 eq. 6. The code's L1 means divide by element counts, which matches the normaliser, but the layer weights differ (see the Hyperparameters table) loss.py.
• The discriminator's patch grid is H/16 by W/16 eq. 2. With the 256x512 training crops, that is a 16x32 grid of patch decisions.
• Model selection on Cityscapes validation (see Paper vs code) can inflate the reported mIoU if the reported checkpoint was picked on that split. The paper does not say.
• The dataset and crop sizes differ between paper and code (see Paper vs code). Use the code's values when reproducing the reported 45.4, and report the difference.
• Compute: 88 GPU hours on one 16 GB V100 §3.3. The code's batch size is 2 with 250,000 iterations train_dise_gta2city.py [code: train_dise_gta2city.py#L83].
• The code loads VGG-19 from torchvision with pretrained=True [code: util/loss.py#L67]. The paper says "pre-trained VGG" without naming a checkpoint or dataset §3.2.
• The code is GPL-3.0 [LICENSE]. Using it as a reference is fine; copying code requires GPL compliance. Prefer writing your own implementation from this card.

Lineage

• Builds on: the output-space adversarial adaptation of Tsai et al. §3.2 tab. 2, a perceptual loss of Johnson et al. §3.2 eq. 3, the channel-mean style loss inspired by AdaIN (Huang et al.) §3.2, and domain separation networks (Bousmalis et al.) and DRIT (Lee et al.) §1. DISE differs from the separation work in making only the structure component invariant, and in translating across domains within one framework §1.
• Compared against: Sankaranarayanan et al., Wu et al., Hong et al., Chen et al., Saleh et al. tab. 2 tab. 3. The related-work table compares their use of distribution alignment, image translation and label transfer tab. 1 §2.
• Superseded by: not stated in this wiki.