Semantics Disentangling for Text-to-Image Generation

In plain words

Describe the same bird to three people and you get three different sentences. A text-to-image model that learns from such captions can read each sentence as a different scene, so the pictures it paints for the same bird can wander from one caption to the next. The image may still look sharp while showing a different bird.

This paper's model, SD-GAN, tackles that in two ways. First, it runs two branches of the picture-making network at once, each fed a different caption. The judge that scores the pictures is trained to make pictures from captions that describe the same thing look alike, and pictures from unrelated captions look different. Second, the picture-making network's internal dials are set by the words of the caption, not only by a single summary of the sentence, so small details such as wing colour can still reach the image.

The model is tested on bird pictures (CUB) and everyday scenes (MS-COCO). Its pictures score higher on a standard automatic image-quality score, and human raters pick its pictures as the best match to the text more often than they pick those of two earlier models.

TL;DR

SD-GAN is a stacked text-to-image GAN with two additions §3. A Siamese setup feeds two captions through the generator and applies a contrastive loss to the discriminator features of the two results §3.1 eq. 1 eq. 2. The generator's batch normalization layers are replaced by semantic-conditioned batch norm (SCBN), whose scale and shift come from the sentence vector and from word-level attention §3.2 eq. 4 eq. 5 eq. 6. Author-reported on CUB, Inception score 4.67 ± .09 vs 4.36 ± .03 for AttnGAN; on MS-COCO 35.69 ± .50 vs 25.89 ± .47 tab. 1. Relevant if you train text-conditioned GANs and care about consistency across paraphrases. No official code was found, and the training optimizer, learning rate and batch size are not stated.

Problem

Captions for the same image vary a lot in wording, and a generator that treats each caption as the sole truth produces unstable appearances that can be far from the reference image §1 fig. 1. Earlier work improved visual quality and resolution through stacked coarse-to-fine generators or attention over words, but kept conditioning on the raw text without separating shared meaning from detail §1 §2. The method assumes that each training image has several captions, and that the captions of one image can be told apart from those of another §4.1.

Method

Overview. The model has a text encoder and three generator-discriminator stages (G0, G1, G2 and D0, D1, D2). Each stage's discriminator is independent of the others and shares no parameters §3.1 fig. 2. The model runs two Siamese branches, one per caption, and combines their features with a contrastive loss §3.1. Generator layers use SCBN, which is applied at the end of each up-sampling layer fig. 3 §3.2.

1. Text encoder. A bi-directional LSTM gives one feature per word, w_t, and one sentence vector, s̄, taken from the last hidden states §3.1.
2. Initial stage. G0 maps a noise vector z, drawn from a standard normal distribution, and s̄ to a 64 × 64 image §3.1 fig. 3.
3. Later stages. G1 and G2 take the previous stage's output and s̄ and produce higher-resolution images §3.1. The final output is 256 × 256 in the supplementary grids app..
4. Siamese branches. Each branch receives one caption of the pair. The contrastive loss is computed on the discriminator feature vectors of the two branches, at each stage §3.1 §4.3.
5. Contrastive loss. Same-image pairs are pulled together until their distance reaches a floor α. Different-image pairs are pushed apart until their distance reaches a margin ε eq. 2 §3.1. The unmodified loss is eq. 1; the floor version replaces it because pairs of the same image should not collapse to the same picture §3.1.
6. SCBN. Batch norm is applied to the feature maps first eq. 3. The scale and shift are then learned parameters plus a conditional term eq. 4. The conditional term comes from the sentence vector through two one-hidden-layer MLPs eq. 5 (sentence cue), or from word-level attention followed by two 1 × 1 convolutions eq. 6 (word cue) §3.2.
7. Other losses. The generator and discriminator losses other than the contrastive term follow AttnGAN §4.1. The text encoder and the Inception feature extractor are pretrained by AttnGAN and kept fixed during training §4.1.

Equations:

Contrastive loss, original form eq. 1:

L_{c} = \frac{1}{2N} \sum_{n=1}^{N} y \cdot d^{2} + (1 - y) \max(\varepsilon-d, 0)^{2}

with d = ||v_1 − v_2||_2 between the two branches' visual feature vectors, y = 1 for the same image and 0 otherwise, N = 256, and ε = 1.0 §3.1.

Contrastive loss, modified form used in the model eq. 2:

L_{c} = \frac{1}{2N} \sum_{n=1}^{N} y \max(d, \alpha)^{2} + (1 - y) \max(\varepsilon-d, 0)^{2}

with α = 0.1 §3.1.

Batch norm eq. 3:

\texttt{BN}(x) = \gamma \cdot \frac{x - \mu(x)}{\sigma(x)} + \beta

Conditional batch norm eq. 4:

\texttt{BN}(x | c) = (\gamma + \gamma_{c}) \cdot \frac{x - \mu(x)}{\sigma(x)} + (\beta + \beta_{c})

Sentence-level cue eq. 5:

\gamma_{c} = f_{\gamma}(\bar{s}), \quad \beta_{c} = f_{\beta}(\bar{s})

Word-level cue eq. 6:

\texttt{vse}_{j} = \sum_{t} \sigma(v_{j}^\top f(w_t)) \, f(w_t)

where σ is the softmax in the experiments §3.2.

Pseudocode for one training step, combining the stages, the Siamese pair and the contrastive term. The adversarial losses are defined by AttnGAN and are not reproduced here §4.1.

# One SD-GAN training step on a pair of captions  [§3.1] [fig. 2]
# Input:  caption_a, caption_b: two captions per example  [§3.1]
#         same: 1 if both captions describe the same image, else 0  [§3.1]
# Output: total loss for the generators and discriminators

def training_step(caption_a, caption_b, same):
    # Encode each caption with the bi-directional LSTM  [§3.1]
    words_a, sentence_a = text_encoder(caption_a)
    words_b, sentence_b = text_encoder(caption_b)
    # Draw one noise vector per branch from N(0, I)  [§3.1]
    noise_a = sample_normal(batch_size, z_dim)
    noise_b = sample_normal(batch_size, z_dim)
    # Generate images at three stages in each branch  [§3.1] [fig. 3]
    images_a = generate_stages(noise_a, sentence_a, words_a)
    images_b = generate_stages(noise_b, sentence_b, words_b)
    # Adversarial losses as in AttnGAN; not defined in this paper  [§4.1]
    gan_loss = adversarial_losses(images_a, images_b)
    # Contrastive term at every stage on the discriminator features  [§4.3]
    contrast = 0
    for stage in (0, 1, 2):
        # Feature vectors from the two Siamese branches  [§3.1] [app.]
        feat_a = discriminator_feature(stage, images_a[stage])
        feat_b = discriminator_feature(stage, images_b[stage])
        distance = l2_norm(feat_a - feat_b)
        contrast = contrast + pair_loss(distance, same)
    # Weight between the two terms is not stated  [§4.1]
    return gan_loss + contrastive_weight * contrast

# Pair loss on one stage, modified form  [eq. 2]
# Input:  distance (B,): feature distance d per pair
#         same (B,): 1 for the same image, 0 otherwise
# Output: scalar

def pair_loss(distance, same):
    # Same-image pairs: pull together, but stop below the floor alpha  [eq. 2]
    pull = same * maximum(distance, alpha) ** 2
    # Different-image pairs: push apart until the margin epsilon  [eq. 2]
    push = (1 - same) * maximum(epsilon - distance, 0) ** 2
    # 1/(2N) with N = 256 read literally; see Pitfalls  [§3.1]
    return sum(pull + push) / (2 * feature_length)

# Semantic-conditioned batch norm with the sentence cue  [eq. 3-5]
# Input:  features (B, C, H, W); sentence (B, D_s)
# Output: modulated features (B, C, H, W)

def scbn_sentence(features, sentence):
    # Normalize each channel over the batch and space  [eq. 3]
    normalized = normalize_per_channel(features)
    # One-hidden-layer MLPs give per-channel modulation  [eq. 5]
    gamma_c = mlp_gamma(sentence)            # (B, C), broadcast over H, W
    beta_c = mlp_beta(sentence)              # (B, C), broadcast over H, W
    # Scale and shift with learned plus conditional terms  [eq. 4]
    return (gamma + gamma_c) * normalized + (beta + beta_c)

# Semantic-conditioned batch norm with the word cue  [eq. 6] [§3.2]
# Input:  features (B, C, H, W); words (B, T, D_w)
# Output: modulated features (B, C, H, W)

def scbn_word(features, words):
    # Normalize each channel over the batch and space  [eq. 3]
    normalized = normalize_per_channel(features)
    # Project each sub-region and each word to a shared size  [§3.2]
    regions = embed_regions(features)        # (B, L, D), L = H * W
    word_keys = perception_layer(words)      # (B, T, D)
    # Attention of each word per sub-region; softmax axis not stated  [eq. 6]
    weights = softmax(regions @ word_keys.transpose(0, 2, 1))
    # Weighted sum of word vectors per sub-region  [eq. 6]
    vse = weights @ word_keys                # (B, L, D)
    # Two 1x1 convolutions give the modulation maps  [§3.2]
    gamma_c = conv_gamma(reshape_to_map(vse))   # (B, C, H, W)
    beta_c = conv_beta(reshape_to_map(vse))     # (B, C, H, W)
    return (gamma + gamma_c) * normalized + (beta + beta_c)

Hyperparameters

Values are as stated in the paper unless a row says otherwise.

Results

Author-reported. Setup: CUB (200 bird species) and MS-COCO, Inception score §4.1. Human study: ranking of three methods by 50 raters §4.1 tab. 2. Baseline numbers are as printed in the paper. The paper does not say whether they were re-run or copied from the original papers tab. 1.

Table 1, Inception score, mean ± std tab. 1. Underline and bold are the paper's own marks; "-" means no value reported:

Table 2, human ranking, share of rankings in first place tab. 2. The three rows sum to 100% in each column:

Table 3, adding the two components to AttnGAN tab. 3:

Table 4, component ablation of SD-GAN tab. 4:

Table 5, contrastive loss ablation tab. 5. Rows 1–4 vary α; rows 5–7 remove the loss at some stages:

Table 6, SCBN versus plain BN, both without Siamese tab. 6:

Qualitative results: SD-GAN versus StackGAN and AttnGAN on CUB and MS-COCO fig. 5. Component examples fig. 6. Editing one word or phrase changes the bird colour or scene in the expected way fig. 7. Extra grids of 400 samples per dataset are in the supplementary material app. fig. 10 fig. 11.

Paper vs code

No official code.

Searched, in order:

1. Abstract page and arXiv comments: no code link. The comment reads only "14 pages, 11 figures, accepted as an ORAL at CVPR2019".
2. Paper body and LaTeX source: no repository or project-page link.
3. GitHub repository search, WebFetch against the API: "SD-GAN" with "text image generation" and "semantics disentangling text-to-image" gave no matches; "SDGAN" gave 34 results, and the first 20 were checked. Two candidates are not used: tlwzzy/SDGAN ("Code for SDGAN at CVPR2019"), whose owner is not an author and whose README says the code will be released later and states no license; and ianong1/SDGAN ("for paper"), whose owner is not an author and which links no paper.
4. An organisation-scoped query for SenseTime-Research returned an HTTP 422 error, so that search was not completed.

Because no official repository was found, nothing in this card is checked against code. Every hyperparameter above comes from the paper alone.

Pitfalls

• Sum bound and normalization of the contrastive loss eq. 1 eq. 2. The paper sets N = 256 as the feature length, but the sum in the loss runs over n = 1 to N, and the factor 1/(2N) is written with the same N. Whether the sum runs over feature dimensions or over pairs in the batch is not stated. A standard reading sums over pairs; the normalization then needs a choice. Decide and document it.
• Appendix and main text disagree. The main text uses one distance term eq. 2. The appendix writes a second contrastive term on a conditional branch, with distance d_c between features from the sentence-conditioned branch, added to the first without a stated weight app.. The main text says its contrastive loss is "Eq.2 in the main paper", the version with one term. Whether the second term is used in the reported results is not stated.
• Where the contrastive features come from. The appendix says the visual features are taken from the first output branch of each discriminator and that the contrastive loss uses a feature vector app.. The layer, pooling and whether the vector is normalized are not stated. The margin ε = 1.0 depends on the feature scale, so these details change the effect of ε.
• Floor α. With max(d, α)², same-image pairs closer than 0.1 receive no gradient. This is the intended effect §3.1, but it means the pull stops early on unnormalized features.
• Pair construction. The paper says two captions go to the two branches §3.1 but does not say how the pairs are sampled in a batch, whether both branches share generator weights, or how batch norm statistics are computed when the two branches run in the same batch.
• Total loss weight. The paper says the contrastive losses are added to the AttnGAN losses §4.1 but gives no weight. Choosing it is the largest unstated setting for reproduction.
• Symbol clashes in the word-level cue eq. 6. The sum index runs from t = 0 to T − 1, but the word set is defined from t = 1 to T §3.2. The symbol σ is also used for the standard deviation in eq. 3 and for the softmax in eq. 6. The softmax axis (over words, per sub-region) is implied but not written.
• Sentence cue shape. The scale and shift from the sentence vector are per-channel values eq. 5. The text says they are extended to the size of x §3.2, which means broadcasting over height and width. The sentence vector is the last hidden states of a bi-directional LSTM; how the two directions are combined is not stated.
• Initialization of the conditional terms. If γ_c and β_c start at zero, SCBN starts as plain BN. The initialization is not stated.
• Inconsistent SCBN numbers. Table 4 row 3 (no Siamese, with SCBN) reports 4.49 on CUB. Table 6 "SCBN, word cue" is the same setting and reports 4.45, with the same MS-COCO value 29.79. The cue level used in Table 4 is not stated.
• Cost of the Siamese setup. The paper runs ablations on the variant without the Siamese structure "due to the less computational cost" §4.3. The Siamese setup processes two captions per step, so its cost is higher; the paper does not give timings or memory use.
• Evaluation details. The Inception score is computed with the Inception model from StackGAN §4.1; the number of generated samples is not stated, and scores from different Inception models are not comparable. The paper reports only the Inception score and a human ranking, with no FID or caption-consistency metric. The COCO evaluation set is called validation in §4.1 and test in §4.2.
• Baselines. Whether baseline rows were re-run or copied is not stated tab. 1.
• Human study. Raters are shown the same captions and ranked three images §4.1. The per-row shares sum to 100% tab. 2, so the ranking is forced. The study does not report agreement between raters.
• No official code means no reference for the training loop. The optimizer, learning rate, batch size, number of epochs and the AttnGAN loss definitions must come from the AttnGAN paper, which is not summarized in this wiki.

Lineage

• Builds on: AttnGAN (Xu et al., 2017) for the adversarial losses, the pretrained text encoder and Inception features §4.1; StackGAN and StackGAN++ (Zhang et al.) for the stacked stages §3.1; the Siamese contrastive loss of Hadsell et al., 2006 §3.1; and conditional batch norm of Dumoulin et al., 2017, which SCBN extends §3.2. None of these has a card in this wiki, so none is linked.
• Compared against: GAN-INT-CLS, GAWWN, StackGAN, StackGAN++, PPGN, AttnGAN, HDGAN, Cascaded and Recurrent C4Synth, LayoutSynthesis and SceneGraph tab. 1; StackGAN and AttnGAN in the human study tab. 2.
• Superseded by: not known from content already in this wiki.