Conditional batch normalization

Overview

Conditional batch normalization (cBN) keeps ordinary batch normalization and adds a term that depends on a conditioning input. A generator layer can then be steered by text without a separate branch for each caption.

Only one card on this wiki covers the concept: SD-GAN, which calls its version semantic-conditioned batch norm (SCBN) 1904.01480 §3.2. This page therefore describes that one paper. The shared structure is two steps. First, each channel is normalized over the batch and spatial positions 1904.01480 eq. 3. Second, the scale and shift are a learned value plus a conditional value that comes from the input 1904.01480 eq. 4.

Variants

Sentence-level cue

The conditional scale and shift come from the sentence vector through two one-hidden-layer MLPs, one for each 1904.01480 eq. 5. The result is one value per channel, applied across all spatial positions 1904.01480 §3.2. Author-reported, without the Siamese branch: CUB Inception score 4.39 ± .06 and MS-COCO 28.81 ± .53 1904.01480 tab. 6. Card: Semantics Disentangling for Text-to-Image Generation.

Word-level cue

The conditional terms come from attention between each sub-region of the feature map and the caption's word features 1904.01480 eq. 6 1904.01480 §3.2. Two 1×1 convolutions turn the attended vectors into scale and shift maps with the feature shape 1904.01480 §3.2. Author-reported, without the Siamese branch: CUB 4.45 ± .06 and MS-COCO 29.79 ± .61 1904.01480 tab. 6. Card: Semantics Disentangling for Text-to-Image Generation.

When to use which

The paper tests these choices in one setting only (text-to-image GANs on CUB and MS-COCO), so the table ranks them on that setting's numbers.

Implementation sketch

The shared core normalizes, then modulates. Each cue function returns a conditional scale and shift. The sentence and word cues differ only there.

# Conditional batch norm, shared core  [eq. 3] [eq. 4]
# Input:  x (B, C, H, W); cond: sentence (B, D) or words (B, T, D)
# Output: (B, C, H, W)

def conditional_bn(x, cond, cue_fn):
    # Normalize each channel over batch and space  [eq. 3]
    x_hat = (x - mean(x)) / std(x)
    # Learned scale and shift, plus the conditional terms  [eq. 4]
    gamma_c, beta_c = cue_fn(x, cond)
    return (gamma + gamma_c) * x_hat + (beta + beta_c)

# Sentence cue  [eq. 5] [§3.2]
# Input:  x (B, C, H, W); sentence (B, D)
# Output: gamma_c, beta_c (B, C, 1, 1), broadcast over H and W

def sentence_cue(x, sentence):
    # One-hidden-layer MLPs give one value per channel  [eq. 5]
    gamma_c = mlp_gamma(sentence)[:, :, None, None]
    beta_c = mlp_beta(sentence)[:, :, None, None]
    return gamma_c, beta_c

# Word cue  [eq. 6] [§3.2]
# Input:  x (B, C, H, W); words (B, T, D_w)
# Output: gamma_c, beta_c (B, C, H, W)

def word_cue(x, words):
    # Project sub-regions and words to a shared size D  [§3.2]
    regions = embed_regions(x)                  # (B, L, D), L = H * W
    keys = project_words(words)                 # (B, T, D)
    # Softmax over words, per sub-region; axis implied, not written  [eq. 6]
    weights = softmax(regions @ keys.transpose(0, 2, 1), axis=-1)
    attended = weights @ keys                   # (B, L, D)
    # Two 1x1 convolutions give the modulation maps  [§3.2]
    feat = reshape_to_map(attended)             # (B, D, H, W)
    return conv_gamma(feat), conv_beta(feat)

Contradictions

Three different numbers appear for "no Siamese branch, with SCBN". The paper does not say which cue each table uses:

• Table 3, AttnGAN with SCBN: 4.48 ± .08 on CUB and 29.42 ± .45 on MS-COCO 1904.01480 tab. 3.
• Table 4, row 3, SD-GAN without Siamese: 4.49 ± .06 on CUB and 29.79 ± .61 on MS-COCO 1904.01480 tab. 4.
• Table 6, word cue without Siamese: 4.45 ± .06 on CUB and 29.79 ± .61 on MS-COCO 1904.01480 tab. 6.

The gap is small on CUB (about 0.04) and larger on MS-COCO (about 0.37). The paper gives no reason for the differences, and it does not say which cue Table 4 uses.

Papers

• Semantics Disentangling for Text-to-Image Generation: SD-GAN, which adds semantic-conditioned batch norm to a stacked text-to-image GAN and a Siamese contrastive loss.