Adaptive instance normalization

Overview

Adaptive instance normalization (AdaIN) is a feature step. For each channel it removes the channel's own mean and standard deviation, then applies the mean and standard deviation of a reference. The step has no learned parameters of its own 2103.16877 app. B.

Three cards on this wiki use AdaIN or the same scale-and-shift form, in three different ways:

• Feature transfer. ArtFlow takes the statistics from a style image's features and moves the content features to them 2103.16877 §4.1 2103.16877 app. B.
• Style injection. StarGAN v2 takes the scale and shift of each normalisation from a style code. The code comes from a mapping network or a style encoder, not from a style image's statistics 1912.01865 §2.1 model.py.
• Conditional scale and shift. Drive-OccWorld layer-normalises each memory embedding, then scales and shifts it with values derived from a condition 2408.14197 §3.2 2408.14197 eq. 5. It uses layer normalisation over the embedding, not per-channel reference statistics, so it is not AdaIN in the strict sense. It is included because the scale-and-shift form is the same.

The paper that introduced AdaIN has no card here. This page describes AdaIN as the cards use it and assumes nothing else about the original method.

In the feature-transfer use, the output keeps the content's spatial layout and takes the style's channel statistics. ArtFlow's proof of "unbiasedness" uses a bilinear model of content and style factors, and the authors show AdaIN satisfies it 2103.16877 §4.3 2103.16877 app. B. For a decoder-based pipeline, the authors attribute AdaIN's content leak to reconstruction error and to decoder training, not to the AdaIN arithmetic 2103.16877 §3.2.

Variants

AdaIN between a learned encoder and decoder (the baseline)

In the universal methods that ArtFlow compares against, AdaIN sits between a fixed VGG19 encoder and a trained decoder that maps the transferred features back to pixels 2103.16877 §3.1. The decoder is trained with a content term and a style term, the latter matching channel means and standard deviations of VGG features 2103.16877 eq. 2 2103.16877 §3.2.

The authors attribute content leak in this setup to reconstruction error and to the style term biasing the decoder. They show it by feeding each output back in as the next content image for 20 rounds 2103.16877 §3.1 2103.16877 §3.2. Author-reported at 512×512: SSIM 0.29, content loss 3.10 and 0.064 s per image 2103.16877 tab. 1.

AdaIN inside a reversible flow (ArtFlow)

ArtFlow keeps the same AdaIN step but changes the path around it. Content and style images go through the forward pass of an invertible network, AdaIN runs on the resulting latent features, and the reverse pass returns an image 2103.16877 §4.1. The forward pass is lossless by design, so the network has no reconstruction error to accumulate 2103.16877 §4.1 2103.16877 §4.2.

The invertible network is a Glow-style stack with two blocks of eight flows. Each flow is an activation normalization, a learned channel-mixing 1×1 convolution and an additive coupling step 2103.16877 §4.2 2103.16877 §5.1. The authors attribute its higher cost to the network having no pooling layers 2103.16877 §5.2.

Author-reported at 512×512: SSIM 0.45, content loss 2.58 and 0.408 s per image 2103.16877 tab. 1. With 20 rounds of repeated transfer, the authors report no visible content leak 2103.16877 §5.3. This is a qualitative result; the paper gives no metric for it.

Card: ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows.

AdaIN as style injection in a multi-domain generator (StarGAN v2)

StarGAN v2 uses one generator for all domains. Its decoder blocks are normalised with AdaIN, and the style code sets the scale and shift of each block 1912.01865 §2.1 1912.01865 app. E. Down-sampling blocks use instance normalization instead 1912.01865 §2.1. The official code computes the scale as one plus a learned gamma, and adds a learned beta model.py. The paper says the scale biases are set to one 1912.01865 app. B; the two forms agree in effect at initialisation utils.py.

The style code comes from a mapping network that takes a random latent and a domain, or from a style encoder that takes a reference image and its domain 1912.01865 §2.1. Each domain has its own output head, so each domain has its own style space 1912.01865 app. E. The style code then drives the generator through a diversity term, a style-reconstruction term and a cycle-consistency term 1912.01865 §2.2 1912.01865 eq. 5.

Author-reported latent-guided FID is 13.7 on CelebA-HQ and 16.2 on AFHQ 1912.01865 tab. 2. Reference-guided FID is 23.8 and 19.8 1912.01865 tab. 3. These are image-quality and diversity metrics for multi-domain translation, not the SSIM and content-loss measures used for the ArtFlow comparison, so the two sets of numbers are not comparable.

Card: StarGAN v2: Diverse Image Synthesis for Multiple Domains.

Conditional layer normalisation in a driving world model (Drive-OccWorld)

Drive-OccWorld keeps a memory queue of past BEV embeddings. Each entry is layer-normalised with no learned affine parameters, then scaled and shifted by values derived from a condition 2408.14197 §3.2 2408.14197 eq. 5. The paper describes three conditions: semantic labels, ego motion and agent motion. Each one is encoded into its own pair of scale and shift values 2408.14197 §3.2. For ego motion, the 3×4 ego transform between two timestamps is flattened and passed through an MLP 2408.14197 §3.2.

The paper's ablation is run without action conditions and reports 1 s future mIoU. Author-reported: 26.4 with no condition, 26.6 with semantic labels only, 27.1 with agent motion only, and 28.3 with ego motion, alone or together with the other two 2408.14197 tab. 6. The ablation covers all three conditions, so it does not describe the released configuration (see Contradictions).

Card: Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving.

When to use which

The feature-transfer rows, the StarGAN row and the Drive-OccWorld row answer different questions. Choose by the constraints above, not by a single ranking. The ArtFlow user study gives it 314 votes as one method, without splitting the AdaIN and WCT variants, and it has no stated exclusion rule for the 799 effective votes 2103.16877 §6 2103.16877 tab. 2. StarGAN v2 needs labelled domains and takes about 3 days on one V100 for 100K iterations at batch size 8 1912.01865 app. B.

Implementation sketch

The transfer step itself is short. Everything else depends on the variant above. The small epsilon in the variance is not in the paper; it is a numerical guard that the released code adds.

# AdaIN on one content and one style feature map, shape (C, H, W)  [app. B]
# Statistics are per channel, over the spatial positions
def adain(content, style):
    # Per-channel mean and standard deviation of each input  [app. B]
    mu_c = mean(content, over=(H, W))
    sigma_c = std(content, over=(H, W))
    mu_s = mean(style, over=(H, W))
    sigma_s = std(style, over=(H, W))
    # Remove the content statistics, then apply the style statistics  [app. B]
    # The epsilon guards against zero variance (code only, not in the paper)
    return sigma_s * (content - mu_c) / (sigma_c + 1e-5) + mu_s

In ArtFlow the same function runs on the latent features, between the forward and reverse passes of the flow 2103.16877 §4.1. The training objective that matches style statistics uses the same mean and standard deviation terms 2103.16877 eq. 2.

In StarGAN v2 the statistics are not taken from a style image. The normalised input is scaled and shifted by parameters that a learned layer produces from the style code 1912.01865 §2.1 model.py. The code form is one plus a learned gamma, times the normalised input, plus a learned beta.

In Drive-OccWorld the normalisation is layer normalisation without affine parameters. The scale and shift come from the condition, and the result replaces the memory embedding 2408.14197 §3.2 2408.14197 eq. 5.

Contradictions

No disagreement between the cards. Their numbers measure different things (SSIM and content loss for ArtFlow, FID for StarGAN v2, mIoU for Drive-OccWorld), so they cannot be ranked against each other.

Within each card, the paper and the released code differ in the places below. These are paper-vs-code differences, not disagreements between papers. Check the cited notes before implementing.

• ArtFlow content target. Eq. 1 uses the AdaIN output as the target. The code compares the stylized image's relu4_1 features with those of the original content image 2103.16877 eq. 1 net.py train.py. The distance is L2 in the paper and mean squared error in the code 2103.16877 eq. 1 net.py.
• StarGAN v2 AdaIN bias. The paper sets the scale bias to one. The code uses one plus a learned gamma 1912.01865 app. B model.py.
• Drive-OccWorld conditions. The paper describes three conditions. The released configs turn on only the ego-motion one; the semantic and agent-motion normalisation are switched off in the configs and their losses are commented out in the forward pass 2408.14197 §3.2 action_condition_GMO.py drive_occworld.py.

Papers

• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows: applies AdaIN and WCT to the latent features of a reversible flow, and reports less content leak than plain AdaIN with a decoder.
• StarGAN v2: Diverse Image Synthesis for Multiple Domains: uses AdaIN to inject a per-domain style code into one generator for many domains, with diverse outputs from random latents.
• Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving: layer-normalises each memory embedding and scales and shifts it with values from ego-motion, semantic or agent-motion conditions.