StarGAN v2: Diverse Image Synthesis for Multiple Domains

In plain words

Image-to-image translation takes a picture and redraws it as a different visual category, such as a face with another hairstyle or a cat turned into a dog. The usual approach trains one translator for every pair of categories, which becomes unwieldy as the number of categories grows. Other approaches use one model for all categories but give only one fixed answer per input.

StarGAN v2 uses a single generator for all categories. It is told how the output should look by a "style" vector. That vector can be made from random noise, so the same input can come out in several looks, or it can be copied from a reference photo, so the output takes on that photo's look. Each category has its own style vectors, so a hairstyle for one category does not get confused with a pattern for another.

Three extra training penalties keep the model honest. One rewards different noise samples for producing visibly different images. One checks that the translated image can be turned back into the original, so the basic pose and structure survive. One pulls the style back out of the output, so the generator actually uses the style it was given. The authors also release a dataset of animal faces to test translation between very different categories.

TL;DR

• One generator covers all domains. A domain-specific style code, from a mapping network (random latent) or a style encoder (reference image), replaces the fixed domain label of StarGAN §1 §2.1.
• Training adds a style-reconstruction term, a diversity term whose weight decays linearly to zero, and a cycle-consistency term to the adversarial loss §2.2 eq. 5.
• Author-reported latent-guided FID: 13.7 on CelebA-HQ (MSGAN 33.1) and 16.2 on AFHQ (MSGAN 61.4) tab. 2.
• Reference-guided FID: 23.8 on CelebA-HQ and 19.8 on AFHQ, against MSGAN's 39.6 and 69.8 tab. 3.
• The official code is CC BY-NC 4.0. Use the paper, not the code, for anything commercial.

Problem

Translating between K domains with separate models needs K(K-1) generators, one per ordered pair of domains §1. StarGAN fixes this with one generator, but each domain label gives one deterministic output per input, so it cannot capture multi-modal output distributions §1. Prior multi-modal methods inject a random latent code, but in a multi-domain setting a latent code encodes domain-shared styles (for example colour) rather than domain-specific ones (for example hairstyle) §3.1. The only supervision is the domain label; other attributes are learned without labels §3.

Method

The model has four modules and one shared generator §2.1 fig. 2:

1. Generator G(x, s) translates an input image x using a style code s. Style is injected into every decoder block through adaptive instance normalization (AdaIN). Down-sampling blocks use instance normalization §2.1 app. E.
2. Mapping network F takes a latent z, sampled from a standard Gaussian app. E, and a domain y, and returns s = F_y(z). It is an MLP with one output head per domain, so each domain gets its own style space §2.1 app. E.
3. Style encoder E takes an image x and its domain y, and returns s = E_y(x). It has one output head per domain and shares its residual layers across domains. It does not use global average pooling, to keep fine style detail §2.1 app. E.
4. Discriminator D is a multi-task discriminator, with one real/fake output per domain. It has no feature normalisation and no PatchGAN head §2.1 app. E.

Training combines five terms. Let ỹ be a target domain, z a latent code and s̃ = F_ỹ(z) §2.2.

1. Adversarial loss eq. 1, with the discriminator branch chosen by domain:

\mathcal{L}_{adv} = \mathbb{E}_{x,y}[\log D_y(x)] + \mathbb{E}_{x,\tilde{y},z}[\log(1 - D_{\tilde{y}}(G(x, \tilde{s})))] eq. 1

2. Style reconstruction eq. 2. The encoder must recover the style code that was used, so the generator is forced to use it:

\mathcal{L}_{sty} = \mathbb{E}_{x,\tilde{y},z}[\lVert \tilde{s} - E_{\tilde{y}}(G(x, \tilde{s})) \rVert_1] eq. 2

3. Diversity-sensitive loss eq. 3. The paper drops the denominator of the original diversity-sensitive loss, which made training unstable, and keeps the same intuition: two different latent codes must yield different images §2.2:

\mathcal{L}_{ds} = \mathbb{E}_{x,\tilde{y},z_1,z_2}[\lVert G(x, \tilde{s}_1) - G(x, \tilde{s}_2) \rVert_1] eq. 3

4. Cycle consistency eq. 4. The input is reconstructed from the translated image using the input's own estimated style ŝ = E_y(x). This preserves domain-invariant content such as pose §2.2:

\mathcal{L}_{cyc} = \mathbb{E}_{x,y,\tilde{y},z}[\lVert x - G(G(x, \tilde{s}), \hat{s}) \rVert_1] eq. 4

5. Full objective eq. 5. The generator, mapping network and encoder minimise, and the discriminator maximises. The diversity term enters with a minus sign:

\min_{G,F,E}\max_D \; \mathcal{L}_{adv} + \lambda_{sty}\mathcal{L}_{sty} - \lambda_{ds}\mathcal{L}_{ds} + \lambda_{cyc}\mathcal{L}_{cyc} eq. 5

The paper trains this objective with style codes from random latents and, in a further variant, with style codes from reference images §2.2. The official code alternates both within each iteration solver.py. The values of λ are in the hyperparameter table.

The core of one update, in the official code's form, looks like this. The paper gives the losses but no update schedule, so the order of steps comes from the code solver.py.

# Style code from a random latent via the mapping network  [§2.1] [app. E]
def mapping_style(latent, domain):
    # Four shared layers, width 512, ReLU
    # [app. E] [code: core/model.py#L193-L199]
    hidden = shared_layers(latent)                 # (B, 512)
    # One head per domain: three ReLU layers, then linear to 64  [app. E]
    head_outputs = stack([head(hidden) for head in heads], dim=1)  # (B, K, 64)
    return head_outputs[arange(B), domain]         # (B, 64)

# Style code from a reference image via the style encoder  [§2.1] [app. E]
def encoder_style(image, domain):
    # Six residual blocks shared by all domains
    # [app. E] [code: core/model.py#L229-L233]
    features = residual_stack(image)               # (B, 512, 4, 4)
    # 4x4 convolution to 512-d, then one linear head per domain  [app. E]
    features = conv4x4(features).flatten()         # (B, 512)
    return head_outputs(features)[arange(B), domain]  # (B, 64)

# Discriminator loss on real and fake images  [§2.2] [eq. 1] [app. B]
def discriminator_loss(real, source_domain, fake, target_domain):
    # Real images should be scored real in their own domain  [eq. 1]
    real_logit = discriminator(real, source_domain)           # (B,)
    # Generated images should be scored fake in the target domain  [eq. 1]
    fake_logit = discriminator(stop_gradient(fake), target_domain)  # (B,)
    # Logistic losses plus R1 penalty on real images, gamma=1  [app. B]
    return (logistic(real_logit, 1)
            + logistic(fake_logit, 0)
            + r1(real_logit, real))

# Generator, mapping and encoder loss  [§2.2] [eq. 1]-[eq. 5]
def generator_loss(real, source_domain, target_domain, style, style_2):
    # Translate with the target style  [§2.1]
    fake = generator(real, style)
    # Adversarial term: fool D in the target domain  [eq. 1]
    adv = logistic(discriminator(fake, target_domain), 1)
    # Style reconstruction: the encoder must recover the style  [eq. 2]
    sty = mean_abs(encoder(fake, target_domain) - style)
    # Diversity: compare with a second style; detach the second image  [eq. 3]
    fake_2 = stop_gradient(generator(real, style_2))
    ds = mean_abs(fake - fake_2)
    # Cycle: translate back with the input's own estimated style  [eq. 4]
    cyc = mean_abs(generator(fake, encoder(real, source_domain)) - real)
    # Weighted sum; lambda_ds decays to 0 by 100K steps  [eq. 5] [app. B]
    return adv + lambda_sty * sty - lambda_ds * ds + lambda_cyc * cyc

Steps the paper states explicitly, with the value it gives:

1. Latent codes are sampled from a standard Gaussian app. E. Batch size is 8 and training runs for 100K iterations app. B.
2. Adam is used with β1 = 0 and β2 = 0.99. Learning rate is 1e-4 for G, D and E, and 1e-6 for F app. B.
3. The diversity weight λ_ds is decayed linearly to zero over the 100K iterations app. B.
4. Exponential moving averages of all modules except D are used for evaluation app. B.
5. Weights are He-initialised, all biases zero, except the AdaIN scale biases, which are one app. B.

Architecture details, from the appendix app. E:

• Generator: four down-sampling residual blocks (instance norm), four intermediate blocks, four up-sampling blocks (AdaIN). For CelebA-HQ, one extra down- and up-sampling block is added, and the shortcuts of the up-sampling blocks are removed, with skip connections from an adaptive-wing heatmap app. E.
• Mapping network: latent of 16, hidden width 512, style of 64. Four shared fully connected layers, then four per-domain fully connected layers app. E.
• Style encoder and discriminator: six pre-activation residual blocks with leaky ReLU (slope 0.2 in code model.py), then a 4x4 convolution to 1x1. The encoder has one linear head per domain; the discriminator has one real/fake output per domain app. E.

Figures are described here, not reproduced: the four-module overview fig. 2, the ablation images fig. 3, reference-guided results on CelebA-HQ fig. 4, latent-guided results fig. 5, reference-guided results fig. 6, and the FFHQ generalisation test with a CelebA-HQ model fig. 7. Figure numbers follow source order and were not checked against the compiled PDF.

Hyperparameters

Results

Author-reported. Setup: CelebA-HQ (two domains, male and female) and AFHQ (three domains, cat, dog and wildlife), trained at 256 x 256 §3 app. A. AFHQ has 15,000 images at 512 x 512, 5,000 per domain, with 500 test images per domain app. A. Baselines are MUNIT, DRIT and MSGAN, each trained for every pair of domains, and StarGAN, which uses one generator §3. Metrics are FID and LPIPS. Each source image gets 10 outputs, and FID is computed against training images in the target domain app. C. Model size: not stated. Hardware: one Tesla V100, about 3 days app. B.

Ablation on CelebA-HQ, cumulative configurations tab. 1:

Latent-guided synthesis, random latent codes tab. 2:

Reference-guided synthesis, ten reference images tab. 3:

Human preference, percentage of votes for the most preferred method tab. 4. Each comparison used 100 questions with 10 workers each. There were 76 valid workers §3.2:

No confidence intervals or significance tests are reported for any of these numbers.

Paper vs code

Code: official repository clovaai/stargan-v2 at commit 875b70a150609e8a678ed8482562e7074cdce7e5, linked from the abstract. Its LICENSE is Attribution-NonCommercial 4.0 LICENSE. This card cites the code but quotes none of it.

The following differences were found when comparing the paper with the code at that commit:

• Loss normalisation. The paper writes the L1 norm in eq. 2, eq. 3 and eq. 4. The code takes the mean absolute error over all elements: solver.py, solver.py, solver.py. This divides each term by the number of elements (for example 3 x 256 x 256 for an image). If you implement the norms as sums, the λ values in app. B no longer give the same balance.
• Target domain. The paper says the target domain is sampled at random §2.2. The code takes the target domain from the labels of a separate reference batch, drawn from a class-balanced sampler. It does not check that the target differs from the source: solver.py, data_loader.py, data_loader.py.
• Update schedule. The paper says only that the model is also trained "in the same manner" with reference images §2.2. The code runs two discriminator steps and two generator steps per iteration. The mapping network and the encoder are stepped only in the latent-guided generator step; the reference-guided generator step updates the generator alone: solver.py.
• Diversity term gradient. The code detaches the second generated image in the diversity term, so gradients flow only through the first sample: solver.py. The paper does not say this.
• R1 convention. The code computes 0.5 times the squared gradient norm, summed over pixels and averaged over the batch, weighted by λ_reg = 1: solver.py main.py. The paper states γ = 1 but not the factor 1/2.
• Weight decay. The code uses Adam weight decay of 1e-4 main.py solver.py. The paper does not mention weight decay.
• EMA decay. The paper uses EMA app. B but gives no decay value. The code uses 0.999 solver.py. The code keeps EMA copies of G, F and E only, which matches the paper.
• Data augmentation. The paper does not mention augmentation. The training loader applies a random resized crop (scale 0.8 to 1.0, ratio 0.9 to 1.1) with default probability 0.5, then a random horizontal flip data_loader.py. It also samples classes with a balanced sampler data_loader.py.
• First convolution. The paper's generator and encoder tables list a 1x1 first convolution app. E. The code uses 3x3 convolutions for the generator, the encoder and the discriminator: model.py model.py model.py.
• AdaIN bias. The paper says the AdaIN scale biases are set to one app. B. The code instead computes (1 + gamma) times the normalised input plus beta, and He initialisation sets every linear bias to zero: model.py utils.py. The two are equivalent in effect, but a plain gamma-times-x implementation needs its scale bias set to one.
• Hidden width flag. --hidden_dim (default 512) is defined but not passed to the mapping network, which hard-codes 512: main.py model.py model.py. Changing the flag has no effect.
• High-pass skip. The default for --w_hpf is 1 main.py, so the CelebA-HQ path with heatmap skips is the default. The README sets 0 for the three-domain AFHQ run README.md. When w_hpf > 0 the code removes the up-sampling shortcuts and adds a high-pass-filtered mask skip at 32, 64 and 128 pixels: model.py model.py. The mask comes from a pretrained FAN checkpoint at expr/checkpoints/wing.ckpt main.py model.py. The paper does not say how that checkpoint was trained.

Consistent between paper and code:

• Latent dimension 16, style dimension 64, hidden width 512 app. E main.py.
• Four shared and four per-domain fully connected layers in the mapping network app. E model.py.
• Six residual blocks in the encoder and no global average pooling app. E model.py.
• Batch size 8, 100K iterations, learning rates 1e-4 and 1e-6, Adam betas (0, 0.99) app. B main.py.
• The linear decay of λ_ds over the same 100K iterations: app. B solver.py.
• The λ values in the README match app. B README.md.
• Ten outputs per source image for evaluation app. C main.py eval.py.

Not checked: the FID and LPIPS implementations were not compared line by line with app. C.

Pitfalls

• Unstated defaults. The paper does not give the EMA decay, weight decay, augmentation, balanced class sampling, or the order of updates. Implementing only the paper gives a different model from the one whose numbers are reported.
• Loss scale. λ values only make sense together with the normalisation. Using sums where the code uses means scales those terms up by the number of elements, which changes their balance against the adversarial loss. Pick one convention and keep it.
• Diversity term. The paper writes the diversity term as a plain expectation. The code detaches the second sample. A design that backpropagates through both samples would change the gradient, and the paper does not say which is intended.
• Reference-guided branch. The style encoder is updated only in the latent-guided generator step. An implementation that updates the encoder in both steps will differ from the code. The paper does not state which is intended.
• Same-domain targets. Nothing prevents a target domain equal to the source domain data_loader.py. How often this happens depends on the batch composition. The paper does not describe it.
• Real-image FID is not a ceiling. The paper's real-image FIDs (14.8 CelebA-HQ, 12.9 AFHQ) are computed between training and test sets, and the caption says they may not be optimal because the test set is small tab. 2. StarGAN v2's CelebA-HQ FID of 13.7 is below that reference. Do not read it as beating real images.
• LPIPS is not monotone in quality. On AFHQ, MSGAN has higher latent-guided LPIPS (0.517) than StarGAN v2 (0.450). The authors attribute the baselines' high LPIPS to spurious artifacts §3.2 tab. 2. Diversity and quality should be read together.
• Text rounds the ratio. The text says FIDs are about 1.5 times better than the previous leading method on CelebA-HQ §3.2. The table gives 39.6 / 23.8, which is about 1.66 times tab. 3. The AFHQ ratio (69.8 / 19.8, about 3.5 times) matches the text.
• Small human-study sample. Each comparison used 100 questions, each answered by 10 workers, from 76 valid workers §3.2. No significance test is given.
• Dataset licence. The AFHQ images come from Flickr and Pixabay under "permissive licenses". The paper does not list per-image licences app. A. Do not treat AFHQ as CC-licensed. This card cites the dataset only.
• Code licence. The official code is CC BY-NC 4.0 LICENSE. Its terms rule out commercial use. This card does not quote it, and an implementation written from the paper avoids that restriction.
• Compute. Training takes about 3 days on one V100 for 100K iterations at batch size 8 app. B. Compute for the baselines, which were trained per domain pair, is not stated.
• Table and figure numbers. Numbers follow source order in the LaTeX, not the compiled PDF. Appendix tables are cited by section, not number.

Lineage

• Builds on: StarGAN §1 §3, which uses one generator with a fixed domain label. StarGAN v2 replaces the label with a learned style code.
• Builds on: AdaIN and StyleGAN, used to inject the style §2.1.
• Builds on: the multi-task discriminator and R1 regularisation [mescheder2018r1reg], as cited in §2.1 and §3.1.
• Builds on: the diversity term of MSGAN and DSGAN §2.2.
• Compared against: MUNIT, DRIT, MSGAN (each trained per domain pair) and StarGAN §3.
• Superseded by: not stated in this wiki.