Invertible Conditional GANs for image editing

In plain words

A GAN can draw convincing pictures from random noise, but it cannot take a photo you already have and tell you which noise would have produced it. Without that, you cannot take a real face and change one thing about it, such as adding a smile, while keeping the rest.

This paper adds a second part to a conditional GAN, one that can also take labels such as "smiling" or "male" as input. Two small encoder networks work backwards from a real image. One estimates the hidden code for the rest of the picture (pose, lighting, background). The other estimates the labels. Once you have both, you can feed the code back in and change only the labels, which gives a new version of the same photo with the chosen attributes altered.

The authors test this on handwritten digits and on face photos, and report that edited faces generally show the requested attribute §5.4. They also report which encoder design works best and where the label input should go inside the network.

TL;DR

IcGAN = a conditional GAN §3 §4 plus an encoder E = (E_z, E_y) trained after the GAN §4.1. E_z regresses the noise vector z from generated images, and E_y regresses the attribute vector y from real images eq. 3 eq. 4. Reconstruction is x' = G(z, y), and editing keeps z and changes y §4 §5.4. The best encoder configuration is two independent networks (IND) §4.1 §5.3. Author-reported on CelebA: E_z+E_y loss L_e of 0.429 (IND), 0.432 (IND-COND), 0.500 (shared SNG) §5.3. Results are qualitative for MNIST and for attribute edits, and the attribute-classifier accuracy ranges from 74% to 90% across the cGAN configurations tab. 3. Use it when you need attribute edits on real images with a plain DCGAN backbone. Code is public (Torch7/Lua).

Problem

A GAN has a generator G(z) and no inference network, so it cannot find the latent z for a real image x §1 §4.1. Without that inverse, you cannot reconstruct or modify a real image. Variational autoencoders have an inverse but blur their outputs because of a pixel-wise reconstruction loss §2. Previous GAN encoders (Dumoulin et al., Donahue et al., Reed et al.) were unconditional or did not use the label information §2 §4. The paper asks two questions: how to build the encoder in a conditional setting, and which cGAN design choices (where to insert y, how to sample y') work best §1 §4.2.

Method

Conditional GAN §3. The minimax objective is eq. 1 for an unconditional GAN and eq. 2 for the conditional one. The discriminator sees (x, y) pairs and the generator takes (z, y') where y' is sampled as described below.

\min_{g} \max_{d} v(\theta_g,\theta_d) = \mathbb{E}_{x\sim p_{data}} [\log D(x)] + \mathbb{E}_{z\sim p_z}[ \log(1-D(G(z)))] eq. 1

\min_{g} \max_{d} v(\theta_g,\theta_d) = \mathbb{E}_{x,y\sim p_{data}} [\log D(x,y)] + \mathbb{E}_{z\sim p_z, y'\sim p_y}[ \log(1-D(G(z,y'),y'))] eq. 2

z is drawn from a known prior p_z = N(0,1) §4.2. The paper calls the variation controlled by y "constrained" and the variation in z "unconstrained" §3.

Encoder §4.1. Train E after the cGAN has finished. E = (E_z, E_y), with two outputs:

• E_z maps an image to z. It is trained on images G produces, using a squared reconstruction loss eq. 3.
• E_y maps an image to y. It is trained on real images and their labels, because the authors found generated images were noisier than real ones and this improved E_y eq. 4.

\mathcal{L}_{ez}=\mathbb{E}_{z\sim p_z, y'\sim p_y}\|z-E_z(G(z,y'))\|^2_2 eq. 3

\mathcal{L}_{ey}=\mathbb{E}_{x,y\sim p_{data}}\|y-E_y(x)\|^2_2 eq. 4

Three ways to couple the two sub-encoders were tried §4.1:

• SNG: one encoder with shared layers and two output heads.
• IND: two independent encoders, trained separately. The best one §5.3 fig. 4a.
• IND-COND: two encoders, but the z-encoder takes the y-encoder's output as an extra input.

cGAN design choices §4.2.

• Conditional position. In G, y is concatenated with z at the input level. In D, y is replicated spatially and concatenated after the first convolutional layer tab. 1 fig. 3. The paper argues that inserting y earlier gives the network more interactions with it §4.2. Table 3 supports this for the generator (input level) and for the discriminator (layer 1) §5.2.
• Conditional sampling. For training, y' is drawn directly from the training labels (p_y = p_data). Kernel density estimation and interpolation between labels are rejected because the attributes are binary, so interpolated vectors are not plausible labels §4.2. The authors note this sampling could overfit in principle, but argue it does not here, because binary attribute vectors are shared by many images §4.2.

Architecture §4.3 tab. 1 tab. 2. The cGAN follows DCGAN and is trained with the matching-aware discriminator method of Reed et al. §4.3.

• Generator: input is the concatenation of z and y'. Five transposed convolutions with 4×4 kernels produce 512, 256, 128, 64 and 3 channels. BN and ReLU follow every layer except the output, which uses tanh and no BN tab. 1. The first layer is listed with stride 2; see Paper vs code.
• Discriminator: 4×4 convolution with stride 2 and 64 filters, LeakyReLU, no BN tab. 1. y is replicated and concatenated. Then three stride-2 convolutions with 128, 256 and 512 filters, BN and LeakyReLU. The last layer is a 4×4 convolution to one unit with a sigmoid tab. 1.
• Encoder (IND shown): four 5×5 stride-2 convolutions with 32, 64, 128 and 256 filters, each with BN and ReLU tab. 2. Then two fully connected layers, the first with BN and ReLU (4096 units for z, 512 for y), the second linear (100 units for z, n_y for y). The last layer has no BN and no activation, so its output is comparable to N(0,1) §4.3 tab. 2.

The paper's stated encoder training uses the same optimiser and batch size as the cGAN §4.3. The code does not do this; see Paper vs code.

Training recipe (condensed; the paper gives it in prose across §4 and §4.3). Train the cGAN first. Then run the generator to produce pairs (z, y'), train E_z on them, and train E_y on real images.

# Two-stage IcGAN training  [§4] [§4.1] [§4.2]
# Input:  real images x (n, 3, 64, 64) and attribute vectors y (n, n_y)
# Output: generator G, encoders E_z and E_y

def train_icgan(images, labels):
    # 1. Train the cGAN first; the encoders come afterwards  [§4]
    generator, discriminator = train_cgan(images, labels)

    # 2. E_z learns z from images the generator produced  [§4.1] [eq. 3]
    # z_dim is not stated in the paper; the code uses 100
    # [code: cfg/mainConfig.lua#L13]
    z_batch = sample_normal(0, 1, shape=(n, z_dim))  # p_z = N(0, 1)  [§4.2]
    y_batch = sample_training_labels(n)  # y' ~ p_data  [§4.2]
    fake_images = generator(z_batch, y_batch)
    encoder_z = fit(encoder_z, input=fake_images, target=z_batch)  # [eq. 3]

    # 3. E_y learns y from REAL images, since generated ones are noisier  [§4.1]
    encoder_y = fit(encoder_y, input=images, target=labels)  # [eq. 4]

    return generator, encoder_z, encoder_y


def reconstruct_and_edit(
    image, new_attributes, generator, encoder_z, encoder_y
):
    # Invert the real image into z and y  [§4]
    z = encoder_z(image)
    y = encoder_y(image)
    # Reconstruction keeps the estimated y; an edit replaces it  [§4] [§5.4]
    reconstruction = generator(z, y)
    edited = generator(z, new_attributes)  # z (pose, lighting) kept  [§5.4]
    return reconstruction, edited

Discriminator loss with mismatched labels (the paper cites Reed et al. for this but gives no formula; the code implements it, see Pitfalls):

# Discriminator loss with mismatched labels  [code: trainGAN.lua#L203-L262]
# Gradient weights: real+matching 1, real+mismatched 0.5, generated 0.5

def discriminator_loss(real, labels, wrong_labels, fake, fake_labels):
    # Real image, matching label: target 1  [code: trainGAN.lua#L215-L219]
    loss_real = bce(D(real, labels), target=1)
    # Real image, mismatched label: target 0  [code: trainGAN.lua#L223-L233]
    loss_wrong = 0.5 * bce(D(real, wrong_labels), target=0)
    # Generated image, sampled label: target 0  [code: trainGAN.lua#L252-L256]
    loss_fake = 0.5 * bce(D(fake, fake_labels), target=0)
    return loss_real + loss_wrong + loss_fake

Hyperparameters

Results

Author-reported, with the setup as stated. The accuracies (74% to 90%) measure how well an attribute classifier (Anet) recognises the requested attributes in generated images. They do not measure image quality. The repository has no Anet training script; it only loads a saved network [code: evaluation/evaluateModel.lua#L19].

cGAN evaluation on CelebA (generator uses Anet predictions as the metric) §5.2 tab. 3:

Bold marks the best value in each column, as in the source. The authors chose y at the generator input and the discriminator's layer 1 for IcGAN §5.2.

Encoder evaluation on 150K CelebA generated test images, by reconstruction loss L_e §5.3:

The text names this configuration "IND-CND" in one place §5.3. The configuration itself is IND-COND §4.1. Visual comparison in fig. 4a agrees with the loss: IND gives the most faithful reconstructions.

Qualitative results (no numbers reported):

• MNIST: reconstructions keep the handwriting style of real test digits and transfer it to other digits §5.4 fig. 5a.
• CelebA: reconstructions generally match the requested attribute. Faces with unusual conditions (e.g., looking away, off-centre) are noisier. Mustache is often not generated, especially on women §5.4 fig. 5b.
• The paper reports reconstructions from encoders that fill occluded parts of faces, like a hand over the face §5.3 fig. 4b, and that GAN reconstructions are not blurry as VAE reconstructions are §5.3.
• Latent interpolation between two reconstructed CelebA faces gives smooth, plausible transitions §5.4 fig. 6a.
• Attribute swap between pairs of faces: the paper finds z encodes pose, illumination and background, and y encodes distinctive face features §5.4 fig. 6b.

Paper vs code

Code: https://github.com/Guim3/IcGAN, commit 45c5fd9e4a5d620e460bef15668a30cbb2a07bc1. The paper's front-page footnote links it p. 1. The README cites the same arXiv id README.md.

Matches:

• Generator and discriminator filter counts and strides (except the first layer, below) match tab. 1 trainGAN.lua.
• Encoder conv layers and FC sizes match tab. 2 trainEncoder.lua.
• cGAN Adam settings, learning rate, batch size and 25 epochs match §4.3 mainConfig.lua.
• CelebA keeps 18 of 40 attributes §5.1 donkey_celebA.lua.
• The E_z generated set excludes the 19,961 test images, so the encoder never sees the test partition during training generateEncoderDataset.lua.

Differences:

• First generator layer stride. The table says stride 2 tab. 1. The code uses default stride 1 with padding 0 on a 1×1 input trainGAN.lua. From a 1×1 input both settings give a 4×4 output with identical arithmetic, so this is a notational difference, not a functional one.
• Encoder optimiser. The paper says the encoder uses the cGAN's settings §4.3. The code uses learning rate 0.0001 and β1=0.1 for both encoders, with 15 epochs mainConfig.lua mainConfig.lua.
• Loss normalisation. Eq. 3 and 4 use the squared L2 norm, which sums over dimensions eq. 3 eq. 4. The code uses MSECriterion, which averages over elements trainEncoder.lua. The gradients differ by a factor of 1/dimension (100 for z, n_y for y). Adam is largely insensitive to a constant gradient scale, so the effect is probably small, but this was not tested here.
• Generator loss. Eq. 2 is the minimax form. The code trains G with the "real" label, which is the non-saturating form trainGAN.lua.
• Mismatched labels in D. The paper cites Reed et al. for the matching-aware discriminator §4.3 and gives no formula. The code adds a real-image/mismatched-label term at half weight trainGAN.lua and a generated-image term at half weight trainGAN.lua.
• E_y training set. The paper says E_y is trained on the official training partition, 182K images §5.1. The code splits the preprocessed file, which contains every image in the folder preprocess_celebA.lua, at 66% mainConfig.lua. That gives about 133,715 training images out of 202,599 (floor(0.66 × 202,599)), not 182K. The paper does not state the encoder training set size.
• E_z test set size. The paper reports a test set of 150K generated images §5.3. The code generates 182,638 samples and splits them 66/34 [code: cfg/mainConfig.lua#L86] mainConfig.lua. That leaves 62,097 for testing. The two numbers cannot both be right; the paper is not explicit.
• Code-only details. The LeakyReLU slope, weight init, and [-1, 1] scaling are not in the paper and appear only in code (see Hyperparameters).
• Variants. SNG, IND-COND and the other encoder experiments are in other/alternativeEncoders/, not in the main training path [code: other/alternativeEncoders/]. The README does not run them.

The paper also has an internal name mismatch: IND-COND is defined in §4.1 but called IND-CND in §5.3.

Pitfalls

• Optimiser for E_z and E_y. The paper says the encoder uses the cGAN's optimiser §4.3, but the code uses a 5× lower learning rate and different β1 mainConfig.lua. Use the code's values for a faithful reproduction, and note the paper does not state them.
• Encoder training set for E_y may be about 133K images, not the full 182K training partition, because of the 66% split in the code (see Paper vs code). Pick one and state it.
• Overfitting claim. The paper says the cGAN does not memorise training data because it reconstructs unseen test images §5.3. The evidence is visual; no held-out metric is given.
• Anet is not documented. The 74% to 90% accuracies depend on a classifier the paper describes only by architecture §5.2, and the repo has no training script for it. Compare numbers only within this paper's table.
• Table 3 duplicate. Two generator rows (layers 2 and 3) both show 82.49% accuracy as printed tab. 3. Treat that value as possibly a typo.
• Text vs table. The text says "overall accuracy ∼86%" §5.2, but the best generator value in the table is 89.83% tab. 3.
• Mismatched-label term. It is in the code, but the paper gives no formula §4.3. Implement it from the code if you want the same behaviour, and mark that choice as code-derived.
• Non-saturating G loss. The paper's Eq. 2 is minimax, but the code uses the non-saturating loss trainGAN.lua. Training may behave differently at the start of training.
• E_y needs real labels. Training E_y on generated images gave noisier results, so it uses real images §4.1. Do not swap this to save data collection.
• Choice of y' sampling. Sampling y' directly from training labels can overfit when attributes are unique per image. The paper argues binary attribute vectors avoid this §4.2, but this is an argument, not a test.
• Interpolating y is not supported. The paper rejects interpolated binary y because it is not a plausible label §4.2. Attribute edits are done by setting y to a chosen binary vector §5.4.
• Licence of code. The repo's LICENSE.md is BSD text that names dcgan.torch and Facebook (2015), the baseline the authors built on [code: LICENSE.md#L1-L5]. No separate IcGAN licence statement was found. The card records BSD-3-Clause because that is the licence text in the repo, but check with the authors before relying on it for IcGAN-specific code.
• Memory and disk. The README says the CelebA training loads the whole dataset into RAM, about 10 GB, and needs about 25 GB of disk [code: README.md#L53]. The config defaults to GPU mode mainConfig.lua.
• Pretrained model. The authors link a pretrained CelebA model (generator plus both encoders) on Google Drive README.md. It is not checked here.

Lineage

• Builds on: DCGAN, the baseline architecture and training recipe §4.3 README.md (arXiv:1511.06434, linked in README.md).
• Builds on: conditional GANs with label input §3 (Mirza and Osindero, arXiv:1411.1784, cited in the paper's bibliography).
• Uses: the matching-aware discriminator of Reed et al. §4.3 (no arXiv id in the source; not cited here by id).
• Compared against or related: unconditional GAN encoders by Donahue et al. (arXiv:1605.09782), Dumoulin et al. (arXiv:1606.00704), Makhzani et al. (arXiv:1511.05644), and VAE/GAN hybrids §2. The paper says its encoder work is independent of Dumoulin and Donahue and that its IND approach "performs nearly equally" to Donahue's §4.1.
• Superseded by: not known from content on this wiki.