GAN inversion

Overview

A GAN generator turns a noise vector z into an image, but it has no encoder. For a real photo you do not know which z would have produced it, so you cannot reconstruct the photo or change one thing about it 1611.06355 §1 §4.1. Inversion means finding that missing map from image back to a code, or a code plus a tuned generator, that reproduces the image 2106.05744 §1.

The inversion papers differ in what they train and what they optimise for each image:

• a pair of encoders, trained once after the GAN 1611.06355 §4
• a per-image optimisation of a W+ code and per-layer noise maps, with the generator fixed 1911.11544 §4 §5
• a per-image optimisation of a W code, then a short fine-tune of the whole generator 2106.05744 §3
• a per-image optimisation of a 512-value texture code inside a 3D face model, with shape, camera and lighting fitted in the same loop 1902.05978 §3.1 §3.4

Two further papers use a GAN after an inversion step. StyleCLIP edits by text. For a real photo it first needs an inverted code, which it obtains with e4e 2103.17249 §4. DragGAN edits by dragging points, and for a real photo it first inverts the image with PTI 2305.10973 §4.1.

Every route that edits ends with the same kind of step: change a code, or a region of it, and decode it again. The exceptions are StyleCLIP's global direction, which edits a style code without an input photo, and GANFIT, which fits a 3D face and does not describe an edit 1902.05978 §3.

Variants

Two encoders after a conditional GAN (IcGAN)

The generator is a conditional GAN with labels concatenated to the noise. Its training follows DCGAN and uses the matching-aware discriminator of Reed et al. 1611.06355 §4.3. Once it is trained, two encoders are fitted:

• E_z maps an image to z. It is trained on images the generator produced from known z and labels, with a squared loss 1611.06355 §4.1 eq. 3.
• E_y maps an image to its attribute vector. It is trained on real images, because the authors found that generated images gave noisier results 1611.06355 §4.1 eq. 4.

Three ways of coupling the two encoders were compared 1611.06355 §4.1:

• SNG: one network with shared layers and two output heads.
• IND: two independent networks. This was the best option in the paper.
• IND-COND: two networks, with the z-encoder also taking the output of the y-encoder as input.

On 150K generated CelebA test images, the reconstruction loss L_e was 0.429 for IND, 0.432 for IND-COND and 0.500 for SNG 1611.06355 §5.3. The paper also shows IND reconstructions visually 1611.06355 fig. 4a.

• Reconstruction: encode the image to (z, y) and decode with the same y 1611.06355 §4.
• Editing: keep the z from the encoder and replace y with the target labels 1611.06355 §5.4. The paper reports that z tends to hold pose, illumination and background, and y holds distinctive face features 1611.06355 §5.4 fig. 6b.
• Qualitative results: on CelebA, reconstructions generally show the requested attribute. Faces in unusual conditions, such as looking away or off-centre, are noisier, and a moustache is often not generated, especially for women 1611.06355 §5.4 fig. 5b. On MNIST, the handwriting style of a real digit carries over to other digits 1611.06355 §5.4 fig. 5a.

Optimised W+ code with noise maps (Image2StyleGAN++)

The generator is a pretrained StyleGAN whose weights stay fixed 1911.11544 §5. The image is embedded in two stages. First a W+ code is optimised. Then per-layer noise maps are optimised with W+ held fixed. Optimising both at once gives high-frequency artifacts 1911.11544 §4.2 fig. 2. The reported schedule is 5000 W+ steps and 3000 noise steps, with a mean face code as the start for faces and a random code otherwise 1911.11544 §7.1.

Reconstruction reaches 39 to 45 dB PSNR, against about 20 dB for the earlier W+-only method. Both numbers are author-reported 1911.11544 §1 §7.1 fig. 4. The same optimiser supports local edits:

• Masked embedding: the loss covers only part of the image, so a missing region can be filled in, or a rough scribble can steer a region 1911.11544 §7.3 §7.4.
• Activation mixing: the GAN's intermediate activations are copied from one embedding into another under two masks. This gives edits, such as swapping features, that blend into the rest of the photo 1911.11544 §5 eq. 5 §7.6.

For inpainting at 256, 512 and 1024 px, the paper reports better SSIM, MSE and PSNR than partial and gated convolution 1911.11544 tab. 1. The evaluation uses 10 ground-truth images and 7 masks 1911.11544 §9.1.

Pivotal tuning (PTI)

The pivot is a code in StyleGAN2's native W space, found with the generator frozen. The paper uses W rather than W+, because the tuning step that follows removes the distortion 2106.05744 §3.1. The inversion optimises w and the noise maps against an LPIPS loss, and only w is kept eq. 1.

Then the whole generator is tuned for up to 350 steps, so that the fixed pivot reproduces the input image 2106.05744 §3.2 §4.5. A locality term keeps the tuned generator close to the original on random codes near the pivot, which limits side effects on other faces 2106.05744 §3.3 eq. 7. Edits then run on the tuned generator, with existing edit directions and StyleCLIP 2106.05744 §4.2.

On CelebA-HQ the paper reports LPIPS 0.09 against 0.34 for the best baseline, and identity similarity 0.9 against 0.85 2106.05744 tab. 1. Both are author-reported. Tuning takes under 3 minutes per identity on one RTX 2080 2106.05744 §4.5 §5.

GAN texture prior in a 3D face model (GANFIT)

GANFIT fits a 3D morphable face to one photo, and the GAN supplies only the skin texture 1902.05978 §3.1. The GAN is a progressive-growing network trained on about 10,000 UV textures at 512 × 512. Its 512-value latent is one of the parameters that the fit optimises 1902.05978 §3.1 §4.1 eq. 5. The shape model has 158 identity and 29 expression bases [arxiv:1902.05978 §4.1]. Camera (7 values) and lighting (9 values) are optimised too 1902.05978 §3.2.

The fit runs in two stages. The first stage uses the landmark loss alone and moves shape, expression and camera so the mesh overlays the face. The second stage optimises all parameters with a weighted sum of an identity loss on a face-recognition network, a content loss on its intermediate layers, a full-resolution pixel loss, a landmark loss and a regulariser [arxiv:1902.05978 §3.4] eq. 12. The reported weights are 2.0 for identity, 50.0 for content, 1.0 for pixels and 0.001 for landmarks, with Adam at learning rate 0.01 1902.05978 §4.1.

On the MICC Florence benchmark, the author-reported mean error is 0.94 to 0.95 for the method, against 1.48 to 1.50 for the best baseline in each setting 1902.05978 tab. 1. Fitting takes about 30 s per image on a GTX 1080 Ti 1902.05978 §4.1. The card records that the released repository holds only the MICC evaluation scripts, so the hyperparameters come from the paper alone; see the card.

Text-driven edits on a real face (StyleCLIP)

StyleCLIP is not an inversion method. It edits StyleGAN2 images from a text prompt, scoring each result with the CLIP cosine distance to the prompt 2103.17249 §4 eq. 1. Real photos are first inverted with e4e 2103.17249 §4 §5 §6. e4e is a separate encoder, and this wiki has no card for it: Tov et al., e4e. The three routes below therefore take their input codes from a step outside this page.

All three routes use a frozen StyleGAN2 generator 2103.17249 §4 fn. 1.

• Per-image latent optimisation. Gradient descent on the W+ code of one photo, for each prompt, for 200 to 300 iterations 2103.17249 §4. The loss adds an L2 penalty to the source code and an optional identity term eq. 1 eq. 2. The table reports 98 s of inference per edit on one GTX 1080Ti tab. 1. The text says "several minutes" §4 §5. The authors call the method sensitive to its parameters and leave it out of the quantitative comparison §5 §7.
• Latent mapper, one network per prompt. Three small networks, one for each group of W+ layers (coarse, medium and fine), output a step that is added to the input code 2103.17249 §5 eq. 3. Training takes 10 to 12 h, and one edit then takes 75 ms on a GTX 1080Ti tab. 1. The authors report that it does better than the global direction on identity-linked prompts such as "Trump" §7 fig. 10.
• Global direction in style space S. One direction per prompt, found from per-channel CLIP relevance 2103.17249 §6 eq. 6 eq. 7. It does not depend on the input image. Pre-processing takes 4 h, and an edit takes 72 ms tab. 1. The edit moves the style code to G(s + αΔs), so α is a strength dial §6. The authors report that it removes wrinkles where the mapper does not §7 fig. 10.

Point dragging after inversion (DragGAN)

DragGAN is an editing method, not an inversion method. The user places handle points on a StyleGAN2 image and gives a target for each. The optimisation then moves each handle toward its target 2305.10973 §3. It takes one optimiser step on the first six layers of the W+ code per iteration. The loss pushes the block-6 features near each handle toward the target 2305.10973 §3.2 eq. 1. After each step, every handle is moved to the nearest feature match in a small square around it 2305.10973 §3.3 eq. 2. No tracking network is used. The loop stops when each handle is within a tolerance of its target, and usually takes 30 to 200 iterations 2305.10973 §3.1 §3.4.

For a real photo, the image is first inverted with PTI, and then the same loop runs 2305.10973 §4.1. The paper gives no inversion settings beyond the citation 2305.10973 §4.1 fig. 5, so results depend on the inversion. On face landmarks, the author-reported mean distance for one point is 2.44, against 11.64 for UserControllableLT 2305.10973 tab. 1. The unit of that distance is not stated 2305.10973 §4.2.

When to use which

The constraints below come from each card's setup. The cards time different hardware, or do not state the hardware, so run times are not compared across variants.

Costs the cards record:

• IcGAN: the GAN and both encoders are trained in sequence, and the encoders need labelled real images for E_y 1611.06355 §4.1. The paper trains on CelebA with 18 of 40 attributes 1611.06355 §5.1.
• Image2StyleGAN++: a per-image optimisation over a frozen generator, run on one 32 GB V100. The run time per image is not stated 1911.11544 §6.
• PTI: a full copy of the generator per identity, and a tuning run per image 2106.05744 §4.5.
• GANFIT: about 30 s per image on one GTX 1080 Ti 1902.05978 §4.1. The GAN's own training time is not stated 1902.05978 §4.1.
• StyleCLIP: the mapper trains for 10 to 12 h, the global direction needs 4 h of pre-processing, and the per-image route runs 200 to 300 steps per edit 2103.17249 tab. 1 §4. The card's timings are on one GTX 1080Ti. The e4e inversion is not costed.
• DragGAN: 2.0 s per edit for one point 2305.10973 tab. 1. The hardware for those times is not stated, although the paper names a single RTX 3090 2305.10973 §1.

Implementation sketch

The common core is the same for every route: estimate a code, optionally tune the generator, then edit the code and decode. The blocks below show the parts where the variants differ.

# Shared shape of all routes: image -> code -> (optional tuning) -> edit
def invert_and_edit(image, G, edit_fn, tune_generator):
    # Stage A: estimate a code for the real image  [see variants above]
    code = estimate_code(image, G)

    # Stage B (PTI only): tune G so the fixed code reproduces the image
    if tune_generator:
        G = pivotal_tune(G, code, image)  # [arxiv:2106.05744 §3.2]

    # Stage C: edit the code, or only a region of it, and decode
    # [arxiv:1611.06355 §5.4] [arxiv:1911.11544 §5]
    return G(edit_fn(code))

# W+ then noise embedding, as in reconstruction  [arxiv:1911.11544 §7.1]
def embed_w_plus_then_noise(image, G):
    # Start from the mean face code, or a random code for non-faces  [§7.1]
    w = initial_w_plus(image)
    n = standard_normal_noise()  # noise maps, N(0, I)  [§7.1]
    # Stage 1: W+ only, noise fixed. Perceptual + masked MSE  [§4.2] [eq. 2]
    w = optimise(w, steps=5000, loss=w_plus_loss(G, w, n, image))
    # Stage 2: noise only, W+ fixed. Masked MSE  [§4.2] [eq. 3]
    n = optimise(n, steps=3000, loss=noise_loss(G, w, n, image))
    return w, n

# Pivotal tuning: invert into W, then tune G  [arxiv:2106.05744 §3]
def pivotal_tune(G_pre, w_pivot, image):
    # The pivot is fixed; only the generator weights change  [§3.2]
    G_t = copy(G_pre)
    for step in range(350):  # upper bound  [§4.5]
        # Reconstruct the image at the pivot  [eq. 2]
        loss = lpips(image, G_t(w_pivot)) + l2(image, G_t(w_pivot))
        # Locality: a random code at distance alpha = 30 from the pivot
        # (alpha and the weight 0.1 from [§4.5])
        w_r = random_code_at_distance(w_pivot, alpha=30)
        target = G_pre(w_r)  # original generator is the target  [eq. 5]
        loss += 0.1 * (lpips(target, G_t(w_r)) + l2(target, G_t(w_r)))
        G_t = adam_step(G_t, loss, lr=3e-4)  # learning rate  [§4.5]
    return G_t

Notes on where the variants differ:

• IcGAN has no per-image loop. After training, each new image needs only the two encoder passes 1611.06355 §4.1.
• The optimisation routes run a loop for each image. Image2StyleGAN++ keeps the noise maps and PTI discards them after inversion 1911.11544 §4 2106.05744 §3.1.
• GANFIT's Stage A is a joint fit, not an encoder pass. A landmark-only stage comes first, then shape, camera, lighting and the texture latent are updated together 1902.05978 §3.4. Its output is a 3D face, so there is no Stage C edit.
• DragGAN replaces Stage C with an iterative loop. It optimises the first six W+ layers, re-locates each handle, and repeats until the handles reach their targets 2305.10973 §3.1 §3.3. For a real photo, Stage B is PTI 2305.10973 §4.1.
• StyleCLIP's global direction has no Stage A. It edits a style code as G(s + αΔs) and does not depend on the input image 2103.17249 §6 tab. 1. Its per-image and mapper routes start from an e4e code for the photo, as described in its variant above.
• The locality term is part of the method 2106.05744 §3.3, but the released code leaves it switched off by default. The card lists this and other code-versus-paper differences; see the card.

Contradictions

The IcGAN paper contains three internal inconsistencies. No other card covers the same results, so they cannot be checked against another source:

• The coupling of the two encoders is named IND-COND in its definition 1611.06355 §4.1 but IND-CND in the results text [arxiv:1611.06355 §5.3].
• The text reports an overall discriminator accuracy near 86% [arxiv:1611.06355 §5.2], while the best generator accuracy in the table is 89.83% [arxiv:1611.06355 tab. 3].
• Two generator rows in the table (layers 2 and 3) both show 82.49% 1611.06355 tab. 3. This may be a typo.

The paper says encoder training uses the same optimiser as the GAN 1611.06355 §4.3. The card's Paper vs code section notes that the released code uses different encoder optimiser settings. See the card for the details.

Across papers, the optimisation routes disagree about the same step:

• Image2StyleGAN++ says optimising W+ and noise maps together gives high-frequency artifacts, because the perceptual loss does not suit the noise maps 1911.11544 §4.2 fig. 2.
• PTI optimises w and the noise maps together in its inversion, against an LPIPS loss 2106.05744 §3.1 eq. 1. It then keeps only w.
• GANFIT fits its texture latent jointly with shape, camera and lighting, after a landmark-only stage 1902.05978 §3.4. It has no noise maps. Its ablation covers the loss terms, not the stages 1902.05978 §4.4, so it does not test the staging question.

The papers do not settle this. They also pick different starting spaces. PTI says W+ reduces distortion but weakens edits 2106.05744 §1 §2. Its table gives an edit magnitude of 11.15 for W+ against 14.86 for its own method 2106.05744 tab. 2, with units not stated. Image2StyleGAN++ builds its embedding on W+ 1911.11544 §4, and its card reports no edit comparison against W. StyleCLIP also edits in W+ and says learned directions there are harder to disentangle 2103.17249 §3. Its card reports no comparison against W.

The reconstruction numbers are not comparable. Image2StyleGAN++ reports PSNR in dB 1911.11544 §1 fig. 4. PTI reports LPIPS, MSE, MS-SSIM and identity similarity 2106.05744 tab. 1. GANFIT reports a point-to-plane shape error in millimetres on 3D scans 1902.05978 tab. 1, and DragGAN reports a landmark distance whose unit is not stated 2305.10973 tab. 1. None of these measures a 2D reconstruction on the same scale.

PTI's table marks MS-SSIM with a down arrow, but its own value is the one the caption calls best. The card treats this row as unclear [arxiv:2106.05744 tab. 1]. It is not a disagreement between papers.

StyleCLIP's card gives two times for its per-image route: "several minutes" in the text 2103.17249 §4 §5, and 98 s in its table tab. 1. This page quotes the table, and the text figure is not resolved.

Papers

• Invertible Conditional GANs for image editing: trains a conditional GAN, then two encoders for noise and attributes, to reconstruct and edit real face images.
• Image2StyleGAN++: How to Edit the Embedded Images?: embeds a photo into StyleGAN's W+ space and then into noise maps, with masks for local edits and inpainting.
• Pivotal Tuning for Latent-based Editing of Real Images: inverts a face into W, then fine-tunes StyleGAN2 so the code reproduces it, with a locality term.
• GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction: fits a 3D face model to one photo, with a GAN supplying the skin texture and a face-recognition network the identity loss.
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery: edits StyleGAN2 images from text prompts, by per-image optimisation, a per-prompt latent mapper, or a global direction in style space.
• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold: moves handle points of a StyleGAN2 image to targets by optimising the latent code, with nearest-neighbour point tracking and no tracker network.