Latent-space editing

Overview

Editing in latent space means three steps: find a code that reproduces the input, move the code, and decode it again. The hard part is the middle step. Naive vector arithmetic in a GAN's latent space can change the target attribute, but it also pushes the code away from the codes the decoder was trained on and can change the person's identity 2008.02401 §1.

The ten covered papers take different routes. Eight edit images, one works on human motion, which uses the same kind of latent space, and one trains a generator from text alone rather than editing a given image:

• A trained encoder pair on a conditional GAN. Two encoders, learned after the GAN, map a real image to its noise vector and its attribute vector. Changing the attribute vector edits the image 1611.06355 §4 §4.1.
• A learned map on a pretrained GAN's latent. A normalizing flow, conditioned on attributes, decides where an edited code should go 2008.02401 §3.
• Optimisation in a pretrained decoder's latent grid, with no training. A joint image-text model scores the decoded image against a prompt, and the grid moves to raise the score 2204.08583 §1 §2.
• Changing the generator instead of the code. A pretrained GAN is fine-tuned from two text prompts alone, so the new generator makes the target domain. The paper claims the latent space stays the source's, so latent-space methods still apply 2108.00946 §4.1 §5.2.
• Tuning the generator to one photo. The photo is inverted into StyleGAN2's W space, then the whole generator is fine-tuned so that this fixed code reproduces the photo 2106.05744 §3.1 §3.2.
• Optimising noise as well as the latent. A StyleGAN photo is embedded into W+, then into per-layer noise maps. Masks limit the embedding to one region, so local edits and fills are possible 1911.11544 §4 §5.
• A conditional autoencoder for face age. An encoder gives a short identity code, and a generator renders it at a chosen age bin. Changing the age label with the code held fixed changes the age 1702.08423 §4.1 §5.2.
• Text prompts on a pretrained StyleGAN2. CLIP's cosine distance to a prompt steers the generator through three routes: per-image latent optimisation, a small per-prompt mapper, and one global direction in style space 2103.17249 §4 §5 §6.
• Dragging handle points on a StyleGAN2 image. Each step optimises the first six W+ layers so that features under each handle move toward its target. A nearest-neighbour search then re-finds each handle, with no tracking network 2305.10973 §3.
• A motion model rather than an image model. A transformer auto-encoder on human motion is aligned to CLIP's text and image spaces, so a text prompt can be decoded directly into a motion 2203.08063 §3 §4.2.

Variants

Trained encoder pair on a conditional GAN (IcGAN)

The method adds a learned inverse to a conditional GAN. The GAN is trained first. Then E_z maps an image to the noise vector z, and E_y maps a real image to its attribute vector y 1611.06355 §4.1. Editing keeps z and replaces y, so the generator redraws the same photo with other attributes 1611.06355 §5.4. Inversion is one forward pass through each encoder, so no per-image optimisation is needed 1611.06355 §4.1.

• Training: the cGAN follows DCGAN. It uses Adam at learning rate 0.0002, β1 0.5, batch size 64 and 25 epochs 1611.06355 §4.3. The paper's encoder optimiser is the same as the cGAN's, but the released code uses other values; see the card's Paper vs code section.
• Encoder design: of three ways to couple the two encoders, two independent networks (IND) gave the lowest reconstruction loss L_e on 150K generated CelebA test images: 0.429, against 0.432 for IND-COND and 0.500 for a shared-layer encoder 1611.06355 §5.3.
• Where the label enters: the attribute vector goes in at the generator's input and after the first convolution of the discriminator. The authors chose these positions from their comparison of insertion points [arxiv:1611.06355 §5.2] tab. 3.
• Labels for E_y: E_y is trained on real images, not generated ones, because generated images gave noisier attribute estimates 1611.06355 §4.1.
• Reported results: edits are judged qualitatively. Reconstructions generally show the requested attribute, and unusual poses are noisier [arxiv:1611.06355 §5.4]. The generator table reports attribute accuracy of 74% to 90%, measured with a classifier the paper describes only by architecture [arxiv:1611.06355 §5.2] tab. 3.

Card: IcGAN. The card's Paper vs code section lists the differences between the paper and the released code, including the encoder optimiser and the loss normalisation.

Attribute-conditioned flow on StyleGAN W (StyleFlow)

The method fits one conditional continuous normalizing flow between StyleGAN's W codes and a standard normal. The attributes enter the flow's velocity field, so one network covers all attributes jointly 2008.02401 §5.1 §5.2. The training set is 10k generated samples, labelled with an attribute classifier 2008.02401 §5.1. The paper trains with Adam at learning rate 1e-3 and batch size 5 for 10 epochs 2008.02401 §5.3.

• Sampling: draw noise, run the flow in reverse with the target attributes, and decode the resulting W code 2008.02401 §6.1.
• Editing: project the image to W+, measure its attributes, invert the flow under the source attributes, and decode under the target attributes. Only the W+ rows that the edit covers are replaced; the rest come from the source 2008.02401 §6.2 §6.2.3.
• Sequential edits: a fast mode skips re-projection after each edit and the paper says it can produce sudden jumps. An accurate mode re-projects all 18 vectors after every edit 2008.02401 §6.2.
• Reported results: on sequential pose, light and expression edits of generated images, FID is 53.15 for this method, against 64.69 for GANSpace, 67.08 for InterFaceGAN and 82.49 for Image2StyleGAN 2008.02401 tab. 1. Baselines were partly retrained by the authors 2008.02401 §7.3.
• Cost: inference is 0.61 s with four flow blocks and 0.21 s with six, on a Titan XP. The paper does not say what one timed call covers 2008.02401 §5.3 tab. 4.

Card: StyleFlow.

CLIP-guided latent optimisation (VQGAN-CLIP)

A pretrained VQGAN decoder turns a grid of codebook vectors into an image. CLIP embeds the image's augmented crops and the prompt, and the loss is a squared spherical distance between them 2204.08583 §2 §2.2 §2.3. Each iteration decodes the grid, averages the loss over many random crops, and takes one Adam step on the grid. The paper uses learning rate 0.15 and 400 iterations 2204.08583 §2.

• Editing: the start point is the encoded input image instead of random codes. The paper's only editing comparison with Open-Edit is visual, with no metric reported 2204.08583 §2 §4.1.
• Regularisation: an optional L2 penalty on the latent is described for a regularised variant. The paper gives no numeric weight, and names a decay of 0.005 that the released repository does not reproduce 2204.08583 §2.4.
• Reported results: a human study on 12 prompts gave a mean of 4.6 out of 5, against 3.3 for the best GLIDE baseline. The baselines were picked best of five, this method was best of one 2204.08583 tab. 1 §3.2.
• Cost: no training, so 0 V100-hours 2204.08583 tab. 2. Runtime is 188.3 s per image on a V100 and 654.3 s on a P100 2204.08583 tab. 2. A Colab run takes under 3 minutes and under 11 GB of GPU memory 2204.08583 §6.1.

Card: VQGAN-CLIP. The card's Paper vs code section lists where the released defaults differ from the paper's values.

Text-only generator adaptation (StyleGAN-NADA)

This route does not edit a given image. It retrains a copy of a pretrained StyleGAN2 to draw a new domain. A frozen copy supplies the source images, and the trained copy is pushed so that its change in CLIP space points along the text direction from source to target prompt. The loss is one minus the cosine between the two directions 2108.00946 §4.1 eq. 2. For shape changes, only a few layers are updated per iteration, chosen by a short latent optimisation 2108.00946 §4.2.

• Training cost: texture-type changes take about 300 iterations, roughly 3 minutes on one V100. Animal changes take 2000 iterations plus a latent mapper, about 6 hours in total 2108.00946 app. I.
• Reuse of latent-space tools: the paper says inversion, latent-direction editing and conditional encoders built for the source generator work on the adapted one without retraining 2108.00946 §5.2 fig. 8 fig. 9 fig. 10. This is the link to the other variants.
• Reported results: dog to cat FID with 10 target images is 41.22 with zero-shot pre-training, against 45.13 for the best few-shot baseline shown 2108.00946 tab. 2. A user study is reported in the same paper §5.1 fig. 12. These are author-reported.

Card: StyleGAN-NADA. The card's Paper vs code section lists where the released defaults differ from the paper's values, including the layer-selection and mapper settings.

Per-photo generator tuning (PTI)

The method inverts a real face into StyleGAN2's native W space with the generator frozen. That code, the pivot, is then held fixed while the whole generator is fine-tuned from its pretrained weights until the pivot reproduces the photo 2106.05744 §3.1 §3.2. The paper runs 450 inversion steps and up to 350 tuning steps 2106.05744 §4.5. Edits then run on the tuned copy.

• Locality term: each tuning step also samples a random code, moves the pivot a fixed distance towards it, and asks the tuned generator to match the original there. The weight is 0.1 and the distance is 30 [arxiv:2106.05744 §3.3] §4.5. In the paper's sweep, a distance of 2 gives little regularisation and 120 gives unrealistic images [arxiv:2106.05744 app. Locality Regularization].
• Pivot quality: random or mean codes as the pivot fail to converge, so the inversion step is what the method depends on 2106.05744 §3.2 §4.4.
• Reported results: on CelebA-HQ reconstruction, LPIPS is 0.09 against 0.34 for the best baseline, SG2 W+. ID similarity is 0.9 against 0.85 2106.05744 tab. 1. For single edits of smile, pose and age, ID similarity is 0.9 against 0.85 2106.05744 tab. 2. These are author-reported.
• Cost: under 3 minutes per identity on one RTX 2080, with inversion about 1 minute 2106.05744 §4.5 §5. Each identity keeps its own full copy of the generator, so storage grows with the number of identities. The card's Pitfalls section gives the details.
• Scope: the method is shown only for StyleGAN2 on faces. Other generators are named as future work 2106.05744 §5.

Card: PTI. The card's Paper vs code section lists where the released code differs from the paper, including the inversion learning rate and whether the locality term is on by default.

Noise-map embedding with masks (Image2StyleGAN++)

The method embeds a photo in two stages. First it optimises a W+ code, with the noise maps fixed. Then it optimises the per-layer noise maps, with the code fixed. Optimising both together gives high-frequency artifacts 1911.11544 §4.2 fig. 2. The paper gives the noise maps the high-frequency detail, and the W+ code the semantic content [arxiv:1911.11544 §4]. Masks confine the loss to one region, and copying intermediate activations from a second embedding blends the two 1911.11544 §5 §7.

• Reconstruction: the authors report 39 to 45 dB PSNR with both stages. The W+ stage alone gives 19 to 22 dB, against about 20 dB for the previous best 1911.11544 §1 §7.1 fig. 4. The full schedule is 5000 W+ steps and 3000 noise steps 1911.11544 §7.1.
• Local edits: the paper applies the same operations to inpainting, two-photo crossover, scribble-guided edits and local style transfer [arxiv:1911.11544 §7.2] §7.3 §7.4 §7.5. Its inpainting table reports SSIM, MSE and PSNR at 256, 512 and 1024 px. At 1024 px the method scores 0.9176 SSIM, against 0.8957 for partial convolution and 0.8693 for gated convolution 1911.11544 tab. 1.
• Cost: one 32 GB V100 1911.11544 §6. The paper gives no run time per image.
• Evaluation scale: inpainting uses 10 ground-truth images and 7 masks, and the edit-realism test uses one face-attribute classifier [arxiv:1911.11544 §9.1] §9.3. The paper's own limitations include lighting, age and skin colour 1911.11544 §9.1.

Card: Image2StyleGAN++. The card's Pitfalls section notes that the number of pixels that the loss divides by is not defined in the paper.

Motion latent space (MotionCLIP)

This variant is not an image editor. The paper trains a transformer auto-encoder on SMPL human motion. Its latent vector is pushed to match CLIP's embedding of the motion's text label and of one rendered frame, using cosine distance 2203.08063 §3. CLIP stays frozen, so a text prompt can be encoded by CLIP and decoded straight into a motion with no text-motion training 2203.08063 §4.2.

• Latent operations: the paper shows linear interpolation between codes, and arithmetic in which the CLIP vector of a style word is added to a content code. It also reports action recognition by comparing the code with the embeddings of 60 class names 2203.08063 §4.3. These latent results are qualitative.
• Text-to-motion: in user studies, MotionCLIP is preferred over JL2P on 76.7% of in-domain prompts and 75.3% of out-of-domain prompts 2203.08063 tab. 1. The paper does not state the number of participants or samples.
• Style transfer: against Aberman et al., which takes motion inputs, the average preference for MotionCLIP is 37.9% 2203.08063 tab. 2.
• Action recognition: top-1 is 40.9% against 41.14% for 2s-AGCN, while top-5 is 57.71% against 73.18% 2203.08063 tab. 3. Without the text loss, top-1 falls to 4.54% 2203.08063 tab. 3.

Card: MotionCLIP. The card's Paper vs code section lists differences from the released code, including the loss weights. Its data licences are not verified and need checking before use.

Conditional adversarial autoencoder for face age (CAAE)

The method learns from faces without paired photos of one person at different ages. An encoder maps a 128 x 128 face to a vector z, and a generator renders the vector with an age label appended 1702.08423 §4.1. The age label is one-hot over 10 bins 1702.08423 §5.1. A discriminator on z pushes the codes toward a uniform prior, and an image discriminator judges realism conditioned on the age label §4.3 §4.4. At test time only the encoder and generator run, and the input's true age is not needed §5.2. Keeping z fixed and changing only the age label is the editing step §3 fig. 2.

• Training data: about 10,670 images from Morph, CACD and crawled sources, with ages and genders estimated from captions or an age estimator 1702.08423 §5.1. The paper sets lambda = 100 and gamma = 10 on the reconstruction and total-variation terms §5.2. The text does not state the latent size; the figure shows 50 fig. 3.
• Reported results: a user study, not a recognition score. Against prior work, 52.77% of votes favour CAAE, 28.99% favour the prior work and 18.24% judge them equal 1702.08423 §5.3. Against ground truth, 48.38% judge the output the same person §5.3. The paper reports no measure of whether the requested age appears.
• Vote counts: the paper reports 3,208 votes from 63 volunteers, who answered 45 questions each. That allows at most 2,835 votes, and the paper does not explain the gap 1702.08423 §5.3.
• Cost: not stated in the paper. The repository README gives about 2.5 hours for 50 epochs on UTKFace on one TITAN X README.md. That run uses a dataset other than the paper's.

Card: CAAE. The card's Paper vs code section lists the differences from the released code. The repository has no LICENSE file, so its code cannot be reused on that basis.

Text-driven StyleGAN editing with CLIP (StyleCLIP)

All three routes share one pretrained StyleGAN2 and a CLIP model. They push the image toward the prompt with CLIP's cosine distance, and keep it near the source with a penalty on the latent 2103.17249 §4 eq. 1 eq. 2. They differ in what runs per image and what runs per prompt.

• Route 1, per-image optimisation: the W+ code is optimised for each image and prompt, with 200 to 300 iterations 2103.17249 §4. Inference takes 98 s on a GTX 1080Ti 2103.17249 tab. 1. The paper calls the method sensitive to its parameters §5 §7.
• Route 2, latent mapper: one small network per group of W+ layers (coarse, medium and fine) predicts a residual, and the edited code is the input plus that residual 2103.17249 §5 eq. 3. Each prompt needs its own training run of 10 to 12 h, after which an edit takes 75 ms 2103.17249 tab. 1.
• Route 3, global direction in style space S: a text direction is built from prompt templates, and only the style channels that respond most to it are moved 2103.17249 §6 eq. 6 eq. 7. The step is G(s + α Δs), so α is a dial. Pre-processing takes 4 h, and each edit takes 72 ms, with no dependence on the input image 2103.17249 tab. 1.
• Reported results: qualitative. The paper gives no image-quality, identity or user-study metric 2103.17249 §7. It says the global direction is similar to StyleSpace, which changes only the target attribute, and that TediGAN fails on all three of its comparison examples §7 app. B.

Card: StyleCLIP. The card's Paper vs code section lists where the released code differs from the paper, including the mapper's step scale and the global channel selection. The code is MIT-licensed, but the pretrained weights are downloaded separately and their licence is not covered here.

Point dragging with latent optimisation (DragGAN)

The user places handle points on a StyleGAN2 image and gives each a target. Each iteration takes one optimiser step on the latent code. The loss pulls the features after the sixth generator block, near each handle, one step toward its target. Then each handle moves to the location whose feature best matches its original feature 2305.10973 §3.1 §3.2 §3.3. The loop stops when every handle is within a pixel tolerance of its target: 1 px for up to five handles, 2 px for more §3.4. The paper reports 30 to 200 iterations in typical use §3.1. The paper says no optical-flow or tracking network is used §1 §3.

• Latent space: only the first six layers of a W+ code are optimised, and the rest stay fixed, which the paper says keeps the appearance unchanged 2305.10973 §3.2. W is available for a more constrained result §3.2 fig. 16.
• Masks: a user-drawn binary mask keeps unmasked features near their initial values, with weight λ = 20 2305.10973 §3.2 §3.4.
• Real images: a real photo is first inverted with PTI, and the same loop then runs 2305.10973 §4.1. See the PTI card for that step.
• Reported results: on a face-landmark test of 1000 cases with one handle, the mean landmark distance is 2.44, against 11.64 for UserControllableLT. FID is 9.28, against 25.32 2305.10973 tab. 1. The distance has no unit in the paper. Reconstruction error is also lower than the baseline's on all four datasets in the paper's table 2305.10973 tab. 2.
• Cost: 2.0 s per edit for one point, against 0.03 s for UserControllableLT 2305.10973 tab. 1. The paper names one RTX 3090 "in most cases", but does not say that the timings were measured on it 2305.10973 §1 tab. 1.
• Tracker choice matters: on the same test, RAFT tracking gives 13.43 for one point, which is worse than the baseline's 11.64. PIPs tracking gives 2.98 2305.10973 tab. 1.
• Limits: the authors report artifacts for out-of-distribution human poses, and handles in texture-less regions drift 2305.10973 §4.3 fig. 14.

Card: DragGAN. The card's Paper vs code section records that the authors' repository carries a research-only licence, so the card uses the paper alone. Its Pitfalls section lists details the paper leaves open, such as how the loss sums are normalised and the units of the stopping tolerance.

When to use which

Attribute control on real photos adds an inversion step. The StyleFlow paper uses an Image2StyleGAN-style projection into W+ 2008.02401 §3 §6.2. Plan for that step separately, because it sets the identity ceiling for every later edit. IcGAN replaces the per-image projection with trained encoders 1611.06355 §4.1. Its edit results are qualitative, so the paper gives no measure of how well identity survives an edit 1611.06355 §5.4. PTI and Image2StyleGAN++ keep per-photo optimisation instead. PTI adds a tuning step after inversion, and Image2StyleGAN++ adds noise maps to the W+ embedding 2106.05744 §3.2 1911.11544 §4.2. StyleCLIP's per-image route also inverts first, with e4e, and its mapper and global routes work on that inverted code 2103.17249 §4 §5 §6. CAAE has no per-image step: its encoder runs once, as in IcGAN 1702.08423 §5.2. DragGAN also inverts a real photo first, with PTI, and then optimises per edit 2305.10973 §4.1.

Implementation sketch

Common core of the editing routes. Variant-specific parts are marked in the comments. The text-only adaptation route trains a generator and has no editing step, so it is not in this sketch. The motion route is not an image editor and is not in this sketch either.

# Latent-space editing, common core  [arxiv:2008.02401 §6.2] [arxiv:2204.08583 §2]
# Input: optional source image, edit specification
# Output: edited image

def edit(image, edit_spec):
    # Start point: a latent that reproduces the input, or a random one
    # StyleFlow projects to W+; VQGAN-CLIP encodes into the latent grid
    # IcGAN instead runs trained encoders E_z and E_y [arxiv:1611.06355 §4.1]
    # PTI inverts to W, then tunes the generator weights [arxiv:2106.05744 §3.2]
    # Image2StyleGAN++ adds per-layer noise maps [arxiv:1911.11544 §4.2]
    # StyleCLIP route 1 inverts with e4e, then optimises W+ [arxiv:2103.17249 §4]
    # CAAE: one encoder pass gives z; the age label is then changed [§5.2]
    if image is not None:
        z = invert(image)
    else:
        z = sample_latent()
    # Move the latent toward the edit
    # StyleFlow re-decodes under the target attributes [§6.2]
    # VQGAN-CLIP takes gradient steps on a CLIP score [§2]
    # DragGAN steps W+ layers 0-5 toward handle targets, then re-tracks
    # the handles [arxiv:2305.10973 §3.2] [§3.3]
    # IcGAN keeps z and swaps the attribute vector y [arxiv:1611.06355 §5.4]
    z_edit = change_latent(z, edit_spec)
    # Decode; StyleFlow copies untouched W+ rows back from the source [§6.2.3]
    # IcGAN has no such copy step; it decodes G(z, y_new) directly
    # PTI decodes with its tuned generator, not the original [§3.2]
    return decode(z_edit, keep=z)

The inversion, the update and the output step differ most between the editing routes. Keep those three in mind when you pick an implementation, and check the cards' Paper vs code and Pitfalls sections before you copy any default.

Contradictions

No contradictions between the papers' claims are known. Their numbers do not measure the same thing and must not be compared. StyleFlow reports FID and face identity on face data 2008.02401 tab. 1 tab. 2. IcGAN reports reconstruction loss and an attribute classifier's accuracy on CelebA 1611.06355 §5.3 tab. 3. VQGAN-CLIP reports human ratings on 12 text prompts 2204.08583 tab. 1. StyleGAN-NADA reports FID on 10 and 100 target images and a user study on a different task 2108.00946 tab. 2 §5.1. PTI reports LPIPS, MSE and identity similarity on CelebA-HQ 2106.05744 tab. 1. Image2StyleGAN++ reports PSNR on its own reconstructions and inpainting metrics on its own test set [arxiv:1911.11544 §7.1] tab. 1. MotionCLIP reports user preference and BABEL-60 accuracy on motion data 2203.08063 tab. 1 tab. 3. CAAE reports vote shares from a user study on identity 1702.08423 §5.3. StyleCLIP reports timings and visual comparisons only 2103.17249 tab. 1 §7. DragGAN reports its landmark distance in a unit the paper does not state, and reconstruction MSE and LPIPS on its own test set 2305.10973 tab. 1 tab. 2.

Inside one paper:

• PTI's reconstruction table marks MS-SSIM with a down arrow and gives its own method as best at 0.21. Standard MS-SSIM rises with similarity, so the arrow or the values may be inverted 2106.05744 tab. 1. Neither reading is resolved here.
• VQGAN-CLIP gives the GLIDE training cost as 19,200 V100-hours in the text, but 20,200 in its break-even discussion and in the Table 2 caption 2204.08583 §6.2 tab. 2. Neither figure is resolved here.
• VQGAN-CLIP gives a runtime for an RTX 2080 Ti in the text, but the runtime table has no such row 2204.08583 §6 tab. 2.
• StyleCLIP gives its own per-image time as "several minutes" in the text, but 98 s in its timing table 2103.17249 §4 §5 tab. 1. Budget for the larger figure until you measure your own setup.
• CAAE's vote count, 3,208, is larger than its stated volunteers and questions allow 1702.08423 §5.3. Neither figure is resolved here.
• IcGAN's text gives an overall generator accuracy of about 86%, but the best generator value in its table is 89.83% 1611.06355 §5.2 tab. 3. Neither figure is resolved here.

Papers

• IcGAN: a conditional GAN with two trained encoders, so a real image can be inverted and its attributes changed.
• StyleFlow: a conditional flow on StyleGAN's W space for attribute-targeted sampling and editing.
• VQGAN-CLIP: text-guided generation and editing by optimising a VQGAN latent grid against CLIP, with no training.
• StyleGAN-NADA: text-only adaptation of a StyleGAN2 generator to a new domain, so latent-space tools can run on the adapted generator.
• PTI: inverts a face into StyleGAN2's W space, then tunes the generator so the fixed code reproduces the photo.
• Image2StyleGAN++: embeds a photo into W+ and noise maps, with masks for local edits and inpainting.
• MotionCLIP: aligns a human-motion latent space with CLIP, so text prompts decode to motion.
• CAAE: a conditional adversarial autoencoder that changes a face's age by changing an age label, trained without paired ages.
• StyleCLIP: CLIP-driven edits of StyleGAN2 images in three routes: per-image optimisation, a per-prompt latent mapper, and a global style-space direction.
• DragGAN: drags chosen points of a StyleGAN2 image to targets by optimising its latent, with nearest-neighbour point tracking and no tracking network.