StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

In plain words

Image generators such as StyleGAN can produce very realistic faces, cars and animals, but changing one of them in a chosen way usually means finding a control for it first. Earlier methods needed a human to hunt for useful directions or a labelled dataset for each attribute. This paper lets the user type the change instead, such as "mohawk hairstyle" or "surprised", and applies it to a picture.

The link between the two models is CLIP, a model trained to match pictures with their captions. The paper uses CLIP's sense of whether a picture matches a sentence as the signal that steers the generator. It offers three ways to do this, from the most flexible but slowest to the fastest but more limited. The slowest one searches for a new setting of the generator's input for each picture and sentence. The fastest one trains a small helper network for one sentence, so that any picture can be edited in a single step. The third finds one global change for a sentence, which can be dialled up or down and made more or less targeted.

The authors show many edits, from hairstyles and expressions to changes between animal species, and compare them with earlier methods. Their comparisons are visual; the card's results section lists the timing numbers they report.

TL;DR

StyleCLIP drives a pretrained StyleGAN2 with CLIP's cosine distance to a text prompt §4 eq. 1. It offers three routes: per-image latent optimisation §4, a small per-prompt residual mapper in W+ that gives one latent step per input §5 eq. 3, and a global direction in style space S derived from per-channel CLIP relevance §6 eq. 6 eq. 7. Use it when you need text-described edits to StyleGAN images and can accept a per-prompt training run (mapper) or a per-channel scan (global direction). The paper reports no quantitative image-quality metric.

Problem

Existing StyleGAN editing needs either manual inspection of the latent space or annotated data for each attribute, so new edits are costly to add §1. Edits along preset directions are also limited to what was prepared in advance §1. Learned directions in the W+ latent space are harder to disentangle, so fine-grained edits affect other attributes §3.

CLIP can express far more visual concepts in words, but a generator edit driven by CLIP has to balance three goals: reach the text, keep the input identity, and avoid unrelated changes §4 §5. The paper's own limitations are that edits stay inside the domains of StyleGAN and CLIP, and that prompts describing sparsely covered concepts may not match the text §7.

Method

Overview. Three methods share one pretrained StyleGAN2 generator and a CLIP model §4 footnote 1. Real images are first inverted into the latent space with e4e §4 §5 §6. The table below summarises the three routes as the authors report them tab. 1.

Route 1: latent optimisation (per image, per prompt) §4.

1. Start from the source latent code w_s in W+.
2. Run gradient descent on w. Each step renders the generator output, scores it with the CLIP cosine distance to the prompt, adds the L2 distance to the source latent and, optionally, an identity term eq. 1 eq. 2.
3. The paper runs 200–300 iterations §4. The loss weights depend on the edit fig. 3.

# Latent optimisation for one (image, prompt) pair  [§4] [eq. 1] [eq. 2]
# Input:  source latent w_s (18, 512) in W+, prompt t
# Output: edited latent w (18, 512)

def edit_by_optimisation(w_s, prompt, lambda_l2, lambda_id, steps):
    # Start the search from the source latent  [§4]
    w = copy(w_s)
    for step in range(steps):
        # Render the candidate through the frozen generator  [§4]
        image = generator(w)
        # Cosine distance between CLIP embeddings of image and prompt  [§4]
        clip_term = clip_cosine_distance(image, prompt)
        # Distance from the source latent, as written in eq. 1  [eq. 1]
        latent_term = norm(w - w_s)
        # Identity term, only for edits that keep the person  [eq. 2]
        id_term = 1 - cosine(arcface(generator(w_s)), arcface(image))
        loss = clip_term + lambda_l2 * latent_term + lambda_id * id_term
        # Gradient step on w; the optimiser and schedule are not stated  [§4]
        w = w - learning_rate * gradient(loss, w)
    return w

Route 2: latent mapper (one trained network per prompt) §5.

1. The 18 W+ layers are split into three groups: coarse, medium and fine. Each group has its own mapping network with 4 layers, not 8 §5.
2. For an input latent w, the three networks produce a residual M_t(w) eq. 3.
3. The edited latent is w plus the residual §5. The loss combines a CLIP term eq. 4, a penalty on the size of the residual §5, and an identity term that is switched off for edits that change identity eq. 5 §5.
4. Training happens once per prompt. Applying the mapper to a new input is a single forward pass §3 tab. 1.

# Loss for one training step of the text-guided latent mapper  [§5] [eq. 5]
# Input:  latents w (B, 18, 512), prompt t, loss weights
# Output: scalar loss

def mapper_loss(w, prompt, lambda_l2, lambda_id):
    # Split the 18 layers into coarse, medium and fine groups  [§5] [eq. 3]
    w_coarse, w_medium, w_fine = split_layers(w)
    # One small network per group gives the residual  [§5] [eq. 3]
    step = concat(
        mapper_coarse(w_coarse),
        mapper_medium(w_medium),
        mapper_fine(w_fine),
    )
    # The edited latent is the input plus the residual  [§5]
    w_edit = w + step
    image = generator(w_edit)
    # CLIP term: cosine distance to the prompt  [§5] [eq. 4]
    clip_term = clip_cosine_distance(image, prompt)
    # Size of the residual, which keeps the edit local  [§5]
    step_term = norm(step)
    # Identity term; use lambda_id = 0 for identity-changing edits  [§5]
    id_term = 1 - cosine(arcface(generator(w)), arcface(image))
    return clip_term + lambda_l2 * step_term + lambda_id * id_term  # [eq. 5]

Route 3: global direction in style space S (per prompt, input-agnostic) §6.

1. Build a text direction Δt. Encode the target and the neutral prompt with the ImageNet prompt-template bank, average each set of embeddings, and take the normalised difference §6.
2. Estimate each style channel's relevance. Sample style codes s. For channel c, set the channel to its mean plus or minus α times its standard deviation, render the pair, and take the CLIP image difference Δi_c. Average the projection of Δi_c on the target direction over 100 pairs, using α = 5 §6 eq. 6.
3. Keep the channels whose relevance is at least β in absolute value, and set the others to zero §6 eq. 7.
4. Edit by moving the style code: G(s + α Δs). The strength α is the user's dial and β sets how many channels move §6.

# Global text-driven direction in style space S  [§6] [eq. 6] [eq. 7]
# Input:  target and neutral prompts, style codes, beta
# Output: direction delta_s with one entry per style channel

def global_direction(target, neutral, codes, beta):
    # Prompt-ensembled, normalised difference in CLIP text space  [§6]
    text_dir = normalise(
        mean_embedding(target) - mean_embedding(neutral)
    )
    relevance = zeros(num_channels)
    for channel in channels:
        # Perturb one channel by +/- 5 channel standard deviations  [§6]
        diffs = [clip_difference(s, channel, alpha=5) for s in codes]
        # Mean projection, eq. 6. The paper projects onto the image-space
        # direction; the code uses text_dir (see Paper vs code)  [eq. 6]
        relevance[channel] = mean(dot(diffs, text_dir))
    # Zero out channels below the threshold  [eq. 7]
    delta_s = where(abs(relevance) >= beta, relevance, 0)
    return delta_s

Hyperparameters

Values stated in the paper, with the code's defaults where the paper is silent. Code values are shown only for orientation; see Paper vs code.

Results

Author-reported. Setup: StyleGAN2 and StyleGAN2-ada generators for faces, cars, dogs, cats and churches; real images inverted by e4e §4 §5 §6 app. B. Timings are on a single NVIDIA GTX 1080Ti tab. 1.

Timing and inputs tab. 1:

Mapper direction similarity across the inverted CelebA-HQ test set tab. 2. Mean and standard deviation of the cosine similarity between the step directions produced for different inputs:

The "Surprised" standard deviation is printed as 0.893 tab. 2. The other values are 0.081–0.145, so this is probably a typo, but the paper does not say.

Qualitative comparisons only. The paper shows no quantitative image-quality, identity or user-study metric. Its own observations are: the global direction is similar to StyleSpace, which changes only the target attribute §7 app. B; the mapper is more successful for complex, identity-linked prompts such as "Trump" §7 fig. 10; the global direction removes wrinkles while the mapper fails to §7 fig. 10; and the global Mohawk edit is slightly less pronounced than the mapper's §7. Comparisons with GANSpace, InterFaceGAN and StyleSpace match the change in a pretrained classifier's logit app. B. The paper reports that TediGAN fails on all three examples in its comparison §7 fig. 10.

Paper vs code

Code: orpatashnik/StyleCLIP at commit f87a47f7fc39e19a1f6c88293875692b303bf168, MIT licence LICENSE. The repository is owned by an author of the paper, and the paper's own footnote links it. The repository ships licence files for two third-party components, stylegan2-pytorch and CLIP LICENSE-stylegan2-pytorch LICENSE-CLIP. The pretrained weights are downloaded separately and are not covered here.

Discrepancies, each with both citations:

• Mapper step scale. The paper defines the edited latent as w + M_t(w) §5. The training code uses w + 0.1 · M(w) coach.py, and the validation loop does the same coach.py.
• Mapper size penalty. The paper penalises the L2 norm of the step §5. The code takes the mean squared error between the edited and the input latent, which is a mean rather than a norm and includes the 0.1 factor coach.py coach.py.
• Optimisation latent penalty. Eq. 1 uses the (non-squared) L2 norm of w − w_s eq. 1. The code uses the sum of squared differences run_optimization.py.
• Global relevance. The paper projects the per-channel CLIP image difference onto the image-space direction Δi §6 eq. 6. The code projects onto the normalised text direction instead MapTS.py MapTS.py. It also normalises each image embedding before differencing, normalises each difference, averages over the pairs and renormalises MapTS.py.
• Global thresholding. Eq. 7 keeps the raw relevance above β §6 eq. 7. The code sets the values below β to zero, then divides by the largest remaining absolute value so that the maximum is 1, and only then multiplies by the channel standard deviation MapTS.py.
• Text direction sign. The paper takes the difference between the target and neutral embeddings without fixing the sign §6. The code computes the first prompt minus the second MapTS.py. Its example lists the neutral prompt first MapTS.py.
• CLIP variant. The paper does not name the CLIP model §4 §5 §6. The code loads ViT-B/32 clip_loss.py MapTS.py.
• Identity loss. The paper uses a cosine similarity for the ArcFace term eq. 2. The code's ArcFace backbone returns L2-normalised features, so a dot product is the cosine model_irse.py.

Pitfalls

• Two different "L2" penalties. The paper's λ_L2 weights the norm of the step §5 and the norm of the latent offset eq. 1. The code applies λ_L2 to a mean of squares in the mapper and to a sum of squares in the optimiser. A mean of squares is much smaller than the norm of the same vector, so the code's λ defaults (0.8 for the mapper, 0.008 for optimisation) do not carry over to a norm-based loss. Re-tune λ if you change the penalty form.
• Undocumented 0.1 step factor. The mapper's trained residual is multiplied by 0.1 in the code and the paper does not mention it coach.py. The network can learn to compensate, but the effective step size and the penalty's effect differ from the paper's description.
• Time figures disagree. The text says optimisation takes "several minutes" per edit §4 §5, while Table 1 gives 98 s tab. 1. Budget for the larger figure until you measure your own setup.
• Optimisation is sensitive. The paper calls the method sensitive to its parameters and excludes it from the quantitative comparison for that reason §5 §7.
• The identity loss is a per-edit choice. The paper turns it off for edits that should change identity §5. Nothing in the method decides this automatically, so the implementer must set λ_ID per prompt.
• Global direction depends on the channel statistics. The step is scaled by each channel's standard deviation, which the paper does not describe. The code computes it from 100,000 samples GetCode.py, so the step size for a given α depends on that choice.
• β is not a single value. The paper gives no default. Figures use values between 0.11 and 0.16 fig. 6, the gender examples use 0.2–0.4 app. C, and the code's example uses 0.13 MapTS.py. Choose β per prompt and check how many channels remain active.
• Check the sign of the global direction. Confirm visually that positive α moves toward the target prompt. The paper does not fix the sign of Δt §6, and the code's example lists the neutral prompt first MapTS.py, so the sign may be the opposite of what you expect.
• Global directions fail on large domain gaps. The paper reports that tiger-to-wolf transforms, with a larger shape change, fail in some cases app. B.
• Mapper training data is not stated. The paper analyses the CelebA-HQ test set §5 but does not say what the mapper was trained on. The README points to an e4e-inverted CelebA-HQ training set README.md.
• Prompts that name people. The paper says prompts that map to sparsely covered areas of CLIP space may not yield faithful edits §7. Prompts naming a person, such as "Trump" or "Obama", behave differently from generic attributes, and the paper treats the identity term for them as a separate choice §5 §7.

Lineage

• Builds on: CLIP for the text-image signal §2 §4; StyleGAN2 as the generator §4 footnote 1; e4e for inversion of real images §4; ArcFace for the identity loss §4; StyleSpace, which the global method works in §6.
• Compared against: TediGAN, GANSpace, InterFaceGAN and StyleSpace §7 app. B; StyleFlow app. B.
• Superseded by: not known from content in this wiki.