Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold

In plain words

You edit a picture that a GAN (a generator network that paints realistic images) has made. You click a few start points on the picture and say where each one should go. The method gently changes the generator's internal code so that the image content at each start point slides toward its target. After each change it looks for where that point has moved to, and it repeats until the points arrive. You can paint a mask over parts of the picture that should stay still.

Because the change happens inside the generator, the result tends to look like a plausible photo rather than a stretched image. The method also works on real photos, if they are first mapped into the generator. It does not need a separate tracking network.

TL;DR

DragGAN moves chosen points of a pretrained StyleGAN2 image to chosen targets. Each iteration takes one optimiser step on the latent code, using a loss on the features of the sixth generator block, and then re-locates each handle by nearest-neighbour search in those same features. No optical-flow or tracking network is used. The authors report a mean landmark distance of 2.44 with one point, against 11.64 for UserControllableLT, at about 2.0 s per edit.

Problem

Controlling a GAN has usually needed manually annotated data or a prior 3D model. Both limit flexibility, precision and generality §1. Text guidance is imprecise for spatial edits and cannot, for example, move an object by a set number of pixels §1. The closest earlier method, UserControllableLT, supports dragging but does not handle more than one point well and does not require handles to reach their targets §1.

The paper's bet is that intermediate GAN features are discriminative enough to serve two jobs: supervising a movement, and tracking a point across steps §1, §3.2. Edits stay on the generator's learned image manifold, so the paper expects plausible results, such as hallucinated occluded teeth or a bent horse leg, rather than the stretching that plain warping produces §1.

Method

The method assumes a pretrained StyleGAN2 generator. A 512-dimensional latent z maps to an intermediate code w through a mapping network. The code w is fed to the generator's layers, and it can be a single code (space W) or one code per layer (space W+) §3.

The user gives n handle points p_i and n target points t_i, and may draw a binary mask M that marks the movable region §3.1. The optimisation alternates two sub-steps §3.1:

1. Motion supervision: one optimiser step on w, using a loss that pushes features near each handle toward its target §3.2 eq. 1.
2. Point tracking: each handle is moved to the location whose feature best matches its original feature §3.3 eq. 2.

The loop stops when every handle is within a tolerance of its target §3.4. The paper reports 30 to 200 iterations in typical use. The user can stop at any step, and can then add new handle and target pairs and continue §3.1. The exact length of each motion step is not known in advance, because it depends on the object and the part §3.1.

Motion supervision uses the feature maps F after the sixth block of StyleGAN2, resized to the image resolution by bilinear interpolation §3.2. The loss is:

\mathcal{L} = \sum_{i} \sum_{q_i \in \Omega_1(p_i, r_1)} \lVert F(q_i) - F(q_i + d_i) \rVert_1 + \lambda \lVert (F - F_0) \cdot (1 - M) \rVert_1 eq. 1

with d_i = (t_i − p_i) / ‖t_i − p_i‖_2 (d_i = 0 when t_i = p_i) and F_0 the features of the initial image eq. 1 §3.2. The gradient is not passed back through F(q_i), so the loss moves the handle and not the target §3.2. The second term keeps unmasked pixels at their initial features §3.2.

Only the first six layers of w are optimised. The remaining layers stay fixed, which keeps the appearance unchanged §3.2. The authors favour W+ because it makes out-of-distribution edits easier; W is available for a more constrained result §3.2 fig. 16.

# DragGAN edit loop  [§3.1]
# generator: pretrained StyleGAN2; w: latent code (layers, 512) in W+ [§3.2]
# handles: (n, 2) start points as (x, y); targets: (n, 2) end points
# mask: (H, W) with 1 = movable, 0 = fixed, or None  [§3.1]
def drag(generator, w, handles, targets, mask):
    # Features of the initial image: F_0, and the reference feature f_i  [§3.3]
    feats_init = block6_features(generator, w)
    refs = [feats_init[:, y, x] for x, y in handles]
    feats = feats_init
    # Repeat until every handle is within d of its target  [§3.4]
    while not reached(handles, targets, threshold):
        # Motion supervision: one step; only layers 0-5 of w move  [§3.2]
        loss = motion_loss(feats, feats_init, handles, targets, mask)
        w = optimiser_step(w, loss)  # Adam, see Hyperparameters  [§3.4]
        # Features after the update: F'  [§3.3]
        feats = block6_features(generator, w)
        # Point tracking: move each handle to its nearest feature match  [§3.3]
        handles = track(handles, refs, feats)
    return generator_image(generator, w), w

Motion loss, as written in eq. 1, step by step:

# Motion supervision loss  [eq. 1]  [§3.2]
# feats:      block-6 features of the current latent, (C, H, W)
# feats_init: the same features before any edit, F_0, (C, H, W)
# mask:       (H, W), 1 = movable, 0 = fixed; None if not given  [§3.1]
def motion_loss(feats, feats_init, handles, targets, mask):
    loss = 0.0
    for handle, target in zip(handles, targets):
        # Unit direction from handle to target; zero if they coincide  [§3.2]
        direction = unit(target - handle)
        # Every pixel q whose distance to the handle is below r1  [§3.2] [§3.4]
        for q in pixels_within(handle, r1):
            # Feature one step along the direction, bilinear sample  [§3.2]
            shifted = sample(feats, q + direction)
            # No gradient through q's feature, so only the handle moves  [§3.2]
            loss += sum(abs(stop_gradient(feats[q]) - shifted))
    if mask is not None:
        # Penalise change in the fixed region, weight lambda = 20  [§3.2] [§3.4]
        loss += lam * sum(abs((feats - feats_init) * (1 - mask)))
    return loss

Point tracking, as written in eq. 2, step by step:

# Point tracking by nearest neighbour in a square patch  [eq. 2]  [§3.3]
# feats: block-6 features after the latest step, F'  [§3.3]
# refs:  reference feature f_i = F_0(p_i) for each handle  [§3.3]
# r2:    patch half-width in pixels  [§3.4]
def track(handles, refs, feats):
    tracked = []
    for handle, ref in zip(handles, refs):
        # Candidate pixels in the square around the handle  [§3.3]
        candidates = square_around(handle, r2)
        # Candidate with the nearest feature to the reference, L1  [eq. 2]
        best = min(candidates, key=lambda q: sum(abs(feats[q] - ref)))
        tracked.append(best)
    return tracked

Real images are edited by first inverting them into the generator, which the paper does with PTI, and then applying the same loop §4.1 fig. 5.

Hyperparameters

Results

Author-reported. Setup: StyleGAN2 pretrained on FFHQ, AFHQCat, SHHQ, LSUN Car, LSUN Cat, Landscapes HQ, a microscope set, and the self-distilled Lion, Dog and Elephant models §4. Main baseline: UserControllableLT §4. Tracking variants replace the tracker with RAFT or PIPs §4.

Qualitative results: side-by-side edits against UserControllableLT fig. 4 §4.1; tracking against RAFT and PIPs, where the other trackers drift off the handle fig. 6 §4.1; and face-landmark edits, where the method opens the mouth and reshapes the jaw to match the target fig. 7 §4.2.

Face-landmark manipulation, 1000 tests, mean distance (MD) between edited and target landmarks. Table 1 times and FID use the 1-point setting tab. 1 §4.2. The unit of MD is not stated.

Paired image reconstruction, following UserControllableLT, averaged over 1000 samples. Values are MSE (×10²) and LPIPS (×10), lower is better tab. 2 §4.2. Each cell reads MSE / LPIPS.

Ablation on the feature block, MD for one-point face landmarks tab. 3 §4.2:

Ablation on r1 (MD) tab. 4 §4.2: r1 = 1, 2, 3, 4, 5 gives 2.49, 2.51, 2.44, 2.45, 2.46. The table does not state the number of landmark points. Its r1 = 3 value (2.44) equals the one-point figure in tab. 3. The authors say performance is not very sensitive to r1 §4.2.

Limits and discussion: a mask keeps the background fixed fig. 8 fig. 15. Out-of-distribution edits, such as an extremely open mouth or an enlarged wheel, are possible §4.3 fig. 9. Limits: a human pose outside the training distribution can produce artifacts §4.3 fig. 14a, and handles in texture-less regions drift more in tracking §4.3 fig. 14b–c.

Paper vs code

No code comparison was made. The authors' repository, XingangPan/DragGAN, is linked from the project page. Its LICENSE is the NVIDIA Source Code License for StyleGAN3, which permits research and evaluation use only [code_license]. wikillm publishes with advertising, so this card uses the paper alone and does not use or cite the repository's code. The code, code_commit and code_license fields record the repository for reference only. Anyone who wants the released implementation should read it under the NVIDIA terms.

Pitfalls

• Index in eq. 1. The outer sum runs from i = 0 to n, but handles are numbered 1 to n §3.1 eq. 1. Read it as 1 to n.
• Sums or means. Eq. 1 writes sums over pixels and handles, and sets λ = 20 §3.4. The paper does not say whether the sums are normalised. That choice changes the real weight of the mask term and the size of the gradient, and λ = 20 is only meaningful once it is fixed.
• Two different "6"s. The features come from the 6th block §3.2, and the optimised part of w is the first 6 layers §3.2. Neither the block index base (0 or 1) nor the layer count of W+ is stated.
• Norm and window in tracking. Eq. 2 uses the L1 norm eq. 2. Ω2 is written with a strict inequality, |x − x_p| < r2 §3.3, so with integer offsets the square is 2·r2 − 1 pixels across, not 2·r2 + 1.
• Undefined norm in Ω1. The "distance" in the motion radius is not given a norm §3.2. Euclidean is the natural reading; the paper does not say so.
• Rounding of radii. r1 = round(3/512 × size) gives 1.5 at 256 px §3.4. The rounding rule for halves is not stated, so r1 at 256 px can be 1 or 2. r2 = 6 at 256 px is exact.
• Stopping threshold. The text uses 1 px for at most 5 handles and 2 px for more §3.4. The paper does not say whether the distance is measured in feature-grid cells or image pixels.
• Motion step length is unknown. The paper says each motion step moves a handle by a small step whose exact length is not known §3.1. The 30 to 200 iteration count is an observation, not a stopping rule.
• Zero direction. When a handle equals its target, d_i = 0 and that handle gets no motion term; only the mask term still acts on it §3.2.
• "If needed" in tracking. Tracking features are 256 × 256 and are resized "if needed" §3.3. The paper does not say when this applies.
• Units of MD. The landmark MD has no unit in the paper. "Pixel" appears only in the stopping tolerance §3.4 tab. 1.
• Timing hardware. The single RTX 3090 is named in the introduction §1. It is not stated that Table 1's times were measured on it.
• Cost per step (inferred). Each iteration runs the generator and back-propagates to w §3.2. That is inferred from the method, not stated in numbers. The paper claims "a few seconds" per edit §1 and 2.0 s for one point tab. 1.
• Reconstruction protocol depends on another paper. Latent perturbation and flow sampling follow UserControllableLT §4.2; this card does not restate those settings. The user input is 32 flow-sampled pixels §4.2.
• Real images depend on inversion. The paper uses PTI and gives no inversion settings beyond citing it §4.1 fig. 5. Results on real images depend on the inversion quality.
• Texture-less handles drift. The authors advise texture-rich handles §4.3 fig. 14.
• Out-of-distribution edits. Edits can leave the training distribution, and the limitations paragraph reports artifacts for out-of-distribution human poses §4.3 fig. 14a. The authors mention adding latent regularisation as a possible control, but it is not part of the method and is not tested §4.3.
• Unstated details. Random seed for the starting image, truncation, noise mode, batch size, brush size, mask resolution, and how two handles that pull in opposite directions are resolved are not stated.
• Misuse. The authors warn that the method can create images of real people with fake poses or expressions, and that its use must respect personality rights and privacy law §4.3.

Lineage

• Builds on: StyleGAN2 §3; the style-mixing idea behind optimising only some layers §3.2; PTI for real-image inversion §4.1.
• Compared against: UserControllableLT §4; point trackers RAFT and PIPs §4.
• Superseded: not known from this wiki.