To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now

In plain words

Text-to-image models such as Stable Diffusion can draw pictures that should not be made. Safety work tries to make a model forget a bad concept, such as nudity, a painter's style or a kind of object, so that it stops drawing it. This paper asks whether that forgetting holds up when someone adds a crafted piece of text to the prompt.

The attack puts a few invented tokens in front of the normal prompt. They are not meaningful words; they are picked to push the model back toward the content it was meant to forget. To pick them, the authors use the model itself as a judge: they show it a picture containing the forgotten content and check which tokens make the model fit that picture best. No second model and no external image classifier is needed.

The work is a stress test. It measures how much a forgetting fix really removes, and the authors report that several such fixes can be bypassed this way. The attack is also faster than the earlier method they compare against.

TL;DR

• Attack on text-to-image diffusion models that were unlearned to refuse a concept (nudity, a style or an object). The adversary is white-box and prepends 3 to 5 token-level perturbations to the prompt §3.
• The objective is the victim model's own denoising error on a target image containing the forgotten content §4. Because of this, no auxiliary diffusion or classifier model is needed, which the authors contrast with P4D §4.
• Optimisation is discrete: a distribution over the vocabulary for each inserted slot, updated by projected gradient steps §4, Remark 3 §5.1.
• Author-reported: on I2P nudity prompts against ESD, attack success rises from 20.42% (no attack) to 76.05% (P4D: 69.71%) tab. 2.
• The official code (MIT) matches the method in most places, but it differs on the optimiser, the loss normalisation and the way the final prompt is read out. See "Paper vs code" before reimplementing.

Problem

Latent diffusion models generate an image by denoising a latent, conditioned on a text prompt c §3. Safety-driven unlearning modifies a pretrained model so that a prompt, a style or an object class is no longer generated §3. The paper calls these models unlearned DMs and studies five of them: ESD, FMN, AC, UCE and SLD. tab. 1 lists which unlearning task (concept, style or object) each one targets §5.1.

The research question is how robust these models are when the adversary can change the text prompt §1 §3. The threat model is white-box: the adversary has the victim's weights §3. The adversarial prompt is written as [APs] + [original prompt], with 3 to 5 token-level perturbations §3. The task is to find a perturbed prompt c' that makes the unlearned model generate content related to the original c §3.

Reported success is split into a pre-attack rate (the model already produces the content for the original prompt) and a post-attack rate (the perturbed prompt produces it). The reported ASR is the sum of the two §5.1. Prompts that already succeed without an attack are therefore counted in ASR too.

Method

The method rests on one idea: a diffusion model can act as a classifier over prompts §4. The classifier probability of a prompt c_i given an image x is proportional to an exponential of the negative expected denoising error under c_i, normalised over the prompts §4, eq. 3:

p_{\theta}(c_i \mid \mathbf{x}) \propto \frac{\exp\{-\mathbb{E}_{t,\epsilon}[\|\epsilon - \epsilon_{\theta}(\mathbf{x}_t \mid c_i)\|_2^2]\}} {\sum_j \exp\{-\mathbb{E}_{t,\epsilon}[\|\epsilon - \epsilon_{\theta}(\mathbf{x}_t \mid c_j)\|_2^2]\}} §4, eq. 3

The attack then treats c' as the label to predict for a target image x_tgt that contains the forgotten content §4, eq. 4. Two simplifications turn this into a loss the authors can optimise §4:

1. Classification needs only relative errors between prompts, so the normalisation can be rewritten as a difference of errors §4, eq. 5.
2. Using the convexity of exp (Jensen's inequality), the objective is bounded above by a term that depends on c' and a constant that does not §4, eq. 7.

Dropping the constant and the monotone exp leaves a single objective, the expected denoising error of the target latent under the perturbed prompt. The method minimises this over c' §4, final objective, tagged "UnlearnDiffAtk":

\min_{c'} \; \mathbb{E}_{t,\epsilon}\big[\|\epsilon - \epsilon_{\theta^*}(\mathbf{x}_{\mathrm{tgt},t} \mid c')\|_2^2\big] §4, final objective

Here θ* is the unlearned model, and x_tgt,t is the target image encoded to the latent space and noised at step t §4. The target can come from the unlearned model's own generation under the original prompt, or from an image found elsewhere §4 fig. 3. The paper contrasts this with P4D, which minimises the difference between the unlearned model's and the original model's noise predictions on the original prompt. That needs a second diffusion pass §4, eq. 8.

The objective is over discrete text, so the paper uses projected gradient descent in a token-distribution space §4, Remark 3. Each of the k inserted slots holds a probability vector over the vocabulary. The steps are:

# Procedure for one prompt (k = 5 for concept, k = 3 for style and object)
# Input:  unlearned model; target latent x_tgt (4, 64, 64); prompt c
# Output: k adversarial tokens, or nothing if no attack is needed

def attack_prompt(model, x_tgt, prompt, k):
    # Stop early if the original prompt already gives unsafe output
    # [code: src/attackers/text_grad_.py#L196-L197]
    if is_unsafe(generate(model, prompt)):
        return None
    # Uniform start over the vocabulary; the paper does not state an init
    # [code: src/attackers/text_grad_.py#L68-L76]
    slot_probs = full((k, vocab_size), 1 / vocab_size)   # (k, V)
    optimizer = Adam(slot_probs, lr=0.01, weight_decay=0.1)
    # 50 evenly spaced timesteps: 10, 30, ..., 990 [§5.1]
    # [code: src/tasks/classifier_.py#L69-L70]
    for timestep in timesteps_50:
        # 40 projected-gradient iterations per timestep [§5.1]
        for step in range(40):
            # Sample one token per slot from its distribution
            # [code: src/attackers/text_grad_.py#L240-L246]
            one_hot = sample_one_hot(slot_probs)         # (k, V)
            slot_embeds = one_hot @ token_embeddings     # (k, d)
            # Denoising error of x_tgt under [APs] + c [§4]
            loss = denoise_error(model, x_tgt, timestep, slot_embeds, prompt)
            # Straight-through gradient to the slot distributions
            # [code: src/attackers/text_grad_.py#L254]
            slot_probs.grad = straight_through_grad(loss, one_hot)
            optimizer.step()
            # Project each slot back onto the capped simplex [§4, Remark 3]
            # [code: src/attackers/text_grad_.py#L59-L66]
            slot_probs = project_capped_simplex(slot_probs)
        # Read out discrete tokens by argmax and test the image
        # [code: src/attackers/text_grad_.py#L164-L171]
        tokens = argmax(slot_probs, axis=-1)             # (k,)
        image = generate(model, tokens, prompt)
        # Stop at the first unsafe image
        # [code: src/attackers/text_grad_.py#L232-L233]
        if is_unsafe(image):
            break
    return tokens

For SLD, the victim is sampled with safety guidance switched on; the loss is unchanged classifier_.py classifier_.py.

Hyperparameters

Hardware: the attack time was measured on one NVIDIA RTX A6000 GPU, per prompt §5.2. The paper does not say whether the time includes the intermediate image generations (see "Pitfalls").

Results

Author-reported. The setup is Stable Diffusion v1.4 victims, the I2P prompt sets for concept unlearning (142 nudity, 756 violence, 727 illegal activity prompts), Van Gogh style prompts (50), and 50 prompts for each of four Imagenette-style object classes §5.1 tab. 2 tab. 4. Values are attack success rates in percent; "No attack" uses the original prompts. The best value per column is bold in the paper, and it is not marked here.

Concept unlearning tab. 2:

Artistic style (Van Gogh), Top-1 and Top-3 ASR tab. 3:

Object unlearning tab. 4:

Target image source (ESD, SD v1.4 victim) tab. 5: a target generated by SD v1.4 gives 76.05% nudity ASR, and one generated by SD v2.1 gives 73.94%. For Van Gogh the Top-1 / Top-3 values are 32.00 / 76.00 (v1.4) and 34.00 / 82.00 (v2.1). For Church both give 60.00%.

Ablations and baselines (nudity, ESD, appendix) app.:

• Prefix placement is best for UnlearnDiffAtk: prefix 76.05, suffix 66.90, middle 68.31, insert 73.94 tab. A1.
• Length 3 to 9 tokens: 71.13, 73.24, 76.05, 74.65, 71.13, 73.94, 74.65. P4D: 70.42, 71.13, 69.71, 70.42, 71.13, 65.49, 73.24 tab. A2.
• Transfer to other models (generated on ESD, tested on SD and FMN): UnlearnDiffAtk vs P4D on SD v1.4 90.27 vs 84.07, SD v2.0 42.48 vs 38.94, SD v2.1 54.87 vs 46.02, FMN v1.4 81.69 vs 83.80, FMN v2.0 39.44 vs 40.84, FMN v2.1 49.30 vs 47.18 tab. A3.
• Random attacks against ESD (nudity): no attack 20.42, random seed 14.01, random text 57.75, P4D 69.71, UnlearnDiffAtk 76.05 tab. A4.

Cost. Attack time per prompt is 26.29 min (concept), 38.87 min (style) and 31.32 min (object) for UnlearnDiffAtk, against 34.70, 50.79 and 43.65 min for P4D tab. 2 tab. 3 tab. 4.

Paper vs code

The official repository is OPTML-Group/Diffusion-MU-Attack, linked from the paper's abstract. The README names the paper and links the authors' project page README.md, and the code is MIT-licensed LICENSE. The implementation in the repo is named text_grad, not UnlearnDiffAtk attack.py.

• Optimiser. Paper: AdamW §5.1. Code: torch.optim.Adam with weight_decay=0.1 text_grad_.py. Adam applies weight decay as an L2 term inside the gradient, while AdamW decouples it. The two produce different updates.
• Timesteps. Paper: "sample 50 diffusion time steps" §5.1. Code: a fixed grid of 50 values, 10 to 990 in steps of 20 classifier_.py.
• Loss scale. Paper: squared L2 norm of the noise error §4. Code: MSELoss with mean reduction over all elements classifier_.py classifier_.py. This scales the gradient by a constant relative to the paper's formula, which changes how strong the weight decay is relative to the loss gradient.
• Noise. Paper: expectation over epsilon. Code: one Gaussian draw per loss call, shared across the batch classifier_.py.
• Discrete step. Paper: a projection that "maps the token embedding to discrete texts" §4, Remark 3. Code: during optimisation a token is sampled from each slot's distribution, and the straight-through gradient is passed back text_grad_.py. The prompt that is tested is the argmax over the projected distributions, which is not the sampled token sequence text_grad_.py text_grad_.py.
• Sampling randomness. The sampled tokens depend on the random state. The code seeds the global RNG from the config attack.py.
• Stopping. Paper: no stopping rule stated. Code: the attack stops at the first timestep block whose image is unsafe, and skips prompts that already succeed without an attack text_grad_.py text_grad_.py.
• Evaluation frequency. The code generates a full image after every timestep block, so up to 50 generations per prompt text_grad_.py.
• Prompts per evaluation. Paper: 50 prompts per style or object class §5.1. Code: one attack_idx per run; the README loops over indices text_grad_.py README.md.
• Initialisation. Not stated in the paper. Code: uniform 1/V over the vocabulary, with random init as an option (off by default) text_grad_.py.
• Adversarial prompt length. The code splits embeddings at a hard-coded total of 76 tokens text_grad_.py. A long original prompt plus k must fit in that budget; the paper does not mention a limit.
• Style success. The code's vangogh branch computes a top-10 list but leaves the success flag commented out classifier_.py. The Top-1 and Top-3 numbers are computed elsewhere, in scripts/analysis/style_analysis.py README.md.
• Source of the style classifier. The config refers to a relative checkpoint path, results/checkpoint-2800 text_grad_esd_vangogh_classsifier.json. The paper describes a ViT-base fine-tuned on WikiArt §5.1. Whether that checkpoint is published was not verified.

No other discrepancies were found in the hyperparameters listed above: k, iterations, learning rate and the 50 timesteps match the paper's numbers.

Pitfalls

• Optimiser and weight decay. Use AdamW if you follow the paper's wording, Adam with coupled L2 if you follow the code. The paper gives no weight decay. Results may depend on it, and the paper does not report a sensitivity study.
• Loss normalisation. The paper's squared norm and the code's mean-reduced MSE differ by a constant. With weight decay present, the constant changes the balance between the loss and the decay term.
• Sampled vs read-out prompt. The gradient is computed for randomly sampled tokens, but the tested prompt is the argmax of the distributions, so the two can differ. Reimplementations should test the argmax prompt, as the code does.
• Hidden cost. The paper's time per prompt is 26 to 39 min on one A6000 §5.2. The code runs 50 timesteps × 40 steps of U-Net forward and backward passes, and generates an image after each timestep block. The paper does not say whether the generations are included in the reported time.
• Memory. Each slot holds a full distribution over the tokenizer's vocabulary, so the optimised parameters are k × V floats.
• Outputs are not words. The optimised tokens are often nonsense strings. The paper's examples include strings such as "sales rotagra rugged zee" fig. 3. Filters that look for real words will not catch them.
• Target image matters. The paper states the target need not match the prompt, only the erased concept §4. Results with a target from SD v2.1 are close to those from SD v1.4 (nudity 73.94 vs 76.05) tab. 5.
• Early stopping. The reported prompt is the first unsafe checkpoint in the timestep sweep, of up to 50. The ASR therefore depends on this checkpoint budget. Compare methods only under the same budget.
• Claims in the text that the tables do not support. Check these before quoting:
  • Object: the text says UnlearnDiffAtk "consistently achieves a higher ASR than P4D". Table 4 has FMN Church 96% vs 98% and FMN Garbage truck 98% vs 98%.
  • Style: the text says Top-3 ASR "still maintains a performance level exceeding 80%" for UnlearnDiffAtk. Table 3 gives 76.00% for ESD Top-3.
  • Concept: the text says UnlearnDiffAtk outperforms P4D "in most cases". Of the nine ESD/FMN/SLD cells in table 2, UnlearnDiffAtk is higher in five, equal in one and lower in three.
  • Time: the text gives a reduction of "approximately 23.5%" without naming the table. Table 2 gives (34.70 - 26.29) / 34.70 = 24.2%. The 23.5% matches the style table: (50.79 - 38.87) / 50.79.
  • Transfer: "typically outperforms P4D" holds in four of six cells in tab. A3.
  • Placement: "prefix yields the highest ASR" holds for UnlearnDiffAtk only. For P4D, insert (70.42) is above prefix (69.71) tab. A1.
  • Random seeds: the appendix reports 14.01% for random seeds, below the 20.42% of no attack. The text does not discuss this value.
• Binary derivation typo. In the appendix's binary-classification derivation, the first exponent subtracts the c_2 term from itself, which is identically zero. The intended term is probably the c' self-term, also constant, so the final objective is unaffected app., Derivation section.
• Evaluation depends on external classifiers. Nudity uses NudeNet with a 0.45 threshold classifier_.py, concept harm uses Q16 §5.1, objects use ResNet-50 top-1 classifier_.py, and styles use a separate ViT. A different classifier or threshold changes the ASR, and the paper gives no sensitivity analysis.
• Mixed model versions. The LaTeX source is in the ECCV 2024 format. The equation and table numbers here follow that source. The arXiv PDF was not compared, so numbering may differ slightly.
• Scope of evaluation. Results cover five unlearned models (ESD, FMN, AC, UCE, SLD) on SD v1.4, plus transfer to other SD versions. The paper evaluates only models whose code was public and whose unlearning reproduced §5.1. Other unlearning methods are not covered.

Lineage

• Builds on: the diffusion classifier idea, which the paper attributes to Li et al., 2023 and Chen et al., 2023 §4 (no card in this wiki).
• Compared against: P4D, Chin et al., 2023, arXiv:2309.06135 (no card in this wiki). P4D optimises a prompt to match the unlearned model's noise predictions to the original model's §4, eq. 8.
• Victim model family: ESD, Gandikota et al., 2023, arXiv:2303.07345 (no card in this wiki), one of the unlearned models evaluated §5.1.
• Discrete optimisation follows a text-gradient method for NLP attacks, cited in the paper as Hou et al., 2022 §4, Remark 3.