Adversarial text generation

Overview

Three families use this name, and they differ in what the attacker changes. In the training family, a generator writes text and a discriminator is trained to label whether a text came from a human or from the generator. The generator is rewarded when the discriminator is fooled. The shared invariant is that the reward comes from a learned judge, not a hand-written metric 1701.06547 §1. Because sampled tokens are discrete, the discriminator's error cannot be backpropagated into the generator, so the reward is applied as a policy-gradient signal 1701.06547 §2; the long-text paper uses policy gradient for both of its generator networks 1709.08624 §Training of G. For unconditional text, SeqGAN applies the same policy-gradient step with a discriminator that scores finished sequences, and estimates each token's value from Monte Carlo roll-outs 1609.05473 §Method, eq. 3–4.

In the evasion family nothing is trained. An attacker changes a given input so that a fixed detector or classifier stops giving the right answer. The change can be a whole-passage rewrite by a paraphraser, which the detector paper applies over several rounds 2303.11156 §2.1, or a few word swaps that keep the meaning, as in the classifier attack 1907.11932 §Method, Step 2. The attacker keeps the change that gets past the victim 2303.11156 §1 1907.11932 alg. 1. The detector paper also covers the reverse direction, where human text is written to be flagged as AI text 2303.11156 §3. A later classifier attack asks a masked language model for the replacement or insertion words, so the substitutes depend on the sentence around them rather than on a fixed synonym table 2004.01970 §2.

The third family optimises a prompt against a model's behaviour. One method is white-box: the adversary prepends a few invented tokens to a text-to-image prompt, chosen so that a model that was unlearned to refuse some content draws that content again. The judge is the victim model itself: its denoising error on a target image containing the forgotten content 2310.11868 §4. The tokens are often not real words 2310.11868 Pitfalls. The other method is black-box: an attacker language model writes natural-language prompts, reads the target's reply and a judge's score, and refines the prompt over a few rounds 2310.08419 §3.1.

The training variants differ in when the generator gets its signal. One reward per finished sequence is the default. Per-prefix rewards and intermediate discriminator features give the generator guidance earlier. Most training variants also mix in a maximum-likelihood update on human text. The dialogue paper reports that training diverged without it 1701.06547 §3.2; the long-text paper interleaves supervised epochs too 1709.08624 §Interleaved Training.

Ten sources cover this concept so far: three on training a generator (dialogue reply generation, unconditional text with roll-outs, and long text generation), three on evading a classifier or detector (AI-text paraphrasing, and two word-level classifier attacks), two on optimising prompts against a model (an unlearned text-to-image model and a black-box chatbot), one benchmark platform that packages several of the training methods with shared metrics 1802.01886 §3, and one 2021 review that restates a classifier attack second-hand 2103.11943 p. 7–8.

Variants

Whole-sequence adversarial REINFORCE

The generator gets one reward per complete sampled reply, the discriminator's probability that the reply is human, minus a learned baseline. Every token in the reply receives this same reward 1701.06547 §3.1, eq. 1–2. This is the simplest form and is the baseline that the next variant improves on. Card: Adversarial Learning for Neural Dialogue Generation.

Per-step rewards with Monte Carlo rollouts (REGS)

Each prefix of the reply gets its own reward. The reward is the mean discriminator score over N = 5 random completions of that prefix, and the baseline is also computed per prefix, so early words get credit or blame for how the rest of the reply turns out 1701.06547 §3.2, eq. 3. The paper reports this as slightly better than the whole-sequence variant on its automatic evaluator (AdverSuc 0.098 vs 0.080) [arxiv:1701.06547 tab. 3]. The cost is large: each prefix needs its own set of rollouts, and the authors parallelised sampling over 8 GPUs 1701.06547 §3.2, footnote 4.

Partial-sequence discriminator

A cheaper per-step option: the discriminator is trained on random prefixes of human and generated replies, one positive and one negative prefix per pair, so no rollouts are needed. The paper reports it is much faster than Monte Carlo rollouts but less accurate, and that Monte Carlo performs better when training time is not a constraint 1701.06547 §3.2.

Monte Carlo roll-outs for unconditional sequences (SeqGAN)

The generator is an LSTM policy with no conditioning input, and the discriminator sees only finished sequences 1609.05473 §Method, eq. 1. The value of a prefix is the mean discriminator score over N roll-outs that finish the sequence with a roll-out policy. For the last token the value is the score of the finished sequence 1609.05473 eq. 3–4. The generator takes policy-gradient steps, and the discriminator is retrained on fresh negatives from the current generator at every discriminator step 1609.05473 alg. 1. The released code uses no baseline in the value estimate 1609.05473 §Pitfalls.

The paper's results are author-reported. On synthetic data the oracle NLL at length 20 is 8.736 for SeqGAN against 9.038 for MLE 1609.05473 tab. 1. On Chinese poems BLEU-2 is 0.7389 for SeqGAN against 0.6670 for MLE 1609.05473 tab. 2. Card: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.

Leaked discriminator features with a hierarchical generator (LeakGAN)

The generator reads the discriminator's intermediate CNN features for the prefix written so far, not only its final score. A Manager LSTM turns those features into a unit-length goal vector at every step, and a Worker LSTM combines the goal with its own output to pick the next word [arxiv:1709.08624 §Hierarchical Structure of G, eq. 2–6]. The Manager is trained by policy gradient, with the change in discriminator features over the next c steps as its target direction 1709.08624 §Training of G, eq. 7. The Worker is trained by REINFORCE on an intrinsic reward: the mean cosine between recent feature changes and the past goals 1709.08624 eq. 8–9. The paper uses c = 4 and k = 16 [arxiv:1709.08624 §Experiment, GAN Setting; app. Experiment Settings].

The paper's comparisons are against SeqGAN and RankGAN, and against an MLE-trained LSTM. Its synthetic-oracle negative log-likelihood is 7.038 at length 20, against 8.736 for SeqGAN and 8.247 for RankGAN, and 7.191 at length 40, against 10.310 and 9.958 1709.08624 tab. 1. On WMT News, BLEU-2 is 0.956, against 0.859 for SeqGAN and 0.778 for RankGAN 1709.08624 tab. 2. In its Turing-style test the share of WMT sentences judged real is 0.554, against 0.236 for SeqGAN and 0.651 for ground truth 1709.08624 tab. 5. The authors report that the gain over the baselines grows as sentences get longer, up to 45 words 1709.08624 fig. 3. These are author-reported numbers from one paper. Card: Long Text Generation via Adversarial Training with Leaked Information.

Recursive paraphrasing against a fixed detector

No generator is trained. The attacker sends AI output through a paraphraser, feeds each result back in, and repeats for five rounds. The main paraphraser is DIPPER, an 11B model that rewrites a whole passage in one call. A LLaMA-2-7B-Chat paraphraser is the second option 2303.11156 §2.1, app. B.2. The authors chose DIPPER because its two quality knobs gave a clearer trend across rounds 2303.11156 §2.3.

On a soft-watermarked OPT-13B target, TPR at 1% FPR falls from 99.8% to 80.7% after one DIPPER round and to below 20% after five 2303.11156 §2.3. The LLaMA-2 paraphraser gives 54.6% after one round 2303.11156 §2.3. Human raters judged 70.4% of DIPPER outputs to keep the content at 4 or 5 out of 5, and 89.1% to be grammatical at 4 or 5 2303.11156 tab. 1. Perplexity under OPT-13B rises from 5.2 in the table to 8.7 with DIPPER 2303.11156 tab. 3. Against a retrieval detector, five rounds bring detection below about 60% 2303.11156 §2.4.

If the attacker can query the detector's scores, keeping the lowest-scoring round brings watermark TPR to 4.0% on OPT-1.3B, at a cost of +1.5 perplexity 2303.11156 §2.3. Against a RoBERTa-based detector, ten paraphrases from ten queries, keeping the best, leave about 80% TPR 2303.11156 §2.4.

The paper also spoofs detectors. It estimates proxy green lists for a watermarked model from about one million queries on nonsense text built from a list of 181 words 2303.11156 app. D.1. Human text written to use those word pairs is flagged by the watermark detector, and the spoofed AUROC falls from 99.8% to 1.3% 2303.11156 §3. Card: Can AI-Generated Text be Reliably Detected?.

Word substitution against a text classifier (TextFooler)

The attacker sees only the victim's predicted label and confidence, not its weights or training data 1907.11932 §Method, Threat Model. It ranks the words of the input by how much removing each one shifts the prediction 1907.11932 eq. 2. Starting from the most important word and skipping stop words, it replaces each word with an embedding neighbour that has the same part of speech and passes a sentence-similarity check. It stops at the first replacement that flips the label 1907.11932 alg. 1. Unlike the paraphrase attack, each step changes one word and the rest of the text is kept as it is. A 2021 review restates the same four stages and the BERT rows, and it gives no value for the similarity threshold 2103.11943 p. 7–8. The review is second-hand, so its numbers agree with the primary card and add no new evidence.

The reported BERT accuracies fall from 97.0 to 6.6 on Yelp, with 13.9% of words changed 1907.11932 tab. 3, and from 89.4 to 4.0 on SNLI, with 18.5% changed 1907.11932 tab. 4. On IMDB with an LSTM the paper reports a 99.7% success rate with 5.1% of words changed, against 86.7% with 6.9% for TextBugger 1907.11932 tab. 5. Two human checks support the adversarial text: 92% of MR examples and 85% of SNLI examples keep the original label, on 100 sentences per task 1907.11932 §Results, Human Evaluation. Dropping the similarity check makes the attack stronger (MR accuracy 6.2 against 11.5) but lowers meaning similarity (0.58 against 0.65) 1907.11932 tab. 10. Adding adversarial examples to training raised MR accuracy from 11.5 to 18.7, in one run, and SNLI accuracy from 4.0 to 8.3 1907.11932 tab. 12 2103.11943 p. 8. Card: Is BERT Really Robust?.

Adversarial prompt tokens against an unlearned text-to-image model

The victim is a Stable Diffusion v1.4 model that was unlearned to refuse a concept, a style or an object class. The adversary is white-box and prepends k token slots to the prompt: k = 5 for concept unlearning and k = 3 for style and object 2310.11868 §3, §5.1. The loss is the victim's expected denoising error on a target latent that contains the forgotten content, so no second model is needed 2310.11868 §4, final objective. The paper's optimiser works on a distribution over the vocabulary for each slot, with a projection back onto the capped simplex 2310.11868 §4, Remark 3. Its reported attack success on nudity against ESD is 76.05%, against 69.71% for P4D and 20.42% with no attack 2310.11868 tab. 2. The paper reports 26.29 minutes per prompt against 34.70 for P4D, on one RTX A6000 2310.11868 §5.2. These are author-reported numbers. Card: To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images.

Black-box prompt refinement by an attacker LLM (PAIR)

Three language models take part: an attacker, the target and a judge 2310.08419 §3. Several conversations, called streams, run in parallel. In each round the attacker writes a prompt from its own history, the target answers it with no earlier turns, and the judge scores the pair. A stream stops when the judge scores a jailbreak or when it reaches its depth 2310.08419 alg. 1. The paper's experiments use 30 streams and depth 3, so a behaviour takes at most 90 queries 2310.08419 §4. The released code stops all streams as soon as any one succeeds, which the paper does not describe 2310.08419 §Paper vs code.

The author-reported jailbreak rates on JBB-Behaviors are 88% on Vicuna-13B (10.0 queries per success), 73% on Gemini-Pro, 51% on GPT-3.5 and 48% on GPT-4, but only 4% on Llama-2 and 0 to 3% on the Claude models 2310.08419 tab. 2. On Vicuna the paper reports 34 seconds per behaviour, against 1.8 hours for the token-level GCG attack 2310.08419 tab. 4. The judge is Llama Guard in the main experiments 2310.08419 §4 Evaluation. On a 100-item annotated set, Llama Guard misses 47% of the jailbreaks, while GPT-4-0613 misses 7% and wrongly flags 16% 2310.08419 tab. 1. Under SmoothLLM the Vicuna rate falls from 88 to 39, but the defence numbers reuse prompts found against undefended models, so they do not test an adaptive attacker 2310.08419 tab. 5. Card: Jailbreaking Black Box Large Language Models in Twenty Queries.

Shared benchmark for generators (Texygen)

Texygen is a platform, not a new method. It packages seven text generators (MLE, SeqGAN, MaliGAN, RankGAN, TextGAN, GSGAN and LeakGAN) with five metrics: BLEU, EmbSim, NLL-oracle, NLL-test and Self-BLEU [arxiv:1802.01886 §2.1, §2.2]. Its synthetic benchmark draws 10,000 sentences of length 20 from a random LSTM oracle over 5,000 words 1802.01886 §4.1. On COCO captions, LeakGAN has the highest test BLEU at every order from 2 to 5, with BLEU-4 at 0.355, while MLE has the lowest Self-BLEU at every order, with Self-BLEU-4 at 0.583 [arxiv:1802.01886 tab. 2, tab. 3]. LeakGAN also has the highest Self-BLEU-4, at 0.848, so the model with the best BLEU is also the least diverse by that metric. The paper names this quality-diversity trade-off 1802.01886 §1, §2.2.3. Card: Texygen.

When to use which

Implementation sketch

Our own pseudocode for the reported dialogue setting; the paper's algorithm box is 1701.06547 fig. 1. Comments name the variant each step belongs to. LeakGAN changes the reward step: its generator is split into a Manager and a Worker, and the Manager's target comes from discriminator features rather than from the final score 1709.08624 §Hierarchical Structure of G.

# Pretrain generator G by maximum likelihood [arxiv:1701.06547 §3.3]
G = train_mle(G, dialogues)
# Pretrain discriminator D on human vs generated replies [arxiv:1701.06547 §3.3]
D = train_classifier(D, human_pairs, generated_pairs)
loop:
    # Five discriminator steps per generator step [arxiv:1701.06547 fig. 1]
    repeat 5 times:
        # Human reply positive, sampled reply negative
        D = update(D, human_pair, sampled_pair)
    # Sample a reply from G for context x [arxiv:1701.06547 §3.1]
    y_hat = G.sample(x)
    # Reward: D's probability that the reply is human [arxiv:1701.06547 §3.1]
    reward = D.prob_human(x, y_hat)
    # Policy-gradient step; REGS uses one reward per prefix [arxiv:1701.06547 §3.2]
    G = policy_gradient_update(G, x, y_hat, reward)
    # Teacher-forced step on the human reply [arxiv:1701.06547 §3.2]
    G = mle_update(G, x, y)

Notes: the paper's reward is a probability in [0, 1], not a log-probability. When the discriminator is strong, rewards for all samples approach zero, which is the collapse that the teacher-forced step guards against 1701.06547 §3.2. The relative weight of the policy-gradient and maximum-likelihood steps is not stated in the paper, so it must be tuned in any reimplementation.

For LeakGAN, the discriminator's features are the input to the generator, so they must be recomputed as the discriminator trains. The paper's appendix reports that a fully connected layer in place of the hierarchy was numerically unstable 1709.08624 app. Discussions. Its rewards are rescaled by within-batch rank 1709.08624 eq. 11; the card's paper-vs-code notes question whether the code's ranking matches the paper, so treat that step as unverified.

SeqGAN's loop differs from the dialogue sketch in two ways. It draws fresh negatives from the current generator at every discriminator step, and it keeps a roll-out copy of the generator that is reset once per iteration 1609.05473 alg. 1. The released code instead keeps that copy as an exponential average of the generator 1609.05473 §Paper vs code. Its reward is used directly as the action value, with no baseline subtracted 1609.05473 §Pitfalls.

The evasion loop is shorter. Each round feeds the previous output back in, and the optional step keeps the lowest-scoring round when detector scores are available 2303.11156 §2.1, §2.3. The released code is under a non-commercial research notice, so this sketch is our own and copies no code 2303.11156 §Paper vs code.

# Recursive paraphrase against a fixed detector [arxiv:2303.11156 §2.1]
# Input: AI passage S, detector score function score(), rounds = 5
current = S
for round in 1..5:
    # Rewrite the whole current passage in one call [arxiv:2303.11156 §2.1]
    current = paraphraser(current)
    # Keep every round so the lowest score can be selected [arxiv:2303.11156 §2.3]
    record(round, current, score(current))

The classifier attack and PAIR share one outer loop: propose a change, query the victim, keep the change if it succeeds. They differ in what proposes the change, a synonym table with a part-of-speech filter 1907.11932 alg. 1, or an attacker LLM that reads the history 2310.08419 alg. 1. The released classifier code uses a synonym threshold of 0.5, not the paper's 0.7, and measures similarity on a different scale 1907.11932 §Paper vs code.

Contradictions

Few direct contradictions appear across the nine papers, because most of them measure different things. The one exception is the SeqGAN and Texygen pair, given below. The dialogue paper scores responses with AdverSuc, a judge's accuracy at telling them from human replies 1701.06547 §4.1. The long-text paper scores samples with an oracle's negative log-likelihood, BLEU and a Turing-style test 1709.08624 §Experiment, §Evaluation Metrics. The detector paper scores detectors with TPR at 1% FPR and AUROC 2303.11156 §2.3. The classifier paper reports accuracy after attack and the share of changed words 1907.11932 tab. 3. The jailbreak paper reports a rate judged by a separate judge model 2310.08419 §4 Evaluation. Their numbers should not be compared across papers. The prompt-attack paper on text-to-image models counts an attack success rate as the sum of a pre-attack rate and a post-attack rate, so it counts prompts that already succeed without an attack 2310.11868 §5.1. Its numbers are not comparable with the other eight either.

Within the classifier paper, the text and its table disagree. The text gives BERT's Yelp accuracy as falling from 95.6% to 6.8%, while Table 3 gives 97.0% to 6.6% 1907.11932 §Results, Automatic Evaluation [arxiv:1907.11932 tab. 3]. The concept page uses the table.

Masked-LM replacement and insertion (BAE)

The attacker is black-box and soft-label: it may query the classifier for output probabilities but has no weights, gradients or training data 2004.01970 §2. Each token is scored by how much deleting it lowers the probability of the true label. Tokens are then visited from most to least important. At each position the attacker masks the word and takes the top 50 predictions of a BERT base masked language model [arxiv:2004.01970 §2, app. A]. A candidate survives if its sentence-level similarity to the input passes a threshold of 0.8 under the Universal Sentence Encoder, and, for replacements only, if it has the same part of speech 2004.01970 app. A. The attack stops when the label flips 2004.01970 alg. 1. The paper defines four modes: replace only (BAE-R), insert only (BAE-I), a per-token choice between the two (BAE-R/I), and replace followed by insert at the same position (BAE-R+I) 2004.01970 §2. Card: BAE: BERT-based Adversarial Examples for Text Classification.

On seven datasets, author-reported test accuracy with a word-LSTM target falls from 88.0 to 4.0 on Amazon for BAE-R+I, against 31.0 for TextFooler 2004.01970 tab. 1. On a BERT target, BAE-R+I gives 16.0 on IMDB, against 32.0 for TextFooler and 46.0 for BAE-R 2004.01970 tab. 1. On the TREC question-type set, the word-CNN target reaches 8.4 under BAE-R+I 2004.01970 tab. 2. The test sets for Amazon, Yelp and IMDB have only 100 items each, and the paper reports no variance or repeated runs 2004.01970 tab. 5. In a small human check on a BERT target, raters scored BAE-R outputs as more natural than TextFooler outputs on Amazon, IMDB and MR, on a 1 to 5 scale 2004.01970 tab. 3, app. C. The authors' own text and table disagree on one point, which is noted under Contradictions.

Within the jailbreak paper, the headline figures depend on which table is read. The default Mixtral-to-Vicuna setting reads 88% in Table 2 and 93% in Table 7. The text rounds GPT-3.5 and GPT-4 to 50%, where Table 2 gives 51% and 48% 2310.08419 §4 2310.08419 tab. 2 2310.08419 tab. 7. The concept page uses Table 2.

Within the detector paper, the introduction and §2.4 report different numbers for the same detectors. The introduction gives a zero-shot AUROC of 25.2% after attack, while §2.4 reports 59.8% for DetectGPT with the T5 paraphraser 2303.11156 §1 2303.11156 §2.4. The introduction also gives a RoBERTa-based TPR of 60%, while §2.4 reports about 92% for one round. The paper does not say which setting the introduction quotes 2303.11156 §Pitfalls. The concept page reports the §2.4 values only.

Within the long-text paper, the paper and its released code disagree on several settings: learning rate, discriminator dropout, the interleaved supervised schedule and the rescaling constants. The concept page does not resolve these; the card lists each one with both sides cited 1709.08624 §Paper vs code.

Within the BAE paper, the text calls TextFooler sometimes stronger than BAE-R on a BERT target, but the tables give TextFooler the lower accuracy on six of seven datasets, and BAE-R the lower one only on TREC 2004.01970 §3 2004.01970 tab. 1 2004.01970 tab. 2. The concept page reports the tables.

The SeqGAN paper and Texygen disagree on whether SeqGAN beats MLE on real text. SeqGAN reports higher BLEU-2 for SeqGAN than for MLE on Chinese poems, 0.7389 against 0.6670 1609.05473 tab. 2. Texygen reports the two as close on COCO captions, with BLEU-2 at 0.745 for SeqGAN and 0.731 for MLE. On the same captions MLE has the higher BLEU-4 (0.305 against 0.294) and BLEU-5 (0.189 against 0.180), and lower Self-BLEU at every order, so its output is more diverse 1802.01886 tab. 2, tab. 3. The datasets, BLEU settings and training budgets differ, so the concept page reports both sides and does not resolve the gap.

Papers

• Adversarial Learning for Neural Dialogue Generation: trains a Seq2Seq dialogue generator with a learned discriminator as the reward, using policy gradient, per-prefix Monte Carlo rewards and interleaved maximum-likelihood updates.
• Long Text Generation via Adversarial Training with Leaked Information: LeakGAN, a text GAN whose generator takes the discriminator's intermediate features through a Manager–Worker hierarchy and is trained with policy gradient on long sequences.
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient: trains an LSTM text generator as a policy, with a CNN discriminator's score on finished sequences as the reward, and Monte Carlo roll-outs to give each token a value.
• Texygen: A Benchmarking Platform for Text Generation Models: an open platform that packages seven text-generation baselines and five metrics on a synthetic oracle and COCO captions, and reports how BLEU and Self-BLEU trade off.
• BAE: BERT-based Adversarial Examples for Text Classification: a black-box attack that ranks words by importance and fills each position with BERT's masked-LM predictions, replacing or inserting words under sentence-similarity and part-of-speech filters.
• Can AI-Generated Text be Reliably Detected?: stress-tests AI-text detectors with recursive paraphrasing and spoofing, and bounds any detector's AUROC by the total variation distance between human and AI text.
• To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now: UnlearnDiffAtk, which prepends a few optimised tokens to a prompt and scores them with the unlearned model's own denoising error, to bypass concept, style and object unlearning.
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment: TextFooler, a black-box word-substitution attack that ranks words by influence and swaps in embedding synonyms that pass similarity and part-of-speech checks.
• Jailbreaking Black Box Large Language Models in Twenty Queries: PAIR, a black-box method in which an attacker LLM refines a jailbreak prompt from the target's replies and a judge's scores.
• BERT: A Review of Applications in Natural Language Processing and Understanding: a 2021 review of BERT that restates the TextFooler attack and its BERT numbers second-hand; used here only as a cross-check.