---
kind: concept
slug: adversarial-text-generation
title: Adversarial text generation
summary: >-
  Producing text that fools a learned judge or breaks a safety behaviour. Three families: training a generator
  against a discriminator (policy gradient handles discrete tokens), changing an input to fool a fixed detector or
  classifier (paraphrasing, word substitution), and optimising prompts against a model (gradient-based tokens or
  an attacker LLM). Ten sources.
tags: [text-generation, reinforcement-learning, gan, long-text, ai-text-detection, paraphrasing, diffusion, machine-unlearning, red-teaming, word-substitution, masked-language-model, jailbreak, black-box, benchmark]
papers: ["1701.06547", "1609.05473", "1709.08624", "1802.01886", "2303.11156", "2310.11868", "1907.11932", "2004.01970", "2310.08419", "2103.11943"]
related: [policy-gradient, dialogue-generation, adversarial-evaluation, generative-adversarial-network, ai-generated-text-detection, adversarial-robustness, benchmark-validity, diffusion-models, machine-unlearning, red-teaming, text-classification, masked-language-modeling, llm-as-a-judge]
status: draft
updated_at: 2026-10-19
---

## Overview

Three families use this name, and they differ in what the attacker changes. In the training family, a generator
writes text and a discriminator is trained to label whether a text came from a human or from the generator. The
generator is rewarded when the discriminator is fooled. The shared invariant is that the reward comes from a
learned judge, not a hand-written metric [arxiv:1701.06547 §1]. Because sampled tokens are discrete, the
discriminator's error cannot be backpropagated into the generator, so the reward is applied as a policy-gradient
signal [arxiv:1701.06547 §2]; the long-text paper uses policy gradient for both of its generator networks
[arxiv:1709.08624 §Training of G]. For unconditional text, SeqGAN applies the same policy-gradient step with a
discriminator that scores finished sequences, and estimates each token's value from Monte Carlo roll-outs
[arxiv:1609.05473 §Method, eq. 3–4].

In the evasion family nothing is trained. An attacker changes a given input so that a fixed detector or
classifier stops giving the right answer. The change can be a whole-passage rewrite by a paraphraser, which the
detector paper applies over several rounds [arxiv:2303.11156 §2.1], or a few word swaps that keep the meaning, as
in the classifier attack [arxiv:1907.11932 §Method, Step 2]. The attacker keeps the change that gets past the
victim [arxiv:2303.11156 §1] [arxiv:1907.11932 alg. 1]. The detector paper also covers the reverse direction,
where human text is written to be flagged as AI text [arxiv:2303.11156 §3]. A later classifier attack asks a
masked language model for the replacement or insertion words, so the substitutes depend on the sentence around
them rather than on a fixed synonym table [arxiv:2004.01970 §2].

The third family optimises a prompt against a model's behaviour. One method is white-box: the adversary prepends
a few invented tokens to a text-to-image prompt, chosen so that a model that was unlearned to refuse some content
draws that content again. The judge is the victim model itself: its denoising error on a target image containing
the forgotten content [arxiv:2310.11868 §4]. The tokens are often not real words [arxiv:2310.11868 Pitfalls]. The
other method is black-box: an attacker language model writes natural-language prompts, reads the target's reply
and a judge's score, and refines the prompt over a few rounds [arxiv:2310.08419 §3.1].

The training variants differ in when the generator gets its signal. One reward per finished sequence is the
default. Per-prefix rewards and intermediate discriminator features give the generator guidance earlier. Most
training variants also mix in a maximum-likelihood update on human text. The dialogue paper reports that training
diverged without it [arxiv:1701.06547 §3.2]; the long-text paper interleaves supervised epochs too
[arxiv:1709.08624 §Interleaved Training].

Ten sources cover this concept so far: three on training a generator (dialogue reply generation, unconditional
text with roll-outs, and long text generation), three on evading a classifier or detector (AI-text paraphrasing,
and two word-level classifier attacks), two on optimising prompts against a model (an unlearned text-to-image model
and a black-box chatbot), one benchmark platform that packages several of the training methods with shared
metrics [arxiv:1802.01886 §3], and one 2021 review that restates a classifier attack second-hand
[arxiv:2103.11943 p. 7–8].

## Variants

### Whole-sequence adversarial REINFORCE

The generator gets one reward per complete sampled reply, the discriminator's probability that the reply is
human, minus a learned baseline. Every token in the reply receives this same reward
[arxiv:1701.06547 §3.1, eq. 1–2]. This is the simplest form and is the baseline that the next variant
improves on. Card: [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547).

### Per-step rewards with Monte Carlo rollouts (REGS)

Each prefix of the reply gets its own reward. The reward is the mean discriminator score over N = 5 random
completions of that prefix, and the baseline is also computed per prefix, so early words get credit or blame
for how the rest of the reply turns out [arxiv:1701.06547 §3.2, eq. 3]. The paper reports this as slightly
better than the whole-sequence variant on its automatic evaluator (AdverSuc 0.098 vs 0.080) [arxiv:1701.06547
tab. 3]. The cost is large: each prefix needs its own set of rollouts, and the authors parallelised sampling
over 8 GPUs [arxiv:1701.06547 §3.2, footnote 4].

### Partial-sequence discriminator

A cheaper per-step option: the discriminator is trained on random prefixes of human and generated replies,
one positive and one negative prefix per pair, so no rollouts are needed. The paper reports it is much faster
than Monte Carlo rollouts but less accurate, and that Monte Carlo performs better when training time is not
a constraint [arxiv:1701.06547 §3.2].

### Monte Carlo roll-outs for unconditional sequences (SeqGAN)

The generator is an LSTM policy with no conditioning input, and the discriminator sees only finished sequences
[arxiv:1609.05473 §Method, eq. 1]. The value of a prefix is the mean discriminator score over N roll-outs that
finish the sequence with a roll-out policy. For the last token the value is the score of the finished sequence
[arxiv:1609.05473 eq. 3–4]. The generator takes policy-gradient steps, and the discriminator is retrained on fresh
negatives from the current generator at every discriminator step [arxiv:1609.05473 alg. 1]. The released code uses
no baseline in the value estimate [arxiv:1609.05473 §Pitfalls].

The paper's results are author-reported. On synthetic data the oracle NLL at length 20 is 8.736 for SeqGAN against
9.038 for MLE [arxiv:1609.05473 tab. 1]. On Chinese poems BLEU-2 is 0.7389 for SeqGAN against 0.6670 for MLE
[arxiv:1609.05473 tab. 2]. Card: [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](/papers/1609.05473).

### Leaked discriminator features with a hierarchical generator (LeakGAN)

The generator reads the discriminator's intermediate CNN features for the prefix written so far, not only
its final score. A Manager LSTM turns those features into a unit-length goal vector at every step, and a
Worker LSTM combines the goal with its own output to pick the next word [arxiv:1709.08624 §Hierarchical
Structure of G, eq. 2–6]. The Manager is trained by policy gradient, with the change in discriminator
features over the next c steps as its target direction [arxiv:1709.08624 §Training of G, eq. 7]. The Worker is
trained by REINFORCE on an intrinsic reward: the mean cosine between recent feature changes and the past
goals [arxiv:1709.08624 eq. 8–9]. The paper uses c = 4 and k = 16 [arxiv:1709.08624 §Experiment, GAN Setting;
app. Experiment Settings].

The paper's comparisons are against SeqGAN and RankGAN, and against an MLE-trained LSTM. Its synthetic-oracle
negative log-likelihood is 7.038 at length 20, against 8.736 for SeqGAN and 8.247 for RankGAN, and 7.191 at
length 40, against 10.310 and 9.958 [arxiv:1709.08624 tab. 1]. On WMT News, BLEU-2 is 0.956, against 0.859 for
SeqGAN and 0.778 for RankGAN [arxiv:1709.08624 tab. 2]. In its Turing-style test the share of WMT sentences
judged real is 0.554, against 0.236 for SeqGAN and 0.651 for ground truth [arxiv:1709.08624 tab. 5]. The
authors report that the gain over the baselines grows as sentences get longer, up to 45 words
[arxiv:1709.08624 fig. 3]. These are author-reported numbers from one paper. Card:
[Long Text Generation via Adversarial Training with Leaked Information](/papers/1709.08624).

### Recursive paraphrasing against a fixed detector

No generator is trained. The attacker sends AI output through a paraphraser, feeds each result back in, and
repeats for five rounds. The main paraphraser is DIPPER, an 11B model that rewrites a whole passage in one call.
A LLaMA-2-7B-Chat paraphraser is the second option [arxiv:2303.11156 §2.1, app. B.2]. The authors chose DIPPER
because its two quality knobs gave a clearer trend across rounds [arxiv:2303.11156 §2.3].

On a soft-watermarked OPT-13B target, TPR at 1% FPR falls from 99.8% to 80.7% after one DIPPER round and to
below 20% after five [arxiv:2303.11156 §2.3]. The LLaMA-2 paraphraser gives 54.6% after one round
[arxiv:2303.11156 §2.3]. Human raters judged 70.4% of DIPPER outputs to keep the content at 4 or 5 out of 5, and
89.1% to be grammatical at 4 or 5 [arxiv:2303.11156 tab. 1]. Perplexity under OPT-13B rises from 5.2 in the
table to 8.7 with DIPPER [arxiv:2303.11156 tab. 3]. Against a retrieval detector, five rounds bring detection
below about 60% [arxiv:2303.11156 §2.4].

If the attacker can query the detector's scores, keeping the lowest-scoring round brings watermark TPR to 4.0%
on OPT-1.3B, at a cost of +1.5 perplexity [arxiv:2303.11156 §2.3]. Against a RoBERTa-based detector, ten
paraphrases from ten queries, keeping the best, leave about 80% TPR [arxiv:2303.11156 §2.4].

The paper also spoofs detectors. It estimates proxy green lists for a watermarked model from about one million
queries on nonsense text built from a list of 181 words [arxiv:2303.11156 app. D.1]. Human text written to use
those word pairs is flagged by the watermark detector, and the spoofed AUROC falls from 99.8% to 1.3%
[arxiv:2303.11156 §3]. Card: [Can AI-Generated Text be Reliably Detected?](/papers/2303.11156).

### Word substitution against a text classifier (TextFooler)

The attacker sees only the victim's predicted label and confidence, not its weights or training data
[arxiv:1907.11932 §Method, Threat Model]. It ranks the words of the input by how much removing each one shifts
the prediction [arxiv:1907.11932 eq. 2]. Starting from the most important word and skipping stop words, it
replaces each word with an embedding neighbour that has the same part of speech and passes a sentence-similarity
check. It stops at the first replacement that flips the label [arxiv:1907.11932 alg. 1]. Unlike the paraphrase
attack, each step changes one word and the rest of the text is kept as it is. A 2021 review restates the same four
stages and the BERT rows, and it gives no value for the similarity threshold [arxiv:2103.11943 p. 7–8]. The review
is second-hand, so its numbers agree with the primary card and add no new evidence.

The reported BERT accuracies fall from 97.0 to 6.6 on Yelp, with 13.9% of words changed [arxiv:1907.11932 tab. 3],
and from 89.4 to 4.0 on SNLI, with 18.5% changed [arxiv:1907.11932 tab. 4]. On IMDB with an LSTM the paper reports
a 99.7% success rate with 5.1% of words changed, against 86.7% with 6.9% for TextBugger [arxiv:1907.11932 tab. 5].
Two human checks support the adversarial text: 92% of MR examples and 85% of SNLI examples keep the original
label, on 100 sentences per task [arxiv:1907.11932 §Results, Human Evaluation]. Dropping the similarity check
makes the attack stronger (MR accuracy 6.2 against 11.5) but lowers meaning similarity (0.58 against 0.65)
[arxiv:1907.11932 tab. 10]. Adding adversarial examples to training raised MR accuracy from 11.5 to 18.7, in one
run, and SNLI accuracy from 4.0 to 8.3 [arxiv:1907.11932 tab. 12] [arxiv:2103.11943 p. 8]. Card: [Is BERT Really Robust?](/papers/1907.11932).

### Adversarial prompt tokens against an unlearned text-to-image model

The victim is a Stable Diffusion v1.4 model that was unlearned to refuse a concept, a style or an object class.
The adversary is white-box and prepends k token slots to the prompt: k = 5 for concept unlearning and k = 3 for
style and object [arxiv:2310.11868 §3, §5.1]. The loss is the victim's expected denoising error on a target latent
that contains the forgotten content, so no second model is needed [arxiv:2310.11868 §4, final objective]. The
paper's optimiser works on a distribution over the vocabulary for each slot, with a projection back onto the
capped simplex [arxiv:2310.11868 §4, Remark 3]. Its reported attack success on nudity against ESD is 76.05%,
against 69.71% for P4D and 20.42% with no attack [arxiv:2310.11868 tab. 2]. The paper reports 26.29 minutes per
prompt against 34.70 for P4D, on one RTX A6000 [arxiv:2310.11868 §5.2]. These are author-reported numbers.
Card: [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images](/papers/2310.11868).

### Black-box prompt refinement by an attacker LLM (PAIR)

Three language models take part: an attacker, the target and a judge [arxiv:2310.08419 §3]. Several
conversations, called streams, run in parallel. In each round the attacker writes a prompt from its own history,
the target answers it with no earlier turns, and the judge scores the pair. A stream stops when the judge scores
a jailbreak or when it reaches its depth [arxiv:2310.08419 alg. 1]. The paper's experiments use 30 streams and
depth 3, so a behaviour takes at most 90 queries [arxiv:2310.08419 §4]. The released code stops all streams as
soon as any one succeeds, which the paper does not describe [arxiv:2310.08419 §Paper vs code].

The author-reported jailbreak rates on JBB-Behaviors are 88% on Vicuna-13B (10.0 queries per success), 73% on
Gemini-Pro, 51% on GPT-3.5 and 48% on GPT-4, but only 4% on Llama-2 and 0 to 3% on the Claude models
[arxiv:2310.08419 tab. 2]. On Vicuna the paper reports 34 seconds per behaviour, against 1.8 hours for the
token-level GCG attack [arxiv:2310.08419 tab. 4]. The judge is Llama Guard in the main experiments
[arxiv:2310.08419 §4 Evaluation]. On a 100-item annotated set, Llama Guard misses 47% of the jailbreaks, while
GPT-4-0613 misses 7% and wrongly flags 16% [arxiv:2310.08419 tab. 1]. Under SmoothLLM the Vicuna rate falls from
88 to 39, but the defence numbers reuse prompts found against undefended models, so they do not test an adaptive
attacker [arxiv:2310.08419 tab. 5]. Card: [Jailbreaking Black Box Large Language Models in Twenty Queries](/papers/2310.08419).

### Shared benchmark for generators (Texygen)

Texygen is a platform, not a new method. It packages seven text generators (MLE, SeqGAN, MaliGAN, RankGAN, TextGAN,
GSGAN and LeakGAN) with five metrics: BLEU, EmbSim, NLL-oracle, NLL-test and Self-BLEU [arxiv:1802.01886 §2.1,
§2.2]. Its synthetic benchmark draws 10,000 sentences of length 20 from a random LSTM oracle over 5,000 words
[arxiv:1802.01886 §4.1]. On COCO captions, LeakGAN has the highest test BLEU at every order from 2 to 5, with BLEU-4
at 0.355, while MLE has the lowest Self-BLEU at every order, with Self-BLEU-4 at 0.583 [arxiv:1802.01886 tab. 2,
tab. 3]. LeakGAN also has the highest Self-BLEU-4, at 0.848, so the model with the best BLEU is also the least
diverse by that metric. The paper names this quality-diversity trade-off [arxiv:1802.01886 §1, §2.2.3]. Card:
[Texygen](/papers/1802.01886).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Want the simplest adversarial loop, no rollouts, and can accept one reward per reply | Whole-sequence REINFORCE, with teacher-forced updates | No rollouts needed, unlike REGS. AdverSuc 0.080 against 0.073 for MMI+p(t\|s) [arxiv:1701.06547 tab. 3] |
| Want per-word credit and have large compute | MC REGS (N = 5 rollouts per prefix) | Reported as slightly better than vanilla REINFORCE; rollouts were parallelised over 8 GPUs [arxiv:1701.06547 §5.1, §3.2 footnote 4] |
| Compute or wall-clock is tight | Partial-sequence discriminator | Reported much faster, at the price of a less accurate discriminator [arxiv:1701.06547 §3.2] |
| Output is long (the paper's experiments go to 40 words of synthetic data and 45 of WMT News) and you control the discriminator's architecture | LeakGAN | Reported lower NLL and higher BLEU than SeqGAN and RankGAN at both lengths [arxiv:1709.08624 tab. 1, tab. 2]. Needs a discriminator whose features can be exposed and a second, adversarial training loop |
| Any training setup | Keep a maximum-likelihood update on human text | Reported as needed for dialogue: without it perplexity exploded in both variants [arxiv:1701.06547 §3.2]. The long-text paper uses it too, with a different schedule [arxiv:1709.08624 §Interleaved Training] |
| Need to judge the result | Report machine-vs-random accuracy next to AdverSuc | Sampling decoding reached AdverSuc 0.372 but machine-vs-random only 0.679 [arxiv:1701.06547 tab. 3] |
| Target task is machine translation | Not established by this source | The authors report no clear gain in preliminary tests [arxiv:1701.06547 §6] |
| Want a reference implementation | Not established by the LeakGAN source | Its code has no license, so it may be read but not copied; its stack is old (TensorFlow r1.2.1, Python 2.7) [arxiv:1709.08624 §Paper vs code] |
| Stress-test a detector you operate, with no access to its scores, and can host an 11B paraphraser | Recursive DIPPER paraphrasing, five rounds | The pure five-round attack does not query the detector; watermark TPR below 20% [arxiv:2303.11156 §2.3]. About 36 s per 300-token passage for five rounds [arxiv:2303.11156 app. A.2]. The DIPPER checkpoint is not in the released code [arxiv:2303.11156 §Paper vs code] |
| Only a 7B-class paraphraser is available | LLaMA-2-7B-Chat paraphraser | Weaker in the paper: 54.6% TPR after one round against 80.7% for DIPPER, and below 50% after two rounds in all settings [arxiv:2303.11156 §2.3] |
| Detector scores are exposed and you can afford several queries per passage | Keep the lowest-scoring round, or the best of several paraphrases | Needs black-box score access; gives 4.0% TPR on OPT-1.3B [arxiv:2303.11156 §2.3] |
| Want a ceiling on any detector's power for two text distributions | The AUROC bound | AUROC ≤ 1/2 + TV − TV²/2 for total variation TV [arxiv:2303.11156 §4 Thm. 1]. At TV 0.2 the bound is about 0.68 [arxiv:2303.11156 §4 Thm. 1]. The TV estimates for real text are rough [arxiv:2303.11156 app. C.5] |
| Test whether a watermark detector can be spoofed | Learned proxy green lists | AUROC falls to 1.3% after spoofing [arxiv:2303.11156 §3]. Needs about one million queries to the watermarked model [arxiv:2303.11156 app. D.1] |
| Attack a text classifier you can only query for labels and scores, and can afford one similarity-encoder call per candidate word | Word substitution with similarity and part-of-speech checks (TextFooler) | BERT accuracy falls to 6.6% on Yelp and 4.0% on SNLI [arxiv:1907.11932 tab. 3, tab. 4]. Queries grow with text length, up to 4,403 per example for BERT on the Fake news set [arxiv:1907.11932 tab. 3]. Results are single runs [arxiv:1907.11932 §Pitfalls] |
| Stress-test a text-to-image model you have white-box access to, after an unlearning fix | Optimised adversarial prompt tokens, k = 3 to 5 | Reported to beat P4D on nudity against ESD (76.05% vs 69.71%) and to be faster (26.29 vs 34.70 min per prompt) [arxiv:2310.11868 tab. 2]. The threat model needs the victim's weights [arxiv:2310.11868 §3] |
| Attack a text classifier with soft-label access, and want substitutes that fit the sentence rather than a synonym table | Masked-LM replacement and insertion (BAE), with the R+I mode for the strongest reported attack | Word-LSTM and word-CNN accuracy falls to 2.0 to 31.1 across the seven datasets under BAE-R+I [arxiv:2004.01970 tab. 1, tab. 2]. Costs one masked-LM pass per position and one classifier query per candidate, with no timing reported [arxiv:2004.01970 alg. 1]. On a BERT target, BAE-R is the weaker attack on six of seven datasets [arxiv:2004.01970 tab. 1, tab. 2] |
| Need the attack to avoid a second diffusion pass | The victim's own denoising error on a target image | The paper contrasts this with P4D, which needs a second pass of the original model [arxiv:2310.11868 §4, eq. 8] |
| Need the adversarial string to read as normal text past a word filter | Not established by this source | The optimised tokens are often nonsense strings [arxiv:2310.11868 fig. 3] |
| Probe a deployed chatbot through its ordinary chat interface, with a budget of about 90 queries per behaviour, and accept a judge model | Black-box attacker LLM with a judge (PAIR) | 10.0 queries per success on Vicuna and 56.0 on Llama-2 [arxiv:2310.08419 tab. 2]. Results depend strongly on the judge [arxiv:2310.08419 tab. 1]. Low on Llama-2 and the Claude models, so do not average those cells [arxiv:2310.08419 tab. 2] |
| Need a defence evaluated against adaptive attackers | Not established by these sources | The PAIR defence numbers reuse prompts found against undefended models [arxiv:2310.08419 tab. 5] |
| Choosing a generator for diversity as well as quality | Report BLEU and Self-BLEU together, never BLEU alone | On COCO, LeakGAN has the top BLEU-4 (0.355) and the top Self-BLEU-4 (0.848), while MLE has the lowest Self-BLEU-4 (0.583) [arxiv:1802.01886 tab. 2, tab. 3]. The paper names this trade-off [arxiv:1802.01886 §1] |
| Compare several generator families on one synthetic oracle and one caption set | Texygen | Ships seven baselines and five metrics [arxiv:1802.01886 §2.1, §2.2]. Its code is TensorFlow 1 [arxiv:1802.01886 §Pitfalls]. The oracle's initialisation is not given in the paper, so NLL values are comparable only with runs that rebuild the same oracle [arxiv:1802.01886 §Pitfalls] |
| Want a per-token reward on unconditional text and can afford roll-outs | SeqGAN | Reported oracle NLL 8.736 against 9.038 for MLE at length 20 [arxiv:1609.05473 tab. 1]. At T = 20 and N = 16 the released code runs 304 partial roll-outs and 320 discriminator passes per 64-sequence generator step [arxiv:1609.05473 §Pitfalls]. The repository has no license, so it may be read but not copied [arxiv:1609.05473 §Paper vs code] |
| Need evidence that an adversarial method beats MLE on real text | Not settled by these sources | SeqGAN's poem and speech results favour SeqGAN, while Texygen's COCO results favour MLE on BLEU-4 and BLEU-5 [see Contradictions] |

## Implementation sketch

Our own pseudocode for the reported dialogue setting; the paper's algorithm box is
[arxiv:1701.06547 fig. 1]. Comments name the variant each step belongs to. LeakGAN changes the reward step:
its generator is split into a Manager and a Worker, and the Manager's target comes from discriminator features
rather than from the final score [arxiv:1709.08624 §Hierarchical Structure of G].

```pseudocode
# Pretrain generator G by maximum likelihood [arxiv:1701.06547 §3.3]
G = train_mle(G, dialogues)
# Pretrain discriminator D on human vs generated replies [arxiv:1701.06547 §3.3]
D = train_classifier(D, human_pairs, generated_pairs)
loop:
    # Five discriminator steps per generator step [arxiv:1701.06547 fig. 1]
    repeat 5 times:
        # Human reply positive, sampled reply negative
        D = update(D, human_pair, sampled_pair)
    # Sample a reply from G for context x [arxiv:1701.06547 §3.1]
    y_hat = G.sample(x)
    # Reward: D's probability that the reply is human [arxiv:1701.06547 §3.1]
    reward = D.prob_human(x, y_hat)
    # Policy-gradient step; REGS uses one reward per prefix [arxiv:1701.06547 §3.2]
    G = policy_gradient_update(G, x, y_hat, reward)
    # Teacher-forced step on the human reply [arxiv:1701.06547 §3.2]
    G = mle_update(G, x, y)
```

Notes: the paper's reward is a probability in [0, 1], not a log-probability. When the discriminator is strong,
rewards for all samples approach zero, which is the collapse that the teacher-forced step guards against
[arxiv:1701.06547 §3.2]. The relative weight of the policy-gradient and maximum-likelihood steps is not stated
in the paper, so it must be tuned in any reimplementation.

For LeakGAN, the discriminator's features are the input to the generator, so they must be recomputed as the
discriminator trains. The paper's appendix reports that a fully connected layer in place of the hierarchy was
numerically unstable [arxiv:1709.08624 app. Discussions]. Its rewards are rescaled by within-batch rank
[arxiv:1709.08624 eq. 11]; the card's paper-vs-code notes question whether the code's ranking matches the
paper, so treat that step as unverified.

SeqGAN's loop differs from the dialogue sketch in two ways. It draws fresh negatives from the current generator at
every discriminator step, and it keeps a roll-out copy of the generator that is reset once per iteration
[arxiv:1609.05473 alg. 1]. The released code instead keeps that copy as an exponential average of the generator
[arxiv:1609.05473 §Paper vs code]. Its reward is used directly as the action value, with no baseline subtracted
[arxiv:1609.05473 §Pitfalls].

The evasion loop is shorter. Each round feeds the previous output back in, and the optional step keeps the
lowest-scoring round when detector scores are available [arxiv:2303.11156 §2.1, §2.3]. The released code is
under a non-commercial research notice, so this sketch is our own and copies no code
[arxiv:2303.11156 §Paper vs code].

```pseudocode
# Recursive paraphrase against a fixed detector [arxiv:2303.11156 §2.1]
# Input: AI passage S, detector score function score(), rounds = 5
current = S
for round in 1..5:
    # Rewrite the whole current passage in one call [arxiv:2303.11156 §2.1]
    current = paraphraser(current)
    # Keep every round so the lowest score can be selected [arxiv:2303.11156 §2.3]
    record(round, current, score(current))
```

The classifier attack and PAIR share one outer loop: propose a change, query the victim, keep the change if it
succeeds. They differ in what proposes the change, a synonym table with a part-of-speech filter
[arxiv:1907.11932 alg. 1], or an attacker LLM that reads the history [arxiv:2310.08419 alg. 1]. The released
classifier code uses a synonym threshold of 0.5, not the paper's 0.7, and measures similarity on a different
scale [arxiv:1907.11932 §Paper vs code].

## Contradictions

Few direct contradictions appear across the nine papers, because most of them measure different things. The one
exception is the SeqGAN and Texygen pair, given below. The dialogue paper
scores responses with AdverSuc, a judge's accuracy at telling them from human replies [arxiv:1701.06547 §4.1].
The long-text paper scores samples with an oracle's negative log-likelihood, BLEU and a Turing-style test
[arxiv:1709.08624 §Experiment, §Evaluation Metrics]. The detector paper scores detectors with TPR at 1% FPR and
AUROC [arxiv:2303.11156 §2.3]. The classifier paper reports accuracy after attack and the share of changed words
[arxiv:1907.11932 tab. 3]. The jailbreak paper reports a rate judged by a separate judge model
[arxiv:2310.08419 §4 Evaluation]. Their numbers should not be compared across papers. The prompt-attack paper on
text-to-image models counts an attack success rate as the sum of a pre-attack rate and a post-attack rate, so it
counts prompts that already succeed without an attack [arxiv:2310.11868 §5.1]. Its numbers are not comparable
with the other eight either.

Within the classifier paper, the text and its table disagree. The text gives BERT's Yelp accuracy as falling from
95.6% to 6.8%, while Table 3 gives 97.0% to 6.6% [arxiv:1907.11932 §Results, Automatic Evaluation] [arxiv:1907.11932
tab. 3]. The concept page uses the table.

### Masked-LM replacement and insertion (BAE)

The attacker is black-box and soft-label: it may query the classifier for output probabilities but has no
weights, gradients or training data [arxiv:2004.01970 §2]. Each token is scored by how much deleting it lowers the
probability of the true label. Tokens are then visited from most to least important. At each position the
attacker masks the word and takes the top 50 predictions of a BERT base masked language model [arxiv:2004.01970 §2,
app. A]. A candidate survives if its sentence-level similarity to the input passes a threshold of 0.8 under the
Universal Sentence Encoder, and, for replacements only, if it has the same part of speech [arxiv:2004.01970 app. A].
The attack stops when the label flips [arxiv:2004.01970 alg. 1]. The paper defines four modes: replace only (BAE-R),
insert only (BAE-I), a per-token choice between the two (BAE-R/I), and replace followed by insert at the same
position (BAE-R+I) [arxiv:2004.01970 §2]. Card: [BAE: BERT-based Adversarial Examples for Text Classification](/papers/2004.01970).

On seven datasets, author-reported test accuracy with a word-LSTM target falls from 88.0 to 4.0 on Amazon for
BAE-R+I, against 31.0 for TextFooler [arxiv:2004.01970 tab. 1]. On a BERT target, BAE-R+I gives 16.0 on IMDB,
against 32.0 for TextFooler and 46.0 for BAE-R [arxiv:2004.01970 tab. 1]. On the TREC question-type set, the
word-CNN target reaches 8.4 under BAE-R+I [arxiv:2004.01970 tab. 2]. The test sets for Amazon, Yelp and IMDB have
only 100 items each, and the paper reports no variance or repeated runs [arxiv:2004.01970 tab. 5]. In a small human
check on a BERT target, raters scored BAE-R outputs as more natural than TextFooler outputs on Amazon, IMDB and MR,
on a 1 to 5 scale [arxiv:2004.01970 tab. 3, app. C]. The authors' own text and table disagree on one point, which is
noted under Contradictions.

Within the jailbreak paper, the headline figures depend on which table is read. The default Mixtral-to-Vicuna setting
reads 88% in Table 2 and 93% in Table 7. The text rounds GPT-3.5 and GPT-4 to 50%, where Table 2 gives 51% and 48%
[arxiv:2310.08419 §4] [arxiv:2310.08419 tab. 2] [arxiv:2310.08419 tab. 7]. The concept page uses Table 2.

Within the detector paper, the introduction and §2.4 report different numbers for the same detectors. The
introduction gives a zero-shot AUROC of 25.2% after attack, while §2.4 reports 59.8% for DetectGPT with the T5
paraphraser [arxiv:2303.11156 §1] [arxiv:2303.11156 §2.4]. The introduction also gives a RoBERTa-based TPR of
60%, while §2.4 reports about 92% for one round. The paper does not say which setting the introduction quotes
[arxiv:2303.11156 §Pitfalls]. The concept page reports the §2.4 values only.

Within the long-text paper, the paper and its released code disagree on several settings: learning rate,
discriminator dropout, the interleaved supervised schedule and the rescaling constants. The concept page does
not resolve these; the card lists each one with both sides cited [arxiv:1709.08624 §Paper vs code].

Within the BAE paper, the text calls TextFooler sometimes stronger than BAE-R on a BERT target, but the tables give
TextFooler the lower accuracy on six of seven datasets, and BAE-R the lower one only on TREC
[arxiv:2004.01970 §3] [arxiv:2004.01970 tab. 1] [arxiv:2004.01970 tab. 2]. The concept page reports the tables.

The SeqGAN paper and Texygen disagree on whether SeqGAN beats MLE on real text. SeqGAN reports higher BLEU-2 for
SeqGAN than for MLE on Chinese poems, 0.7389 against 0.6670 [arxiv:1609.05473 tab. 2]. Texygen reports the two as
close on COCO captions, with BLEU-2 at 0.745 for SeqGAN and 0.731 for MLE. On the same captions MLE has the higher
BLEU-4 (0.305 against 0.294) and BLEU-5 (0.189 against 0.180), and lower Self-BLEU at every order, so its output
is more diverse [arxiv:1802.01886 tab. 2, tab. 3]. The datasets, BLEU settings and training budgets differ, so the
concept page reports both sides and does not resolve the gap.

## Papers

- [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547): trains a Seq2Seq dialogue generator with a learned discriminator as the reward, using policy gradient, per-prefix Monte Carlo rewards and interleaved maximum-likelihood updates.
- [Long Text Generation via Adversarial Training with Leaked Information](/papers/1709.08624): LeakGAN, a text GAN whose generator takes the discriminator's intermediate features through a Manager–Worker hierarchy and is trained with policy gradient on long sequences.
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](/papers/1609.05473): trains an LSTM text generator as a policy, with a CNN discriminator's score on finished sequences as the reward, and Monte Carlo roll-outs to give each token a value.
- [Texygen: A Benchmarking Platform for Text Generation Models](/papers/1802.01886): an open platform that packages seven text-generation baselines and five metrics on a synthetic oracle and COCO captions, and reports how BLEU and Self-BLEU trade off.
- [BAE: BERT-based Adversarial Examples for Text Classification](/papers/2004.01970): a black-box attack that ranks words by importance and fills each position with BERT's masked-LM predictions, replacing or inserting words under sentence-similarity and part-of-speech filters.
- [Can AI-Generated Text be Reliably Detected?](/papers/2303.11156): stress-tests AI-text detectors with recursive paraphrasing and spoofing, and bounds any detector's AUROC by the total variation distance between human and AI text.
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](/papers/2310.11868): UnlearnDiffAtk, which prepends a few optimised tokens to a prompt and scores them with the unlearned model's own denoising error, to bypass concept, style and object unlearning.
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](/papers/1907.11932): TextFooler, a black-box word-substitution attack that ranks words by influence and swaps in embedding synonyms that pass similarity and part-of-speech checks.
- [Jailbreaking Black Box Large Language Models in Twenty Queries](/papers/2310.08419): PAIR, a black-box method in which an attacker LLM refines a jailbreak prompt from the target's replies and a judge's scores.
- [BERT: A Review of Applications in Natural Language Processing and Understanding](/papers/2103.11943): a 2021 review of BERT that restates the TextFooler attack and its BERT numbers second-hand; used here only as a cross-check.