Can AI-Generated Text be Reliably Detected?

In plain words

Many tools try to tell whether a piece of text was written by a language model. Some add a hidden statistical pattern while the model writes (a watermark), some train a classifier, some check how "surprised" a language model is by the text, and some search a database of past model outputs. The authors ask a simple question: what happens when the text is deliberately changed to fool them?

Their main attack is to run the AI text through a paraphrasing model, then run the output through the paraphraser again, several times. Each round keeps the meaning mostly intact but scrambles the surface patterns the detectors rely on. The detectors that look at the text's statistics and the watermark detector both lose much of their accuracy this way. A second attack goes the other way: someone who writes plain human text can make it look AI-written, either by learning the hidden watermark pattern from a model or by mixing in an AI-flagged passage. The paper also gives a mathematical limit: how well any detector can separate two text sources depends on how different those sources are, so as AI text becomes more human-like, reliable detection gets harder.

TL;DR

• Recursive paraphrasing (mainly DIPPER; LLaMA-2-7B-Chat as a second paraphraser) weakens soft-watermark, zero-shot, trained and retrieval-based detectors on XSum news passages §2.3 §2.4.
• Paraphrase quality stays usable on average, by the authors' human and perplexity measures, though the authors report it degrades §2.2 tab. 1 tab. 3.
• Watermarks can be spoofed: a proxy of the hidden green lists learned from a target model lets a human write text the watermark detector flags §3.
• A theorem bounds any detector's AUROC by the total variation distance between the human and AI text distributions §4 Thm. 1.

Problem

AI text detectors are used to find plagiarism, spam and fake news. Their reported accuracy often assumes a non-adversarial setting. The paper asks how these detectors behave when the author of the AI text (evasion) or the accuser (spoofing) tries to beat them §1. It names four detector classes: soft watermarking, neural-network classifiers (for example a RoBERTa-based GPT-2 detector), zero-shot detectors that use log-probabilities (for example DetectGPT), and retrieval-based detectors that store past model outputs §1 §2.

The span-replacement attack from the watermark paper is treated here as a weaker attack: it replaces random tokens in place and does not remove the watermark §2.3 §1.

Method

The paper has three parts: an evasion attack, a spoofing attack, and a theoretical bound.

Evasion attack: recursive paraphrasing §2.1. An adversary takes the AI output S and sends it through a paraphraser P. The main attack is recursive: the output of round i becomes the input to round i+1. Round 1 is pp1, round 5 is pp5. DIPPER (11B) paraphrases the whole passage in one call and can take an extra context prompt C. The T5-based Parrot paraphraser (222M) works sentence by sentence §2.1. LLaMA-2-7B-Chat is prompted with a fixed system prompt that asks for a diverse, meaning-preserving rewrite app. B.2. The authors chose DIPPER as the main attack because its quality-controlling knobs give a clearer trend over rounds §2.3.

# Recursive paraphrasing attack  [§2.1] [§2.3]
# Input:  AI passage S (about 300 tokens), optional context C
# Output: list of paraphrases pp1 .. pp5
def recursive_paraphrase(passage, context, rounds=5):
    # Each round feeds the previous output back in as input  [§2.1]
    paraphrases = []
    current = passage
    for round_index in range(rounds):
        # One call rewrites the whole current passage  [§2.1]
        paraphrase = dipper(current, context)  # knobs: see Hyperparameters
        paraphrases.append(paraphrase)
        current = paraphrase
    return paraphrases

The DIPPER call in the released code differs from this sketch in two ways: it sets the diversity knobs per round and passes the previous input, not C, as the prefix. See Paper vs code.

Watermark evasion §2.3. The soft watermark of Kirchenbauer et al. seeds a pseudo-random generator from the previous token to define a green list, and the model prefers green tokens. The detector flags a passage with many green tokens §1. The attack paraphrases the watermarked output. The authors measure TPR at a fixed 1% FPR, and AUROC.

Non-watermarked evasion §2.4. For trained and zero-shot detectors the authors use GPT-2-Medium outputs and the Parrot paraphraser. For the retrieval detector they use DIPPER over OPT-1.3B outputs. They also test a multi-query setting: the attacker queries the detector ten times, generates ten paraphrases, and keeps the one with the lowest detection score §2.4. A similar "best of" selection for watermarks needs black-box access to the detector scores §2.3.

Spoofing §3.

• Watermark: the attacker queries the watermarked model about one million times on nonsense sentences built from a list of 181 common words, counts word pairs, and derives a proxy green list for each word app. D.1 §3. A human then writes text that uses those green-list pairs.
• Retrieval: the adversary gets the target LLM to paraphrase a human text, which stores the paraphrase in the database. The original is then flagged as AI text §3.
• Zero-shot and trained detectors: a human-written passage that the detector already flags is prepended to other human passages §3.

# Estimate proxy green lists from a watermarked model  [app. D.1]
# Input:  word list of size N = 181, target watermarked LLM
# Output: count matrix M of shape (N, N)
def estimate_green_lists(words, llm, pair_budget=10**6):
    # Zero counts for every ordered pair of listed words  [app. D.1]
    counts = zeros((N, N))
    while total_pairs(counts) < pair_budget:
        # Prompt with nonsense built from the listed words  [app. D.1]
        text = llm(random_nonsense(words))
        # Count adjacent pairs where both words are on the list  [app. D.1]
        for first, second in adjacent_pairs(text, words):
            counts[index(first)][index(second)] += 1
    # A word often seen after another is likely on its green list [app. D.1]
    return counts

Theory §4. Model a detector as a score D(s) with a threshold. The paper shows that the AUROC of any detector is at most one half plus the total variation distance between the AI distribution and the human distribution, minus half of that distance squared. The bound is an upper limit, holds for any two distributions, and is tight for some detector §4 Thm. 1 app. C.1 app. C.3. The authors also estimate the total variation for GPT-2 models after mapping tokens to five meta-tokens, and report that it shrinks as models grow §4 app. C.5.

\mathrm{AUROC}(D) \le \tfrac{1}{2} + \mathrm{TV}(\mathcal{M}, \mathcal{H}) - \tfrac{1}{2}\,\mathrm{TV}(\mathcal{M}, \mathcal{H})^2 \quad [\text{§4 Thm. 1}]

Here \mathcal{M} is the AI text distribution and \mathcal{H} the human one.

Hyperparameters

Values are as stated in the compiled paper unless the Where column names code. "Not stated" means the paper and the code in this card's checkout do not give the value.

Results

Author-reported. Setup: OPT-13B or OPT-1.3B target LLMs, XSum passages of about 300 tokens, DIPPER or LLaMA-2 paraphrasers, TPR at 1% FPR, unless noted.

Paper vs code

Official code: the repository linked in the abstract, owned by the first author, at code_commit. It contains the paraphrasing, theory and spoofing scripts, but not the watermark implementation, the DIPPER checkpoint or the retrieval detector. The README says to clone the external lm-watermarking and ai-detection-paraphrases repositories for those README.md.

• DIPPER knobs. The paper varies two DIPPER knobs without giving values §2.1. The code sets lexical diversity to 60 and order diversity to 0 for rounds 1 to 4, and to 60 for round 5 recursive_paraphrase.py. The DIPPER call passes 100 - diversity as the knob value recursive_paraphrase.py.
• Context and prefix. The paper's attack takes an optional context C §2.1. The code uses an empty prefix in round 1 and the previous round's input as the prefix afterwards recursive_paraphrase.py recursive_paraphrase.py. The code never passes the original prompt, so the released runs do not match the formula pp1(S) = f(S, C) in the text §2.1.
• Watermark defaults. The paper says "default parameters" app. A.2 and does not state gamma or delta. The code defaults are gamma 0.25 and delta 2.0 utils.py. The paper's z threshold of 4 app. D.1 matches the code default utils.py.
• Length. The paper uses about 300-token passages §2.3. The code's default max_new_tokens is 200 utils.py.
• Parrot knobs. The released code runs Parrot with thresholds 0.9 minus a step that grows by 0.1 rephraser.py, and the evaluation script uses one minus five knobs do_all_rephrase.py. The source bundle's unused appendix.tex lists different values (1.0, 0.96, 0.92, 0.84, 0.75) app. A.2 in source only. Neither is in the compiled text.
• Spoofing word list. The paper uses N = 181 words §3. The code's comment says "about 100 most common words plus 50 most used nouns" wordlist.py. The released list is not counted here, so the exact N is unverified.
• Spoofing budget. The paper says a million prompts app. D.1. The code stops at a million observed word pairs get_matrix.py.
• Code license. The README grants research and non-profit use only and says commercial use needs a licence from the university. There is no LICENSE file README.md. The code is therefore non-commercial. This card cites pseudocode only and quotes no code.

The DIPPER sentence window (3 per call) and the spoof pair counting match the paper's description, so they are not listed as discrepancies.

Pitfalls

• Internal inconsistencies in the text. The introduction says zero-shot AUROC drops from 96.5% to 25.2% §1, but §2.4 reports 59.8% for DetectGPT with the T5 paraphraser §2.4. The introduction also says the RoBERTa detector's TPR drops to 60% §1, while §2.4 reports about 92% for a single round §2.4. Both settings differ, but the paper does not say which setting the introduction quotes.
• Perplexity baseline. The text gives AI-text perplexity 5.5 §2.2, but the table gives 5.2 tab. 3. Use the table value if you calibrate on it.
• Table 2 spread. The "All ppi" average content rating is 4.05 ± 0.2 tab. 2, while the per-round standard deviations are 0.63 to 0.78. The ± 0.2 looks like an error in the table, so do not reuse it as a confidence interval.
• Best-of-rounds needs detector queries. The 99.8% to 4.0% watermark result selects the worst-scoring paraphrase. That needs black-box detector access §2.3. The pure five-round attack does not.
• Retrieval attack is only partly reported. The text says the detector still flags almost all outputs after one round §2.4. The drop to about 60% is given only for five rounds; the values for rounds 2 to 4 appear only in the figure §2.4.
• Theory limits. The bound holds for any two distributions and any detector §4 Thm. 1, but the paper cannot estimate the total variation for real text exactly. Its estimates use five meta-tokens and GPT-2 outputs §4 app. C.5. Read the bound as a direction and a ceiling, not a measurement.
• Numbers from the bound. For AUROC 0.9 the bound needs TV of about 0.55. The text says "TV > 0.5 based on the figure" §4. For TV 0.2 the bound gives AUROC 0.68, so the text's "below 0.7" holds.
• Appendix not in the compiled PDF. The source bundle includes an appendix.tex that main.tex does not include. It contains the full experimental setup, including perplexity with OPT-2.7B, the PEGASUS paraphraser and hardware details. Those statements are not part of the compiled paper and are not used on this card.
• Quality measures are rough. Quality is judged by MTurk raters (3 per paraphrase, 20 passages), perplexity under OPT-13B, and QA accuracy on 200 SQuAD items. A paraphrase rated 4 out of 5 may still need manual editing §2.2.
• Trained detector claims are specific to the setup. The authors hypothesise that trained detectors look more robust because their training data includes human text from the same source; the MAGE dataset includes XSum passages §2.4. That is a hypothesis, not a result.
• Cost. Five DIPPER rounds take about 36 seconds per 300-token passage, and DIPPER is an 11B model app. A.2 §2.1.

Lineage

• Builds on: Kirchenbauer et al. soft watermarking, the source of the green-list scheme attacked here §1 §2.3; Krishna et al.'s DIPPER paraphraser and retrieval detector §2.1 §2.4; Mitchell et al.'s DetectGPT §2.4; OpenAI's RoBERTa-based GPT-2 detector §2.4.
• Compared against: the span-replacement watermark attack of Kirchenbauer et al., which the paper calls weaker §2.3; Zhang et al. (2024) and SICO, which the paper calls substitution-based attacks with different goals §1.
• Followed by: Gu et al. (2024) build on the spoofing attack by distilling a watermarked model into a student §1.
• Superseded by: not stated in this paper.