---
kind: concept
slug: adversarial-evaluation
title: Adversarial evaluation
summary: >-
  Score model outputs with a separate classifier or LLM judge, or measure how far bounded input perturbations push a model's output. Variants: AdverSuc for dialogue, judge-scored attack success for multimodal models, a white-box perturbation test, model-in-the-loop filtering and rounds of adversarial data collection. Pair scores with reliability checks; human-agreement evidence is weak or missing.
tags: [evaluation, dialogue, discriminator, metrics, red-teaming, benchmark, robustness, dynamic-data-collection]
papers: ["1701.06547", "1905.04071", "2012.15761", "2101.02235", "2308.10741", "2311.17600"]
related: [adversarial-text-generation, adversarial-robustness, dialogue-generation, dialogue-evaluation, llm-evaluation, multimodal-language-model, dataset-curation, toxicity-detection, inter-annotator-agreement]
status: draft
updated_at: 2026-11-01
---

## Overview

Human ratings of generated text are slow and expensive, so an automatic judge is attractive. Adversarial evaluation uses a classifier or a model as the judge. In the dialogue setting, the classifier is trained to label texts as human-written (positive) or machine-written (negative). Its accuracy on held-out data is turned into a score: AdverSuc is one minus that accuracy, so higher is better and 0.5 means the classifier cannot tell the two apart [arxiv:1701.06547 §4.1].

The evaluator must not share parameters with any discriminator used to train the generator. Otherwise the model is graded by a judge it was optimized to fool [arxiv:1701.06547 §4]. The paper applies the idea to dialogue responses, using the two preceding utterances as context [arxiv:1701.06547 §3, §5.1].

A high AdverSuc alone does not show good output. Random or degenerate text can also confuse a classifier, so the paper pairs the score with two reliability checks, described below [arxiv:1701.06547 §4.2, §4.3].

A dialogue survey describes the same kind of metric: an adversarial discriminator is trained to tell generated from real dialogue, and its accuracy is used as the score. The survey places it next to the Turing test as a measure of human-likeness [arxiv:1905.04071 §4.3.1].

The word is also used for four other patterns. One feeds a model adversarial inputs and has a judge label each answer, as in the multimodal safety benchmark [arxiv:2311.17600 §3.3]. A second perturbs the input inside a small norm ball and scores the output with reference-based metrics rather than a judge [arxiv:2308.10741 §4, §5]. The third uses models to reject items during benchmark construction, so that the kept items are harder to solve by shortcuts [arxiv:2101.02235 §3.1]. The third pattern measures the data, not the model's output. The fourth asks annotators to write items that fool the current target model, in rounds, and reports how often they succeed [arxiv:2012.15761 §5, §6.1]. It builds data like the third pattern, but its success rate is measured against the model in the loop.

## Variants

Six cards cover this concept. The method paper [arxiv:1701.06547] defines the classifier score and its checks. The survey [arxiv:1905.04071] places the same metric among other dialogue metrics and reports the human-agreement evidence below. The multimodal benchmark [arxiv:2311.17600] uses a judge-scored attack success rate. The robustness paper [arxiv:2308.10741] attacks a vision-language model with bounded perturbations. The question-answering benchmark [arxiv:2101.02235] uses model-in-the-loop filtering. The hate-speech dataset paper [arxiv:2012.15761] runs dynamic data collection against a series of target models.

### Held-out classifier score (AdverSuc)

Train an evaluator on human and model responses, then test it on held-out pairs. The paper compares four evaluator designs: an SVM on unigrams, a concatenated-sentence neural model, a hierarchical neural model, and an SVM that adds features from the hierarchical model and from language-model likelihoods [arxiv:1701.06547 §5.1]. Its reliability error, lower is better, is 0.232, 0.209, 0.193 and 0.152 in the same order [arxiv:1701.06547 tab. 2]. Card: [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547).

### Reliability checks

- **Evaluator reliability error (ERE).** The mean absolute gap between the evaluator's AdverSuc and a gold value over four constructed tasks: human against human, machine against machine, human against randomly chosen responses, and human against a response replaced by the utterance that followed the true one. Gold values are 0.5, 0.5, 0 and 0 [arxiv:1701.06547 §4.2].
- **Machine-vs-random accuracy.** How well the evaluator separates model responses from randomly sampled human responses. High values mean the evaluator is not simply fooled by random-looking text [arxiv:1701.06547 §4.3].

The paper's table of results shows why both checks matter. Sampling-based decoding reaches an AdverSuc of 0.372, far above the other systems, but its machine-vs-random accuracy is only 0.679 [arxiv:1701.06547 tab. 3]. Greedy MLE decoding scores 0.049 and 0.945 on the same two measures [arxiv:1701.06547 tab. 3].

### Discriminator as a human-likeness metric (survey view)

The survey groups the adversarial discriminator with the other learned and human-likeness metrics for chatbots, and names the Turing test as the classic idea [arxiv:1905.04071 §4.3.1]. It reports that a discriminator reached 62.5% accuracy in the first adversarial study it cites. It also reports that a cited study compared discriminator and human judgements on three corpora and found low human agreement. Human accuracy on the same task was 64% to 67.7%, and on Switchboard the Fleiss π was 0.3 for human judges and 0.07 for discriminator against human judges [arxiv:1905.04071 §4.3.1]. Card: [Survey on Evaluation Methods for Dialogue Systems](/papers/1905.04071).

### LLM judge and attack success rate (multimodal safety)

Build adversarial inputs, run each model zero-shot, and have a judge label each answer safe or unsafe. The benchmark pairs 1,680 harmful questions across 13 scenarios with three generated images each: a Stable Diffusion image, a typography image, and both stacked. Attack success rate (ASR) is the share of questions whose answer engages with the malicious query. Refusals and off-topic answers count as zero [arxiv:2311.17600 §3.2] [arxiv:2311.17600 §3.3.1]. The judge is GPT-4, and the label rule depends on the scenario. Most scenarios are unsafe if any harmful content appears. Politics and government scenarios are safe only on a clear refusal to take part. Legal, financial and health scenarios are safe only when the answer says the model is unqualified and gives a risk warning [arxiv:2311.17600 §3.3].

The authors report that typography raises LLaVA-1.5-7B's average ASR from 41.0 (text only) to 71.5, and that a one-sentence safety prefix cuts the stacked-image ASR on the default set from 72.1 to 17.1 [arxiv:2311.17600 tab. 3] [arxiv:2311.17600 tab. 4]. Card: [MM-SafetyBench](/papers/2311.17600).

### White-box perturbation test with reference metrics (robustness view)

Perturb the input inside a small l-infinity ball, optimise the perturbation against the model's own output likelihood, and measure how far the output moves from a reference. The attacker needs all weights of the model under test, so the attack is white-box [arxiv:2308.10741 §4]. The paper attacks OpenFlamingo 9B, bounding the query image by radius 1/255 or 4/255 and optionally the few-shot context images too, and optimises with APGD [arxiv:2308.10741 §4, §5]. The pixel scale of the radius is not stated, so the radius means nothing until the input scale is fixed [arxiv:2308.10741 §5].

The score is not a classifier or a judge. Untargeted runs use CIDEr for captions and accuracy for VQA. Targeted runs use the share of outputs that contain the chosen target caption, plus BLEU-4 [arxiv:2308.10741 §5]. Zero-shot at 4/255 with 500 steps, COCO CIDEr falls from 84.01 to 1.69, and the target "Please reset your password" succeeds on 100% of images [arxiv:2308.10741 tab. 3] [arxiv:2308.10741 tab. 2]. The authors note that 1.69 is below the mean CIDEr of random permutations of 1000 COCO captions (3.01) [arxiv:2308.10741 §2] [arxiv:2308.10741 §6].

Results depend on the step count. At 1/255 on 100 COCO images, targeted success is 0.0 at 10 steps, 56.0 at 500 and 83.0 at 5000 [arxiv:2308.10741 tab. 1]. Table 1 and Table 3 give different zero-shot CIDEr for the same 1/255 setting (15.72 and 9.59), so compare numbers only within one table [arxiv:2308.10741 tab. 1] [arxiv:2308.10741 tab. 3]. Card: [On the Adversarial Robustness of Multi-Modal Foundation Models](/papers/2308.10741).

### Model-in-the-loop item filtering (benchmark construction)

Write candidate items, then let models try them. Items that the models solve too easily are rejected. In the question-answering benchmark, five pretrained solvers reject a question if at least four answer it correctly. Three solvers fine-tuned during collection reject it if all three answer correctly [arxiv:2101.02235 §3.1]. The kept set has 2,780 yes/no questions, and the collection pipeline is described in [arxiv:2101.02235 §3, §4]. The card reports agreement on the final labels (κ = 0.73 for implicit reasoning, 0.65 for multi-step), but no ablation of the filter itself [arxiv:2101.02235 §4.2]. Card: [Did Aristotle Use a Laptop?](/papers/2101.02235).

### Dynamic adversarial data collection (in-loop error rate)

Run the collection in rounds. Annotators write new items that the current target model gets wrong; each round's data is validated, and the next target is trained on the accumulated data. The in-loop error rate of a round is the share of annotator entries that tricked the target [arxiv:2012.15761 §6.1]. In the hate-speech study, the round-1 target is wrong on 54.7% of entries, and the round-4 target on 27.7% [arxiv:2012.15761 tab. 3]. Over the same rounds, the authors report higher macro F1 on each round's test set and higher HateCheck accuracy (60% for the first target, 95% for the fourth) [arxiv:2012.15761 tab. 4] [arxiv:2012.15761 §6.3]. Half of rounds 2 to 4 are label-flipping perturbations written offline, without model feedback, so they are outside the in-loop rate [arxiv:2012.15761 §5.2] [arxiv:2012.15761 tab. 3]. Later rounds also contain harder content, and the authors say the falling error rate understates how much harder they became [arxiv:2012.15761 §6.1]. Error rates are therefore not comparable across rounds as a measure of progress. Card: [Learning from the Worst](/papers/2012.15761).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You have human reference text for your domain and can hold out part of it | Held-out classifier score (AdverSuc) | The score is defined on held-out human and machine samples [arxiv:1701.06547 §4.1] |
| You will compare systems with a single number | Report AdverSuc together with machine-vs-random accuracy | Sampling reaches a high AdverSuc with low machine-vs-random accuracy, so one number can mislead [arxiv:1701.06547 tab. 3] |
| You cannot yet trust your evaluator | Run the four constructed ERE tasks before using the score | These are the method paper's checks of evaluator reliability [arxiv:1701.06547 §4.2] |
| Compute or data for a strong evaluator is limited | Start with a simpler evaluator and measure its ERE | In the paper's table, the stronger evaluators have lower ERE, but the paper does not report their cost [arxiv:1701.06547 tab. 2] |
| You need evidence of quality beyond the classifier | Add human pairwise judgments | The paper's main quality evidence for its own model is a human study of 200 items with three judges per item [arxiv:1701.06547 §5.2] |
| You want the score to stand in for human judgement of single replies | Measure agreement with humans on your own data first | The survey's cited study found low human agreement with discriminator judgements [arxiv:1905.04071 §4.3.1] |
| You test a model's refusals against harmful prompts with attack inputs | Judge-scored ASR, with the scenario's own label rule | ASR counts engagement with the malicious query, and the label rule differs by scenario [arxiv:2311.17600 §3.3.1] [arxiv:2311.17600 §3.3] |
| You have white-box access to a model and want to know how far a bounded image change can steer its output | White-box perturbation test with reference metrics | This is the robustness paper's threat model and its only setting [arxiv:2308.10741 §4] |
| You do not hold the weights of the model you test | Do not use the white-box variant | The attack needs all weights, and the paper says nothing about transfer to other models or to deployed APIs [arxiv:2308.10741 §4] [arxiv:2308.10741 §5] |
| Compute for attack steps is limited | Fix and report the step count; expect weaker attacks at fewer steps | Targeted success rises from 56.0 at 500 steps to 83.0 at 5000 [arxiv:2308.10741 tab. 1], and the authors call 5000 steps costly [arxiv:2308.10741 §5] |
| You want a single robustness number per model | Report CIDEr or success per target, with radius, steps and shots | The paper reports one model, with no seeds or variance [arxiv:2308.10741 §5] |
| Your judge is an external model | Record its version and sampling settings with every score | The benchmark does not state them, so its labels cannot be reproduced from the paper alone [arxiv:2311.17600 §3.3] [arxiv:2311.17600 app. E.2] |
| You build a benchmark and want items that current models do not solve easily | Model-in-the-loop filtering | Items answered correctly by enough solvers are rejected [arxiv:2101.02235 §3.1] |
| You need to know whether the filter helps | Run an ablation on your own pipeline | The card reports no ablation of the filter [arxiv:2101.02235 §3.1] |
| Your test data must keep pace with a model you are improving | Dynamic adversarial data collection | A fixed dataset cannot show new weaknesses in time, so the paper collects data against the current target each round [arxiv:2012.15761 §1] [§2] |
| You run rounds of collection and need a per-round difficulty signal | Report the in-loop error rate per round, with the target model named | The paper's rate is defined against the target of that round and is not comparable across rounds [arxiv:2012.15761 §6.1] |
| Your annotators are few or your data must generalise to new annotators | Hold out annotators in the test set and report their results separately | The paper found held-out annotators' performance dropped substantially in round 2 [arxiv:2012.15761 §5.2] |
| You need the labels to be reliable | Measure annotator agreement per round | Krippendorff's alpha is 0.815 for round-2 originals and 0.55 and 0.52 for rounds 3 and 4 with flagged entries counted as incorrect [arxiv:2012.15761 §5.2–§5.4] |
| You want to compare it with word-overlap or learned chatbot metrics | Run both on the same data | The cards list BLEU, ΔBLEU, ADEM and RUBER correlations with humans [arxiv:1905.04071 §4.3.1], but do not score adversarial evaluation against them |

The evidence comes from one method paper on dialogue with OpenSubtitles data [arxiv:1701.06547 §3.3], one survey that reports other studies' numbers [arxiv:1905.04071 §4.3.1], one multimodal benchmark whose numbers are the authors' own and rely on a single GPT-4 judge [arxiv:2311.17600 §4.2], one white-box robustness paper that tests a single open model [arxiv:2308.10741 §5], one question-answering dataset paper that reports dataset-quality agreement rather than evaluator quality [arxiv:2101.02235 §4.2], and one hate-speech dataset paper whose round-by-round gains and HateCheck results are author-reported and use one target architecture, RoBERTa-base, and report batch size and other training settings as not stated [arxiv:2012.15761 §6] [arxiv:2012.15761 app. D]. The reliability numbers in the method paper are author-reported. Do not assume they hold for other tasks or domains without measuring them.

## Implementation sketch

Python-flavoured outline of the classifier-score evaluation. The helper functions are placeholders for the parts the paper describes. The judge-based variant replaces the classifier with a judge call per answer and keeps the same ASR count.

```pseudocode
# Train evaluator E on labeled human (1) vs model (0) replies, held out
# from any discriminator used during generator training [§4]
E = train_evaluator(human_train, model_train)

# AdverSuc = 1 - accuracy on held-out human vs model replies [§4.1]
acc = accuracy(E, human_heldout, model_heldout)
adversuc = 1 - acc      # 0.5 means indistinguishable

# Machine-vs-random: model replies vs random human replies [§4.3]
mvr = accuracy(E, model_heldout, random_human_heldout)

# ERE: mean absolute error over four constructed tasks [§4.2]
gold = [0.5, 0.5, 0.0, 0.0]
scores = [score_task(E, t) for t in constructed_tasks]
ere = mean(abs(s - g) for s, g in zip(scores, gold))
```

## Contradictions

No direct contradiction, because the cards measure different things. The method paper checks its evaluator with constructed tasks (ERE) and machine-vs-random accuracy. Its human study compares pairs of replies with three crowd judges per item [arxiv:1701.06547 §4.2, §4.3] [arxiv:1701.06547 §5.2]. The survey reports a cited study in which discriminator and human judgements agreed poorly on three corpora [arxiv:1905.04071 §4.3.1]. The multimodal benchmark uses a GPT-4 judge and states the sample size and agreement statistic for its manual check as not stated [arxiv:2311.17600 §3.3.3], so it gives no human-agreement number to compare. The question-answering paper's agreement figures measure annotator agreement on dataset labels, not an evaluator's agreement with humans [arxiv:2101.02235 §4.2]. The hate-speech paper's Krippendorff's alpha values also measure annotator agreement, and fall to 0.52 to 0.55 in rounds 3 and 4 when flagged entries count as incorrect [arxiv:2012.15761 §5.3] [arxiv:2012.15761 §5.4]. The robustness paper scores outputs with reference-based metrics and reports no human check, so it neither supports nor contradicts the survey's finding on human agreement [arxiv:2308.10741 §5]. Both sides are stated here; the evidence does not settle whether the score tracks human judgement on a given task.

## Papers

- [Adversarial Learning for Neural Dialogue Generation](/papers/1701.06547): introduces AdverSuc, ERE and machine-vs-random accuracy for dialogue responses, alongside an adversarially trained generator.
- [Survey on Evaluation Methods for Dialogue Systems](/papers/1905.04071): surveys dialogue evaluation, including the adversarial discriminator as a metric, and reports the human-agreement numbers cited here.
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](/papers/2101.02235): uses solver models to reject easy questions while building a yes/no question-answering dataset.
- [On the Adversarial Robustness of Multi-Modal Foundation Models](/papers/2308.10741): attacks OpenFlamingo with bounded image perturbations and scores the output with CIDEr, BLEU-4 and target-substring success.
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](/papers/2012.15761): collects hate-speech data in four rounds against successive RoBERTa targets and reports the in-loop error rate per round.
- [MM-SafetyBench](/papers/2311.17600): scores multimodal models on adversarial image-and-text harmful queries with a GPT-4 judge and an attack success rate.