Adversarial evaluation

Overview

Human ratings of generated text are slow and expensive, so an automatic judge is attractive. Adversarial evaluation uses a classifier or a model as the judge. In the dialogue setting, the classifier is trained to label texts as human-written (positive) or machine-written (negative). Its accuracy on held-out data is turned into a score: AdverSuc is one minus that accuracy, so higher is better and 0.5 means the classifier cannot tell the two apart 1701.06547 §4.1.

The evaluator must not share parameters with any discriminator used to train the generator. Otherwise the model is graded by a judge it was optimized to fool 1701.06547 §4. The paper applies the idea to dialogue responses, using the two preceding utterances as context 1701.06547 §3, §5.1.

A high AdverSuc alone does not show good output. Random or degenerate text can also confuse a classifier, so the paper pairs the score with two reliability checks, described below 1701.06547 §4.2, §4.3.

A dialogue survey describes the same kind of metric: an adversarial discriminator is trained to tell generated from real dialogue, and its accuracy is used as the score. The survey places it next to the Turing test as a measure of human-likeness 1905.04071 §4.3.1.

The word is also used for four other patterns. One feeds a model adversarial inputs and has a judge label each answer, as in the multimodal safety benchmark 2311.17600 §3.3. A second perturbs the input inside a small norm ball and scores the output with reference-based metrics rather than a judge 2308.10741 §4, §5. The third uses models to reject items during benchmark construction, so that the kept items are harder to solve by shortcuts 2101.02235 §3.1. The third pattern measures the data, not the model's output. The fourth asks annotators to write items that fool the current target model, in rounds, and reports how often they succeed 2012.15761 §5, §6.1. It builds data like the third pattern, but its success rate is measured against the model in the loop.

Variants

Six cards cover this concept. The method paper 1701.06547 defines the classifier score and its checks. The survey 1905.04071 places the same metric among other dialogue metrics and reports the human-agreement evidence below. The multimodal benchmark 2311.17600 uses a judge-scored attack success rate. The robustness paper 2308.10741 attacks a vision-language model with bounded perturbations. The question-answering benchmark 2101.02235 uses model-in-the-loop filtering. The hate-speech dataset paper 2012.15761 runs dynamic data collection against a series of target models.

Held-out classifier score (AdverSuc)

Train an evaluator on human and model responses, then test it on held-out pairs. The paper compares four evaluator designs: an SVM on unigrams, a concatenated-sentence neural model, a hierarchical neural model, and an SVM that adds features from the hierarchical model and from language-model likelihoods 1701.06547 §5.1. Its reliability error, lower is better, is 0.232, 0.209, 0.193 and 0.152 in the same order 1701.06547 tab. 2. Card: Adversarial Learning for Neural Dialogue Generation.

Reliability checks

• Evaluator reliability error (ERE). The mean absolute gap between the evaluator's AdverSuc and a gold value over four constructed tasks: human against human, machine against machine, human against randomly chosen responses, and human against a response replaced by the utterance that followed the true one. Gold values are 0.5, 0.5, 0 and 0 1701.06547 §4.2.
• Machine-vs-random accuracy. How well the evaluator separates model responses from randomly sampled human responses. High values mean the evaluator is not simply fooled by random-looking text 1701.06547 §4.3.

The paper's table of results shows why both checks matter. Sampling-based decoding reaches an AdverSuc of 0.372, far above the other systems, but its machine-vs-random accuracy is only 0.679 1701.06547 tab. 3. Greedy MLE decoding scores 0.049 and 0.945 on the same two measures 1701.06547 tab. 3.

Discriminator as a human-likeness metric (survey view)

The survey groups the adversarial discriminator with the other learned and human-likeness metrics for chatbots, and names the Turing test as the classic idea 1905.04071 §4.3.1. It reports that a discriminator reached 62.5% accuracy in the first adversarial study it cites. It also reports that a cited study compared discriminator and human judgements on three corpora and found low human agreement. Human accuracy on the same task was 64% to 67.7%, and on Switchboard the Fleiss π was 0.3 for human judges and 0.07 for discriminator against human judges 1905.04071 §4.3.1. Card: Survey on Evaluation Methods for Dialogue Systems.

LLM judge and attack success rate (multimodal safety)

Build adversarial inputs, run each model zero-shot, and have a judge label each answer safe or unsafe. The benchmark pairs 1,680 harmful questions across 13 scenarios with three generated images each: a Stable Diffusion image, a typography image, and both stacked. Attack success rate (ASR) is the share of questions whose answer engages with the malicious query. Refusals and off-topic answers count as zero 2311.17600 §3.2 2311.17600 §3.3.1. The judge is GPT-4, and the label rule depends on the scenario. Most scenarios are unsafe if any harmful content appears. Politics and government scenarios are safe only on a clear refusal to take part. Legal, financial and health scenarios are safe only when the answer says the model is unqualified and gives a risk warning 2311.17600 §3.3.

The authors report that typography raises LLaVA-1.5-7B's average ASR from 41.0 (text only) to 71.5, and that a one-sentence safety prefix cuts the stacked-image ASR on the default set from 72.1 to 17.1 2311.17600 tab. 3 2311.17600 tab. 4. Card: MM-SafetyBench.

White-box perturbation test with reference metrics (robustness view)

Perturb the input inside a small l-infinity ball, optimise the perturbation against the model's own output likelihood, and measure how far the output moves from a reference. The attacker needs all weights of the model under test, so the attack is white-box 2308.10741 §4. The paper attacks OpenFlamingo 9B, bounding the query image by radius 1/255 or 4/255 and optionally the few-shot context images too, and optimises with APGD 2308.10741 §4, §5. The pixel scale of the radius is not stated, so the radius means nothing until the input scale is fixed 2308.10741 §5.

The score is not a classifier or a judge. Untargeted runs use CIDEr for captions and accuracy for VQA. Targeted runs use the share of outputs that contain the chosen target caption, plus BLEU-4 2308.10741 §5. Zero-shot at 4/255 with 500 steps, COCO CIDEr falls from 84.01 to 1.69, and the target "Please reset your password" succeeds on 100% of images 2308.10741 tab. 3 2308.10741 tab. 2. The authors note that 1.69 is below the mean CIDEr of random permutations of 1000 COCO captions (3.01) 2308.10741 §2 2308.10741 §6.

Results depend on the step count. At 1/255 on 100 COCO images, targeted success is 0.0 at 10 steps, 56.0 at 500 and 83.0 at 5000 2308.10741 tab. 1. Table 1 and Table 3 give different zero-shot CIDEr for the same 1/255 setting (15.72 and 9.59), so compare numbers only within one table 2308.10741 tab. 1 2308.10741 tab. 3. Card: On the Adversarial Robustness of Multi-Modal Foundation Models.

Model-in-the-loop item filtering (benchmark construction)

Write candidate items, then let models try them. Items that the models solve too easily are rejected. In the question-answering benchmark, five pretrained solvers reject a question if at least four answer it correctly. Three solvers fine-tuned during collection reject it if all three answer correctly 2101.02235 §3.1. The kept set has 2,780 yes/no questions, and the collection pipeline is described in 2101.02235 §3, §4. The card reports agreement on the final labels (κ = 0.73 for implicit reasoning, 0.65 for multi-step), but no ablation of the filter itself 2101.02235 §4.2. Card: Did Aristotle Use a Laptop?.

Dynamic adversarial data collection (in-loop error rate)

Run the collection in rounds. Annotators write new items that the current target model gets wrong; each round's data is validated, and the next target is trained on the accumulated data. The in-loop error rate of a round is the share of annotator entries that tricked the target 2012.15761 §6.1. In the hate-speech study, the round-1 target is wrong on 54.7% of entries, and the round-4 target on 27.7% 2012.15761 tab. 3. Over the same rounds, the authors report higher macro F1 on each round's test set and higher HateCheck accuracy (60% for the first target, 95% for the fourth) 2012.15761 tab. 4 2012.15761 §6.3. Half of rounds 2 to 4 are label-flipping perturbations written offline, without model feedback, so they are outside the in-loop rate 2012.15761 §5.2 2012.15761 tab. 3. Later rounds also contain harder content, and the authors say the falling error rate understates how much harder they became 2012.15761 §6.1. Error rates are therefore not comparable across rounds as a measure of progress. Card: Learning from the Worst.

When to use which

The evidence comes from one method paper on dialogue with OpenSubtitles data 1701.06547 §3.3, one survey that reports other studies' numbers 1905.04071 §4.3.1, one multimodal benchmark whose numbers are the authors' own and rely on a single GPT-4 judge 2311.17600 §4.2, one white-box robustness paper that tests a single open model 2308.10741 §5, one question-answering dataset paper that reports dataset-quality agreement rather than evaluator quality 2101.02235 §4.2, and one hate-speech dataset paper whose round-by-round gains and HateCheck results are author-reported and use one target architecture, RoBERTa-base, and report batch size and other training settings as not stated 2012.15761 §6 2012.15761 app. D. The reliability numbers in the method paper are author-reported. Do not assume they hold for other tasks or domains without measuring them.

Implementation sketch

Python-flavoured outline of the classifier-score evaluation. The helper functions are placeholders for the parts the paper describes. The judge-based variant replaces the classifier with a judge call per answer and keeps the same ASR count.

# Train evaluator E on labeled human (1) vs model (0) replies, held out
# from any discriminator used during generator training [§4]
E = train_evaluator(human_train, model_train)

# AdverSuc = 1 - accuracy on held-out human vs model replies [§4.1]
acc = accuracy(E, human_heldout, model_heldout)
adversuc = 1 - acc      # 0.5 means indistinguishable

# Machine-vs-random: model replies vs random human replies [§4.3]
mvr = accuracy(E, model_heldout, random_human_heldout)

# ERE: mean absolute error over four constructed tasks [§4.2]
gold = [0.5, 0.5, 0.0, 0.0]
scores = [score_task(E, t) for t in constructed_tasks]
ere = mean(abs(s - g) for s, g in zip(scores, gold))

Contradictions

No direct contradiction, because the cards measure different things. The method paper checks its evaluator with constructed tasks (ERE) and machine-vs-random accuracy. Its human study compares pairs of replies with three crowd judges per item 1701.06547 §4.2, §4.3 1701.06547 §5.2. The survey reports a cited study in which discriminator and human judgements agreed poorly on three corpora 1905.04071 §4.3.1. The multimodal benchmark uses a GPT-4 judge and states the sample size and agreement statistic for its manual check as not stated 2311.17600 §3.3.3, so it gives no human-agreement number to compare. The question-answering paper's agreement figures measure annotator agreement on dataset labels, not an evaluator's agreement with humans 2101.02235 §4.2. The hate-speech paper's Krippendorff's alpha values also measure annotator agreement, and fall to 0.52 to 0.55 in rounds 3 and 4 when flagged entries count as incorrect 2012.15761 §5.3 2012.15761 §5.4. The robustness paper scores outputs with reference-based metrics and reports no human check, so it neither supports nor contradicts the survey's finding on human agreement 2308.10741 §5. Both sides are stated here; the evidence does not settle whether the score tracks human judgement on a given task.

Papers

• Adversarial Learning for Neural Dialogue Generation: introduces AdverSuc, ERE and machine-vs-random accuracy for dialogue responses, alongside an adversarially trained generator.
• Survey on Evaluation Methods for Dialogue Systems: surveys dialogue evaluation, including the adversarial discriminator as a metric, and reports the human-agreement numbers cited here.
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies: uses solver models to reject easy questions while building a yes/no question-answering dataset.
• On the Adversarial Robustness of Multi-Modal Foundation Models: attacks OpenFlamingo with bounded image perturbations and scores the output with CIDEr, BLEU-4 and target-substring success.
• Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection: collects hate-speech data in four rounds against successive RoBERTa targets and reports the in-loop error rate per round.
• MM-SafetyBench: scores multimodal models on adversarial image-and-text harmful queries with a GPT-4 judge and an attack success rate.