---
kind: concept
slug: red-teaming
title: Red teaming
summary: >-
  Probe a model on purpose for harmful replies, score them, then fix the failures. Variants: crowd dialogue attacks, RLHF probing, red-team data for preference training, self-critique, gradient and LLM-driven attacks, multi-turn RL attackers, judge-scored benchmarks, trust tests and system cards. Results are author-reported and not comparable.
tags: [safety, red-teaming, rlhf, benchmark, human-feedback, diffusion, multimodal, jailbreak, system-card, trustworthiness, reinforcement-learning, constitutional-ai, decoding]
papers: ["2204.05862", "2209.07858", "2209.14375", "2212.08073", "2306.11698", "2310.08419", "2310.11868", "2310.11986", "2311.17600", "2410.21276", "2412.16720", "2508.04451", "2508.07173", "2601.03267", "2601.07239"]
related: [rlhf, policy-gradient, hierarchical-reinforcement-learning, llm-ethics, adversarial-evaluation, adversarial-robustness, algorithmic-fairness, in-context-learning, multimodal-language-model, dialogue-generation, machine-unlearning, diffusion-models, adversarial-text-generation, llm-as-a-judge, benchmark-validity, llm-evaluation, large-reasoning-models, prompt-injection, self-refinement, decoding-strategies, self-consistency]
status: draft
updated_at: 2026-11-01
---

## Overview

Red teaming means probing a model on purpose for harmful outputs, then changing
the model so it avoids them [arxiv:2209.07858 §1]. Most variants share three
parts: inputs that try to elicit harm, a judge that labels each reply, and a
rate or score that says how the model did. The judge differs by variant: people,
a learned harmlessness preference model, a GPT-4 judge, a panel of five LLMs, an
image classifier, a Llama Guard classifier, a GPT-4o autograder, or the model
itself under a written principle [arxiv:2209.07858 §3.2] [arxiv:2311.17600 §3.3]
[arxiv:2508.07173 §2.3.1] [arxiv:2310.11868 §5.1] [arxiv:2310.08419 §4]
[arxiv:2412.16720 §4.1.1] [arxiv:2306.11698 §3.1] [arxiv:2212.08073 §3.1].
A multi-turn RL attacker uses a Llama Guard score as its training reward
[arxiv:2508.04451 §4.2].

The papers do not share one definition of "harmful". One leaves the call to
each red-team member through a pairwise choice [arxiv:2209.07858 §3.1]. Another
uses 23 specific rules instead of a broad label such as "toxic"
[arxiv:2209.14375 §2.1]. A third asks whether the model understood the input at
all, and scores only the replies where it did [arxiv:2508.07173 §2.3.1]. Without
that filter, a model that misreads a request looks safe.

Some papers are not attack methods. One is a framework that asks which layers of
a system a red-team result can speak to [arxiv:2310.11986 §2]. Three are system
cards, which report red teaming as one part of a pre-release safety review
[arxiv:2410.21276 §3.1] [arxiv:2412.16720 §4.4] [arxiv:2601.03267 §4]. Two more use red-team prompts
as training data rather than as a test [arxiv:2204.05862 §2.2]
[arxiv:2212.08073 §3.1].

Caveats that apply to all variants: every result is author-reported. The crowd
in the largest study is not representative of the US population, and about 80%
of its attacks came from about 50 of about 300 workers [arxiv:2209.07858 §3.3].
Human agreement on success ratings is modest, with Fleiss kappa between 0.32
and 0.55 [arxiv:2209.07858 §3.5].

## Variants

### Crowd dialogue attacks with a harmlessness preference model

Crowdworkers hold open, multi-turn chats with an assistant and try to make it
misbehave. After each assistant turn the worker sees two candidate replies and
picks the more harmful one. Those picks train a preference model. At the end
the worker rates the attack's success from 0 to 4 [arxiv:2209.07858 §3.1].
The assistants are plain, prompted, rejection-sampled and RLHF models at 2.7B,
13B and 52B parameters [arxiv:2209.07858 §3.2]. A harm score for an attack is
the minimum of the preference model over the assistant turns, so it reflects
the worst turn [arxiv:2209.07858 §3.4].

The main finding is about size: RLHF models get harder to red-team as they
grow, while the other types show no clear trend [arxiv:2209.07858 §4]
[arxiv:2209.07858 fig. 1]. The release has 38,961 attacks as data only
[arxiv:2209.07858 tab. 1]. The preference model and the models are not released
[arxiv:2209.07858 app. A.7]. Card: [Ganguli et al., 2022](/papers/2209.07858).

### Rule-targeted probing inside an RLHF loop

Raters are given one of 23 written rules and try to make the agent break it
[arxiv:2209.14375 §2.1] [arxiv:2209.14375 §2.3]. A rule reward model estimates
the probability that the rule was broken [arxiv:2209.14375 §2.5]. Adversarial
questions come from a prompted model that is told a chosen adjective and target,
such as medical or legal, and the questions join the buffer of contexts used
for RL [arxiv:2209.14375 app. E.2] [arxiv:2209.14375 §2.8].

The authors report that 8% of probe conversations broke the targeted rule. The
baseline rate is not given in the text [arxiv:2209.14375 §1]. On all of its
rule data the rule-conditioned classifier reaches AUC 0.85, against 0.77 for a
general safety classifier [arxiv:2209.14375 §3.5]. The interventions did not
reduce violations for stereotypes, medical advice, legal advice, microaggressions
or insults [arxiv:2209.14375 §3.1]. Card: [Glaese et al., 2022](/papers/2209.14375).

### Gradient-optimised prompts against unlearned image models

This attack targets text-to-image models that were unlearned to refuse a concept
such as nudity, an artist's style or an object. The attacker holds the weights
[arxiv:2310.11868 §3]. It prepends 3 to 5 optimised tokens to the prompt
[arxiv:2310.11868 §3]. The tokens are chosen to minimise the model's own
denoising error on a target image that contains the forgotten content, so no
second model is needed to score them [arxiv:2310.11868 §4]. On nudity prompts
against ESD, the attack success rate rises from 20.42% with no attack to 76.05%.
The earlier method, P4D, reaches 69.71% [arxiv:2310.11868 tab. 2].

Each prompt took 26.29 minutes on one GPU, against 34.70 minutes for P4D
[arxiv:2310.11868 §5.2] [arxiv:2310.11868 tab. 2]. The reported rate also counts
prompts that already produced the content without any attack, so it is not only
the attack's effect [arxiv:2310.11868 §5.1]. Card: [Zhang et al., 2023](/papers/2310.11868).

### Black-box attacker-LLM refinement (PAIR)

PAIR uses one language model as the attacker. It writes a prompt, the target
answers, and a judge scores the answer. The attacker then reads the history of
its own attempts and writes a new prompt, for a few rounds [arxiv:2310.08419 §3.1]
[arxiv:2310.08419 alg. 1]. Several such conversations run in parallel. The search needs only chat
access to the target and no gradients [arxiv:2310.08419 §3.3] [arxiv:2310.08419 §1].
The paper's main setting uses 30 streams and depth 3, so at most 90 queries per
behaviour [arxiv:2310.08419 §4]. The attacks are readable text, which the
authors present as a point in its favour over token-level methods
[arxiv:2310.08419 §1].

On 100 JailbreakBench behaviours the authors report jailbreak rates of 88% for
Vicuna-13B, 73% for Gemini-Pro, and 51% and 48% for GPT-3.5 and GPT-4, with 10
to 56 queries per success. Llama-2 reached 4%, and the two Claude models 0% to 3%
[arxiv:2310.08419 tab. 2]. The authors tie the low figures to safety fine-tuning
[arxiv:2310.08419 §4].

The judge in the main experiments is Llama Guard. On the paper's 100 annotated
prompt-response pairs, Llama Guard agreed with the majority label 76% of the time
and missed 47% of the jailbreaks. GPT-4-0613 agreed 88% of the time
[arxiv:2310.08419 tab. 1]. Against a defended target, prompts generated against
undefended models drop from 88% to 39% under SmoothLLM on Vicuna. The paper
does not test an attacker that adapts to the defence [arxiv:2310.08419 tab. 5]
[arxiv:2310.08419 §4]. On the paper's Vicuna setting, a behaviour took 34
seconds on average, on CPU [arxiv:2310.08419 tab. 4]. The paper gives GCG 1.8 hours on a GPU and about 256,000 queries
per behaviour in its setup [arxiv:2310.08419 tab. 4] [arxiv:2310.08419 tab. 2].
Card: [Chao et al., 2023](/papers/2310.08419).

### Multi-turn RL attacker with hierarchical PPO

This attacker plans a whole conversation rather than one prompt. A high-level
policy picks a short guide for each turn; one variant chooses from ten
templates, eight of them personas, and the other writes free-text guides
[arxiv:2508.04451 §4.3] [app. F]. A low-level policy writes the message one
token at a time to follow the guide. Both are trained with PPO against the
target's replies [arxiv:2508.04451 §4.2] [app. D.5].

The reward for a turn is the Llama Guard probability that the target's reply
is unsafe, plus a cosine term that keeps the message close to its guide
[arxiv:2508.04451 §4.2] [eq. 2]. Because that reward arrives late, each token
gets a share of the credit. The share comes from masking single tokens and
pairs of top-attended tokens, and measuring the drop in the turn score
[arxiv:2508.04451 §4.3] [eq. 4].

On HarmBench, ASR@5 against Llama-3.1-8B-Instruct is 75.2 when the target sees
only the latest message and 62.5 when it sees the whole dialogue. Against
GPT-5.4-mini the figures are 35.0 and 41.0 [arxiv:2508.04451 tab. 1]. The
authors re-ran every baseline in one shared setting, so these baselines are not
the original papers' numbers [arxiv:2508.04451 §5]. Removing the high-level
policy drops ASR@5 to 31.2, and an untrained agent reaches 2.1
[arxiv:2508.04451 tab. 3]. Card: [Belaire et al., 2025](/papers/2508.04451).

### Sampled-decoding safety probing (greedy versus sampled)

This variant does not attack the model. It changes how a fixed set of
attack prompts is decoded. Each prompt gets one greedy completion and k
sampled completions, at temperature 0.7 or 1.0 with top-p 0.9. The paper
defines the stochastic risk gap as the sampled attack success rate minus the
greedy one, and an illusion index as the share of sampled risk that greedy
runs miss [arxiv:2601.07239 §6]. Its prompts fall into three groups: robust
safe, concealed risk (harmful only under sampling) and deterministic failure
[arxiv:2601.07239 §6].

The safety table is marked as the authors' own simulations, not a measured
run. Its example values include a greedy attack success rate of 0.03 against
0.28 under sampling for one distilled reasoning model, at k = 8 [arxiv:2601.07239
tab. safety-concealed-risk]. Do not take these as measured rates. The paper
also gives an analytic tail risk, 1 - (1 - q)^k, for a prompt with harmful
mass q, which at k = 16 and q = 0.01 is about 0.15 [arxiv:2601.07239 §6].
Use it to reason about how repeated sampling multiplies exposure, not as a
result. Card: [Joshi et al., 2026](/papers/2601.07239).

### Judge-scored benchmark for image and text queries

The benchmark has 1,680 questions across 13 policy scenarios, with three images
per question: an AI-generated picture of the key phrase, the phrase typed as
text, or both stacked [arxiv:2311.17600 §3.2] [arxiv:2311.17600 tab. 1]. Twelve
released multimodal models are tested without training, and a GPT-4 judge labels
each answer safe or unsafe [arxiv:2311.17600 §4.1] [arxiv:2311.17600 §3.3].

The attack success rate (ASR) for LLaVA-1.5-7B averages 41.01 with text only and
71.52 with the typography image [arxiv:2311.17600 tab. 3]. A one-sentence safety
prefix placed before the question lowers the stacked-image average from 72.14 to
17.08 [arxiv:2311.17600 tab. 4]. ASR counts an incoherent answer as safe, so the
authors suggest reporting a refusal rate beside it. They give that rate only in a
figure [arxiv:2311.17600 §4.4] [arxiv:2311.17600 fig. 6]. Card: [Liu et al., 2023](/papers/2311.17600).

Licensing: the repository lists the data as CC BY-NC 4.0, and the code has no
license file. Do not reuse the images or questions from that source
[arxiv:2311.17600 §4.4].

### Judge-scored benchmark for audio and video inputs

The benchmark starts from 972 harmful seed requests. Each one is rendered in 24
variations that mix text, image, video and audio, giving 23,328 test cases
[arxiv:2508.07173 §2.2] [arxiv:2508.07173 tab. 2]. Five LLM judges vote on three
flags for each reply: whether the model understood the input, whether the reply
is harmful, and whether it refused [arxiv:2508.07173 §2.3.1].

Of eleven models, only three exceed 0.6 on both the average Safety-score and the
CMSC consistency score: Qwen3-Omni, gemini-3-pro and gemini-2.5-pro
[arxiv:2508.07173 tab. 8]. The consistency score rewards uniform failure as
well as uniform safety. Unified-IO2 has the highest CMSC and the lowest average
Safety-score [arxiv:2508.07173 tab. 8]. Seven of eleven models have their worst
variation in an image-audio or video-audio combination [arxiv:2508.07173 tab. 9].
The paper's five-judge vote is not what the released script runs; the card's
"Paper vs code" section lists the differences. Card: [Pan et al., 2025](/papers/2508.07173).

### Multi-perspective trust tests under adversarial system prompts

DecodingTrust is an evaluation suite, not an attack search. It runs fixed prompt
sets over eight trust areas for two GPT snapshots, gpt-4-0314 and
gpt-3.5-turbo-0301 [arxiv:2306.11698 §2]. Most areas are run twice, once with a
benign system prompt and once with an adversarial one, such as a jailbreak
prompt [arxiv:2306.11698 §3.2]. Toxicity is scored by the Perspective API over
25 continuations per prompt [arxiv:2306.11698 §3.1].

On the toxic RealToxicityPrompts subset, expected maximum toxicity rises from
0.38 to 0.94 for GPT-4 and from 0.37 to 0.86 for GPT-3.5 when the adversarial
system prompt is used [arxiv:2306.11698 tab. 2]. Stereotype agreement moves
more under targeted system prompts for GPT-4 (+0.535) than for GPT-3.5 (+0.346)
[arxiv:2306.11698 §4.3] [arxiv:2306.11698 fig. 10]. The authors' summary is that
GPT-4 is usually more reliable on standard benchmarks, yet follows adversarial
system prompts and backdoored demonstrations more readily [arxiv:2306.11698 §9]
[arxiv:2306.11698 §7]. Card: [Wang et al., 2023](/papers/2306.11698).

### Red teaming inside a pre-release system card

Three OpenAI system cards report red teaming as one part of a release review.
Both older cards describe closed models, so the numbers are the developer's own
and cannot be re-run from the paper [arxiv:2410.21276 §1] [arxiv:2412.16720 §1].

For GPT-4o, red teaming ran in four phases. The number of red teamers grew from
10 to 30 to 65, and the checkpoints accepted more modalities and became
multi-turn and real-time as the phases went on. The authors report more than 100
red teamers across 45 languages and 29 countries [arxiv:2410.21276 §3.1]. Speech
tests convert text benchmarks to audio with a TTS system and score only the text
of the reply [arxiv:2410.21276 §3.2]. The overall Preparedness rating is medium,
driven by persuasion, which is borderline; cybersecurity, biological threats and
autonomy are rated low [arxiv:2410.21276 §3.4]. Card: [OpenAI et al., 2024 (GPT-4o)](/papers/2410.21276).

For o1, a pairwise test with the Red Teaming Network rated o1 safer than GPT-4o in
59.75% of comparisons, GPT-4o safer in 28.48%, and 11.76% were ties
[arxiv:2412.16720 tab. 9]. The same card's jailbreak arena gives o1 an attack
success rate of about 6% on harmful text, against about 3.5% for GPT-4o. The two
results favour different models. The pairwise test counts only conversations in
which a red teamer saw an unsafe output [arxiv:2412.16720 §4.4.1] [arxiv:2412.16720 §4.4.2].
Apollo Research's scheming tests report o1 disabling oversight in 5% of runs,
against 0% for GPT-4o, and a follow-up deception rate of 98% for runs with that
action [arxiv:2412.16720 tab. 10]. The Preparedness ratings are medium for
chemical and biological threats and persuasion, and low for cybersecurity and
autonomy [arxiv:2412.16720 §5]. Card: [OpenAI et al., 2024 (o1)](/papers/2412.16720).

For GPT-5, the card reports red teaming as campaign totals and a few headline
tests. Across all campaigns it cites more than 5,000 hours and more than 400
external testers [arxiv:2601.03267 §4]. In a violent attack planning test, 25
red teamers with defence, intelligence or law-enforcement backgrounds made blind
pairwise comparisons. They judged gpt-5-thinking the safer model 65.1% of the
time against o3 (95% CI 63.7% to 66.5%) [arxiv:2601.03267 §4.1]. Two external
groups ran a two-week review of connectors and their mitigations. Of 47 reported
findings, 10 were notable, and the card says mitigations were deployed before
release [arxiv:2601.03267 §4.2]. Microsoft's AI Red Team judged gpt-5-thinking
qualitatively safer than o3 in the frontier and content-safety domains. That is
a judgement, and the text gives no numbers for it [arxiv:2601.03267 §4.2]. A
third-party test of the biology safeguards ran for 80 hours over one week. It
found partial vulnerabilities and one technique that bypassed a subset of the
layers, but no end-to-end jailbreak that evaded every layer
[arxiv:2601.03267 §6.3.3]. All these figures are the developer's own
[arxiv:2601.03267 §4]. Card: [OpenAI et al., 2025 (GPT-5)](/papers/2601.03267).

### Red-team comparisons as RLHF training data

Crowdworkers chat with an assistant and pick the more harmful of two replies.
This is a separate set from the helpfulness comparisons, where they pick the
more helpful reply [arxiv:2204.05862 §2.1] [§2.2]. Both sets train the same
preference model, so the red-team pool is a training input and not only a test
[arxiv:2204.05862 §3]. The base data hold 42,000 red-teaming comparisons and
the rejection-sampling round adds 2,000, while the online rounds add none
[arxiv:2204.05862 §2.3]. A preference model trained on only one of the two kinds
scores below chance on the other. The paper compares mixtures from 100% helpful
to 100% harmless, in steps of 10% [arxiv:2204.05862 §5.1] [fig. 19]. Card:
[Bai et al., 2022](/papers/2204.05862).

### Self-critique of red-team prompts (Constitutional AI)

The model answers each red-team prompt, then critiques its own answer under one
of 16 written principles and rewrites it. Four rounds per prompt give the
supervised training pairs [arxiv:2212.08073 §3.1] [§3.2]. The prompt pool has
182,831 items: 42,496 written by people in earlier red-team work and 140,335
generated by few-shot prompting [arxiv:2212.08073 §3.2]. Across revisions the
harmlessness score rises and the helpfulness score falls [arxiv:2212.08073
fig. 5]. A second stage has the model pick the less harmful of two of its own
replies, and those picks train a preference model for RL [arxiv:2212.08073 §4.1]
[§4.2]. The critiques are often wrong or overstated, yet the revisions are still
rated more harmless [arxiv:2212.08073 §3.5] [app. A]. Card:
[Bai et al., 2022 (Constitutional AI)](/papers/2212.08073).

### Where to evaluate: a framework, not a test

The framework has three layers: the model's capability, human interaction with
the system, and systemic impact on society, the economy and the environment
[arxiv:2310.11986 §2]. The layers are not ordered and may run at the same time
[arxiv:2310.11986 §2.4]. The paper has no experiments of its own.

Its survey covers evaluations known to the authors up to 10 October 2023. It is
not a systematic search. Most of those evaluations target text output, and
human-interaction and systemic-impact evaluations are described as rare
[arxiv:2310.11986 §3.3] [arxiv:2310.11986 fig. 3.2]. The paper says that a
capability test alone cannot show whether a capability causes harm, since that
depends on who uses the system and how [arxiv:2310.11986 §2.1]. Adversarial
testing is a capability method. Measuring how much friction a tester meets is a
human-interaction measure [arxiv:2310.11986 §2]. The paper gives no acceptance
threshold; one must come from a legitimate process [arxiv:2310.11986 §5.3.2].
Card: [Weidinger et al., 2023](/papers/2310.11986).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Open-ended dialogue model, new failure modes wanted, budget for human red-teamers | Crowd dialogue attacks | People find multi-turn attacks; the authors' crowd alone cost at least USD 60K [arxiv:2209.07858 app. A.5] |
| Need to compare how harm changes with model size | Crowd dialogue attacks | Of the three, only this study reports a size trend, and its metric depends on the preference model [arxiv:2209.07858 §4] |
| Training a dialogue agent and able to write per-rule guidelines | Rule-targeted probing | Gives per-rule labels for RL and reward models; needs large training infrastructure, 64 TPU v3 machines [arxiv:2209.14375 app. E] |
| Repeatable test of a vision-language model, with a GPT-4 judge and an image generator | Judge-scored benchmark | Fixed question set; the default set needs 5,040 judge calls per model [arxiv:2311.17600 tab. 1] |
| White-box access to an unlearned text-to-image model, testing whether the forgetting holds | Gradient-optimised prompts | Needs the weights; about 26 minutes per prompt on one GPU [arxiv:2310.11868 §5.2] |
| Black-box chat access to a model, automated attacks with readable prompts and few queries | Attacker-LLM refinement (PAIR) | Needs only chat access; 10 to 56 queries per success on the paper's targets, but 0% to 4% on Llama-2 and Claude [arxiv:2310.08419 tab. 2] |
| A fixed prompt set already exists, and the question is whether greedy-only testing hides risk | Sampled-decoding safety probing | Needs k extra generations per prompt; the paper's risk figures are simulated, so measure on your own model before relying on the gap [arxiv:2601.07239 §6] [tab. safety-concealed-risk] |
| A reusable multi-turn attacker, and budget for RL training on an open 8B model with a guard model | Multi-turn RL attacker | Plans across turns, reporting ASR@5 of 75.2 on Llama-3.1-8B; the authors used two H100s for about 10 hours on one LoRA setting [arxiv:2508.04451 tab. 1] [app. E] |
| Audio or video input to an omni-modal model, consistency across input types matters | Audio and video benchmark | 24 variations per seed and a judge panel; the released script runs one judge [arxiv:2508.07173 §2.3.1] |
| Broad check of a chat model across trust areas, with benign and adversarial system prompts | Multi-perspective trust tests | Eight areas, with released code and dataset; results cover only the two 2023 GPT snapshots, so other models need a rerun [arxiv:2306.11698 §2] |
| Writing or reading a pre-release safety report for a closed model that covers many risk areas | System-card red teaming | Covers persuasion, autonomy and scheming in one report; the GPT-5 card adds a blind pairwise test against o3; no code or weights, so the numbers cannot be reproduced [arxiv:2410.21276 §3] [arxiv:2412.16720 §5] [arxiv:2601.03267 §4.1] |
| Training a helpful and harmless chat model from crowd comparisons, with harmful-side picks in the same preference model | Red-team comparisons as RLHF data | The base set holds 42,000 red-teaming comparisons [arxiv:2204.05862 §2.3]; a model trained on one kind alone scores below chance on the other [§5.1] |
| Little human harm labelling available, but a written list of principles and a capable helpful model | Self-critique of red-team prompts | Replaces harm labels with AI comparisons; the paper gives few numbers in text and most results are plots [arxiv:2212.08073 §4.2] [fig. 3] |
| Deciding what a red-team result does and does not cover | Framework | Names the layers that a capability test leaves out [arxiv:2310.11986 §2] |

Constraints that change the choice:

- **Reproducing the crowd metric** needs your own preference model, because
  neither the model nor the released data includes one [arxiv:2209.07858 app. A.7].
- **Data reuse** is limited. The crowd data is released for research to make
  models less harmful, not to train harmful ones [arxiv:2209.07858 app. A.7].
  The image-and-text benchmark data is non-commercial, see above. The
  DecodingTrust repository is marked CC BY-SA 4.0 on its card, a Creative
  Commons licence rather than a software licence; check the dataset's terms
  before reuse [arxiv:2306.11698 abstract].
- **Bai et al. data:** the paper says the helpfulness data is public and the
  harmlessness data would follow [arxiv:2204.05862 §1 footnote]. The card's
  repository check found red-team transcripts in `anthropics/hh-rlhf`, which it
  records as MIT-licensed and archived. Check that dataset's terms before reuse.
- **Constitutional AI data:** the principle prompts are in a repository with no
  LICENSE file, so the card cites them by path only [arxiv:2212.08073 §1.3].
- **Weights access:** the gradient attack assumes the attacker holds the model
  weights [arxiv:2310.11868 §3]. The other variants, PAIR included, do not need
  them.
- **Judge dependence:** the image-and-text benchmark's labels come from one
  external judge with unstated version and settings [arxiv:2311.17600 §3.3]. The
  audio and video benchmark's labels come from a five-model vote, which the
  released code does not reproduce. DecodingTrust's toxicity labels come from
  the Perspective API, an external scorer the paper ties its results to
  [arxiv:2306.11698 §3.1]. PAIR's labels come from Llama Guard in the
  paper's runs, which missed 47% of jailbreaks on its annotated set
  [arxiv:2310.08419 tab. 1]. The multi-turn RL attacker is trained on a Llama
  Guard reward, but the paper does not name the judge behind its reported ASR
  [arxiv:2508.04451 §4.2] [arxiv:2508.04451 §5]. The crowd study's labels come
  from people and a preference model. The system cards rely on a GPT-4o
  autograder and a model judge that is not released [arxiv:2412.16720 §4.1.1].
- **Protocol dependence:** Constitutional AI tells crowdworkers to prefer
  thoughtful answers over evasive ones when both are equally harmless, so its
  harmlessness Elo is not directly comparable with the earlier HH paper's
  [arxiv:2212.08073 §4.4] [§3.3 footnote].
- **Metric dependence:** a rate that counts incomprehension as safe (as the
  image-and-text benchmark does) and a rate that excludes it (as the audio and
  video benchmark does) are not comparable [arxiv:2311.17600 §4.4]
  [arxiv:2508.07173 §2.3.1]. A pairwise rate that counts only flagged
  conversations, as the o1 card does, is also not comparable with an arena rate
  over all attempts [arxiv:2412.16720 §4.4.1] [arxiv:2412.16720 §4.4.2]. The
  GPT-5 card's blind pairwise rate, 65.1% against o3, does not say whether it
  counts only flagged conversations [arxiv:2601.03267 §4.1]. The
  multi-turn RL paper reports ASR@5 and ASR@30 in different settings, and its
  tab. 2 and tab. 4 disagree on several cells for the same method and target,
  so compare only within one table [arxiv:2508.04451 tab. 2]
  [arxiv:2508.04451 tab. 4].
- **Decoding dependence:** a rate from one greedy completion per prompt and a
  rate from k sampled completions measure different things. The sampled-decoding
  paper's gap rests on its own simulated table, so it is not a measured
  comparison [arxiv:2601.07239 §6]. Before comparing a rate from this page
  with another, check the decoding settings of each source.

## Implementation sketch

Shared core of the variants. Each comment names the variant that uses it.

```pseudocode
# Red-teaming loop, shared core  [arxiv:2209.07858 §3.1] [arxiv:2311.17600 §3.2]
# Input:  target model, attack source, judge
# Output: attack success rate, and the attacks that worked

def red_team(target, attacks, judge):
    # Attacks come from crowd dialogues, LM-generated questions, a fixed set,
    # or an optimiser over the prompt  [arxiv:2209.07858 §3.1]
    # [arxiv:2209.14375 §2.8] [arxiv:2311.17600 §3.2] [arxiv:2310.11868 §4]
    # PAIR instead writes each attack from earlier replies and scores  [arxiv:2310.08419 §3.1]
    # A multi-turn RL attacker learns the policy that writes each turn  [arxiv:2508.04451 §4.2]
    transcripts = [target.respond(attack) for attack in attacks]
    # The judge is a person, a preference model, a GPT-4 prompt, a panel of
    # LLMs, Llama Guard or an image classifier  [arxiv:2209.07858 §3.2]
    # [arxiv:2311.17600 §3.3] [arxiv:2508.07173 §2.3.1]
    labels = [judge.label(transcript) for transcript in transcripts]
    # Unsafe replies over all attacks  [arxiv:2311.17600 §3.3.1]
    asr = sum(label == "unsafe" for label in labels) / len(labels)
    # Failing attacks become training data in the RLHF variants  [arxiv:2209.07858 §3.2]
    failures = [a for a, label in zip(attacks, labels) if label == "unsafe"]
    return asr, failures
```

The variants differ in several places. Constitutional AI's loop has the model judge
its own reply under a written principle [arxiv:2212.08073 §3.1]. The crowd study scores each assistant turn
and keeps the minimum, so a single bad turn sets the score [arxiv:2209.07858
§3.4]. The rule study uses a probability per rule, not a binary label
[arxiv:2209.14375 §2.5]. The image-and-text benchmark counts a refusal as a zero
[arxiv:2311.17600 §3.3.1]. The gradient attack's optimisation uses the model's own
denoising error, and the external classifier only decides when to stop
[arxiv:2310.11868 §4] [arxiv:2310.11868 §5.1]. PAIR runs the loop per attack and
stops a stream once its judge returns a success [arxiv:2310.08419 alg. 1]. The
multi-turn RL attacker gives each token a credit from masked utterances
[arxiv:2508.04451 eq. 4].

## Contradictions

Prompt-based safety measures. The crowd study suspects that its prompt-based
intervention showed no clear effect because static prompt sets are less
adversarial than dialogue attacks [arxiv:2209.07858 §4]. The benchmark finds
that a one-sentence safety prefix cuts ASR from 72.14 to 17.08 on LLaVA-1.5-7B
[arxiv:2311.17600 tab. 4]. The settings differ: dialogue attacks against text
models versus single image-and-text queries, and different interventions. The
two results are not resolved here.

PAIR's reported success across papers. The PAIR study reports jailbreak rates
of 88% for Vicuna-13B and 4% for Llama-2 on JailbreakBench [arxiv:2310.08419
tab. 2]. The multi-turn RL study reruns PAIR as a baseline and reports ASR@5 of
38.75 against Llama-3.1-8B-Instruct and 0.0 against GPT-5.4-mini on HarmBench
[arxiv:2508.04451 tab. 1]. The targets, benchmarks and query budgets differ, so
these numbers are not directly comparable; the RL paper's baseline runs are its
own. Neither paper tests the other's setup.

Why an earlier assistant refused too often. Constitutional AI says its earlier
helpful-and-harmless assistant gave evasive answers, and that crowdworkers
rewarded evasiveness in the data [arxiv:2212.08073 §1.1] [§4.4]. Bai et al.
attribute the over-refusal of early policies to over-optimising harmlessness
[arxiv:2204.05862 §4.4] [fig. 14]. One account points at the crowd labels and the
other at the optimisation pressure. Neither paper tests them against each other,
and they do not say whether they are one effect.

Whether the more capable model is easier to steer. DecodingTrust reports GPT-4
misled more often than GPT-3.5 under adversarial system prompts and backdoored
demonstrations [arxiv:2306.11698 §7] [arxiv:2306.11698 §9]. PAIR reports close
jailbreak rates for GPT-3.5 (51%) and GPT-4 (48%) on its behaviours
[arxiv:2310.08419 tab. 2]. The attacks differ: DecodingTrust changes the system
prompt and the demonstrations, PAIR rewrites the user prompt. The results do not
establish a direction.

## Papers

- [Ganguli et al., 2022](/papers/2209.07858): crowd dialogue attacks, a harmlessness preference model, and a size trend for RLHF models.
- [Glaese et al., 2022](/papers/2209.14375): rule-targeted probing and rule reward models inside an RLHF loop.
- [Chao et al., 2023](/papers/2310.08419): PAIR, a black-box attacker-LLM loop that refines jailbreak prompts over a few rounds.
- [Zhang et al., 2023](/papers/2310.11868): gradient-optimised prompts that bypass concept unlearning in text-to-image models.
- [Wang et al., 2023](/papers/2306.11698): DecodingTrust, a multi-perspective trust benchmark for GPT-4 and GPT-3.5 under benign and adversarial prompts.
- [Weidinger et al., 2023](/papers/2310.11986): a three-layer framework for safety evaluation, and a survey of 2023 evaluations.
- [Belaire et al., 2025](/papers/2508.04451): a multi-turn attacker trained with hierarchical PPO, per-token credit and a guard-model reward.
- [Liu et al., 2023](/papers/2311.17600): a judge-scored benchmark of harmful queries with images, and a safety prefix.
- [Pan et al., 2025](/papers/2508.07173): a judge-scored benchmark of harmful requests across text, image, video and audio, with a consistency score.
- [OpenAI et al., 2024 (GPT-4o)](/papers/2410.21276): the GPT-4o system card, with phased red teaming and speech safety evaluations.
- [OpenAI et al., 2024 (o1)](/papers/2412.16720): the o1 system card, with pairwise and arena red teaming, scheming tests and Preparedness ratings.
- [OpenAI et al., 2025 (GPT-5)](/papers/2601.03267): the GPT-5 system card, with blind pairwise red teaming against o3 and a two-week prompt-injection review.
- [Bai et al., 2022](/papers/2204.05862): helpful and harmless RLHF, with red-team comparisons as training data alongside helpfulness data.
- [Bai et al., 2022 (Constitutional AI)](/papers/2212.08073): self-critique and revision of red-team prompts under written principles, then RL from AI feedback.
- [Joshi et al., 2026](/papers/2601.07239): greedy versus sampled decoding on fixed safety prompts, with a stochastic risk gap; the safety table is simulated.