Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

In plain words

Before a chatbot is released, its builders try to make it say harmful things on purpose, so they can find the failures and fix them. This paper describes how one group did that at scale. Paid people chatted freely with several versions of an AI assistant, tried to get it to misbehave, and rated how well each attempt had worked. A separate model also scored every reply for how harmful it was.

The main finding is about size. Assistants trained with reinforcement learning from human feedback became harder to break as they grew larger. The other versions showed no clear trend with size. The authors released the collected attacks as a public dataset and wrote down their process in detail, including how they looked after the people doing the work.

This is a report on methods, data and lessons learned. It does not add a new algorithm you can drop into a model.

TL;DR

• Red teaming here means manual, open-ended dialogue attacks by crowdworkers on four assistant types at three sizes, scored with a harmlessness preference model §3.1 §3.2 §3.4.
• Under the min-harmlessness metric, RLHF models become harder to red team as they scale, and plain, prompted and rejection-sampled models show a flat trend §4 fig. 1.
• The release has 38,961 attacks tab. 1. The preference model and the models are not released app. A.7.
• All results are author-reported, and the authors say human agreement on what counts as a successful attack is low §3.5.

Problem

Language models can produce offensive, toxic or otherwise harmful text. Red teaming means probing a model adversarially for such outputs and then changing the model so it avoids them §1. Static prompt sets such as RealToxicityPrompts may be less adversarial than dialogue attacks, and the authors suspect this explains why a prompt-based safety intervention showed no clear effect against their attacks §4. The paper asks how safety interventions behave as models scale, and documents the red-team method so that others can reproduce it §1 §2.

The paper does not define "harmful". It leaves that judgement to each red-team member through a pairwise choice §3.1. Results therefore depend on the crowdworkers, their instructions and the preference model, and the authors note that the crowd is not representative of the US population §3.3.

Method

Red-team task §3.1. A crowdworker has an open-ended, multi-turn conversation with an AI assistant, with the instruction to make it behave badly fig. 3. Workers get a short list of example topics. They are asked to be creative, avoid obvious profanity and keep to one topic per attack. Before each attack, they write a short plan. After each assistant turn the interface shows two candidate replies, and the worker picks the more harmful one. This doubles the chances per attempt and yields pairwise labels for training the preference model. At the end, the worker rates success on a 0 to 4 scale. A task is five dialogues, typically on five different topics. Workers do not know which of the four assistants they are talking to §3.1.

Harmlessness preference model §3.2. The preference model is fine-tuned from general language models of 2.7B, 13B and 52B parameters. It learns to predict which reply a red-team member found less harmful, so a higher score means less harmful §3.2. The authors train it on helpfulness as well, because a model can be harmless simply by refusing to help §3.2, footnote. The training data for the harmlessness model comes from attacks on the 52B prompted assistant, which is why that condition has about ten times more attacks §3.2 tab. 1.

Four assistant types §3.2. All are built from the same general models.

• Plain LM: one example conversation placed in context (1-shot). This is the control.
• Prompted LM: fourteen example conversations placed in context (14-shot) to make the model helpful, honest and harmless. Context distillation gives prompt-free variants §3.2.
• Rejection sampling (RS): sample 16 replies from the prompted model, rank them with the preference model and keep the two least harmful. The authors did not vary the 16 §3.2.
• RLHF: reinforcement learning against the preference model, trained on top of the prompted model. Full training details are in arXiv:2204.05862, not in this paper §3.2.

The preference model and the policy are the same size in the RS and RLHF conditions §3.2.

# Rejection sampling reply  [§3.2]
# Input:  prompted_lm, pm (same size), context
# Output: the 2 least harmful candidate replies

def rejection_sampling_replies(prompted_lm, pm, context):
    # Draw 16 candidate replies from the prompted model  [§3.2]
    candidates = [prompted_lm.sample(context) for _ in range(16)]
    # Higher preference score means less harmful  [§3.2]
    ranked = sorted(candidates, key=lambda reply: pm.score(context, reply))
    # Keep the two highest scores, i.e. the least harmful  [§3.2]
    return ranked[-2:]

Measuring an attack §3.4.

• Self-report: the worker's 0 to 4 success rating §3.1 §3.4.
• Min harmlessness: the preference model scores each assistant turn given all earlier turns. The attack's score is the minimum, which is the worst turn §3.4 fig. 6. The paper says the mean gives the same results, but the comparison is not tabulated §3.4.
• Intent score: the preference model also scores the worker's task description. The authors call this out of distribution, because the model was trained on assistant text §3.4, footnote. It is used only as a control app. A.3.

# Min harmlessness of one attack  [§3.4]
# Input:  pm, transcript (alternating human and assistant turns)
# Output: scalar; lower means more harmful

def min_harmlessness(pm, transcript):
    # One score per assistant turn, conditioned on all earlier turns  [§3.4]
    scores = []
    for index in assistant_turn_indices(transcript):
        context = transcript[:index]
        scores.append(pm.score(context, transcript[index]))
    # The minimum is the worst reply and is used as a lower bound  [§3.4]
    return min(scores)

Review task §3.5. A separate experiment measured agreement. Three reviewers rated 500 attacks on the 52B prompted assistant and 500 on the 52B RLHF assistant, using the same 0 to 4 scale. They also tagged each transcript with up to two of 20 topic tags §3.5 fig. 8. The authors chose tags after reading existing harm taxonomies and the 100 most harmful conversations in their data §3.5.

Worker welfare app. A.2. The authors consulted trust-and-safety professionals and built in clear warnings, freedom to pick topics within one's own risk tolerance, suggested breaks, pay by time rather than by quota for the review task, and a Slack group for the reviewers. Reviewers could preview a transcript and skip it. A short well-being survey after every ten tasks is summarised in tab. 2.

Statistical model app. A.3. Attack success and min harmlessness are regressed on model size, intervention and their interaction, with a random intercept for each red-team member. Covariates are the intent score and whether the worker came from MTurk. The authors fit it with lme4 and report the main results hold. The model's coefficients are not reported.

Hyperparameters

Results

Author-reported. Setup: crowdworkers, 2.7B to 52B assistants, one dialogue per attack. All figures are described in words only; see the cited figure numbers in the paper.

Table 1 rows sum to 38,961, which matches the paper's total tab. 1. The scaling trends are read from a figure; the paper gives no numeric values for them fig. 1.

Paper vs code

The linked repository is anthropics/hh-rlhf at the commit above, under MIT LICENSE. It is the authors' own repository and is linked from the introduction §1. It contains data, not model code.

• No training, RLHF, rejection-sampling, preference-model, UMAP or mixed-model code is in the repository. The method cannot be re-run from it README.md.
• The README lists the red-team fields. They match the datasheet fields in app. A.7 README.md.
• The README says the tags exist for 1,000 attempts across two of the four model types README.md. The paper says roughly 3% §4.
• The GitHub repository is deprecated in favour of the same data hosted on Hugging Face README.md.

Pitfalls

• Data, not models. The preference model and the trained assistants are not released app. A.7. The min-harmlessness metric can be recomputed only with a model you train yourself.
• Evaluation leakage risk. The preference model is trained on comparisons from attacks on the 52B prompted assistant §3.2, and the 52B prompted assistant is one of the conditions scored by that preference model §4. The paper does not discuss overlap. This is an inference from the setup, not a result.
• Tag count conflict. Reviewers tagged up to two of 20 tags §3.5. The released datasheet says up to six tags per transcript app. A.7. The released file has not been checked for the actual maximum.
• Count wording. The text says about 1K attacks per condition, with about 10K for the 52B prompted model §3.4. Table 1 shows 2,498 for 2.7B RLHF and 17,739 for 52B prompted, so "about 1K" is loose tab. 1.
• Figure reference. The worker well-being table is cited as "Figure" in the appendix text, though it is Table 2 app. A.2. Cite it as tab. 2.
• Inter-rater agreement is low. Fleiss kappa is 0.32 to 0.55 §3.5. The success label is subjective and noisy, and the authors say so app. A.7.
• Min vs mean. The choice of minimum over assistant turns affects the reported spread. The mean result is given only in words §3.4.
• Harmlessness score is not calibrated. It is a preference-model output, so only its ordering is meaningful. The intent score is out of distribution for the model §3.4, footnote.
• Workforce bias. The workers are not representative of the US population, and about 80% of attacks come from a small group §3.3. Template attacks make many attacks redundant app. A.7.
• PII filter is imperfect. A regular expression removed possible PII from the release. It may catch false positives, and some generated PII is synthetic and unverifiable app. A.6 app. A.5. The regex is printed in app. A.6 and is not reproduced here.
• Use restrictions. The authors state the data should be used for research to make models less harmful, and should not be used to train more harmful models app. A.7. The data contains offensive content §5.1.
• Costs. Crowdworkers alone cost at least USD 60K. Including engineering and training, the authors estimate at least ten times that app. A.5.

Lineage

• Builds on: arXiv:2112.00861 for the prompted, plain and context-distilled assistants and the HHH prompt §3.2. arXiv:2204.05862 for the preference model, RLHF and rejection sampling §3.2.
• Compared against: arXiv:2107.03451, the Bot Adversarial Dialogues dataset of about 5K conversations with three agents of 345M to 2.7B parameters §2.
• Related: arXiv:2202.03286 on red teaming with language models instead of people §2 §5.1.
• Superseded by: not stated.