Constitutional AI: Harmlessness from AI Feedback

In plain words

Assistants like chatbots are usually taught to behave by people who compare pairs of answers and say which is better. Teaching them to avoid harmful answers this way takes a large pile of human judgments. This paper asks how far the model can do that work itself, guided by a short list of written rules, called a constitution.

The training has two stages. In the first, the model answers a provocative question, then reads its own answer, points out what is harmful about it under one of the rules, and rewrites it. The rewritten answers are used to train the model again. In the second stage, the model compares pairs of its own answers and picks the less harmful one, using the same rules. Those picks train a reward model, which then guides reinforcement learning as in the usual recipe.

The result is an assistant that avoids harm but still engages with hard questions and explains why it objects, instead of refusing with a canned reply. Human judgments are still used for helpfulness, so the method removes the human harmlessness labels, not all human feedback.

TL;DR

The paper trains a helpful assistant to be harmless using a list of 16 written principles instead of human harmlessness labels [§1.1, §3.1, §4.1]. Stage one (supervised) finetunes on self-critiqued, self-revised answers §3. Stage two (RLAIF) trains a preference model on AI-labelled comparisons and runs RL against it §4. The authors report that RL-CAI is rated more harmless than RLHF baselines by crowdworkers, and that it is virtually never evasive §4.3, §4.4. The paper gives few numbers in the text; most results are plots, and all results are author-reported.

Problem

Standard RLHF typically uses at least tens of thousands of human feedback labels, and the authors want to reduce human supervision to a small number of written principles §1.1. Their earlier helpful-and-harmless assistant often gave evasive answers to controversial questions, and evasiveness was rewarded by the crowdworkers who labelled its data §1.1, §4.4. The aim is a harmless assistant that is also non-evasive and that explains its objections §1.1.

Method

The method has two stages fig. 1 §1.2. Both use a small constitution of principles, which are written natural-language instructions. Stage one is supervised learning on self-revised responses. Stage two is RL from AI feedback (RLAIF).

Stage 1: critique, revision, supervised learning §3

1. Start from a helpful RLHF model, meaning one trained to follow instructions with only helpfulness labels §1.4 §3.
2. Build the red-team prompt set: 42,496 human-written prompts from prior red teaming work, plus 140,335 prompts generated by few-shot prompting a pre-trained model, for 182,831 in total §3.2.
3. For each prompt, sample an initial response from the helpful model at temperature 1 §3.1 §3.2.
4. In each round, draw one principle at random from the 16 principles §3.1 app. C.
5. Append the principle's critique request to the conversation and sample a critique §3.1.
6. Append the principle's revision request and sample a revised response §3.1.
7. Use the initial prompt and the revised response as the new pair, and repeat from step 4. The paper uses 4 revisions per prompt §3.1 §3.2. The critique text is not part of the pair §3.1.
8. Few-shot examples of critiques and revisions, formatted the same way, are prepended to the requests so the model does not confuse the two roles §3.1 app. E.1.
9. Helpfulness data: 135,296 human-written helpfulness prompts, with 2 responses each sampled from the helpful model at temperature 1 §3.2.
10. Finetune a pre-trained language model (not the RLHF model) for one epoch on all revisions from every round, plus the helpfulness samples §3.1 §3.2. The result is the SL-CAI model.

# Critique-revision data for supervised finetuning  [§3.1] [§3.2]
# Inputs: helpful RLHF model, 16 principles, red-team prompts
# Output: (prompt, revised response) pairs
# Few-shot examples [app. E.1] are prepended to every call; omitted here.

def make_revision_pairs(helpful_model, principles, red_team_prompts):
    pairs = []
    for prompt in red_team_prompts:
        # Sample an initial response at temperature 1  [§3.2]
        response = sample(helpful_model, prompt, temperature=1.0)
        for step in range(4):
            # Draw one principle at random for this round  [§3.1]
            principle = random_choice(principles)
            # Append the critique request and sample a critique  [§3.1]
            context = prompt + response + principle.critique_request
            critique = sample(helpful_model, context)
            # Append the revision request and sample a revision  [§3.1]
            context = context + critique + principle.revision_request
            response = sample(helpful_model, context)
            # The prompt and revision form the next training pair  [§3.1]
            pairs.append((prompt, response))
    return pairs

The paper also tests a variant without critiques, which asks for a revision directly §3.5 fig. 7. The main results use critiqued revisions §3.5.

Stage 2: AI preference labels, preference model, RL §4

1. For each prompt, sample a pair of responses from the SL-CAI model §4.1.
2. Build a multiple-choice question for a feedback model. It contains the conversation, one principle, and the two responses labelled (A) and (B). The paper prepends few-shot examples in the same format, each with its own sampled principle §4.1 app. E.2.
3. Read the normalised probabilities of answers (A) and (B). Use them as soft targets for a comparison example §4.1.
4. Pick the principle at random for each comparison from the 16 principles §4.1.
5. Combine the labels for the preference model: 135,296 human helpfulness comparisons and 182,831 AI harmlessness comparisons §4.2. Train it with the process of prior work §1.2.
6. Run RL against this preference model, starting from the SL-CAI model. The pipeline is otherwise the RLHF pipeline §4.1 §1.2. The result is RL-CAI.

The chain-of-thought variant changes the feedback step. It uses the helpful RLHF model as the feedback model, writes the question as a Human/Assistant dialogue, and asks it to reason step by step before answering. Hand-written few-shot examples with reasoning are prepended §4.1 app. E.2. Because the reasoning usually commits to one answer, the paper clamps the probability to the 40–60% range before using it as a target §4.1 §4.3.

# AI preference labels for harmlessness  [§4.1]
# Inputs: SL-CAI policy, feedback model, 16 principles, prompts
# Output: comparisons with soft targets for the preference model
# Few-shot examples [app. E.2] are prepended to the feedback prompt; omitted.

def make_ai_labels(policy, feedback_model, principles, prompts):
    labels = []
    for prompt in prompts:
        # Sample a pair of responses from the SL-CAI policy  [§4.1]
        response_a = sample(policy, prompt)
        response_b = sample(policy, prompt)
        # Draw one principle at random for this comparison  [§4.1]
        principle = random_choice(principles)
        # Format the conversation and the two options as a question  [§4.1]
        question = format_choice(
            prompt,
            principle,
            response_a,
            response_b,
        )
        # Read the normalised probabilities of answers (A) and (B)  [§4.1]
        p_a, p_b = choice_probabilities(feedback_model, question)
        labels.append((prompt, response_a, response_b, p_a, p_b))
    return labels


# Chain-of-thought variant: clamp the (A) probability  [§4.1] [§4.3]
# The paper does not say whether (B) is clamped separately.
def clamp_cot_label(p_a):
    # Keep the probability between 40% and 60%  [§4.1] [§4.3]
    return min(max(p_a, 0.4), 0.6)

Hyperparameters

Values are as written in this paper. Items the paper defers to prior work or leaves out are marked "not stated".

Results

Author-reported. Setup: crowdworker A/B tests for Elo scores, with 52B-scale models as the main comparison §3.3 §4.3. Hardware and training time are not stated. The text gives few numbers, so most results are read from plots.

Paper vs code

The paper's footnote links an official repository at GitHub, owned by the authors' organisation §1.3. The repository is "supplementary material"; its README says so README.md. It contains the critique and revision instructions, the RL labelling instructions, and few-shot prompts, plus the evaluation sets and sample responses CritiqueRevisionInstructions.json RLMadisonInstructions.json 438HHHEvaluations.jsonl.

Checks that match the paper:

• The evaluation files have 438, 254 and 287 items, matching §2 and app. B.
• Both instruction sets have 16 principles, matching §3.1 and §4.1.

What the repository does not contain: any supervised or RL training code. The training hyperparameters in Hyperparameters therefore cannot be checked against code. The repository has no LICENSE file, so nothing from it is quoted here; it is cited only by path.

No discrepancies found in the parts that can be checked.

Pitfalls

• RL is under-specified here. The paper says RL used the same hyperparameters as prior work §4.2. Learning rate, batch size, KL penalty and step count are not stated in this paper.
• Absolute SL learning rate is missing. It is given only as 0.5 times the pre-training rate §3.2, and the pre-training rate is not stated.
• Critiques are often wrong. The authors found them frequently inaccurate or overstated, while the revisions were still more harmless §3.5 app. A. An implementation should not treat critique text as reliable. It is kept for transparency §3.5.
• Critique text is not in the training pair. The training pair is the prompt and the final revision §3.1.
• Clamp target is ambiguous. The paper says the CoT probabilities are clamped to 40–60% §4.1 but does not say whether (A) alone is clamped or (B) too. Without the clamp, RL-CAI learns more extreme responses §4.1.
• Soft versus hard labels is qualitative. Without CoT, the paper reads labels from the log-probabilities of (A) and (B) and reports soft labels as much better than hard 0/1 labels, with no numbers §4.1 §4.3. With CoT it could not extract soft labels without sampling several CoT outputs per label §4.3.
• Evaluation protocol changed from prior work. Crowdworkers were told to prefer thoughtful answers over evasive ones when both are equally harmless §4.4. Harmlessness Elo scores are therefore not directly comparable with the prior HH paper §3.3 footnote §4.4.
• Over-training causes boilerplate. RL-CAI, when over-trained, can answer with reassurance boilerplate and overly harsh refusals §4.3.
• Preference-model scores lose calibration at high values §3.4. Revision scores in fig. 5 should be read with this in mind.
• Helpfulness falls as revisions increase fig. 5. The harmlessness gain has a helpfulness cost.
• Human labels are not fully removed. Helpfulness still uses human labels. The authors say so §6.
• Principles were written ad hoc. The authors say they were not carefully designed and should be refined by more stakeholders §1.1 app. C.
• Hidden cost. Every red-team prompt needs several rounds of critique and revision. The appendix says these come from the 52B helpful model §3.2 app. A. CoT labels add generated reasoning per comparison §4.1.
• Feedback model size. The text says "typically a pretrained LM" §4.1. The calibration figure's caption gives 52B labels fig. 9. Treat the feedback model size as stated only in that caption.

Lineage

• Builds on: Bai et al. 2022, RLHF for a helpful and harmless assistant. This paper reuses its pipeline and helpfulness labels, and removes the harmlessness labels §1.4 §4.1.
• Uses red-team prompts from: arXiv:2209.07858 (Ganguli et al.) §3.2. No card in this wiki.
• Uses chain-of-thought prompting from: arXiv:2201.11903 §4.1.
• Superseded by: not known from content in this wiki.