Reinforcement learning from AI feedback (RLAIF)

Overview

RLAIF keeps the usual post-training loop of RLHF: collect pairwise preference labels, fit a preference model to them, then run RL against that model 2212.08073 §1.2. The difference is who writes the labels. A language model judges pairs of answers against a short list of written principles, instead of people 2212.08073 §1.1, §4.1. The shared invariant is that the reward model is trained on comparisons, whatever their source.

The paper combines this with a supervised stage in which the model critiques and revises its own answers under the same principles, which gives the starting policy and its training data 2212.08073 §1.2, §3.1.

Variants

Constitutional AI with AI harmlessness labels (RL-CAI)

Sample two answers from the supervised model for each prompt. Ask a feedback model which answer better follows one principle drawn at random, and turn the normalised probabilities of the two options into a soft label [arxiv:2212.08073 §4.1]. Combine these AI harmlessness labels with the usual human helpfulness labels, 135,296 and 182,831 comparisons in the paper, and train the preference model on both 2212.08073 §4.2. Then run RL from the supervised model 2212.08073 §4.1. Card: Constitutional AI: Harmlessness from AI Feedback.

Chain-of-thought feedback labels

Same labelling loop, but the feedback model is prompted to reason step by step before it answers, using hand-written examples with reasoning [arxiv:2212.08073 §4.1]. The reasoning tends to commit to one option, so the paper clamps the probability to 40–60% before using it as a target [arxiv:2212.08073 §4.1, §4.3]. The paper also tried a 20–80% range 2212.08073 §4.3. Card: Constitutional AI: Harmlessness from AI Feedback.

Critique and revision for supervised data (SL-CAI)

Before any AI labels exist, the model answers red-team prompts, critiques its own answer under a random principle, and rewrites it. The final revisions, plus helpfulness samples, are used to finetune a pre-trained model for one epoch 2212.08073 §3.1, §3.2. This is a self-refinement loop applied to training data, not to inference (see Self-refinement). Card: Constitutional AI: Harmlessness from AI Feedback.

Caveats that apply to all variants

• The paper's own harmlessness scores are not directly comparable with its earlier helpful-and-harmless work, because crowdworkers were told to prefer thoughtful answers over evasive ones when both were equally harmless 2212.08073 §3.3, §4.4.
• The RL hyperparameters are "same as prior work" and are not given in the paper 2212.08073 §4.2. Learning rate, batch size, KL penalty and step count are therefore not stated.
• Over-training the RL model produced reassurance boilerplate and overly harsh refusals 2212.08073 §4.3.

When to use which

The paper's main results use models at 52B scale 2212.08073 §3.3. This wiki has no source that tests AI label quality across feedback-model sizes.

Implementation sketch

The labelling loop shared by the RL-CAI variants. The chain-of-thought variant changes only the judge prompt and the clamp.

# One AI harmlessness label  [arxiv:2212.08073 §4.1]
# Inputs: SL policy, feedback model, principles, one prompt
# Output: (prompt, answer_a, answer_b, p_a) soft target

def make_label(policy, judge, principles, prompt):
    # Sample two candidate answers from the supervised policy  [§4.1]
    answer_a = sample(policy, prompt)
    answer_b = sample(policy, prompt)
    # Draw one written principle at random for this comparison  [§4.1]
    rule = random_choice(principles)
    # Ask a two-option multiple-choice question  [§4.1]
    question = build_question(prompt, rule, answer_a, answer_b)
    # Read the judge's probability for options (A) and (B)  [§4.1]
    p_a, p_b = option_probabilities(judge, question)
    # Normalise over the two options and keep it as a soft target  [§4.1]
    p_a = p_a / (p_a + p_b)
    return prompt, answer_a, answer_b, p_a


# Chain-of-thought variant: clamp the soft target  [§4.1] [§4.3]
def clamp_cot_target(p_a):
    # The paper uses 40% to 60%; it does not say whether (B) is clamped too
    return min(max(p_a, 0.4), 0.6)

Next, train the preference model on the labels together with the human helpfulness comparisons, then run RL against it 2212.08073 §4.2.

Contradictions

None known. Only one source in this wiki covers the concept, so there is no cross-paper disagreement to report.

Papers

• Constitutional AI: Harmlessness from AI Feedback: introduces RLAIF with a written constitution. The supervised critique-and-revision stage and the AI-labelled preference model both come from this paper, and its results are author-reported.