---
kind: concept
slug: rlaif
title: Reinforcement learning from AI feedback (RLAIF)
summary: >-
  Replaces human harmlessness labels with AI preference labels. A feedback model picks the less harmful of two answers under a written principle, a preference model is trained on those picks, and RL optimises the policy against it. Helpfulness can keep human labels. Evidence is one paper, author-reported, mostly plots.
tags: [alignment, preference-model, reinforcement-learning, ai-feedback]
papers: ["2212.08073"]
related: [rlhf, preference-optimization, reward-modeling, self-refinement, red-teaming, bradley-terry-model]
status: draft
updated_at: 2026-10-24
---

## Overview

RLAIF keeps the usual post-training loop of RLHF: collect pairwise preference
labels, fit a preference model to them, then run RL against that model
[arxiv:2212.08073 §1.2]. The difference is who writes the labels. A language
model judges pairs of answers against a short list of written principles,
instead of people [arxiv:2212.08073 §1.1, §4.1]. The shared invariant is that
the reward model is trained on comparisons, whatever their source.

The paper combines this with a supervised stage in which the model critiques
and revises its own answers under the same principles, which gives the starting
policy and its training data [arxiv:2212.08073 §1.2, §3.1].

## Variants

### Constitutional AI with AI harmlessness labels (RL-CAI)

Sample two answers from the supervised model for each prompt. Ask a feedback
model which answer better follows one principle drawn at random, and turn the
normalised probabilities of the two options into a soft label [arxiv:2212.08073
§4.1]. Combine these AI harmlessness labels with the usual human helpfulness
labels, 135,296 and 182,831 comparisons in the paper, and train the preference
model on both [arxiv:2212.08073 §4.2]. Then run RL from the supervised model
[arxiv:2212.08073 §4.1]. Card: [Constitutional AI: Harmlessness from AI Feedback](/papers/2212.08073).

### Chain-of-thought feedback labels

Same labelling loop, but the feedback model is prompted to reason step by step
before it answers, using hand-written examples with reasoning [arxiv:2212.08073
§4.1]. The reasoning tends to commit to one option, so the paper clamps the
probability to 40–60% before using it as a target [arxiv:2212.08073 §4.1,
§4.3]. The paper also tried a 20–80% range [arxiv:2212.08073 §4.3]. Card: [Constitutional AI: Harmlessness from AI Feedback](/papers/2212.08073).

### Critique and revision for supervised data (SL-CAI)

Before any AI labels exist, the model answers red-team prompts, critiques its
own answer under a random principle, and rewrites it. The final revisions,
plus helpfulness samples, are used to finetune a pre-trained model for one
epoch [arxiv:2212.08073 §3.1, §3.2]. This is a self-refinement loop applied to
training data, not to inference (see [Self-refinement](/concepts/self-refinement)). Card: [Constitutional AI: Harmlessness from AI Feedback](/papers/2212.08073).

### Caveats that apply to all variants

- The paper's own harmlessness scores are not directly comparable with its
  earlier helpful-and-harmless work, because crowdworkers were told to prefer
  thoughtful answers over evasive ones when both were equally harmless
  [arxiv:2212.08073 §3.3, §4.4].
- The RL hyperparameters are "same as prior work" and are not given in the
  paper [arxiv:2212.08073 §4.2]. Learning rate, batch size, KL penalty and step
  count are therefore not stated.
- Over-training the RL model produced reassurance boilerplate and overly harsh
  refusals [arxiv:2212.08073 §4.3].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You want fewer human harm labels but still have helpfulness labels | RL-CAI with AI harmlessness labels | The paper keeps human helpfulness comparisons and adds AI harmlessness ones [arxiv:2212.08073 §4.2] |
| You have no written rules yet | Budget time to write and refine them | The paper's 16 principles were written ad hoc and the authors say they should be refined [arxiv:2212.08073 §1.1, app. C] |
| You need to audit or explain the judge's choice | Chain-of-thought labels | The judge states reasoning before choosing [arxiv:2212.08073 §4.1]; it costs extra generation per label, and probabilities need clamping [arxiv:2212.08073 §4.3] |
| You want the judge's raw probabilities as the signal | Direct labels, no reasoning | The paper reports soft labels as better than hard 0/1 labels, but gives no numbers [arxiv:2212.08073 §4.1, §4.3] |
| Compute is tight for the supervised data | Direct revision instead of critiqued revision | Critiqued revisions score better for small models; at 52B the two are similar, with critiques slightly ahead [arxiv:2212.08073 §3.5] |
| Feedback-model size is your main question | Check the paper's caption, not the text | The text says only "typically a pretrained LM"; a calibration figure caption gives 52B [arxiv:2212.08073 §4.1, fig. 9] |

The paper's main results use models at 52B scale [arxiv:2212.08073 §3.3]. This
wiki has no source that tests AI label quality across feedback-model sizes.

## Implementation sketch

The labelling loop shared by the RL-CAI variants. The chain-of-thought
variant changes only the judge prompt and the clamp.

```pseudocode
# One AI harmlessness label  [arxiv:2212.08073 §4.1]
# Inputs: SL policy, feedback model, principles, one prompt
# Output: (prompt, answer_a, answer_b, p_a) soft target

def make_label(policy, judge, principles, prompt):
    # Sample two candidate answers from the supervised policy  [§4.1]
    answer_a = sample(policy, prompt)
    answer_b = sample(policy, prompt)
    # Draw one written principle at random for this comparison  [§4.1]
    rule = random_choice(principles)
    # Ask a two-option multiple-choice question  [§4.1]
    question = build_question(prompt, rule, answer_a, answer_b)
    # Read the judge's probability for options (A) and (B)  [§4.1]
    p_a, p_b = option_probabilities(judge, question)
    # Normalise over the two options and keep it as a soft target  [§4.1]
    p_a = p_a / (p_a + p_b)
    return prompt, answer_a, answer_b, p_a


# Chain-of-thought variant: clamp the soft target  [§4.1] [§4.3]
def clamp_cot_target(p_a):
    # The paper uses 40% to 60%; it does not say whether (B) is clamped too
    return min(max(p_a, 0.4), 0.6)
```

Next, train the preference model on the labels together with the human
helpfulness comparisons, then run RL against it [arxiv:2212.08073 §4.2].

## Contradictions

None known. Only one source in this wiki covers the concept, so there is no
cross-paper disagreement to report.

## Papers

- [Constitutional AI: Harmlessness from AI Feedback](/papers/2212.08073): introduces RLAIF with a written constitution. The supervised critique-and-revision stage and the AI-labelled preference model both come from this paper, and its results are author-reported.