OpenAssistant Conversations -- Democratizing Large Language Model Alignment

In plain words

Chatbots get good at answering questions only after they learn from examples of good answers and from people saying which answer is better. Those examples are costly to make, and the big labs usually keep them private. This paper releases a large set of conversations written by volunteers on an open website. Volunteers write opening messages, write replies as the assistant or as the user, label messages for spam and quality, and rank competing assistant replies. Filters and moderators remove bad content. Each conversation is stored as a tree, so one opening message can have several different continuations, and the rankings show which continuation people liked more. The authors also tune open language models on this data and report that they do better than their starting points on several standard tests, with one exception. These results come from the authors, so they should be checked.

TL;DR

• Releases OpenAssistant Conversations, a crowd-written corpus stored as conversation trees, with quality labels and rankings of assistant replies §4.
• The collection is a web app with five task types, a tree state machine, and moderation by peer labels, points and human moderators §3.
• Several rankers' orders of the same replies are merged with ranked pairs (Tideman's method) app. B.
• SFT and RLHF models of Pythia, Falcon and LLaMA trained on the data beat their base models on most reported benchmarks tab. 1. The exception in the table is Pythia SFT on LMEH (60.28 vs 60.33). Author-reported.
• Code is Apache-2.0. The paper calls the data "fully permissive" without naming a license, so check the dataset page before reuse app. L.

Problem

Supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) need high-quality human feedback data §1. That data is expensive, often proprietary, and rare in the open §1. Most open instruction data is generated by language models or limited to a fixed list of task types §1. The method assumes enough volunteers will contribute and that peer review and a few human moderators can keep quality and safety acceptable as volume grows §3.4.

Method

The paper has two parts. First, a crowd-sourcing pipeline that produces conversation trees with labels and rankings §2 §3. Second, SFT, reward-model (RM) and PPO training on that data to show it is usable §6.1 app. H.

Data format §2 fig. 1

• A conversation tree (CT) has a root prompt written by a prompter.
• Nodes are messages. A node's children have the opposite role.
• A thread is any path from the root to a node and is a valid conversation.
• Assistant nodes carry a rank among the replies to the same parent.
• Nodes also store labels, a timestamp and the language §2.

Collection tasks §3.1

Each user gets one unit of work at a time. Five task types exist:

1. Create a prompt. A lottery limits how many new prompts are active.
2. Reply as assistant. This is the most demanding task and carries a larger reward.
3. Reply as prompter. Quality bar is low; diversity is the goal.
4. Label a prompt or reply. Three label groups:
   • spam, where messages flagged by several users are removed;
   • guideline labels (wrong language, personal data, hate speech, sexual content, other inappropriate content), which send the message to human moderators;
   • five-point quality scales (creativity, quality, humor, helpfulness, violence, rudeness) §4. Users may also label any message outside their assigned task.
5. Rank assistant replies to the same parent §3.1.

Tree state machine §3.2 app. G

A tree starts in an initial-prompt review state. If the prompt passes, the tree grows by issuing reply, label and ranking tasks until an end state is reached, which is a maximum message count or another preset criterion. Trees that fail review go to an aborted low-grade state and are removed. Moderators can also halt a tree. Ranking is a separate phase: trees in a backlog ranking state are activated when others finish app. G.

Guidelines and moderation §3.3 §3.4

• Guidelines ask assistant replies to be polite, helpful, concise, friendly and safety-aware. Prompts and prompter replies should be diverse and challenging. The guidelines follow an informed-consent idea: explain risks and limits rather than refuse everything §3.3.
• Points and leaderboards reward contributions. Points for an assistant reply rise or fall with later labels and rankings. A negatively scored "Trollboard" helps moderators find problem users §3.4.
• Users can report messages. Moderators can delete messages or all messages of a user. Deleted messages stay in the database but are not exported §3.4.
• Auto-moderation rules halt trees or delete replies when enough users skip a message or flag it app. G.

Merging rankings: ranked pairs app. B

Each ranking is made by K independent rankers. The paper merges them with a variant of Tideman's ranked-pairs method app. B:

# Ranked-pairs (Tideman) merge of K rankings  [app. B]
# Input:  rankings: K orderings of the same N replies, best first
# Output: one consensus ordering of the N replies

def ranked_pairs(rankings):
    # Count, for every ordered pair (a, b), the votes for a over b  [app. B]
    net_votes = pairwise_net_votes(rankings)          # (N, N)
    # Candidate edges a > b, strongest first  [app. B]
    # [code: backend/oasst_backend/utils/ranking.py#L98-L110]
    candidates = sorted(
        [(a, b) for a in range(N) for b in range(N)
         if a != b and net_votes[a, b] >= 0],
        key=lambda pair: net_votes[pair],
        reverse=True,
    )
    # Keep an edge unless it closes a cycle  [app. B]
    locked_edges = []
    for winner, loser in candidates:
        if not has_cycle(locked_edges + [(winner, loser)]):
            locked_edges.append((winner, loser))
    # Output nodes that have no incoming locked edge, best first  [app. B]
    return order_by_removing_sources(locked_edges)

Training app. H

• Conversation format. A thread P1, R1, P2, R2, ... is serialised as prompter token, prompt, end-of-text token, assistant token, reply, end-of-text token, and so on. The paper gives these tokens as placeholders app. H.
• SFT. Standard language-model training on the thread. Prompt tokens are masked, so only assistant replies are predicted app. H.
• Reward model. The LM head is replaced by a linear layer that outputs one score for the last reply. With K replies to one prompt there are K choose 2 pairwise comparisons. The loss is below. The paper also says an optional extra regularizer may be added app. H.

\mathrm{loss}(\theta) = -\frac{1}{\binom{K}{2}}\, \mathbb{E}_{(x, y_w, y_l)}\Big[\log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big)\Big] app. H

# Pairwise reward-model loss for the K replies to one prompt  [app. H]
# Input:  scores: (K,) reward per reply, best-ranked reply first
# Output: scalar loss

def reward_model_loss(scores, beta):
    # Each pair (i, j) with i ranked above j is one comparison  [app. H]
    losses = []
    for i in range(K):
        for j in range(i + 1, K):
            margin = scores[i] - scores[j]
            # Negative log-likelihood that the preferred reply scores higher
            nll = -log_sigmoid(margin)                              # [app. H]
            # Optional score penalty; code uses beta = 0.001
            # [code: model/model_training/utils/losses.py#L76-L77]
            penalty = beta * 0.5 * (scores[i] ** 2 + scores[j] ** 2)
            losses.append(nll + penalty)
    return mean(losses)

• PPO. The SFT model answers unanswered prompts. The RM scores the answers and PPO updates the policy. A per-token KL penalty to the SFT model is added to limit over-optimisation. The paper names the trlx framework app. H.

Hyperparameters

The paper states collection parameters (Table 5 in the compiled paper, appendix G) but gives no training hyperparameters. The table below mixes the paper's values and the repository's configs. Repository values are labelled "code" and are not verified to be the values behind any released checkpoint.

Results

Author-reported. Base for the card: arXiv v2 (updated 2023-10-31).

Dataset §4 app. L

The authors recommend the variant with all completed trees for instruction tuning and add incomplete trees if more data is needed §4.

Models and benchmarks §6.1 tab. 1. Setup: Pythia-12B, Falcon-40B and LLaMA-30B/65B as base models. Evaluations were run externally with FastEval. LMEH is the average over BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c and OBQA app. I. VEL is the Vicuna Elo rank, OAIE is OpenAI Evals, and HE is HumanEval. Higher is better. Base models are scored on LMEH only.

The authors say the OpenAssistant models beat their baselines consistently. The table shows one exception on LMEH: Pythia SFT (60.28) is below its base (60.33). The authors also say RLHF beats SFT on some benchmarks but not others. Rankings across benchmarks disagree §6.1. The sft-mix model adds other instruction datasets to OASST data app. H.

Moderation check §6.2 tab. 2. Detoxify scores of deleted and retained messages in the languages Detoxify supports. Deleted messages score higher on all six categories: toxicity 4.625% vs 0.988%; insult 2.085% vs 0.715%; identity attack 0.651% vs 0.121%. The authors note that toxicity alone does not decide deletion. Low-quality or ungrammatical messages were deleted too §6.2.

Paper vs code

Code: LAION-AI/Open-Assistant at f1e6ed9526f5817531f3ab85441a40b3671ddccb, Apache-2.0 LICENSE. The paper links this repository in its abstract.

• Collection parameters differ. Paper values (Table 5, app. G) differ from the backend defaults for max active trees (100 vs 10), max tree depth (5 vs 3), max children (2 vs 3), goal tree size (9 vs 12), p full labeling for prompter replies (0.1 vs 0.25), min active rankings per language (20 vs 0) and max prompt lottery waiting (1000 vs 250) config.py config.py config.py config.py. The paper does not say which values produced the release.
• Ranking merge. The paper's worked example counts pairwise wins (A over B 2 times, A over C 3 times) app. B. The code sorts candidate edges by net margin, wins minus losses ranking.py. The code also keeps zero-margin pairs as candidates, so ties are settled by order ranking.py.
• Random winner. The docstring of get_winner says a winner is chosen at random. The code returns the first candidate in sorted order, so it is deterministic ranking.py.
• Reply cap in RM training. The default RM config caps replies per prompt at 5 config_rm.yaml. The collator truncates longer lists ranking_collator.py. The paper describes K distinct replies with no cap app. H.
• RM regularizer. The paper's loss has no penalty term; it says one may be added app. H. The code always adds 0.5·beta·(r_i² + r_j²) with beta 0.001 losses.py. beta comes from the config utils.py.
• Special tokens. The paper uses placeholders app. H. The code uses <|prompter|>, <|assistant|> and the tokenizer's end-of-sequence token formatting.py.
• No discrepancy found for prompt masking in SFT. The code enables label masking by default config.yaml and passes it to the SFT trainer trainer_sft.py.

Pitfalls

• License of the data. The paper says the code and data are released "under a fully permissive licence" [abstract] but does not name it. The code is Apache-2.0, but that does not license the data. The data page is linked in the paper app. L. It was not checked for this card. Do not reuse the data until the dataset license is confirmed.
• Released checkpoints vs configs. The paper names released checkpoints such as pythia-12b-sft-v8-7k-steps tab. 1. The repository's pythia-12b-sft-8 config starts from a 12.5k-step base config.yaml. The paper does not map configs to released checkpoints, so do not assume one.
• Training hyperparameters are not in the paper. Learning rate, epochs, batch and warmup come only from the repository, and the repository's default blocks differ from the named configs. Start from the config that matches the target model and say so in your own notes.
• Tokens. Implementers must use the real special tokens of the base tokenizer. The paper's placeholders are not literal strings app. H.
• Ranking ties and ordering. Ties are settled by pair order ranking.py. This affects which ordering is stored.
• RLHF vs SFT. The gains are mixed. The authors suggest the reward data may explain part of the gap, because the reward model was trained on human-written replies and not on model-written ones §7.
• Compute. GPU type, count and training time are not stated. The acknowledgments name Redmond AI for training compute and Stability AI and Hugging Face for inference compute [ack.].
• Results are author-reported. The benchmarks were run externally on a leaderboard that is updated over time §6.1. Numbers may change. Some table cells are not reported.
• Contributor bias. 89.1% of annotators identify as male, median age 26 §7. A few power users make a large share of the contributions §4 §7. The demographics survey was announced on Discord and so reaches active members only §5.
• Unsafe content and model output. Unsafe content may remain after moderation §7. ChatGPT-written content was filtered with keyword tests and votes, but not all of it was removed app. D. The authors say the released models may be unsafe and vulnerable to prompt injection §7.
• Source-only study. The fetched LaTeX source also contains a user preference study against gpt-3.5-turbo (48.3% win rate). It sits inside \iffalse blocks and is not in the compiled text, so this card does not report it.

Lineage

• Builds on: InstructGPT's three-stage recipe of SFT, reward model and PPO §1. The InstructGPT entry in the bibliography has no arXiv id here and no card, so it is not linked.
• Base models used: Pythia for the 12B models, and LLaMA for the 30B and 65B models §6.1 app. H.
• Compared against: gpt-3.5-turbo in tab. 1.
• Superseded by: not stated.