Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

In plain words

Reinforcement learning from human feedback is the recipe behind many chat assistants: people compare pairs of answers, a model learns to predict those preferences, and the assistant is then tuned to earn high scores from that predictor. This paper is a catalogue of what goes wrong with that recipe.

The people giving feedback can be biased, rushed, misled or simply unable to judge hard answers. The learned predictor can be a poor stand-in for what people actually want, and the assistant will find and exploit its gaps. The tuning step can also make answers more uniform and can make the assistant harder to trust in rare or adversarial cases.

The authors sort these problems into those that better engineering might fix and those that would need a different method altogether. They then list other safety measures that should sit alongside the recipe, and the details a company could publish so outsiders can check its work. The paper reports no new experiments, so its claims are arguments, and the numbers it gives come from other studies.

TL;DR

• A survey and position paper on RLHF, the standard method for tuning LLMs with human preference judgments §1.
• It classifies challenges by stage (feedback, reward model, policy, joint training) and labels each one tractable or fundamental §3 app. B.
• It argues RLHF should be one layer among several safety measures §4 and proposes disclosure items for auditing §5 fig. 4.
• No experiments and no code. Empirical numbers are cited from other work.

Problem

RLHF is used to finetune large language models before deployment, yet deployed models finetuned this way still leak private information, hallucinate, show political bias and sycophancy, and remain open to jailbreaks and prompt injection §1. The paper argues that these failures have had little public systematic treatment §1. It asks which failures are fixable inside RLHF and which need other approaches, and what outside parties should be able to check §1 §5.

Method

The paper has no method of its own. Its contributions are:

1. A formal framework for RLHF, with three steps (feedback collection, reward modelling, policy optimisation) that are often repeated iteratively §2 eq. 1 eq. 2 eq. 3.
2. A taxonomy of open problems: challenges with human feedback §3.1, the reward model §3.2, the policy §3.3 and joint reward-and-policy training §3.4. Each problem is marked tractable or fundamental; the paper defines fundamental as one that could only be overcome by a method that is no longer RLHF §3 app. B.
3. A review of approaches to understand, improve and complement RLHF §4.1 §4.2 §4.3 fig. 3.
4. A governance section with disclosure items for RLHF-trained systems, and concerns about social and economic equity §5 fig. 4.
5. An alternative model of the feedback process that adds sampling humans, context and a rendering function app. A eq. 4.

The formal RLHF loop from the paper's framework §2 eq. 1–3, written in pseudocode. This is the only part of the paper that an implementer could code directly:

# RLHF as formalised by the paper  [§2]
# Input:  base policy (theta), human H, feedback function f, rounds
# Output: reward model (phi), fine-tuned policy (theta_new)

def rlhf(base_policy, human, feedback_fn, rounds):
    # The loop is iterative in practice, not a single pass  [§2]
    for round in range(rounds):
        # Sample examples from the base model; each example is one or
        # more generations  [§2] [eq. 1]
        examples = sample(base_policy)

        # Feedback depends on the human, the example and noise  [§2] [eq. 1]
        feedback = [
            feedback_fn(human, example, noise())
            for example in examples
        ]

        # Fit the reward model to the feedback, with a regulariser  [§2] [eq. 2]
        reward_model = fit_reward(examples, feedback)

        # Maximise expected reward minus a penalty tied to the base
        # policy, e.g. a divergence  [§2] [eq. 3]
        base_policy = optimise_policy(
            reward_model,
            regulariser_to(base_policy),
        )
    return reward_model, base_policy

The paper gives no concrete choice for fit_reward, optimise_policy or the regulariser. Its examples are a cross-entropy loss for pairwise comparisons, or a Bayesian personalised ranking loss §2; the regulariser is "some divergence-based penalty" §2.

Feedback types the paper discusses: binary and k-wise preferences, best-of-k queries, scalar ratings, label selection, corrections and natural-language feedback §3.1.4. Remedies it lists: AI-assisted feedback §4.2.1, fine-grained feedback §4.2.1, process supervision §4.2.1, learning from demonstrations §4.2.1, directly specified rewards §4.2.2, multi-objective oversight §4.2.2, uncertainty over the reward function §4.2.2, pretraining with a reward model §4.2.3, and supervised alternatives §4.2.3.

Hyperparameters

Not applicable. The paper trains no models and states no hyperparameters. The only parameters it names are the regularisers in the formal framework §2 eq. 2 eq. 3, which are left unspecified.

Results

No experiments. The paper reports no new measurements. Its quantitative statements are cited from other work and are listed here as the paper gives them.

The last row is a worked example, not a measurement. The paper's three contributions are qualitative: a taxonomy, a review and a disclosure list §1.

Paper vs code

No official code. The source has no repository link, and no project page is linked. A GitHub search for "rlhf open problems" and "fundamental limitations rlhf" returned two unrelated repositories (casinca/GRPO-classic-RL and DavidWise01/alignment) and no match for the authors. Neither belongs to an author or an author's lab, so neither is used. There is no implementation for the paper to diverge from.

Pitfalls

• Nothing to reimplement. An agent asked to implement "the paper" can only implement the formal loop §2 and the alternative feedback model app. A. The taxonomy is judgment, not an algorithm.
• The tractable/fundamental split is soft. The paper calls it "soft" and says some categories are marginal §3 app. B. Appendix B itself calls several fundamental items "marginal" or "trivial", for example individual values app. B, §3.2.1, misgeneralisation app. B, §3.2.2 and deployment shift app. B, §3.3.2. Treat the labels as the authors' view, not a fact.
• Many claims are hedged. The paper says the cause of the political bias "remains unclear" §3.1.1, that the viability of AI feedback "remains uncertain" §4.2.1, and that the sycophancy cause is "unclear" §3.1.1. An implementer should not treat these as settled.
• Figures are secondhand. The numbers in Results come from other papers, some about models from 2022 and 2023 (GPT-4, Bard, Llama 2-Chat §1). Check the original sources before quoting them.
• Disclosure items are proposals. The list in §5 and fig. 4 is what the authors think should be disclosed. The paper says how such risks should be documented "remains an area of active work" §5.
• Abstract wording differs. The arXiv metadata abstract says "multi-faceted approach"; the LaTeX source says "multi-layered approach" (abstract). This card follows the source. This card follows the source.
• Numbering is inferred. Section, equation and figure numbers come from counting \section, \begin{equation} and \begin{figure} in the LaTeX source, and the compiled PDF was not checked. Figures 1–4 follow source order. Equation 4 is the appendix equation, numbered after the three in the main text.

Lineage

• Builds on: the RLHF methodology popularised by Christiano et al. (2017) and used in Ziegler et al. (2019), Stiennon et al. (2020) and Bai et al. (2022), all cited in §1 and §2. None of these has a card here.
• Discusses as an alternative: direct, reward-model-free preference training §4.2.3 cites Rafailov et al. (2023), DPO, see [/papers/2305.18290].
• Superseded by: not stated. The paper is a 2023 survey and does not claim any successor.