Fine-tuning language models to find agreement among humans with diverse preferences

In plain words

Most language models are tuned toward one imagined "typical" user. This paper asks a different question: when a small group of people disagree about a political question, can a model write a short statement that the whole group could accept? The authors collected written opinions from groups of people on debate questions, then had the model write several candidate statements that try to reflect all of those opinions.

The model does not pick a side or try to persuade anyone. Each candidate is scored by a second model that guesses how much each group member would agree with it, given that person's own opinion. The candidate with the best overall score for the group is the one shown. People who were asked to rate these statements preferred them over statements from prompted models and, for the best of them, over the best individual opinion written by a group member.

TL;DR

• A 70B language model is fine-tuned to produce a consensus statement for a group of three to five people from their written opinions §3.4.
• A reward model trained on human agreement ratings predicts each person's agreement with each candidate; 16 candidates are sampled and the one with the highest Utilitarian welfare is shown §3.3 §4.1.
• Author-reported: raters preferred it over prompted Chinchilla baselines and over human-written opinions §4.1 §4.2.
• Data is from UK-based crowd participants on UK political questions, so generalisation beyond that population is untested §5.1.
• The public repository does not contain the fine-tuning or reward-model training code described here; see Paper vs code.

Problem

Preference tuning for language models usually assumes that preferences are static and shared across people. That assumption works for tasks with a ground truth, such as summarising an article, but not for group decisions where people hold different values §1. The authors frame consensus formation as a test case: produce a statement that a large share of a group can accept, without any single group member's view being imposed §1 §3.3.

Two practical constraints shape the method. The model's prompt length limits how many opinions it can read at once, so groups are kept at three to five people §5.1. Human ratings are expensive, so the pipeline is built to make the most of each collected rating §3.4.

Method

Data pipeline. The authors first generated debate questions with a prompted 70B model, starting from 152 hand-written seed questions, and filtered them manually §3.1 app. A. Questions were clustered by topic into in-distribution training and test sets and an out-of-distribution test set §3.1. Participants wrote an opinion on a question, then rated candidate consensus statements on agreement and quality §3.2 fig. 1.

Group alignment (inference). For each candidate statement, a reward model gives every participant a predicted agreement score, conditioned on that participant's own opinion §3.3. The scores are aggregated with a social welfare function (SWF) over the group. The paper uses a one-parameter family of isoelastic welfare functions, with the Utilitarian case (inequality aversion α = 0) used for the main results §3.3 eq. 1. The reward model is used only to rank candidates; the scalar rewards select the statement at inference app. C.2.

Training loop. The procedure iterates twice §3.4 fig. 1:

1. Generate candidate statements for groups of 3, 4 or 5 opinions, then collect human ratings. Round one uses zero-shot and few-shot prompted Chinchilla; round two uses candidates from the first fine-tuned model §3.4 step 1 app. C.
2. Supervised fine-tuning (SFT) of a pretrained 70B Chinchilla model on candidates rated as high quality. The purpose is to teach the prompt format and raise quality, not to match a demonstration; agreement ratings are not used for filtering so that stances stay diverse §3.4 step 2 app. C.1.2.
3. Train a reward model (RM) from a pretrained Chinchilla model with an extra linear output layer. It reads a question, one opinion and one candidate, and is trained on pairwise agreement preferences with cross-entropy §3.4 step 3 app. C.2.

Candidate generation. At inference the SFT model samples 16 statements with nucleus sampling app. C.1. Each call shuffles the order of the opinions in the prompt app. C.1. The reranked winner is the candidate with the highest welfare under the chosen SWF §4.1.

Baselines. SFT-Base is the fine-tuned model without reranking (one sample). Few-shot and zero-shot are prompted Chinchilla models §4.1 app. C.1.1.

Key equation (the welfare function, eq. 1):

W_{\alpha}(u_1,\dots,u_n) = \left[\frac{1}{n}\sum_{i=1}^{n} u_i^{1-\alpha}\right]^{\frac{1}{1-\alpha}} \quad \alpha \ge 0,\ \alpha \ne 1

W_{1}(u_1,\dots,u_n) = \sqrt[n]{\prod_{i=1}^{n} u_i} eq. 1

Pseudocode for inference, as the paper describes it:

# Consensus selection at inference (SFT-Utilitarian)  [§3.3] [§4.1]
# Input:  question (str); opinions (3 to 5 strings)
# Output: one consensus statement (str)

def select_consensus(question, opinions):
    # Sample 16 candidates, nucleus p = 0.8, temperature 1  [app. C.1]
    # Reshuffle the opinions in every prompt  [app. C.1]
    candidates = []
    for _ in range(16):
        prompt = build_prompt(question, shuffle(opinions))
        candidates.append(sample_nucleus(sft_model, prompt))
    # Predict each person's agreement with each candidate  [§3.3]
    # One reward call per (person, candidate) pair
    utilities = [
        [reward_model(question, opinion, c) for c in candidates]
        for opinion in opinions
    ]
    # Utilitarian welfare, alpha = 0: the mean over people  [§3.3] [eq. 1]
    welfare = [mean(column) for column in transpose(utilities)]
    return candidates[argmax(welfare)]

Pseudocode for one reward-model training example, as described in the paper:

# Reward-model loss on one preference pair  [§3.4] [app. C.2]
# Input:  question; one participant's opinion; a preferred and a rejected
#         candidate, whose ratings differ by 2 or more Likert points
# Output: scalar loss

def pair_loss(question, opinion, preferred, rejected):
    # Scalar reward for each candidate, given this one opinion  [§3.4]
    score_preferred = reward_model(question, opinion, preferred)
    score_rejected = reward_model(question, opinion, rejected)
    # The paper says "standard cross-entropy" over the pair  [app. C.2]
    # Our reading of that: -log(sigmoid(difference)); not stated exactly
    margin = score_preferred - score_rejected
    return -log(sigmoid(margin))

Hyperparameters

Values are from the paper unless the Notes say otherwise. "Not stated" means the arXiv v1 text and appendix do not give it.

Results

Author-reported. Setup: a 70B Chinchilla-based model against prompted Chinchilla baselines and human-written opinions, with crowd-sourced UK participants rating statements on 7-point scales §4 §5.1. Win rates are pairwise comparisons of Likert ratings, ties excluded fig. 2. Bracketed values are 95% bootstrapped confidence intervals unless stated otherwise.

Position statements: about half of the rounds were divisive, meaning at least one participant disagreed with the statement §4.1. Human-written opinions are described as high quality §4.2.

Paper vs code

The official repository is google-deepmind/habermas_machine (Apache-2.0), pinned at code_commit. Its README describes it as the dataset and a prompted version of the method, not the fine-tuned model in this paper README.md. Discrepancies, each with both citations:

• No training code. The paper fine-tunes Chinchilla with SFT and trains a reward model §3.4 app. C. The repository contains no SFT or reward-model training code and no weights; the README says the prompted version "is not fine-tuned on the dataset" README.md. The SFT and RM hyperparameters in this card therefore cannot be checked against code.
• Base model. The paper uses Chinchilla 70B §3.1 §3.4. The code calls a Gemini model through AI Studio; the example notebook sets MODEL = 'gemini-1.5-flash' example_aistudio.ipynb aistudio_client.py.
• Reward model. The paper's RM outputs a scalar and is trained with cross-entropy on pairs app. C.2. The code's reward model asks an LLM to write one full ranking of all candidates in arrow notation (for example B > A > C), parsed into integer ranks where 0 is best [code: habermas_machine/reward_model/cot_ranking_model.py#L32-L80] [code: habermas_machine/reward_model/cot_ranking_model.py#L323-L387].
• Aggregation. The paper picks the candidate with the highest Utilitarian welfare over predicted agreements §3.3 eq. 1. The code aggregates the citizens' rankings with the Schulze method, which uses pairwise defeat counts and strongest paths schulze_method.py schulze_method.py. The winner is the first candidate in the untied Schulze order [code: habermas_machine/machine.py#L242-L249]. The example notebook uses TBRC tie breaking example_aistudio.ipynb. Welfare functions do not appear in the code.
• Sampling. The paper uses nucleus sampling with cut-off 0.8 and temperature 1 app. C.1. The code's default temperature is 0.8 [code: habermas_machine/llm_client/base_client.py#L21]. The AI Studio call passes temperature and stop sequences but no top_p [code: habermas_machine/llm_client/aistudio_client.py#L103-L107].
• Candidate count and group size. The paper uses 16 candidates §3.4 [app. C.1]. The code default is 16 machine.py, but the example notebook uses 4 example_aistudio.ipynb. The code default is five citizens machine.py. The code has no step that drops opinions to create groups of 3 or 4 [code: habermas_machine/machine.py#L172-L176].
• Critique round. The paper has no critique step; its iteration is about training §3.4. The code docstring says "In the paper, we use one opinion and one critique round" machine.py. The README refers to a different, later publication README.md. Treat the critique round as a feature of the code, not of this arXiv version.
• Retries and failures. The paper does not describe error handling. The code retries up to eight times by default [code: habermas_machine/machine.py#L69]. A statement that still fails is returned as an empty string cot_model.py. A ranking that still fails raises an error [code: habermas_machine/machine.py#L148-L150].
• Seeds. The AI Studio client ignores the seed [code: habermas_machine/llm_client/aistudio_client.py#L94], so runs are not reproducible through the seed argument.

Pitfalls

• The paper's method is not in the repository. Anyone implementing this paper must build SFT, the reward model and the welfare step from the description. The repository gives a prompted, different algorithm (see above).
• Welfare needs positive utilities; the paper does not say how to get them. Eq. 1 raises each u_i to 1 − α, which is only defined for u_i > 0 when α is not an integer, and the geometric mean needs u_i > 0 too. The reward model is a linear output with no stated range §3.4 app. C.2. How scores are mapped to positive values is not stated.
• Inequality-aversion distribution is not given. Training draws α from a log-normal distribution §3.4, but its location and scale are not stated. Evaluation uses α = 0.
• The welfare choice may not matter. Reranking under the Utilitarian, Rawlsian and Bernoulli-Nash functions showed no reliable difference in mean or minimum agreement §5.1 app. D.5. The reported gains come from the whole pipeline (fine-tuning plus reranking), not from the welfare function.
• SFT quality threshold conflicts within the paper. §3.4 step 2 keeps candidates with mean quality "6 or higher" §3.4 step 2. Appendix C.1.2 trains on candidates above 5 and validates on those above 6. Implementers should use the appendix values and note the conflict.
• Cluster count conflicts. The main text says 110 topic clusters §3.1. The appendix says the silhouette score converges at 125 clusters and uses those app. A.
• Model-size interval conflicts. The text gives the 70B vs 1.4B win rate as 59.4% with interval [53.7, 64.7] app. D.1. Table A5 gives [52.3, 66.7] tab. A5.
• Reward-model epoch count. Round 2 is stated as 0.8 epochs app. C.2. By our arithmetic, 1113 steps × batch 8 over 10434 pairs is about 0.85 epochs. Check the step and batch values before reproducing.
• Figure caption cross-reference. The appendix training-figure caption cites Sections 4.1 and 4.2 for the within-distribution and out-of-distribution sets app. C.2. Out-of-distribution results are in §4.4.
• Pairwise rating data is filtered hard. Pairs are kept only when ratings differ by two or more points, and ties are dropped app. C.2. The authors chose this stricter filter over dropping only exact ties, because it trained faster and evaluated better app. C.2.
• Reward model sees candidates not written for the rater. Participants also rate candidates produced from other members' opinions §3.4 step 3. The opinion-exclusion analysis is the authors' check that the model uses the opinions it is given §4.3; it covers only candidates built from three or four opinions §4.3.
• Quality may stand in for agreement. Higher quality ratings for the reranked model do not show why the statements are preferred. The authors warn that quality may be a proxy for agreement, which favours confident or authoritative participants §5.1.
• Population. Participants were UK-based crowd workers and the questions were about UK politics. Results may not transfer to other populations, and the authors note that baseline agreement may be inflated by homogeneity in the sample §5.1.
• Scale. Groups are limited to four or five people by prompt length §5.1. Larger groups would need an architecture that aggregates opinion embeddings; the paper does not test one §5.1.
• Cost. Each candidate set is sampled while participants wait app. C.1. Fine-tuning used 64 TPU v3 cores for the language model and 32 for the reward model app. C.1.2 app. C.2. The total study cost was about £46,000 §3.2. Data collection is the main expense, and each training round needs new human ratings §3.4 app. C.
• Factuality and misuse. The model is not fine-tuned for factual accuracy. The authors flag misrepresentation of consensus and persuasion risks §5.2.
• Data release. The arXiv v1 text does not state whether the human ratings are released. The repository links a dataset, but it belongs to the later publication (see Paper vs code).

Lineage

• Builds on: Chinchilla, the 70B base model §3.4; reward-model fine-tuning from human preferences, following Ziegler et al. (2019) §3.4; expert-iteration style sampling and filtering §3.4 step 2 footnote.
• Compared against: SFT-Base (the same fine-tuning without reranking), few-shot and zero-shot prompted Chinchilla §4.1; a 1.4B fine-tuned model app. D.1; naive reward baselines (tf-idf similarity, longest candidate, ROUGE, BLEU; the text calls them six) app. C.2.
• Superseded by: not in this wiki. The code README cites a later Science publication by the same group, which is not covered here.