Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions

In plain words

Public opinion surveys are slow and expensive. A researcher drafting a new survey would like to know, before asking anyone, roughly how different groups of people are likely to answer. Language models could help, but when they are simply told "answer as someone from this group", their answers tend to bunch up on one option or to miss how much people within a group disagree.

The paper's idea is to train the model on real survey results instead of only prompting it. Each training example pairs a question and a group with the share of that group who picked each answer. The model is trained so that its own probabilities for the answer letters match those shares. The authors collected a large set of such questions and groups from public survey data, and they test whether the trained model also works for surveys and groups it did not see during training. The check is how close the predicted spread of answers is to the real spread.

TL;DR

• Fine-tune a pretrained LLM with LoRA so that its probabilities over answer letters match the human answer distribution of a demographic subgroup §3.1 app. B.
• Loss: forward KL from the human distribution to the model distribution, computed over the option tokens §3.1.
• Data: SubPOP-Train, 3,229 multiple-choice questions from Pew ATP waves 61 to 132, and SubPOP-Eval, 133 questions from GSS 2022 §3.2 app. A.3, app. A.4.
• Author-reported: Wasserstein distance (WD) on OpinionQA drops to 0.094 to 0.106 for the four fine-tuned models, against 0.138 to 0.173 for the best zero-shot baseline tab. 1.
• Pitfalls: the paper's WD formula is not normalised the way the code is, the text's percentage reductions do not reproduce from Table 1, and several training settings (epochs, schedule, validation split) are not stated.

Problem

Surveys are costly, and their results depend on calibration and on who responds §1. Prompting an LLM with a group description ("steering") is the usual way to simulate a group, but the authors report that prompting alone rarely reproduces the spread of human answers §1, §4.2. Common failure modes they list are:

• The base model over-represents groups that are common in its pretraining text, and gives stereotyped answers for others §1.
• Instruction-tuned models tend to put most probability on one answer token, which hides the within-group spread §4.2, app. C.2.
• Prompt formats are sensitive, so small wording changes move results §4.2.

The gap the paper targets is a steering method that predicts the whole answer distribution of a group, generalises to new questions, new survey families and new groups §1, §4.2, §4.3.

Method

Train a pretrained causal LLM on pairs of (steering prompt plus question, human distribution over options). At inference, read the model's probabilities for the answer letters, with no sampling §3.1 app. B run_inference.py.

Model output and training objective

For a question q, a subgroup g and answer set A_q, the model defines p_θ(A_q | q, g) over the answer letters. The target is the group-level human distribution p_H(A_q | q, g) computed from weighted survey responses §3.1 app. A.2. Training minimises the forward KL divergence from p_H to p_θ, averaged over (q, g) pairs §3.1. The paper says forward KL is used because it penalises the model for missing an answer that the group often picks §3.1.

The paper compares three ways to represent the target, and explicit probabilities perform best: one-hot (only the most frequent answer), augmentation by response frequency, and explicit probabilities §3.1 app. C.1.

Datasets

• SubPOP-Train (training): 3,229 questions from ATP waves 61 to 132, excluding waves already used in OpinionQA. Questions with more than 10 options, redacted answers, or dependence on earlier questions were removed. Question wording was rewritten by GPT-4o with minimal changes §3.2 app. A.3.
• SubPOP-Eval (out-of-distribution test): 133 questions from the 2022 General Social Survey, selected with the same criteria §3.2 app. A.4.
• OpinionQA (evaluation only): about 500 questions from 14 ATP waves, with ordinal information for each option app. A.2.
• Deduplication: a question pair counts as the same question when the cosine similarity of their text embeddings is at least 0.87. The authors chose this threshold by manual inspection app. A.5.
• Subpopulations: 22 subpopulations are used in the main evaluation, defined by demographic, socioeconomic and ideological traits §1 app. A.2. The paper says training data give each group an equal number of samples §4.3.

Evaluation metric

Answer options are mapped to ordinal values (for example, "Very likely" to 1). The WD between p_H and p_θ is computed over these values, and per-question WDs are averaged over questions and groups §3.3 app. B. The paper argues WD is preferable to one-hot accuracy because it penalises a large miss more than a near miss §3.3.

Reference points:

• Upper bound: uniform distribution over the options §4.1 app. D.
• Lower bound: WD between the human group distribution and bootstrap resamples of its respondents, with R = 1000 bootstraps app. D.

Training step

# One fine-tuning example: (steering prompt + question, human distribution)
# [§3.1] [app. B]
# Input:  prompt_tokens (T,), option_tokens (K,) = ids of " A", " B", ...
#         human_dist (K,) with entries summing to 1
# Output: scalar loss, the forward KL from human_dist to the model

def training_loss(prompt_tokens, option_tokens, human_dist):
    # Next-token logits at the position that predicts the answer letter
    logits = model(prompt_tokens)[-1]              # (vocab,)
    # Keep the logits of the K option letters and renormalise over them
    option_logits = logits[option_tokens]          # (K,)
    model_dist = softmax(option_logits)            # (K,)
    # Forward KL(human || model), summed over options  [§3.1]
    return sum(
        human_dist * (log(human_dist) - log(model_dist))
    )

The code applies a softmax over the whole vocabulary, selects the option tokens and renormalises. This equals the softmax over option logits used above. It adds 1e-8 inside each logarithm train_utils.py.

Ordinal Wasserstein distance

# Ordinal Wasserstein distance between two option distributions
# [§3.3] [app. B] [code: subpop/train/utils/train_utils.py#L79-L150]
# Input:  human (K,), model (K,) probabilities; ordinal (K,) option values
# Output: scalar, divided by the ordinal range (see Pitfalls)

def ordinal_wd(human, model, ordinal):
    # Sort options by ordinal value; ignore leading options with value < 0
    human, model, ordinal = sort_by(ordinal, human, model)
    human, model = normalise(human), normalise(model)
    # Cumulative distributions over the sorted options
    cdf_human = cumsum(human)
    cdf_model = cumsum(model)
    # Gap between neighbouring ordinal values, times the CDF difference
    gaps = ordinal[1:] - ordinal[:-1]
    total = sum(abs(cdf_human[:-1] - cdf_model[:-1]) * gaps)
    return total / (max(ordinal) - min(ordinal))

Inference

Prompt the model with the steering prompt and the question, and read the probability of each option letter from the first generated token. Both the "Answer:A" and "Answer: A" spellings are summed. The probabilities are renormalised over the options run_inference.py app. B.

Hyperparameters

Results

Author-reported. Setup: four fine-tuned base models, OpinionQA and SubPOP-Eval, WD averaged over all questions and groups tab. 1 §4.2.

Table 1 tab. 1: WD for 22 subpopulations (lower is better).

The Modular Pluralism cell is a one-hot error rate taken from the original paper, not a WD tab. 1 caption.

Table 2 tab. 2: per-group results on OpinionQA, Llama-2-7B. "Relative improvement" is the share of the gap between zero-shot WD and the human lower bound that fine-tuning closes. Rows marked * are subpopulations in the training data; the rest are unseen.

The text reports a mean relative improvement of 46.7% (standard deviation 4.4%) across the 22 training subpopulations, and 44.7% for the unseen age groups §4.3. Appendix C.3 gives 43.1% (standard deviation 6.7%) for the other unseen groups app. C.3.

Other results

• Fine-tuning reduces WD by a relative 32 to 46% on OpinionQA and 39 to 42% on SubPOP-Eval against the best baseline, according to the text §4.2. These ranges do not reproduce from Table 1; see Pitfalls.
• Intergroup disagreement (WD between a target group and a source group) for education levels. Human responses show that groups with similar education disagree less. The fine-tuned model reproduces this pattern for education levels not seen in training, while the zero-shot base model does not §4.3, fig. 4.
• Scaling: training on random 25%, 50%, 75% and 87.5% subsets of SubPOP-Train, the models reach 72 to 78% of the full gain at 25%. The authors fit linear trend lines on log-scale axes and extrapolate that about 25 times the current data would give WD 0.07 for Mistral-7B, against the human lower bound of 0.031 §4.4, fig. 5.

Response-distribution modelling (Llama-2-7B, WD) app. C.1:

Chat versus base model (Llama-2-7B) app. C.2:

Per-wave and per-group results on OpinionQA and SubPOP-Eval are in app. E; they are not transcribed here.

Paper vs code

Code at e0569aca48361415779aac7407aed1e2cbb67b4a (BSD-3-Clause).

• Objective. The paper's forward KL §3.1 matches the code's default loss, loss_function_type = "ce", which is the KL term training.py train_utils.py. The name "ce" is misleading, since the term is a KL divergence. The code also has a WD loss option.
• WD normalisation. Appendix B gives the WD as the sum of absolute CDF differences over the options, with no division app. B. The code also multiplies each CDF gap by the spacing between ordinal values and divides by the ordinal range, so the result lies in [0, 1] train_utils.py survey_utils.py. The reported values (uniform upper bound 0.178 and 0.208) fit the normalised version. This is an inference from the magnitudes, not a stated fact.
• Refusal option. The worked example in §3.1 fig. 2 lists option E "Refused" as a response. The data script's docstring says refusal is excluded from the target distribution prepare_finetuning_data.py. The script actually appends the refusal rate as the last entry of the training target prepare_finetuning_data.py. Whether the refusal mass enters the KL depends on whether "Refused" is one of the option tokens; check before implementing.
• Batch size and epochs. The paper's selected batch size is 256 app. B. The code's default is 32 training.py and the README example uses 128 README.md. The code default for epochs is 3 training.py and the README example uses 50. The paper does not state epochs.
• Learning rate. The code default is 1e-4 training.py. The paper's Llama-2-7B setting is 2e-4 app. B.
• Scheduler. The code defaults to a step decay with gamma 0.85 per epoch training.py finetuning.py. The README example uses cosine decay with warmup README.md. The paper does not state which was used for the reported runs.
• LoRA. Rank 8, alpha 32, dropout 0.05 and query and value targets match the paper peft.py app. B.
• Answer tokens. Training uses the letter tokens with a leading space (" A"). Inference sums both the "Answer:A" and "Answer: A" spellings run_inference.py. The paper describes only the option probabilities.
• Validation metric. The validation loop drops samples whose WD is exactly zero before averaging train_utils.py. The paper does not mention this filter.
• Baselines. The Modular Pluralism and few-shot baselines were not checked against code in this repository.

Pitfalls

• Reported reductions do not reproduce. Table 1 gives (best baseline − this method) / best baseline of 32 to 40% on OpinionQA and 34 to 42% on SubPOP-Eval. The text says 32 to 46% and 39 to 42%. Closing the gap to the human lower bound gives 41 to 49% on OpinionQA. Cite Table 1 values directly and do not repeat the text's ranges.
• The "46%" headline is a different measure. The abstract's "up to 46%" and the mean of 46.7% in §4.3 are gap-closure figures per subgroup, not WD ratios. Table 2 uses the gap-closure measure.
• WD scale. Use the normalised version (divide by the ordinal range). The formula in Appendix B is not normalised, so an implementation that follows it literally gives values several times larger for a question with more than two options.
• Modular Pluralism is not in WD units. Its 0.285 and 0.279 are WD values in the table, but the percentages (55.6%, 55.2%) are one-hot error rates from the original paper. Do not compare them as the same metric.
• Unstated training settings. Epochs, schedule, warmup and validation split are not stated. The code's 90/10 question split is the only source prepare_finetuning_data.py. Expect to tune these.
• Llama-3-70B is not tuned. Its learning rate was set heuristically app. B, so its Table 1 numbers are not directly comparable with the other three models.
• Chat models are worse on SubPOP-Eval. Fine-tuning Llama-2-7B-chat gives 0.148 on SubPOP-Eval against 0.121 for the base model app. C.2. The paper uses base models.
• Numerical guards. The code adds 1e-8 inside the logarithms train_utils.py. Human distributions with zero entries are therefore handled by the epsilon, not by masking.
• Ordinal handling. Options with negative ordinal value are dropped only when they come first after sorting train_utils.py. Samples whose WD is exactly zero are excluded from the validation mean, which includes questions whose options all share one ordinal value. Check how your data encodes "Refused" and "Not sure".
• Data and code licences. The paper's SubPOP dataset is distributed on Hugging Face, and the page requires accepting its terms [README.md]. The raw ATP (Pew) and GSS (NORC) files come under those sources' own terms. The question rewrites were produced with GPT-4o app. A.3; regenerating them may carry different terms. The repository's LICENSE is BSD-3-Clause, but several training files carry a Meta "Llama 2 Community License Agreement" header. This card quotes no code, so check the headers before copying any file.
• Cost. Four base models and 22 subpopulations, with a fine-tuning run for each model, need A100-80GB-class GPUs app. B. The paper gives no per-run wall-clock time.

Lineage

• Builds on: OpinionQA, the survey and steering-prompt benchmark of Santurkar et al., "Whose opinions do language models reflect?" (ICML 2023; no arXiv id in the paper's bibliography) app. A.2. LoRA, arXiv:2106.09685 §3.1.
• Response-distribution targets: the one-hot target follows CultureLLM, arXiv:2402.10946 §3.1. Augmentation by response frequency is the other alternative the paper compares against §3.1 app. C.1.
• Compared against: zero-shot QA, BIO and PORTRAY prompting; few-shot prompting with the top five questions by embedding similarity, following arXiv:2305.14929 §4.1, app. D; Modular Pluralism, Feng et al. (2024), which uses one fine-tuned model per community plus a black-box aggregator §4.1, app. D.
• Pretrained base models preferred over chat models, following Virtual Personas (Moon et al., 2024) §4.2.
• Superseded by: not known from this wiki.