---
kind: concept
slug: llm-simulated-participants
title: LLM simulated participants
summary: >-
  Language models as stand-ins for survey respondents. One 2023 study finds that blinded prompts confound a simulated price effect with invented customer details. Stating the randomisation design (unblinding) lowers error against a human survey. Adding covariates to the prompt gives non-monotone error and focalism. Trade-off: prompt-only cost versus fine-tuning on survey data.
tags: [llm-simulation, causal-inference, prompt-design, demand-estimation]
papers: ["2312.15524"]
related: [causal-inference, prompt-engineering, agent-based-simulation, benchmark-validity]
status: draft
updated_at: 2026-10-31
---

## Overview

This page covers one practice: prompting a language model to answer as a
simulated participant in an experiment, such as a purchase-intent survey. The
wiki has one source for it, a 2023 study of a price experiment with 40 consumer
products [arxiv:2312.15524 §1, §2]. Every claim below therefore comes from that
one paper, from one main model (gpt-4o-mini) and from its author-reported
results [arxiv:2312.15524 §2, §3].

The shared aim is that each simulated answer, for a product at a given price,
matches the purchase probability that a human survey gives for the same cell
[arxiv:2312.15524 §5]. The shared problem is that the model fills in customer
details the prompt does not state, and changing the price in the prompt can
change those details too [arxiv:2312.15524 §2]. The variants below differ in
how they handle that problem.

## Variants

### Blinded prompting (baseline)

The simulated customer is told that it is filling in a blank and is not told
that the price belongs to an experiment. Each product-price cell is an
independent call. The purchase probability is the share of purchase answers
over 50 draws at temperature 1 [arxiv:2312.15524 §2, §5]. In the pricing study,
the simulated demand curve is inverted-U, while the human curve slopes
downward [arxiv:2312.15524 §2, fig. 2]. The paper reports an out-of-box mean
absolute error (MAE) of 0.532 against the survey for this prompt [arxiv:2312.15524
tab. 1]. Card: [Gui and Toubia, 2023](/papers/2312.15524).

### Unblinded prompting (randomisation disclosed)

The system prompt adds that the price is drawn at random between a minimum and a
maximum, and that the customer sees only one price and does not know about this
[arxiv:2312.15524 §5]. The two bounds are placeholders in the prompt. Their values
are not stated in the paper [arxiv:2312.15524 §5]. Out-of-box MAE falls from
0.532 to 0.397 [arxiv:2312.15524 tab. 1]. The paper says unblinded prompts
gave lower MAE than blinded ones for every model it tested, with relative gains
from about 1% to over 60%. The per-model values appear only in a figure
[arxiv:2312.15524 §5, fig. 6]. Card: [Gui and Toubia, 2023](/papers/2312.15524).

### Covariate augmentation

The prompt states customer and store attributes, so the model does not have to
invent them. The paper tests two forms. One gives 500 generated personas per
product, with demographics sampled while the price is left blank [arxiv:2312.15524
§3, app. D]. The other adds up to 30 variables in twelve stages, from 14
demographic items to behavioural and psychological measures [arxiv:2312.15524
§3, app. E].

Adding covariates can reduce confounding but can also cause focalism, in which
the model gives too much weight to variables a consumer would usually ignore
[arxiv:2312.15524 §3]. Error does not fall monotonically. In the app. E table,
stage 1 (14 variables) gives 0.2250, stage 2 (15) gives 0.0971, the lowest value,
and stage 11 (25) gives 0.2025. Stage 12 (30) gives 0.1929 [arxiv:2312.15524 app. E].
These MAE values are measured against a replication of the human study by another
group, not against the survey in tab. 1 [arxiv:2312.15524 §3]. Do not compare
them directly with the tab. 1 figures.

### Fine-tuning on survey responses

The model is fine-tuned on the human survey. Evaluation holds out one of four
product-category groups at a time [arxiv:2312.15524 §5]. The prompt type is
chosen separately for fine-tuning and for evaluation, in a 2 × 2 design
[arxiv:2312.15524 §5, tab. 1]. Across the four fine-tuned cells in tab. 1, MAE lies between 0.113 and 0.134,
against 0.532 (blinded evaluation) for the out-of-box model [arxiv:2312.15524
tab. 1]. The best cell is unblinded fine-tuning evaluated with unblinded prompts,
at MAE 0.113. The paper states that this is 15% below the best blinded model,
0.134 [arxiv:2312.15524 §5].

A second round adds 10,000 observational purchase records from an Amazon browsing
dataset to the survey data [arxiv:2312.15524 §5, app. G]. With blinded fine-tuning,
MAE under blinded evaluation rises from 0.134 to 0.233. With unblinded fine-tuning
and unblinded evaluation it moves from 0.113 to 0.120 [arxiv:2312.15524 tab. 1,
tab. 2]. The paper's explanation is that the unblinded prompt lets the model weight the
experimental data correctly and discount the observational data [arxiv:2312.15524 §5].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Only prompting is possible, with no survey data | Unblinded prompt, with the price range stated | Out-of-box MAE 0.397 against 0.532 blinded in the same setup [arxiv:2312.15524 tab. 1]. No training needed. The bounds must be chosen, as they are not stated [arxiv:2312.15524 §5]. |
| A human survey on the same products exists and can be fine-tuned on | Fine-tune on the survey, with the unblinded prompt in training and evaluation | Lowest MAE in tab. 1, 0.113 [arxiv:2312.15524 tab. 1]. The paper describes unblinding as close to cost-free next to acquiring data [arxiv:2312.15524 §5]. |
| Observational data (purchases only) is available alongside the survey | Keep the unblinded prompt; check that blinded fine-tuning is not used | Blinded fine-tuning degraded from 0.134 to 0.233 when the Amazon data was added. Unblinded fine-tuning changed little, from 0.113 to 0.120 [arxiv:2312.15524 tab. 1, tab. 2]. |
| Persona or demographic data is available and the study is on one model | Add covariates in small steps and check each one against held-out human data | Error changed sharply with one added variable and was not monotone at larger sets [arxiv:2312.15524 app. E]. The paper gives no single rule for where to stop [arxiv:2312.15524 §3]. |

Only the first two rows rest on the paper's main comparisons. The last two rest
on one model, and the covariate results use a replication benchmark rather than
the survey, so treat them as cautions, not settled advice.

## Implementation sketch

The common core is one call per product-price cell, averaged over draws, and
scored against the human purchase probability. Variants change only the system
prompt and the covariates [arxiv:2312.15524 §2, §5].

```pseudocode
# Build the simulated purchase probability for one product-price cell  [§2, §5]
# Input:  system prompt, product, price, number of draws
# Output: share of draws that answer "buy"

def purchase_probability(system_prompt, product, price, draws=50):
    # Each draw is an independent call at temperature 1  [§2, §5]
    answers = [ask_model(system_prompt, product, price, temperature=1.0)
               for _ in range(draws)]
    return share_of_purchase(answers)

# Choose the system prompt: blinded or unblinded  [§5]
# The unblinded text states the random price range and that the customer
# does not know about it. The min and max values are not stated  [§5]
def make_prompt(unblinded, price_min=None, price_max=None):
    if not unblinded:
        return BLINDED_PROMPT
    return UNBLINDED_PROMPT.format(min_price=price_min, max_price=price_max)

# Score against the human survey  [§5, tab. 1]
# Mean absolute error over all (product, price) cells
def score(system_prompt, cells):
    errors = []
    for product, price, human_probability in cells:
        model_probability = purchase_probability(system_prompt, product, price)
        errors.append(abs(model_probability - human_probability))
    return sum(errors) / len(errors)
```

## Contradictions

None known. The page has one source, so there are no cross-paper disagreements.
Within that paper, the covariate MAE values (table in app. E) and the survey MAE
values (tab. 1) come from different benchmarks, so they cannot be ranked against
each other [arxiv:2312.15524 §3, app. E, tab. 1].

## Papers

- [Gui and Toubia, 2023](/papers/2312.15524): shows that blinded prompts confound the price effect with invented covariates, and that unblinded prompts and survey fine-tuning lower error against a human survey.