LLM simulated participants

Overview

This page covers one practice: prompting a language model to answer as a simulated participant in an experiment, such as a purchase-intent survey. The wiki has one source for it, a 2023 study of a price experiment with 40 consumer products 2312.15524 §1, §2. Every claim below therefore comes from that one paper, from one main model (gpt-4o-mini) and from its author-reported results 2312.15524 §2, §3.

The shared aim is that each simulated answer, for a product at a given price, matches the purchase probability that a human survey gives for the same cell 2312.15524 §5. The shared problem is that the model fills in customer details the prompt does not state, and changing the price in the prompt can change those details too 2312.15524 §2. The variants below differ in how they handle that problem.

Variants

Blinded prompting (baseline)

The simulated customer is told that it is filling in a blank and is not told that the price belongs to an experiment. Each product-price cell is an independent call. The purchase probability is the share of purchase answers over 50 draws at temperature 1 2312.15524 §2, §5. In the pricing study, the simulated demand curve is inverted-U, while the human curve slopes downward 2312.15524 §2, fig. 2. The paper reports an out-of-box mean absolute error (MAE) of 0.532 against the survey for this prompt [arxiv:2312.15524 tab. 1]. Card: Gui and Toubia, 2023.

Unblinded prompting (randomisation disclosed)

The system prompt adds that the price is drawn at random between a minimum and a maximum, and that the customer sees only one price and does not know about this 2312.15524 §5. The two bounds are placeholders in the prompt. Their values are not stated in the paper 2312.15524 §5. Out-of-box MAE falls from 0.532 to 0.397 2312.15524 tab. 1. The paper says unblinded prompts gave lower MAE than blinded ones for every model it tested, with relative gains from about 1% to over 60%. The per-model values appear only in a figure 2312.15524 §5, fig. 6. Card: Gui and Toubia, 2023.

Covariate augmentation

The prompt states customer and store attributes, so the model does not have to invent them. The paper tests two forms. One gives 500 generated personas per product, with demographics sampled while the price is left blank [arxiv:2312.15524 §3, app. D]. The other adds up to 30 variables in twelve stages, from 14 demographic items to behavioural and psychological measures [arxiv:2312.15524 §3, app. E].

Adding covariates can reduce confounding but can also cause focalism, in which the model gives too much weight to variables a consumer would usually ignore 2312.15524 §3. Error does not fall monotonically. In the app. E table, stage 1 (14 variables) gives 0.2250, stage 2 (15) gives 0.0971, the lowest value, and stage 11 (25) gives 0.2025. Stage 12 (30) gives 0.1929 2312.15524 app. E. These MAE values are measured against a replication of the human study by another group, not against the survey in tab. 1 2312.15524 §3. Do not compare them directly with the tab. 1 figures.

Fine-tuning on survey responses

The model is fine-tuned on the human survey. Evaluation holds out one of four product-category groups at a time 2312.15524 §5. The prompt type is chosen separately for fine-tuning and for evaluation, in a 2 × 2 design 2312.15524 §5, tab. 1. Across the four fine-tuned cells in tab. 1, MAE lies between 0.113 and 0.134, against 0.532 (blinded evaluation) for the out-of-box model [arxiv:2312.15524 tab. 1]. The best cell is unblinded fine-tuning evaluated with unblinded prompts, at MAE 0.113. The paper states that this is 15% below the best blinded model, 0.134 2312.15524 §5.

A second round adds 10,000 observational purchase records from an Amazon browsing dataset to the survey data 2312.15524 §5, app. G. With blinded fine-tuning, MAE under blinded evaluation rises from 0.134 to 0.233. With unblinded fine-tuning and unblinded evaluation it moves from 0.113 to 0.120 [arxiv:2312.15524 tab. 1, tab. 2]. The paper's explanation is that the unblinded prompt lets the model weight the experimental data correctly and discount the observational data 2312.15524 §5.

When to use which

Only the first two rows rest on the paper's main comparisons. The last two rest on one model, and the covariate results use a replication benchmark rather than the survey, so treat them as cautions, not settled advice.

Implementation sketch

The common core is one call per product-price cell, averaged over draws, and scored against the human purchase probability. Variants change only the system prompt and the covariates 2312.15524 §2, §5.

# Build the simulated purchase probability for one product-price cell  [§2, §5]
# Input:  system prompt, product, price, number of draws
# Output: share of draws that answer "buy"

def purchase_probability(system_prompt, product, price, draws=50):
    # Each draw is an independent call at temperature 1  [§2, §5]
    answers = [ask_model(system_prompt, product, price, temperature=1.0)
               for _ in range(draws)]
    return share_of_purchase(answers)

# Choose the system prompt: blinded or unblinded  [§5]
# The unblinded text states the random price range and that the customer
# does not know about it. The min and max values are not stated  [§5]
def make_prompt(unblinded, price_min=None, price_max=None):
    if not unblinded:
        return BLINDED_PROMPT
    return UNBLINDED_PROMPT.format(min_price=price_min, max_price=price_max)

# Score against the human survey  [§5, tab. 1]
# Mean absolute error over all (product, price) cells
def score(system_prompt, cells):
    errors = []
    for product, price, human_probability in cells:
        model_probability = purchase_probability(system_prompt, product, price)
        errors.append(abs(model_probability - human_probability))
    return sum(errors) / len(errors)

Contradictions

None known. The page has one source, so there are no cross-paper disagreements. Within that paper, the covariate MAE values (table in app. E) and the survey MAE values (tab. 1) come from different benchmarks, so they cannot be ranked against each other 2312.15524 §3, app. E, tab. 1.

Papers

• Gui and Toubia, 2023: shows that blinded prompts confound the price effect with invented covariates, and that unblinded prompts and survey fine-tuning lower error against a human survey.