Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

In plain words

Before running an experiment on people, a researcher can ask whether a text model would behave the way people did. This paper proposes a test of that kind, called a Turing Experiment. The model gets a written setup with a made-up participant, such as a named person who has been given a sum of money to split, and it writes what that person does next. The model is run again for many different names, so the results look like a small group of people instead of one imagined individual.

The authors apply this to four classic studies from economics, psycholinguistics and social psychology. They then compare the simulated group with what real people did in the original studies. In three of the four, the simulated trends match the reported human ones. In the fourth, a wisdom-of-crowds estimation task, the newer models give answers that are too accurate in a way people's answers are not.

TL;DR

Turing Experiments simulate the behaviour of many zero-shot participants with a prompted language model and compare the outcomes with published human data. Four experiments are simulated with OpenAI models. The recent models reproduce the human trends in three of them, and show a hyper-accuracy distortion in the wisdom-of-crowds task. Everything is author-reported, and the released code is archived.

Problem

Simulating human behaviour is hard, and human subject studies are costly and slow. A single-persona "Turing Test" says little about which people and which behaviours a model can simulate. The authors want a test that asks which aspects of human behaviour a model reproduces across a representative sample of participants §1. Each Turing Experiment must be zero-shot: neither the prompt nor the training data should contain prior data for that specific experiment, though the authors note this is hard to enforce with models pretrained on large corpora §1.

Method

A Turing Experiment (TE) is a program that builds prompts, queries a language model (LM) and reconstructs a text record of a simulated run. Its inputs are participant details (for example a title and surname) and optional experimental conditions. Its output is a record with the outcome of interest §1 §2.

1. Prompt construction. Each TE writes one or more zero-shot prompts with the participant names and the condition filled in §2 fig. 1. The authors stress validating prompts on the validity rate, before any hypothesis is tested, to avoid p-hacking §2.
2. Querying the LM. Two query types are used §2.
   • Completion: sample a continuation from the LM with temperature 1 and top-p 1 §3.
   • k-choice prompt: score each valid completion c_i after prompt s, and normalize over the valid completions. The normalizing constant Z is the validity rate §2.
3. Record and outcome. The record is the prompt plus the completion, or a sequence of prompts and completions for multi-step TEs §4 §6.
4. Validity. Completions that do not follow the required format are counted as invalid. Table 1 reports the valid share for five models tab. 1.

The four TEs are:

• Ultimatum Game (§4). Two-choice prompts ask whether the responder accepts or rejects an offer out of a $10 endowment, for 11 offers (0 to 10) and 10,000 name pairs §4 fig. 3.
• Garden-path sentences (§5, app. D). Two-choice prompts ask whether a sentence is grammatical or ungrammatical. The paper compares garden-path sentences with their controls app. D.
• Milgram shock (§6, app. F). A multi-step loop over 30 shock levels with free-form subject actions, classified by LM judges, and experimenter prods. Details are in the pseudocode below app. F.1 fig. 17.
• Wisdom of crowds (§7, app. E). Free-response prompts ask for an integer estimate. Estimates are normalized by the true answer fig. 8 app. E.

A novel variant of the Milgram scenario (a driver is "submerged" instead of a learner being shocked) and five authored questions for the wisdom-of-crowds TE test whether the results depend on the original wording §3 §6 app. F.2.

# k-choice scoring: p_i = p(s c_i) / Z  [§2]
def choice_probabilities(prompt, completions, lm):
    # Unnormalized probability of prompt followed by each choice  [§2]
    scores = [lm.sequence_prob(prompt, c) for c in completions]
    # Z: total mass on the valid completions, i.e. the validity rate  [§2]
    validity_rate = sum(scores)
    # Normalize over the valid completions only  [§2]
    return [score / validity_rate for score in scores]

# One stage of the Milgram loop  [app. F.1] [fig. 17]
def run_stage(record, subject, stage, lm):
    # Step 2: append the stage text (voltage and learner response)  [fig. 19]
    record += stage_text(subject, stage)
    prods_given = 0
    while True:
        # Step 3: the subject's name starts the next line  [app. F.1]
        prompt = record + "\n" + subject
        # Step 4: free-form completion, keep only its first sentence  [app. F.1]
        completion = lm.complete(prompt)
        sentence = first_sentence(completion)
        # Step 5: stop when the termination judge gives p > 0.5  [app. F.1]
        if lm.judge_stop(record, sentence) > 0.5:
            return record + sentence, "terminated"
        # Step 6: obedience (the subject shocked) ends the stage  [app. F.1]
        if lm.judge_not_shock(record, sentence) <= 0.5:
            return record + sentence, "obeyed"
        # Disobedience after all four prods ends the run  [fig. 21] [fig. 23]
        if prods_given == 4:
            return record + sentence, "terminated"
        # Otherwise append the next prod and re-attempt the stage  [fig. 21]
        prods_given += 1
        record += sentence + "\n" + prod_text(prods_given)

Hyperparameters

Results

Author-reported. Setup: OpenAI text models through the API, the datasets above and human reference data from the original studies §3 §4 §5 §6 §7.

WoC validity for every model: LM-1 51.0, LM-2 94.4, LM-3 88.0, LM-4 98.0, LM-5 99.0 tab. 1.

Valid share of completions, %; standard error below 0.05% tab. 1

Wisdom of crowds, median (IQR) of estimates, LM-1 to LM-6 tab. 3

The IQR is the difference between the 75th and 25th percentiles. The paper writes the speed of light as "2…" in its table; the exact constant is in Table 4 below tab. 3 tab. 4. Human values are blank where the human study did not include the question tab. 3.

Wisdom of crowds, median (IQR) for davinci, gpt-35-turbo and gpt-4 tab. 4

For the speed of light, most gpt-4 answers are 3 × 10^8 rather than the exact constant, per the paper's note under Table 4 tab. 4. Garden-path ratings, Milgram stage texts and the UG per-offer acceptance rates for all five models are in fig. 9 to fig. 13 and fig. 15 to fig. 23.

Paper vs code

The official repository is microsoft/turing-experiments at commit f00115e793f5f728eccf13044bb299d64901de57, MIT licensed (copyright Microsoft Corporation). The repository is archived on GitHub. The paper names a different repository (GatiAher/...), which holds only a README that links to the Microsoft one. Code that runs experiments lives in scripts/ and src/. Data and prompt templates live in data/. The paper's wording differs from the code in these places. The Garden-path and WoC notebooks were checked only for the points below.

• Judge probability is not normalized (Milgram). The paper defines the k-choice probability with normalization over valid completions §2. The Milgram judges take exp of the summed log-probabilities of the last two tokens, which is the unnormalized probability of "not shock" or "not stop", and compare it with 0.5 simulate_milgram_experiment.py simulate_milgram_experiment.py. The UG notebook divides by the valid mass before analysis Simulate_Ultimatum_Game_Experiment.ipynb. The Milgram code does not, so the 0.5 rule on raw probabilities is not the normalized rule in §2.
• Level 27 (315 V) skips "please continue". The first disobedience at this level gets the Prod 1* text and no prod. The next disobedience gets Prod 2, so Prod 1 is never used at this level simulate_milgram_experiment.py fig. 21. The paper says prods are "always made in sequence" app. F.1, so this is a real choice the paper does not describe.
• Generation settings. Milgram subject turns use text-davinci-002, max_tokens=40, temperature=1, and echo=False simulate_milgram_experiment.py. The paper gives temperature 1 and top-p 1 §3 but no token limit.
• Judge settings. Milgram judges use temperature=0, max_tokens=0 and echo=True simulate_milgram_experiment.py simulate_milgram_experiment.py.
• Level table. The code has 36 rows that map to 30 voltages. A correct learner answer keeps the voltage for the next row, and the loop runs up to row 36 DO_level_information_remote-learner.txt simulate_milgram_experiment.py. The paper describes 30 shock levels app. F.
• First-sentence extraction. The code uses regex (.*?(?<!Mr|Ms)[\.\!\?\;]"?) and replaces a semicolon with a period simulate_milgram_experiment.py. The paper says only "extracted using regex". By reading, a completion with no ., !, ? or ; makes re.search return None and the next line fails. We did not run this.
• Pronouns. Pronouns come from the title only: "Mr." gives he, anything else gives she simulate_milgram_experiment.py. "Mx." is not handled in the Milgram code. The paper uses Mx. only in the WoC TE.
• Wisdom of crowds parsing. The integer extractor rejects completions with more than one number, drops digits after a decimal point (so "1.23" becomes 1), and rejects absolute values of 10^100 or more wisdom_of_crowds.ipynb. The paper says only that commas and spaces are ignored app. E. The default max tokens is 100 wisdom_of_crowds.ipynb, and the human WoC call uses stop token ] wisdom_of_crowds.ipynb.
• Notebook labels. The UG notebook calls text-davinci-001 "LM-3" in one note Simulate_Ultimatum_Game_Experiment.ipynb, but its mapping and heading use "LM-4" Simulate_Ultimatum_Game_Experiment.ipynb Simulate_Ultimatum_Game_Experiment.ipynb. The paper's table uses LM-4 for text-davinci-001 §3 tab. 1. Check the mapping before using labels from the notebook.
• Release state of the Milgram script. GLOBAL_VAR_MOCKED = True makes the script skip LM calls and return mock values simulate_milgram_experiment.py. run_full_experiment also breaks after the first subject simulate_milgram_experiment.py. Both look like test settings. Reproducing the Milgram results needs them changed, which the code does not document.
• Engines. src/openai_handler.py lists five engines, from text-ada-001 to text-davinci-002 openai_handler.py. The paper uses more models, including gpt-35-turbo and gpt-4, through a separate chat path in the WoC notebook wisdom_of_crowds.ipynb.
• Not checked. simulate_milgram_experiment_alternate.py (the novel scenario), the analysis notebooks and the garden-path notebook beyond the normalization cell were not read in full.

Pitfalls

• Validity is the first thing to check. The paper says to tune the prompt until the validity rate is "sufficiently close to 1" before running the full experiment §2. It gives no threshold. Table 1 shows the range: 51.0% for LM-1 on WoC up to 99.5% for LM-5 on UG tab. 1.
• Normalize before comparing with 0.5 (judges). A raw probability of "not shock" can be below 0.5 while the normalized probability is above it, when the other completion has mass too. Use the normalized value, as the paper defines §2, or measure the difference in the code as described in Paper vs code.
• Prompt length. The paper notes that most available LMs cannot handle the Milgram prompt length, which grows with each stage §3. The record is carried forward across all stages app. F.1, so cost and context grow with the level.
• Calls per attempt. Each attempt uses three calls in the code: one generation and two judges simulate_milgram_experiment.py simulate_milgram_experiment.py simulate_milgram_experiment.py. A stage can repeat up to four prods, so cost depends on the subject's path.
• Sampling and repeats are not stated. The paper does not say how many completions were drawn per prompt or whether each name was run once. Treat the reported percentages as one draw per prompt unless the authors say otherwise §4 §6.
• Contamination is a known risk. The authors say models were probably trained on descriptions of these experiments, so they added novel variants for three of four TEs §3 §8 app. F.2. The novel Milgram scenario still follows the same algorithm and shares the Milgram structure app. F.2.
• Hyper-accuracy has an untested cause. The authors suspect alignment training. They test this on other GPT versions, but say the details of those models are not released app. E fig. 15.
• Authors' own claims. Results are author-reported on an API with models that the paper dates to 2022 and 2023. The p-value for the UG gender effect is given as below 1e-16 §4. Reproducing the numbers needs the same engines and prompts, and the paper cannot guarantee the same outputs.
• Garden-path results are qualitative in the text. Only a few exact numbers appear for the garden-path TE. Use the figures for the per-sentence picture fig. 10 fig. 13.
• Milgram data are one-sided. The paper compares 100 simulated subjects with 40 human subjects from Experiment 1 fig. 7 fig. 22. The experiment design differs from the original in several ways the paper lists app. F.1.

Lineage

• Builds on: Turing's Imitation Game as a test of one simulated individual §1; the human studies it replicates are from the original papers cited in §4 §5 §6 §7.
• Compared against: concurrent work on simulating survey respondents from demographic prompts §1. That work is not on the wiki.
• Superseded by: not stated.