User simulation

Overview

A user simulator plays the user, so a system can be tested or trained without real people in the loop. The hard part is knowing whether the simulated user behaves like a real one, and the answer depends on what the simulator is for. The two cards on this page use simulators for different purposes.

The tutoring study measures realism. Its simulator sees the real dialogue history, writes the next student turn, and that turn is compared with the real one. Because each turn is conditioned on real history, errors do not build up across a dialogue 2601.04025 §2.1. Realism over a whole dialogue is left to future work 2601.04025 §5.

The mobile-agent benchmark uses a simulator as the user side of an evaluation. The simulator holds a hidden profile and answers the agent's clarifying questions in character. The score is about the agent, not about how realistic the simulator is 2604.08455 §3.2 2604.08455 §3.3. Its profiles are synthetic, and real-user behaviour is not measured [arxiv:2604.08455 app. D.1].

Both settings share a core: a model receives context and a question, and returns one simulated user reply. They differ in what the simulator is given (only the history, or a hidden profile) and in what is measured (agreement with real turns, or an agent's score). The tutoring results come from one setting: a middle-school student answers a multiple-choice maths question with a human tutor on the Eedi dataset 2601.04025 §3.1. The benchmark covers Android phone tasks 2604.08455 §3.1.

Variants

Prompted LLM simulator

Instruct a general-purpose model to play a student. No training is needed. The paper tests zero-shot prompting, an OCEAN persona, in-context learning with retrieved summaries, an Oracle prompt that receives a dialogue summary, and a reasoning model 2601.04025 §3.2.2 2601.04025 tab. 1. Card: Simulated Students in Tutoring Dialogues: Substance or Illusion?.

Author-reported test results: zero-shot GPT-4.1 reaches 0.4998 on acts and 0.5460 on cosine similarity, and the reasoning baseline with GPT-5 mini reaches 0.5755 on acts 2601.04025 tab. 1. Prompted students write much longer turns than real ones: 10.89 words on average for in-context learning, against 4.11 for real students 2601.04025 §4.3.

Two prompted baselines see the real dialogue. The OCEAN persona is estimated from the full ground-truth dialogue, and the Oracle receives a summary that includes the real student's errors 2601.04025 §3.2.2 [arxiv:2601.04025 §4.1]. The Oracle scores 0.6755 on correctness and 0.1872 on errors 2601.04025 tab. 1, but these rows leak information. Do not compare them with methods that see only the history 2601.04025 §4.1.

Supervised fine-tuned simulator

Train a base instruct model on real student turns with the usual next-token loss, using LoRA adapters 2601.04025 §2.1 2601.04025 app. B.1. The paper uses Llama 3.1 8B and Llama 3.2 3B. The appendix gives LoRA rank 32, alpha 64 and dropout 0.05, 3 epochs, a learning rate of 5e-5 with 10% warm-up, an effective batch of 64 and weight decay 0.01 2601.04025 app. B.1 2601.04025 tab. 5.

Results on the 8B model: acts 0.6671, cosine similarity 0.7383 and ROUGE-L 0.3212, all above every prompted row for those metrics 2601.04025 tab. 1. On correctness it scores 0.5670, below every prompted row [arxiv:2601.04025 tab. 1]. Training took 19 minutes on one GPU, an L40S or an A40 [arxiv:2601.04025 tab. 5] 2601.04025 app. B.1.

Preference-tuned simulator (SFT, then DPO)

After SFT, sample four candidate student turns for each turn, score them, and train with DPO on pairs whose reward gap is above 0.1 2601.04025 §2.3 2601.04025 app. B.1. The reward is the mean of the metric scores that apply to a candidate 2601.04025 §2.3. The appendix gives DPO beta 0.1, a learning rate of 5e-6, and one epoch on a random 20% of the training dialogues. The first five turns of each dialogue are skipped [arxiv:2601.04025 app. B.1] 2601.04025 §2.3.

Results on the 8B model: acts 0.6840, cosine similarity 0.7390 and errors 0.0529, against 0.6671, 0.7383 and 0.0661 for SFT 2601.04025 tab. 1. The gain is small, and training took 81 minutes 2601.04025 tab. 5. In the 3B reward ablation, the averaged reward gives the highest acts score (0.6962). A correctness-only reward gives the best correctness (0.5852) but lower acts than SFT (0.6692 against 0.6795) 2601.04025 tab. 4. An errors-only reward moves errors only from 0.0506 to 0.0562 2601.04025 §4.4 2601.04025 tab. 4.

Human evaluation covers a subset of 190 turns from 38 dialogues, scored by three evaluators. There, DPO 8B reaches 0.7905 on acts against 0.6143 for zero-shot 2601.04025 tab. 2.

Hidden-profile simulator in an agent benchmark

The simulator answers only when the agent calls an ask-user action. Its replies come from a structured profile that the agent never sees: identity, locations, habits, preferences, decision criteria and similar fields [arxiv:2604.08455 §3.2] 2604.08455 tab. 6. The agent sees only a timestamped log of past phone actions, and must infer preferences from that log or ask 2604.08455 §3.2 2604.08455 tab. 7. The prompt is built from the profile, the current state and the dialogue so far, and the paper uses one of four roles: Developer, Grandma, Student or Researcher 2604.08455 §3.2 2604.08455 app. D.1. The paper uses gpt-4o as the simulator [arxiv:2604.08455 §4.1]. Card: KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation.

The simulator is an evaluation component, so its outputs feed the agent's score rather than a training loss. Author-reported results: Claude Sonnet 4.6 has the best overall success of the 11 models, 60.4% 2604.08455 tab. 2. On easy general tasks it reaches 94.4%, and on easy personalized tasks 44.2% 2604.08455 tab. 2. The authors attribute most of its personalized failures to insufficient clarification, not to screen navigation 2604.08455 §4.4.

When to use which

Further points for the choice:

• Judges are proprietary. Correctness and errors are judged by GPT-5 mini with the real turn as reference, and real-turn act labels come from GPT-4.1 2601.04025 §2.2 2601.04025 app. B.1. Re-running needs the same models or a new check against human labels.
• Benchmark scores depend on the judge. The paper does not name the judge model 2604.08455 §3.4. The repository uses the simulator's model setting for it, and gives half credit when a judge call fails base_prefrence_task.py base_prefrence_task.py. The paper's only human check, in its figure of judge agreement, covers the judge; the card records no check of the simulator's replies 2604.08455 §4.3.
• Step budget changes results. The paper gives no step limit 2604.08455 §3.1. The repository runs without a limit unless one is passed eval.py.
• Acts labels are a noisy target. Two human annotators agree on acts with a kappa of 0.4978, while a human and the metric agree at 0.7337 [arxiv:2601.04025 tab. 3]. Read acts gains with that spread in mind.
• The errors metric is sparse. It applies only to incorrect real turns, and every non-Oracle method scores below 0.09 2601.04025 tab. 1.
• Greedy decoding gives very short turns. The authors report that random sampling lowered performance in preliminary tests 2601.04025 app. D.1.
• Reward hacking. Training on LLM-judged metrics can overfit the judge. The authors recommend expert human evaluation whenever such metrics train a model 2601.04025 §2.3.

Implementation sketch

The tutoring variants share a generation step and a scoring step. The pseudocode below follows the paper's description. Their variants differ in the model and in the reward, not in this loop. The benchmark variant reuses the reply step, but its scoring is per task.

# Simulate one student turn from the real history  [§2.1]
# Inputs: history (turns before i), question q, prompt P (may hold a persona)
# Output: one simulated student utterance

def simulate_turn(model, history, question, prompt):
    # Build the context from the prompt, the question and the real history  [§2.1]
    context = build_context(prompt, question, history)
    # Greedy decoding for non-reasoning models in the paper's tests  [app. B.1]
    return model.generate(context, decoding="greedy", max_new_tokens=400)

# Answer one ask_user question from a hidden profile  [§3.2] [app. E.2]
# Inputs: profile P (hidden from the agent), state S, dialogue so far, question q
# Output: one free-text reply in the simulated user's voice

def answer_question(model, profile, state, dialogue, question):
    # The profile and state go into a role-grounded prompt  [§3.2]
    prompt = build_role_prompt(profile, state, dialogue)
    # The agent receives only this reply, never the profile  [§3.2]
    # Temperature is not stated in the paper; the repository uses 0.0
    # [code: src/knowu_bench/tasks/definitions/preference/base_prefrence_task.py#L257]
    return model.generate(prompt + question, temperature=0.0)

# Score one simulated turn against the real turn  [§2.2] [fig. 1]
# Inputs: history, real_turn, sim_turn
# Output: one score per metric; None where a metric does not apply

def score_turn(history, real_turn, sim_turn):
    # Acts: real turns labelled by an LLM, simulated turns by a classifier  [§2.2]
    acts = float(act_label_llm(real_turn) == act_classifier(sim_turn))
    # Correctness and errors: one LLM judge call with the real turn as reference  [§2.2]
    corr, err = llm_judge(history, real_turn, sim_turn)
    # Linguistic: embedding cosine and ROUGE-L  [§2.2]
    cos = cosine(embed(real_turn), embed(sim_turn))
    rouge = rouge_l(real_turn, sim_turn)
    # Knowledge and tutor-response metrics are left out of this sketch  [§2.2]
    return {"acts": acts, "corr": corr, "err": err, "cos": cos, "rouge": rouge}

# Build DPO pairs from the sampled candidates of one turn  [§2.3] [app. B.1]
# Inputs: prompt, candidates (n = 4), rewards (one mean score per candidate)
# Output: list of (prompt, chosen, rejected)

def build_pairs(prompt, candidates, rewards, eps=0.1):
    pairs = []
    for i in range(len(candidates)):
        for j in range(len(candidates)):
            # Keep a pair only when the reward gap is above the threshold  [§2.3]
            if rewards[i] - rewards[j] > eps:
                pairs.append((prompt, candidates[i], candidates[j]))
    return pairs

Contradictions

The two cards measure different things, so their numbers are not comparable. The tutoring study scores simulator turns against real ones 2601.04025 §2.2. The benchmark uses its simulator to score agents 2604.08455 §3.3.

Within the tutoring paper, the text and its released code disagree on the DPO setup. The text says the reward averages all seven metrics, ROUGE-L included 2601.04025 §2.3 2601.04025 §3.3. The code's default reward has six metrics and no ROUGE-L dpo.py. Table 4 uses the same six 2601.04025 tab. 4. The text also says all candidate pairs are formed 2601.04025 §2.3, while the code keeps the six largest gaps per turn dpo.py. Implement the setup you intend, and say which one you chose.

Within the benchmark paper, the text and its released code also disagree. The text gives one weighted sum of rule and judge scores, with a per-task weight 2604.08455 §3.4. The repository hard-codes a different combination in each preference task order_lunch_tradeoff.py. The text names gpt-4o as the simulator 2604.08455 §4.1, while the shipped example environment sets gpt-4.1 .env.example. Neither disagreement is resolved here; the benchmark's numbers should be read with both in mind.

Papers

• Simulated Students in Tutoring Dialogues: Substance or Illusion?: turn-level student simulation for tutoring, with seven reference-based metrics and a comparison of prompting, SFT and DPO on the Eedi dialogues.
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation: an online Android benchmark where a hidden-profile LLM user answers an agent's clarifying questions, scored by rule checks and an LLM judge.