---
kind: concept
slug: user-simulation
title: User simulation
summary: >-
  Simulated users stand in for real ones when testing or training dialogue and agent systems. A 2026 tutoring study finds that fine-tuned student simulators match real turns better than prompted ones, and a preference step adds a little more. A 2026 mobile-agent benchmark uses a hidden-profile LLM user to answer clarifying questions. Oracle-style prompts leak real answers.
tags: [user-simulation, dialogue-systems, llm-simulation, evaluation, education, agents, benchmark]
papers: ["2601.04025", "2604.08455"]
related: [llm-simulated-participants, dialogue-evaluation, dialogue-generation, llm-agent, llm-as-a-judge, preference-optimization, low-rank-adaptation, in-context-learning, benchmark-validity]
status: draft
updated_at: 2026-11-01
---

## Overview

A user simulator plays the user, so a system can be tested or trained without
real people in the loop. The hard part is knowing whether the simulated user
behaves like a real one, and the answer depends on what the simulator is for.
The two cards on this page use simulators for different purposes.

The tutoring study measures realism. Its simulator sees the real dialogue
history, writes the next student turn, and that turn is compared with the real
one. Because each turn is conditioned on real history, errors do not build up
across a dialogue [arxiv:2601.04025 §2.1]. Realism over a whole dialogue is left
to future work [arxiv:2601.04025 §5].

The mobile-agent benchmark uses a simulator as the user side of an evaluation.
The simulator holds a hidden profile and answers the agent's clarifying
questions in character. The score is about the agent, not about how realistic
the simulator is [arxiv:2604.08455 §3.2] [arxiv:2604.08455 §3.3]. Its profiles
are synthetic, and real-user behaviour is not measured [arxiv:2604.08455 app.
D.1].

Both settings share a core: a model receives context and a question, and returns
one simulated user reply. They differ in what the simulator is given (only the
history, or a hidden profile) and in what is measured (agreement with real turns,
or an agent's score). The tutoring results come from one setting: a
middle-school student answers a multiple-choice maths question with a human
tutor on the Eedi dataset [arxiv:2601.04025 §3.1]. The benchmark covers Android
phone tasks [arxiv:2604.08455 §3.1].

## Variants

### Prompted LLM simulator

Instruct a general-purpose model to play a student. No training is needed. The
paper tests zero-shot prompting, an OCEAN persona, in-context learning with
retrieved summaries, an Oracle prompt that receives a dialogue summary, and a
reasoning model [arxiv:2601.04025 §3.2.2] [arxiv:2601.04025 tab. 1]. Card:
[Simulated Students in Tutoring Dialogues: Substance or Illusion?](/papers/2601.04025).

Author-reported test results: zero-shot GPT-4.1 reaches 0.4998 on acts and
0.5460 on cosine similarity, and the reasoning baseline with GPT-5 mini reaches
0.5755 on acts [arxiv:2601.04025 tab. 1]. Prompted students write much longer
turns than real ones: 10.89 words on average for in-context learning, against
4.11 for real students [arxiv:2601.04025 §4.3].

Two prompted baselines see the real dialogue. The OCEAN persona is estimated
from the full ground-truth dialogue, and the Oracle receives a summary that
includes the real student's errors [arxiv:2601.04025 §3.2.2] [arxiv:2601.04025
§4.1]. The Oracle scores 0.6755 on correctness and 0.1872 on errors
[arxiv:2601.04025 tab. 1], but these rows leak information. Do not compare them
with methods that see only the history [arxiv:2601.04025 §4.1].

### Supervised fine-tuned simulator

Train a base instruct model on real student turns with the usual next-token
loss, using LoRA adapters [arxiv:2601.04025 §2.1] [arxiv:2601.04025 app. B.1].
The paper uses Llama 3.1 8B and Llama 3.2 3B. The appendix gives LoRA rank 32,
alpha 64 and dropout 0.05, 3 epochs, a learning rate of 5e-5 with 10% warm-up,
an effective batch of 64 and weight decay 0.01 [arxiv:2601.04025 app. B.1]
[arxiv:2601.04025 tab. 5].

Results on the 8B model: acts 0.6671, cosine similarity 0.7383 and ROUGE-L
0.3212, all above every prompted row for those metrics [arxiv:2601.04025 tab. 1].
On correctness it scores 0.5670, below every prompted row [arxiv:2601.04025
tab. 1]. Training took 19 minutes on one GPU, an L40S or an A40 [arxiv:2601.04025
tab. 5] [arxiv:2601.04025 app. B.1].

### Preference-tuned simulator (SFT, then DPO)

After SFT, sample four candidate student turns for each turn, score them, and
train with DPO on pairs whose reward gap is above 0.1 [arxiv:2601.04025 §2.3]
[arxiv:2601.04025 app. B.1]. The reward is the mean of the metric scores that
apply to a candidate [arxiv:2601.04025 §2.3]. The appendix gives DPO beta 0.1,
a learning rate of 5e-6, and one epoch on a random 20% of the training
dialogues. The first five turns of each dialogue are skipped [arxiv:2601.04025
app. B.1] [arxiv:2601.04025 §2.3].

Results on the 8B model: acts 0.6840, cosine similarity 0.7390 and errors 0.0529,
against 0.6671, 0.7383 and 0.0661 for SFT [arxiv:2601.04025 tab. 1]. The gain
is small, and training took 81 minutes [arxiv:2601.04025 tab. 5]. In the 3B
reward ablation, the averaged reward gives the highest acts score (0.6962). A
correctness-only reward gives the best correctness (0.5852) but lower acts than
SFT (0.6692 against 0.6795) [arxiv:2601.04025 tab. 4]. An errors-only reward
moves errors only from 0.0506 to 0.0562 [arxiv:2601.04025 §4.4]
[arxiv:2601.04025 tab. 4].

Human evaluation covers a subset of 190 turns from 38 dialogues, scored by three
evaluators. There, DPO 8B reaches 0.7905 on acts against 0.6143 for zero-shot
[arxiv:2601.04025 tab. 2].

### Hidden-profile simulator in an agent benchmark

The simulator answers only when the agent calls an ask-user action. Its replies
come from a structured profile that the agent never sees: identity, locations,
habits, preferences, decision criteria and similar fields [arxiv:2604.08455
§3.2] [arxiv:2604.08455 tab. 6]. The agent sees only a timestamped log of past
phone actions, and must infer preferences from that log or ask
[arxiv:2604.08455 §3.2] [arxiv:2604.08455 tab. 7]. The prompt is built from the
profile, the current state and the dialogue so far, and the paper uses one of
four roles: Developer, Grandma, Student or Researcher [arxiv:2604.08455 §3.2]
[arxiv:2604.08455 app. D.1]. The paper uses gpt-4o as the simulator [arxiv:2604.08455
§4.1]. Card: [KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation](/papers/2604.08455).

The simulator is an evaluation component, so its outputs feed the agent's score
rather than a training loss. Author-reported results: Claude Sonnet 4.6 has the
best overall success of the 11 models, 60.4% [arxiv:2604.08455 tab. 2]. On easy
general tasks it reaches 94.4%, and on easy personalized tasks 44.2%
[arxiv:2604.08455 tab. 2]. The authors attribute most of its personalized
failures to insufficient clarification, not to screen navigation
[arxiv:2604.08455 §4.4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Realism of a simulator, with real turns to compare against | Prompted, SFT or DPO, scored turn by turn | The tutoring metrics compare each simulated turn with a real one [arxiv:2601.04025 §2.2]. |
| Real logs, no training pipeline | Prompted, history only (zero-shot or in-context) | No training needed. Zero-shot acts 0.4998 against 0.6671 for SFT 8B [arxiv:2601.04025 tab. 1]. Prompted students over-produce Seek Information [arxiv:2601.04025 §4.3]. |
| Real logs and one GPU for a few hours | SFT with LoRA on an 8B open model | Higher acts, cosine and ROUGE-L than every prompted row, at 19 minutes of training [arxiv:2601.04025 tab. 1] [arxiv:2601.04025 tab. 5]. Correctness is lower than the prompted rows [arxiv:2601.04025 tab. 1]. |
| Real logs and time for a preference step | SFT, then DPO on the averaged reward | Small extra gain: acts 0.6671 to 0.6840 at 8B [arxiv:2601.04025 tab. 1]. Needs judge calls for the rewards and 81 minutes of training [arxiv:2601.04025 tab. 5]. |
| Testing an agent that must ask clarifying questions, with no real users | Hidden-profile simulator with a role prompt | The agent's score depends on what it asks and what it does with the answer, and the profile makes the answers checkable [arxiv:2604.08455 §3.2] [arxiv:2604.08455 §3.4]. Profiles are synthetic [arxiv:2604.08455 app. D.1]. |
| Comparing methods or reporting realism | History-only methods | Oracle and OCEAN use information from the real dialogue [arxiv:2601.04025 §3.2.2]. |

Further points for the choice:

- **Judges are proprietary.** Correctness and errors are judged by GPT-5 mini
  with the real turn as reference, and real-turn act labels come from GPT-4.1
  [arxiv:2601.04025 §2.2] [arxiv:2601.04025 app. B.1]. Re-running needs the same
  models or a new check against human labels.
- **Benchmark scores depend on the judge.** The paper does not name the judge
  model [arxiv:2604.08455 §3.4]. The repository uses the simulator's model
  setting for it, and gives half credit when a judge call fails
  [code: src/knowu_bench/tasks/definitions/preference/base_prefrence_task.py#L241]
  [code: src/knowu_bench/tasks/definitions/preference/base_prefrence_task.py#L291].
  The paper's only human check, in its figure of judge agreement, covers the
  judge; the card records no check of the simulator's replies
  [arxiv:2604.08455 §4.3].
- **Step budget changes results.** The paper gives no step limit
  [arxiv:2604.08455 §3.1]. The repository runs without a limit unless one is
  passed [code: src/knowu_bench/core/subcommands/eval.py#L266].
- **Acts labels are a noisy target.** Two human annotators agree on acts with a
  kappa of 0.4978, while a human and the metric agree at 0.7337 [arxiv:2601.04025
  tab. 3]. Read acts gains with that spread in mind.
- **The errors metric is sparse.** It applies only to incorrect real turns, and
  every non-Oracle method scores below 0.09 [arxiv:2601.04025 tab. 1].
- **Greedy decoding gives very short turns.** The authors report that random
  sampling lowered performance in preliminary tests [arxiv:2601.04025 app. D.1].
- **Reward hacking.** Training on LLM-judged metrics can overfit the judge. The
  authors recommend expert human evaluation whenever such metrics train a model
  [arxiv:2601.04025 §2.3].

## Implementation sketch

The tutoring variants share a generation step and a scoring step. The
pseudocode below follows the paper's description. Their variants differ in the
model and in the reward, not in this loop. The benchmark variant reuses the
reply step, but its scoring is per task.

```pseudocode
# Simulate one student turn from the real history  [§2.1]
# Inputs: history (turns before i), question q, prompt P (may hold a persona)
# Output: one simulated student utterance

def simulate_turn(model, history, question, prompt):
    # Build the context from the prompt, the question and the real history  [§2.1]
    context = build_context(prompt, question, history)
    # Greedy decoding for non-reasoning models in the paper's tests  [app. B.1]
    return model.generate(context, decoding="greedy", max_new_tokens=400)
```

```pseudocode
# Answer one ask_user question from a hidden profile  [§3.2] [app. E.2]
# Inputs: profile P (hidden from the agent), state S, dialogue so far, question q
# Output: one free-text reply in the simulated user's voice

def answer_question(model, profile, state, dialogue, question):
    # The profile and state go into a role-grounded prompt  [§3.2]
    prompt = build_role_prompt(profile, state, dialogue)
    # The agent receives only this reply, never the profile  [§3.2]
    # Temperature is not stated in the paper; the repository uses 0.0
    # [code: src/knowu_bench/tasks/definitions/preference/base_prefrence_task.py#L257]
    return model.generate(prompt + question, temperature=0.0)
```

```pseudocode
# Score one simulated turn against the real turn  [§2.2] [fig. 1]
# Inputs: history, real_turn, sim_turn
# Output: one score per metric; None where a metric does not apply

def score_turn(history, real_turn, sim_turn):
    # Acts: real turns labelled by an LLM, simulated turns by a classifier  [§2.2]
    acts = float(act_label_llm(real_turn) == act_classifier(sim_turn))
    # Correctness and errors: one LLM judge call with the real turn as reference  [§2.2]
    corr, err = llm_judge(history, real_turn, sim_turn)
    # Linguistic: embedding cosine and ROUGE-L  [§2.2]
    cos = cosine(embed(real_turn), embed(sim_turn))
    rouge = rouge_l(real_turn, sim_turn)
    # Knowledge and tutor-response metrics are left out of this sketch  [§2.2]
    return {"acts": acts, "corr": corr, "err": err, "cos": cos, "rouge": rouge}
```

```pseudocode
# Build DPO pairs from the sampled candidates of one turn  [§2.3] [app. B.1]
# Inputs: prompt, candidates (n = 4), rewards (one mean score per candidate)
# Output: list of (prompt, chosen, rejected)

def build_pairs(prompt, candidates, rewards, eps=0.1):
    pairs = []
    for i in range(len(candidates)):
        for j in range(len(candidates)):
            # Keep a pair only when the reward gap is above the threshold  [§2.3]
            if rewards[i] - rewards[j] > eps:
                pairs.append((prompt, candidates[i], candidates[j]))
    return pairs
```

## Contradictions

The two cards measure different things, so their numbers are not comparable.
The tutoring study scores simulator turns against real ones
[arxiv:2601.04025 §2.2]. The benchmark uses its simulator to score agents
[arxiv:2604.08455 §3.3].

Within the tutoring paper, the text and its released code disagree on the DPO
setup. The text says the reward averages all seven metrics, ROUGE-L included
[arxiv:2601.04025 §2.3] [arxiv:2601.04025 §3.3]. The code's default reward has
six metrics and no ROUGE-L [code: sim_student/dpo.py#L359]. Table 4 uses the
same six [arxiv:2601.04025 tab. 4]. The text also says all candidate pairs are
formed [arxiv:2601.04025 §2.3], while the code keeps the six largest gaps per
turn [code: sim_student/dpo.py#L386]. Implement the setup you intend, and say
which one you chose.

Within the benchmark paper, the text and its released code also disagree. The
text gives one weighted sum of rule and judge scores, with a per-task weight
[arxiv:2604.08455 §3.4]. The repository hard-codes a different combination in
each preference task [code: src/knowu_bench/tasks/definitions/preference/order_lunch_tradeoff.py#L83-L99].
The text names gpt-4o as the simulator [arxiv:2604.08455 §4.1], while the
shipped example environment sets gpt-4.1 [code: .env.example#L5]. Neither
disagreement is resolved here; the benchmark's numbers should be read with both
in mind.

## Papers

- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](/papers/2601.04025):
  turn-level student simulation for tutoring, with seven reference-based metrics
  and a comparison of prompting, SFT and DPO on the Eedi dialogues.
- [KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation](/papers/2604.08455):
  an online Android benchmark where a hidden-profile LLM user answers an agent's
  clarifying questions, scored by rule checks and an LLM judge.