Simulated Students in Tutoring Dialogues: Substance or Illusion?

In plain words

Builders of tutoring software want to test it on pretend students before putting it in front of real ones. The usual shortcut is to tell a language model to "act like a student". This paper checks how well that works. For each turn of a real tutoring chat, the model sees the same history and writes the next student message. The message is then compared with the one the real student sent, on several points: what kind of move the student makes (answering, asking, saying they are stuck, chatting), whether the answer is right, what mistake it contains, whether it shows the student learning something, how the wording compares, and whether the tutor would reply the way it actually did.

The main finding is that prompting alone does poorly. Training a model on real student turns does much better, and a further step that rewards the better-scoring replies helps a little more. None of the methods gets close to real students. The data are online chats between middle school students and human tutors about math problems, so the results are for that setting.

TL;DR

The paper defines turn-level student simulation for tutoring dialogues and proposes seven reference-based metrics that compare a simulated student turn with the real one. It benchmarks prompting (zero-shot, persona, Oracle summary, in-context, reasoning), supervised fine-tuning, and DPO on the Eedi dialogue set. Fine-tuned models match real students more closely than prompted ones, though the authors report that all methods remain weak. Every number is author-reported, and the LLM-judged metrics depend on proprietary models.

Problem

Simulated students are used to train and evaluate LLM tutors, often by prompting a model to play a student. The authors state that prompting does not reliably produce realistic student behaviour, and that real students are too costly to use at scale §1. Few metrics exist for judging simulated students turn by turn against a reference; earlier work used population statistics, persona consistency or human ratings §1 app. A.2.

The setting assumes access to a dataset of real tutor-student dialogues, which is used both as training data and as the reference for scoring §3.1. The dataset is Eedi's Question-Anchored Tutoring Dialogues 2k, in which a middle school student solves a multiple-choice math problem with a trained tutor §3.1. The simulator is conditioned on the real dialogue history, so each turn is scored on its own and errors do not compound across turns §2.1. Dialogue-level realism is left to future work §5.

Method

Formulation. A dialogue alternates student turns s_i and tutor turns t_i. For a dataset of real dialogues, a simulator M_S generates a simulated student utterance for each turn, conditioned on the history, the question q and a prompt P that may include a persona §2.1. Three simulators are compared:

1. Prompting a pre-trained LLM, with or without a persona or example §3.2.2.
2. Supervised fine-tuning (SFT) on real student turns, by minimising the negative log-likelihood of the student tokens §2.1 training_utils.py.
3. DPO after SFT, trained on preference pairs whose ranking comes from the metrics below §2.3.

Evaluation metrics. Each simulated turn is compared with the real turn on seven metrics that cover six dimensions §2.2 fig. 1:

• Acts. A five-label dialogue-act set (Math Answer, Not Understanding, Seek Information, Off-Topic, Acknowledge) tab. 6. Real turns are labelled by an LLM (GPT-4.1). A fine-tuned classifier labels simulated turns. The score is 1 when the labels match, else 0 §2.2 eval.py.
• Correctness. Whether the student answers the tutor's question correctly, or is not applicable (na) when no question is asked. The score is 1 when the simulated turn's label matches the real one. GPT-5 mini judges the simulated turn with the real turn as reference, at low reasoning effort §2.2 eval.py eval.py.
• Errors. Only for real turns that are incorrect: 1 if the simulated turn has the same mathematical error, judged in the same GPT-5 mini prompt as correctness §2.2 eval.py.
• Knowledge acquisition. A knowledge-tracing model (LLMKT) estimates mastery of knowledge components (KCs) after each turn. The change in mastery per KC is split into five quantile bins; the score is one minus the mean bin distance divided by four §2.2 app. B.2.
• Linguistic. Cosine similarity between Qwen3-Embedding-8B embeddings of the two turns, and ROUGE-L between them §2.2 eval.py.
• Tutor response. The inverse perplexity of the real next tutor turn, under a fine-tuned tutor model conditioned on the simulated turn §2.2.

Reward for DPO. The reward is the average of the metric scores. Preference pairs are formed only when two candidates differ by more than a threshold epsilon, and the first five turns of each dialogue are skipped §2.3. The steps are in the pseudocode below.

# Reference-based scores for one simulated student turn  [§2.2] [fig. 1]
# Inputs: history (turns before i), real_turn, simulated_turn
# Output: seven scores; None where a metric does not apply

def score_turn(history, real_turn, simulated_turn):
    # Acts: exact match of the act labels  [§2.2]
    act_real = act_label_from_llm(real_turn, history)
    act_sim = act_classifier(simulated_turn, history)
    acts = float(act_real == act_sim)
    # Correctness and errors: one LLM judge call  [§2.2]
    corr, err = llm_judge(history, real_turn, simulated_turn)
    # Knowledge: per-KC mastery change, binned into quantiles  [§2.2]
    knowledge = knowledge_similarity(history, real_turn, simulated_turn)
    # Linguistic: embedding cosine and ROUGE-L  [§2.2]
    cosine = dot(embed(real_turn), embed(simulated_turn))
    rouge = rouge_l(real_turn, simulated_turn)
    # Tutor response: inverse perplexity of the real next tutor turn  [§2.2]
    tutor = inverse_perplexity(next_tutor_turn, simulated_turn, history)
    return {"acts": acts, "corr": corr, "err": err,
            "knowledge": knowledge, "cosine": cosine,
            "rouge": rouge, "tutor": tutor}

# Build DPO preference pairs for one student turn  [§2.3] [app. B.1]
# Inputs: prompt, candidates (n = 4 samples), scores[candidate][metric]
# Output: list of (prompt, chosen, rejected)

def build_pairs(prompt, candidates, scores, epsilon, num_pairs):
    # Average the metrics that apply to each candidate  [§2.3]
    final = [mean_of_defined(scores[c]) for c in candidates]
    # Keep ordered pairs whose averages differ by more than epsilon  [§2.3]
    gaps = []
    for better, worse in all_ordered_pairs(candidates):
        gap = final[better] - final[worse]
        if gap > epsilon:
            gaps.append((gap, better, worse))
    # Keep the num_pairs largest gaps per turn  [code: sim_student/dpo.py#L250]
    gaps = sorted(gaps, reverse=True)[:num_pairs]
    return [(prompt, candidates[b], candidates[w]) for _, b, w in gaps]

Prompts and decoding. Fine-tuned students receive a short system prompt and the dialogue as chat turns prompting.py. The prompting baselines use longer system prompts, which the paper gives in full in its appendix app. G. The OCEAN persona is estimated by an LLM from the full ground-truth dialogue, which leaks information §3.2.2. The Oracle baseline gets an LLM-written summary of the dialogue, which also leaks the answer §3.2.2 §4.1.

Hyperparameters

Values in the paper's appendix are the reference. Code values are added where the repository sets something the paper does not state, and they are marked as code-only.

Results

Author-reported. Setup: Question-Anchored Tutoring Dialogues 2k (Eedi), 382 test dialogues, turn-level prediction of the next student turn with the real history given §3.1 §4.1. Best per column is bold in the paper and not marked here.

Turn-level results on the test set tab. 1:

Human evaluation on a subset tab. 2 §3.4 app. E: 190 turns across 38 dialogues, scored by three evaluators, with 20 turns shared between two of them for agreement. Only three methods were evaluated: DPO (8B), Zero-Shot and Oracle. Evaluators rated linguistic similarity on a 1 to 5 scale app. E, but the table reports values between 0 and 1, and the paper does not say how the scale was converted.

Agreement tab. 3 (Cohen's kappa for Acts, Corr. and Errors; Pearson for Linguistic §4.2):

Reward ablation tab. 4 §4.4: DPO trained on one metric at a time, Llama 3.2 3B, evaluated on a random 20% of the test set. The first row is SFT alone.

Qualitative findings §4.3 fig. 2 fig. 3: fine-tuned models produce much shorter turns than prompted ones (2.28 words for DPO 8B, 10.89 for ICL, 4.11 for real students). Fine-tuned models under-produce the Seek Information act, and prompted models over-produce it. Both families overestimate how often students answer correctly §4.3 app. D.2 fig. 3. Per-turn results are shown by turn pair index, truncated after 15 turn pairs fig. 4 app. D.3.

Training statistics tab. 5 (train and test times in minutes):

The dialogue-act set, with one-line definitions per act tab. 6:

• Math Answer: the tutor asks a maths question and the student attempts it.
• Not Understanding: the student says they do not know or do not understand.
• Seek Information: the student asks a clarifying or conceptual question.
• Off-Topic: greetings, goodbyes or talk unrelated to the problem.
• Acknowledge: the student acknowledges the tutor's previous turn.

Paper vs code

Code: umass-ml4ed/sim-student-eval at 4c94a33e6053fd7acf812ff122deab16dcd3c126, linked from a footnote on the abstract [abstract]. Repository LICENSE: MIT, copyright Alexander Scarlatos. The README cites the paper as an ACL 2026 long paper. The knowledge-tracing model and its training code are in an external repository (dialogue-kt), which this card did not fetch llmkt.py.

Discrepancies, each with both citations:

• Pair formation. The paper forms pairs "between all possible pairs of the n candidate responses" §2.3. The code keeps the 6 largest-gap pairs per turn by default dpo.py dpo.py. The code also has an each_wins strategy that the paper does not describe dpo.py.
• Reward set. The paper says the reward is "the average of all metrics" §2.3, and lists seven metrics including ROUGE-L §3.3. The code's default reward set is cosine, acts, knowledge, correctness, errors and tutor response, with no ROUGE-L dpo.py dpo.py. Table 4 uses the same six rewards tab. 4. The average also skips metrics that do not apply to a candidate dpo.py, so turns where the correctness or errors metric is missing are averaged over fewer metrics.
• Candidates. The paper uses n = 4 candidates per turn app. B.1. The code removes duplicates with list(set(candidates)), so fewer than four remain when samples repeat dpo.py.
• Precision. The paper says models load in fp16 app. B.1. The code loads bf16 unless quantising model.py.
• Knowledge cutoffs. The paper describes five quantile bins "for all" mastery deltas §2.2. The code computes the cutoffs from the real turns only, then applies them to the simulated turns eval.py.
• Knowledge delta reference. The paper writes the delta against z_{i-1} §2.2. The code subtracts the mastery of the turn two indices back, which is the previous student turn eval.py.
• ROUGE-L. The paper calls the metric word-level recall §2.2. The code calls evaluate's rouge module with rougeL and no aggregation eval.py. Whether that returns recall, precision or F-measure is set by the library, which is not in this repository, so the definition was not checked.
• Cosine. The code takes a raw dot product of the sentence-transformer outputs eval.py. That equals cosine similarity only if the embeddings are already normalised, and the code does not normalise them. The model's pipeline is not in the repository, so this was not verified. Without normalisation the value can exceed 1, which would break the [0, 1] assumption behind the average reward.
• Test split. The paper reports test-set results §4.1. The default --test_on value in the test script is val testing.py. Running the script without --test_on test scores the validation set.
• Dataset split. The paper gives 1,147 / 382 for train and validation §3.1. The loader takes int(0.75 x N) of a seeded shuffle data_loading.py, which does not by itself reproduce those counts. The raw counts were not checked.
• Learning-rate schedule. The paper says "linear warmup" for 10% of steps app. B.1. The code sets warm-up only training_utils.py, so the decay after warm-up is the transformers default, which the paper does not name.

Data licence. The repository includes annotated Eedi CSVs under data/annotated/eedi. Its README says the annotations are CC BY-NC-SA 4.0 and the dataset is for non-commercial research [README]. The MIT LICENSE does not cover them. This card uses neither the data nor the annotations.

Pitfalls

• The LLM-judged metrics (correctness, errors and the real-turn labels) depend on GPT-5 mini and GPT-4.1, both proprietary. The authors say smaller open models were "significantly less reliable" in preliminary tests [Limitations]. Re-running them needs the same API models or a new validation against human labels.
• The Errors metric is very sparse: it applies only to incorrect real turns, and all non-Oracle methods score below 0.09 tab. 1. The authors attribute the weak Errors result to a sparse reward signal. Training on the Errors reward alone reached 0.0562, against 0.0506 for SFT §4.4 tab. 4.
• Oracle and OCEAN leak real information: Oracle summarises the dialogue including the real student's errors, which raises correctness and errors scores §4.1. Do not compare those rows with methods that see only the history.
• Greedy decoding makes fine-tuned students very brief. The authors hypothesise that the most probable reply is short, and report that random sampling reduced performance in preliminary tests app. D.1.
• Reward hacking: the authors warn that training on LLM-judged metrics can overfit the judge. They recommend expert human evaluation whenever metrics are used in training §2.3.
• The DPO average depends on which metrics apply to each turn (see Paper vs code), so the effective weights vary per turn.
• Inferred from code: candidate order comes from set(), whose iteration order can change between Python processes. Ties in the pair ranking may therefore resolve differently across runs dpo.py.
• The knowledge-tracing model is weak in absolute terms (validation AUC 0.6557), and the authors say its predictions are only a complement to the correctness metric app. B.2 tab. 5.
• Human evaluation covers three methods on 38 dialogues, with three evaluators, and the conversion of the Likert scale is not given tab. 2 app. E.
• Prompts for the prompting baselines are long and given only in the appendix app. G. Implement them from the appendix text, not from memory.
• DPO candidates are sampled from the SFT model. The paper does not state the sampling settings for them; the code uses temperature 1.0 and top-p 0.95 dpo.py dpo.py.

Lineage

• Builds on: Direct preference optimization for the preference loss, with a score-gap threshold taken from SMART (bib key smart) §2.3. LoRA with rank stabilisation, cited as hu2022lora and kalajdzievski2023rank app. B.1. Knowledge tracing in tutoring dialogues follows dialogue-kt (no arXiv id in the paper) §2.2.
• Compared against: prompting baselines from ETH tutor-RL work (bib key eth-tutor-rl), OCEAN persona prompting (bib key 2024.EDM-long-papers.6) and in-context learning with retrieved summaries (bib key lee-etal-2024-effective) §3.2.2.
• Related: human-expert study of simulated students (eth-sim-student) and a simulator-based realism test (dou-etal-2025-simulatorarena) app. A.2.
• Superseded by: not known.