Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

In plain words

Tutors who teach students by chat often have no expert next to them to suggest what to say when a student gets a problem wrong. Tutor CoPilot is a button inside the tutoring chat. When a tutor presses it, the system reads the recent conversation and proposes several possible replies, each built around a different teaching move, such as asking a guiding question or prompting the student to explain their reasoning. The tutor can edit, regenerate or pick a different move, and the final choice is always theirs.

The authors tested the tool in a randomized trial with tutors from one online tutoring provider. Tutors who had the tool were more likely to have students pass the short exit tickets that check mastery of each topic, and they used more of the high-quality teaching moves. The gains were largest for tutors who had been rated lower before the study. Read the rest of this card for the exact numbers and their limits.

TL;DR

• Tutor CoPilot is a tutor-facing assistant that offers one suggested reply per pedagogical strategy, built from the conversation, the lesson topic and the chosen strategy § Methods fig. 1.
• The suggestions come from GPT-4 prompted with the Bridge method, which was developed in earlier work § Discussion [SI: Tutor CoPilot].
• In a preregistered randomized trial (tutors randomized, n = 783), access to the tool raised the unconditional exit-ticket pass rate by 4.0 p.p. (61.7% to 65.7%) tab. 1.
• The paper reports a cost of about USD 20 per tutor per year [SI: Study Costs].
• Everything is author-reported. The effect on end-of-year test scores was not estimated § Methods.

Problem

Novice tutors often give generic praise or the answer, instead of guiding the student through the problem. Expert-guided training is costly and hard to scale § Introduction. The authors aim to give novices expert-like wording in the moment, while the tutor keeps control over what is sent § Introduction.

The approach assumes that the system can see the live conversation and the lesson topic. The demo notebook states that effectiveness depends on immediate access to that context Tutor_CoPilot_Demo.ipynb. Early pilots also found the tool too slow (over 30 seconds per response), which was fixed before launch [SI: Pilot Studies].

Method

System. The tool is a button in a chat-based tutoring platform. When pressed, it runs the following steps (see the pseudocode). Names are replaced with the placeholders [STUDENT] and [TUTOR] using Edu-ConvoKit. Only the 10 most recent messages are sent to the external language model [SI: Tutor CoPilot]. The tutor sees one suggestion per strategy, can edit, regenerate, or choose another strategy from a list fig. 1 [SI: Tutor CoPilot].

Strategies. The interface offers options such as giving a full solution, a worked example, a small correction, a related problem, a simpler version of the question, confirming a correct answer, and generic encouragement [SI: Tutor CoPilot]. The demo notebook's list uses similar names and ends with an "other" option Tutor_CoPilot_Demo.ipynb.

Prompting. The paper states that Bridge supplies the prompts and that the deployed model is GPT-4 [SI: Tutor CoPilot]. The demo uses a template that begins with an instruction to act as an experienced elementary math teacher, then inserts the lesson topic, the strategy and the conversation, and asks for a reply of at most one sentence Tutor_CoPilot_Demo.ipynb. The paper does not state that prompt text, and the one-sentence limit appears only in the demo.

Trial design (for the outcome-analysis pipeline). Tutors were randomized, not students § Methods. Treatment tutors got the tool from 27 March to 17 May 2024 § Methods. The unit of analysis is the session. The primary outcome model is an intent-to-treat regression:

Y_{ist} = \beta_0 + \beta_1 \text{Treatment}_{t} + X_s\gamma + \omega_{k(s)} + \epsilon_{ist} eq. 1

Here Y_ist is a session outcome, X_s are student covariates (baseline MAP math score, gender, race, free or reduced lunch, special education, limited English proficiency), and omega_k(s) is a school-by-grade fixed effect. The residual is clustered by student-tutor pair § Methods eq. 1.

# Tutor CoPilot, one activation  [§ Methods] [SI: Tutor CoPilot]
# Input:  chat_history (list of messages), lesson_topic, strategies
# Output: one suggested reply per strategy

def suggest_replies(chat_history, lesson_topic, strategies):
    # 1. Replace student and tutor names with placeholders  [SI: Tutor CoPilot]
    deidentified = replace_names(chat_history, "[STUDENT]", "[TUTOR]")

    # 2. Keep only the most recent messages sent to the external model  [SI: Tutor CoPilot]
    context = deidentified[-10:]

    replies = []
    # 3. One prompt per chosen strategy  [SI: Tutor CoPilot]
    # [code: Tutor_CoPilot_Demo.ipynb#L1894-L1898]
    for strategy in strategies:
        prompt = build_prompt(lesson_topic, context, strategy)
        # 4. Single model call; the paper states GPT-4  [SI: Tutor CoPilot]
        replies.append(call_language_model(prompt))
    return replies

Strategy classifiers (for RQ3). The authors labelled 3,000 tutor messages with an LLM first pass (GPT-4) and corrected the labels by hand § Methods [SI: Development of NLP Classifiers]. They fine-tuned a RoBERTa-large model with a sigmoid class-balanced loss [cui2019class] [SI: Development of NLP Classifiers]. Input is the 10 prior messages plus a separator token before the target message [SI: Development of NLP Classifiers]. The split is 6:1:3 for train, validation and test [SI: Development of NLP Classifiers]. The hyperparameter sweep and selected values are not stated in the paper.

Hyperparameters

Results

Author-reported. Setting: a randomized controlled trial with tutors randomized (treatment vs. business as usual), one chat-based tutoring provider, and a U.S. school district. The primary model is the intent-to-treat regression eq. 1. The table reports unconditional results, control means and Romano-Wolf adjusted p-values tab. 1.

Significance: + p<0.1, * p<0.05, ** p<0.01 tab. 1.

Effect on learning (RQ1). Students with treatment tutors were 4 p.p. more likely to pass the exit ticket, from 62% to 66%, p<0.01 § Results tab. 1. Using the tool, instead of only having access to it, was estimated with 2SLS as a 14 p.p. increase, 62% to 76% (p<0.01) [SI: Treatment on the Treated Analysis]. The authors describe this as an estimate for tutors whose use was affected by assignment. The 2SLS table reports 0.14 (SE 0.050) for the unconditional outcome [SI: Treatment on the Treated Analysis].

Heterogeneity (RQ2). Lower-rated tutors' students passed 9 p.p. more often (56% to 65%) fig. 2a. Less experienced tutors' students passed 7 p.p. more often (61% to 68%) fig. 2b. The paper reports the reference point as: treated students of lower-rated or less experienced tutors matched or exceeded control students of higher-rated or more experienced tutors § Results.

Tutoring strategies (RQ3). Treatment tutors used "prompt student to explain" about 10% more often and control tutors used "encourage student in generic way" about 10% more often fig. 3 § Results. Each estimate is from a logit regression eq. 2 on 241,005 classified messages (115,517 treatment, 125,488 control) fig. 3. The strategy taxonomy and classifier F1 scores appear in tab. 2:

Tutor perception (RQ4). Interviews with 18 treatment tutors, one week after the study, found the tool helpful for well-phrased explanations. Tutors also reported that suggestions were sometimes too advanced for the grade level § Results.

Null results. Participation points, the standardized version and all survey outcomes were not significant after correction tab. 1 § Results. The end-of-year MAP effect was not estimated directly. A back-of-the-envelope figure of 0.024 standard deviations for a full year of use is given as a rough calculation § Introduction [SI: Exit Ticket Significance].

Paper vs code

The official repository is https://github.com/rosewang2008/tutor-copilot, linked from the paper's Code Availability section § Code Availability. Commit 25c6eab (full hash in the frontmatter) is the shallow clone we read. It contains only a README, a LICENSE file (Apache-2.0), and a demo notebook [code: Tutor_CoPilot_Demo.ipynb]. It does not contain the trial analysis, the classifiers or the deployed system.

• The paper states the deployed model is GPT-4 with Bridge prompts [SI: Tutor CoPilot]. The demo calls gpt-4 and sets no temperature, top-p or token limit Tutor_CoPilot_Demo.ipynb. The deployed sampling settings are not stated.
• The paper says the external model receives the 10 most recent messages [SI: Tutor CoPilot]. The demo builds the conversation from the rows it is given and shows no window cut Tutor_CoPilot_Demo.ipynb. The deployed cut is therefore not visible in the demo.
• The one-sentence reply limit appears only in the demo prompt [code: Tutor_CoPilot_Demo.ipynb#L341-L346]. The paper does not state it.
• The analysis repository linked in the paper, anatrindaderibeiro/tutorcopilot_analysis, has no license. Its owner is scale-nssa. We did not clone or quote it. The paper's analysis code therefore could not be checked here.
• The classifier models are said to be on Hugging Face (StanfordSCALE) [§ Code Availability]. We did not check them.

Pitfalls

• Conflicting numbers. The arXiv abstract metadata says 900 tutors, 1,800 students and 550,000+ messages. The LaTeX source says more than 700 tutors, 1,000 students and 350,000+ messages, and the analysed sample is 783 tutors and 1,013 students § Methods. This card uses the body text. Check the version you cite.
• Unit of randomization. Tutors were randomized, not students. Students' treatment status changed with the tutor matched to each session § Methods. Do not read the effect as a student-level trial.
• Intent-to-treat vs usage. The 4 p.p. effect is for access. The 14 p.p. effect is an instrumental-variable estimate for tutors whose use changed with assignment [SI: Treatment on the Treated Analysis]. It is not the usage effect of the tool overall.
• Low exposure. Students matched to a tutor with access had little exposure, and the trial lasted about two months § Limitations. Null results on surveys and end-of-year scores may reflect that.
• Multiple outcomes. Several session outcomes were tested. Romano-Wolf p-values are the paper's own correction tab. 1.
• Classifier-based outcomes. The strategy effects depend on RoBERTa classifiers with F1 0.65 to 0.90. Three categories were dropped because F1 was below 0.60 [SI: Development of NLP Classifiers]. The paper's F1 values are author-reported. Strategies that fell below the threshold are not analysed.
• Cost basis. The USD 20 figure divides a USD 1,419.66 API cost over 429 treatment tutors, while the analysed sample has 388 treatment tutors [SI: Study Costs] [§ Methods]. The paper calls it an estimate.
• Latency. Early pilots flagged responses taking over 30 seconds. The fix came from profiling the data pipeline [SI: Pilot Studies]. The paper does not give latency numbers after the fix.
• Human-selected output. Tutors could edit, regenerate or swap strategies. The paper treats this as part of the design, so the output is not fixed text fig. 1 § Limitations.
• Not endorsed. This page is an AI-compiled summary and is not endorsed by the authors.

Lineage

• Builds on: Bridge, a method that turns think-aloud data from experienced educators into instructions for LLMs (Wang et al., 2024, cited as arXiv:2310.10648 in the demo notebook) § Discussion [code: Tutor_CoPilot_Demo.ipynb].
• Compared against: fine-tuning and prompt-engineering baselines were reported in the Bridge work, not in this paper § Discussion.
• Superseded by: not stated.