Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

Rose E. Wang; Ana T. Ribeiro; Carly D. Robinson; Susanna Loeb; Dora Demszky

cs.CLOct 3, 2024

Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

Authors: Rose E. Wang, Ana T. Ribeiro, Carly D. Robinson, Susanna Loeb, Dora Demszky

Organizations: Stanford University

Abstract

Generative AI, particularly Language Models (LMs), has the potential to transform real-world domains with societal impact, particularly where access to experts is limited. For example, in education, training novice educators with expert guidance is important for effectiveness but expensive, creating significant barriers to improving education quality at scale. This challenge disproportionately harms students from under-served communities, who stand to gain the most from high-quality education. We introduce Tutor CoPilot, a novel Human-AI approach that leverages a model of expert thinking to provide expert-like guidance to tutors as they tutor. This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities. Following a preregistered analysis plan, we find that students working with tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master topics (p<0.01). Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by 9 p.p. We find that Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages using classifiers to identify pedagogical strategies, and find that tutors with access to Tutor CoPilot are more likely to use high-quality strategies to foster student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student. Tutor interviews highlight how Tutor CoPilot's guidance helps tutors to respond to student needs, though they flag issues in Tutor CoPilot, such as generating suggestions that are not grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students.

Figures & tables

Figure 1. Illustration of Tutor CoPilot front end interface and back end system operations. Panel (a) Tutor CoPilot is integrated into live contexts as a button that the tutor can activate for real-time assistance during their tutoring sessions. Panel (b) Tutor CoPilot applies user safety and privacy practices , such as automatically de-identifying student and tutor names and limiting the amount of user information sent to external LM services. Panel (c) Tutor CoPilot generates expert-like guidance by leveraging the Bridge method 2 which captures expert decision-making from their verbalized reasoning patterns. Panel (d) Tutor CoPilot is activated , displaying three suggested responses for tutors to choose from in panels that enable user customization by editing ( ), re-generating ( ), or displaying the full list of strategies with seven additional options ( ).

	Panel A. Session outcomes
	Participation	Participation	Exit Tickets	Exit Tickets	Exit Tickets
	Points	Points Standardized	Attempted	Passed (Cond.)	Passed (Uncond.)
Treatment	0.094	0.010	0.019 +	0.031 ∗	0.040 ∗∗
	(0.27)	(0.028)	(0.011)	(0.014)	(0.015)
Control Mean	14.071	0.016	0.843	0.732	0.617
	(0.196)	(0.020)	(0.008)	(0.010)	(0.010)

Table 1. Intent-to-treat analysis on student session-level outcomes. Estimates are from our primary model, which controls for baseline math scores, student demographics, and fixed effects for strata (school

\times

grade). Participation points are standardized within-sample and by grade. Survey items are on a 5-point scale where higher is better.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

correspond to student-tutor pair clustered standard errors in parentheses. Romano-Wolf adjusted p-values are presented in brackets.

Quality	Strategy Name	Definition	Examples	F1
Category	(Frequency)
High	Prompt Student	The tutor prompts the student	“Go ahead and try to explain	0.89
	to Explain	to explain a concept, rule, or	how you got the answer.”
	(3%)	their reasoning.
High	Ask Question	The tutor asks the student a	“What number can we multiply	0.90
	to Guide Thinking	question to help them think	the number 10 to get an equal

Table 2. Taxonomy of high - and low -quality strategies, including their definitions, examples, and frequency over the labelled dataset of 241,066 messages sent by tutors during tutoring sessions. We train binary classifiers to identify these strategies at scale and report their test F1 score as well. A tutor response can include multiple strategies.

Figure 3. Strategies more likely to be used by control tutors (left) vs. treatment tutors (right). Logit regression coefficient results with 95% Confidence intervals based on student-tutor pair standard errors and a sample of 241,005 classified tutor messages (115,517 from treatment and 125,488 from control tutors). Strategies more likely to be used by treatment tutors are shaded in red and strategies more likely to be used by control tutors are shaded in blue ( $p<0.05$ ). Strategies not significantly different in use by either group are shaded in gray ( $p>0.05$ ). See Table 3 in Supplementary Information G.2 .

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	0.095 ∗∗	0.059 +	0.033	0.022	-0.049	0.0012	-0.098 ∗∗∗
	(0.036)	(0.031)	(0.022)	(0.063)	(0.030)	(0.030)	(0.026)
Romano-Wolf p-val	[0.010]	[0.020]	[0.158]	[0.871]	[0.129]	[0.990]	[0.010]
Z	2.614	1.897	1.505	0.345	-1.614	0.039	-3.795
Odds Ratio	1.100	1.060	1.034	1.022	0.952	1.001	0.907
N	241005	241005	241005	241005	241005	241005	241005

Table 3. Log Odds Analysis of Strategy Use Differences Between Treatment and Control Tutors. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Dependent	EOY MAP Math Std (2022-23sy)
Exit Tickets Passed	0.02***
	(0.002)
BOY MAP Math Std (2022-23sy)	0.87***
	(0.02)
R 2	0.728
N	689

Appendix

Table 4. Linear Regression End-of-Year Standardized MAP Math Scores on Exit Tickets Passed and Beginning-of-Year Standardized MAP Math Scores. This regression uses exit ticket and achievement data from students in the same district of our study who received tutoring services from FEV Tutor during the 2022-23 school year, before Tutor Copilot was implemented. Results represent the predictive value of student beginning of the year (BOY) MAP performance and the number of exit tickets successfully completed during the year for the end of the year (EOY) MAP performance. We used standardized BOY and EOY MAP performance measures and robust standard errors.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

Panel A. Elementary Schools
	Grade 3	Grade 4	Grade 5	Total
Total	676	1,828	357	2,861
Panel B. Middle Schools
	Grade 6	Grade 7	Grade 8	Total
Total	1,275	0	0	1,275
Total				4,136

Appendix

Table 5. Session-level Statistics by Grade. We had 8 elementary schools and 1 middle school in our study sample.

		Mean
Panel A: Student Characteristics ( $N=1,013$ )
Gender
	Male	0.46
	Female	0.46
	Missing	0.09
Race/Ethnicity

Appendix

Table 6. Descriptive statistics for students and tutors who participated in the study. We define a participating student as a student who has at least one tutoring session recorded with a treatment or control tutor during the implementation period. We define a participating tutor as a tutor randomized to the treatment or control condition who has at least one tutoring session recorded during the implementation period with a student from the school district of interest.

Figure 4. Number of sessions per day by tutor assignment.

Figure 5. Percentage of treatment sessions that used Tutor CoPilot at least once in their session. About 29% of treatment sessions used Tutor CoPilot during our study.

Panel of Figure 6 — Figure 6. (a) reports the average number of Tutor CoPilot uses, including the sessions that had no usage, and (b) reports the average number of uses but excludes the sessions with no usage. When including the zero-use sessions, tutors use Tutor CoPilot about 3 times per session. When excluding the zero-use sessions, tutors use Tutor CoPilot about 10 times per session.

Moment (Frequency)	Definition	Examples	F1
Start of session	The tutoring session is just starting. The	“Happy to work with you today!”	0.79
(0.9%)	student and tutor have not yet started a problem.
Start of problem	The tutor starts a new problem and/or gives	“Go ahead and start showing your	0.70
(3.2%)	instructions for the new problem.	work for this question.”
During problem attempt	The student is attempting the problem and/or	“Are you working on this problem?”	0.70
(48.4%)	the tutor has not yet given away the answer

Appendix

Table 7. Taxonomy of tutoring moments, including their definitions, examples, and frequency over the labelled dataset of 2,114 messages sent immediately before activation of the Tutor CoPilot tool. We train binary classifiers to identify these moments at scale and report their test F1 score as well. A majority of Tutor CoPilot usage concetrates during the “meat” of student learning: when the student is attempting the problem, or after they have attempted the problem and the tutor is giving them feedback. The classifier for “after exit ticket attempt” scored a low test F1 score, even after tuning the class-imbalance loss, thus we omit its frequency. Note that the frequencies do not sum to 1 because the classifiers are not mutually exclusive.

Dependent Variable	Used	Number of Uses
Gender (Is Female)	0.015	1.36**
	(0.03)	(0.51)
Experience (Months)	-0.002+	-0.031+
	(0.001)	(0.02)
Quality Rating	0.12	1.69
	(0.08)	(1.41)

Appendix

Table 8. Predicting Tutor CoPilot Use From Tutor Characteristics. The parentheses report standard errors clustered by tutor id.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

Figure 7. Histogram of student treatment exposure.

	Math MAP (std) With Imputation			Math MAP (std) Without Imputation
Prop of Treatment Sessions	-0.0044	-0.0016	-0.081	-0.022	-0.020	-0.061
	(0.050)	(0.050)	(0.082)	(0.048)	(0.048)	(0.078)
N Sessions Completed		0.0097	0.0030		0.0053	0.0017
		(0.0071)	(0.0085)		(0.0064)	(0.0078)
			0.022			0.012
			(0.018)			(0.017)

Appendix

Table 9. Student-level Estimates of Treatment Exposure Impact on EOY Standardized MAP Math Scores. The sample of students is restricted to students from the partner district who attended at least one tutoring session with a treatment or control tutor during the implementation period for this study. We report the coefficient estimates for models with and without baseline scores (MOY MAP Math) imputations for students missing this information, and the interaction of the percentage of treated sessions and the number of sessions attended during Tutor CoPilot implementation. Robust standard errors in parentheses. All models include controls for student achievement, characteristics (gender, race/ethnicity, free/reduced lunch, special education, and English learner), and strata (school x grade) fixed effects. +

p<0.10

, ∗

p<0.05

, ∗∗

p<0.01

, ∗∗∗

p<0.001

Dependent	Do you agree or disagree	How confident are you	How effective are you at	How much more or less
Variable	with this statement: “My	at recognizing the kind	helping students fix their	effective are you at
	students learn more by	of mathematical	mistakes?	helping students fix their
	making mistakes.”?	mistakes students are		mistakes now than you
		making?		were three months ago?
Range	1 = Disagree	1 = Not at all confident	1 = Not at all effective	1 = A lot less effective
	2 = Somewhat disagree	2 = Slightly confident	2 = Slightly effective	2 = Slightly less effective

Appendix

Table 10. Tutor-level Survey Outcomes. The parentheses report robust standard error. Estimates are from our primary model, which controls for tutor quality rating, months of experience at the platform, and gender.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Exit Tickets Attempted	Exit Tickets Passed Conditional	Exit Tickets Passed Unconditional
Treatment	0.17 +	0.18 ∗	0.19 ∗∗
	(0.093)	(0.080)	(0.069)
Romano-Wolf p-val	[0.030]	[0.030]	[0.010]
Z	1.803	2.202	2.769
Odds Ratio	1.182	1.193	1.209
N	4105	3518	4133

Appendix

Table 11. ITT Logit Model Results for Binary Outcomes. Controls for baseline math scores, student demographics, and fixed effects for strata (school

\times

grade). Standard errors clustered at the tutor-student pair are shown in parentheses, and Romano-Wolf p-values adjusted for multiple hypotheses are shown in brackets.

	Exit Tickets Attempted	Exit Tickets Passed Conditional	Exit Tickets Passed Unconditional
Treatment	0.18	0.16 +	0.21 ∗
	(0.14)	(0.093)	(0.096)
Z	1.231	1.769	2.203
Odds Ratio	1.195	1.179	1.236
N	4105	3518	4133

Appendix

Table 12. ITT Logit Model Results for Binary Outcomes with Random Effects for Students and Tutors. Controls for baseline math scores, student demographics, and fixed effects for strata (school

\times

grade). Standard errors shown in parentheses.

	Panel A. Session outcomes
	Participation Points .	Participation Points Standardized	Exit Tickets Attempted .	Exit Tickets Passed Conditional	Exit Tickets Passed Unconditional
Treatment	-0.022	-0.0013	0.017	0.028 +	0.037 ∗
	(0.33)	(0.034)	(0.013)	(0.016)	(0.016)
Control Mean	14.035	0.012	0.842	0.733	0.617
	(0.234)	(0.024)	(0.009)	(0.011)	(0.011)
N	4136	4136	4136	3521	4136

Appendix

Table 13. ITT Linear Model Results for Student Session-level Outcomes with Random Effects for Students and Tutors. The parentheses report the standard error. Models include controls for baseline math scores, student demographics, and a fixed effect for strata (school

\times

grade). Participation points are standardized within-sample, by grade. The survey items are on a 5-point scale where higher is better.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Exit Tickets Passed Unconditional	Exit Tickets Passed Unconditional	Exit Tickets Passed Unconditional
Treatment	0.0869 +	0.204 ∗	0.00184
	(0.0449)	(0.0965)	(0.0346)
Treat x Tutor Quality Rating (Cont)	-0.0891	-0.392
	(0.0747)	(0.299)
Treat x Tutor Experience (Cont)	-0.000395	-0.00746
	(0.00123)	(0.00924)

Appendix

Table 14. Regression Results from the Tutor Quality and Experience Heterogeneity Analyses. Column 1 models the interaction between the treatment and tutor quality and experience linear measures, column 2 includes a quadratic term and interactions for these measures, and column 3 shows the results for a model that categorizes the level of quality and experience into "Low", "Medium", and "High", based on tercile grouping. Figure 2 is based on the latter. Estimates include controls for baseline math scores, student demographics, and fixed effects for strata (school

\times

grade). Standard errors clustered for student-tutor pairs in parentheses. +

p<0.10

, ∗

p<0.05

, ∗∗

p<0.01

, ∗∗∗

p<0.001

Figure 8. Log odds Analysis as proposed in Monroe et al. (2008) 7 . Strategies More Likely to be Used by Control Tutors (left) vs. Treatment Tutors (right). Strategies with a z-score below 1 standard deviation are shaded in gray.

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	0.12 ∗	0.039	0.038	0.12	-0.039	-0.0056	-0.10 ∗
	(0.057)	(0.045)	(0.031)	(0.10)	(0.043)	(0.044)	(0.043)
Z	2.107	0.861	1.250	1.176	-0.900	-0.128	-2.416
Odds Ratio	1.127	1.040	1.039	1.126	0.962	0.994	0.901
N	241005	241005	241005	241005	241005	241005	241005

Appendix

Table 15. Alternative Specification - Log Odds Analysis of Strategy Use Differences Between Treatment and Control Tutors. This table reports the estimates from a logit model with random effects for tutors and students separately. The parentheses report the standard error.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	0.19 ∗∗	-0.087	0.12 ∗∗	0.18 +	0.00095	-0.00045	-0.058
	(0.061)	(0.053)	(0.038)	(0.10)	(0.052)	(0.049)	(0.045)
Romano-Wolf p-val	[0.010]	[0.356]	[0.010]	[0.228]	[1.000]	[1.000]	[0.723]
Z	3.199	-1.628	3.063	1.791	0.018	-0.009	-1.293
Odds Ratio	1.214	0.917	1.124	1.198	1.001	1.000	0.944
N	82952	82952	82952	82952	82952	82952	82952

Appendix

Table 16. Low Quality Rating Score Tutors - Log odds analysis of strategy use. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	0.10 +	0.12 ∗	-0.025	-0.12	-0.034	0.036	-0.26 ∗∗∗
	(0.063)	(0.054)	(0.038)	(0.11)	(0.051)	(0.053)	(0.047)
Romano-Wolf p-val	[0.317]	[0.020]	[1.000]	[0.911]	[1.000]	[1.000]	[0.010]
Z	1.675	2.284	-0.667	-1.034	-0.678	0.677	-5.594
Odds Ratio	1.110	1.131	0.975	0.889	0.966	1.037	0.770
N	78358	78358	78358	78358	78358	78358	78358

Appendix

Table 17. Medium Quality Rating Score Tutors - Log odds analysis of strategy use. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	-0.015	0.14 ∗∗	0.012	-0.057	-0.11 ∗	-0.039	0.029
	(0.065)	(0.053)	(0.038)	(0.11)	(0.055)	(0.053)	(0.042)
Romano-Wolf p-val	[1.000]	[0.010]	[1.000]	[1.000]	[0.050]	[0.990]	[1.000]
Z	-0.236	2.740	0.316	-0.493	-2.018	-0.749	0.697
Odds Ratio	0.985	1.156	1.012	0.945	0.895	0.961	1.030
N	79695	79695	79695	79695	79695	79695	79695

Appendix

Table 18. High Quality Rating Score Tutors - Log odds analysis of strategy use. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	-0.027	0.082	0.036	0.065	-0.010	0.0023	-0.13 ∗∗
	(0.063)	(0.053)	(0.036)	(0.11)	(0.050)	(0.049)	(0.041)
Romano-Wolf p-val	[1.000]	[0.406]	[0.931]	[1.000]	[1.000]	[1.000]	[0.010]
Z	-0.436	1.565	0.980	0.578	-0.202	0.047	-3.079
Odds Ratio	0.973	1.086	1.036	1.067	0.990	1.002	0.882
N	88684	88684	88684	88684	88684	88684	88684

Appendix

Table 19. Low Experience Tutors - Log odds analysis of strategy use. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	0.29 ∗∗∗	0.011	0.048	-0.19 +	0.021	-0.012	-0.16 ∗∗∗
	(0.062)	(0.055)	(0.038)	(0.11)	(0.055)	(0.052)	(0.046)
Romano-Wolf p-val	[0.010]	[1.000]	[0.723]	[0.228]	[1.000]	[1.000]	[0.010]
Z	4.636	0.208	1.271	-1.791	0.389	-0.221	-3.558
Odds Ratio	1.331	1.012	1.049	0.827	1.022	0.988	0.851
N	78381	78381	78381	78381	78381	78381	78381

Appendix

Table 20. Medium Experience Tutors - Log odds analysis of strategy use. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Prompt Student to Explain	Ask Question to Guide Thinking	Affirm Student’s Correct Attempt	Ask Student to Retry	Give Away Answer/ Explanation	Give Solution Strategy	Encourage Student in Generic Way
Treatment	0.029	0.077	0.00064	0.20 +	-0.14 ∗∗	0.032	0.0052
	(0.064)	(0.053)	(0.040)	(0.11)	(0.054)	(0.053)	(0.047)
Romano-Wolf p-val	[1.000]	[0.475]	[1.000]	[0.228]	[0.010]	[1.000]	[1.000]
Z	0.454	1.464	0.016	1.795	-2.587	0.610	0.110
Odds Ratio	1.029	1.081	1.001	1.216	0.870	1.033	1.005
N	73940	73940	73940	73940	73940	73940	73940

Appendix

Table 21. High Experience Tutors - Log odds analysis of strategy use. The parentheses report the standard error clustered by student-tutor pairs. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Panel A. Session outcomes
Dependent	Participation	Participation	Exit Ticket	Exit Ticket	Exit Ticket
Variable:	Points	Points	Attempted	Passed	Passed
		(Standardized)		(Conditional)	(Unconditional)
Treatment Effect	0.32	0.035	0.064 +	0.10 ∗	0.14 ∗∗
	(0.93)	(0.095)	(0.037)	(0.047)	(0.050)
Romano-Wolf p-val	[0.990]	[0.990]	[0.129]	[0.040]	[0.010]

Appendix

Table 22. TOT Analysis on Student Session-level Outcomes. The parentheses report the standard error clustered by student-tutor pairs. Estimates are from the second stage of the 2SLS model, which includes controls for baseline math scores, student demographics, and fixed effects for strata (school

\times

grade). Participation points are standardized within-sample and by grade. The survey items are on a 5-point scale where higher is better. Romano-Wolf adjusted p-values are shown in brackets.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Control (N=444)	Treatment (N=427)	p-value
Gender (Is Female)	0.56	0.53	0.27
Experience (Months)	21	22	0.61
Quality Rating	0.41	0.42	0.62

Appendix

Table 23. Balance check of pre-study sample of tutors assigned to treatment and control. Experience reports the number of months the tutor has been with the tutoring provider. Quality Rating reports the tutor’s tutoring quality rating determined prior to the study by the tutoring provider based on session observations & quality rubric scores.

	Session Attrition	Survey Attrition
Treatment	-0.013	0.0077
	(0.021)	(0.022)
Tutor Quality Score	-0.021	-0.088
	(0.056)	(0.063)
Tutor Experience (Months)	-0.00034	-0.0015 +
	(0.00084)	(0.00086)

Appendix

Table 24. Tutor differential attrition on session participation and post-survey response. Robust standard errors in parenthesis. Experience reports the number of months the tutor has been with the tutoring provider. Quality Rating reports the tutor’s tutoring quality rating determined prior to the study by the tutoring provider based on session observations & quality rubric scores.

{}^{+}p<0.1

;

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

	Control (N=394)	Treatment (N=386)	p-value
Gender (Is Female)	0.55	0.52	0.33
Experience (Months)	21	22	0.66
Quality Rating	0.41	0.42	0.61
# Sessions during Study	5	5	0.39
Total Session Time during Study (Minutes)	199	194	0.48

Appendix

Table 25. Balance check on study’s actual tutor sample with sessions (post attrition). Experience reports the number of months the tutor has been with the tutoring provider. Quality Rating reports the tutor’s tutoring quality rating determined prior to the study by the tutoring provider based on session observations & quality rubric scores.

	Control (N=371)	Treatment (N=355)	p-value
Gender (Is Female)	0.55	0.53	0.57
Experience (Months)	21	22	0.45
Quality Rating	0.41	0.42	0.76
# Sessions during Study	5	5	0.77
Total Session Time during Study (Minutes)	202	201	0.91

Appendix

Table 26. Balance check on study’s tutor sample post-survey. Experience reports the number of months the tutor has been with the tutoring provider. Quality Rating reports the tutor’s tutoring quality rating determined prior to the study by the tutoring provider based on session observations & quality rubric scores.

	Ask Question to Guide Thinking	Give Solution Strategy	Prompt Student to Explain	Encourage Student in Generic Way	Affirm Student’s Correct Attempt	Give Away Answer/ Explanation	Ask Student to Retry
Ask Question to Guide Thinking	0.04451	0.00005	0.00000	0.00007	0.00027	0.00076	0.00010
Give Solution Strategy	0.00005	0.00509	0.00000	0.00000	0.00005	0.00047	0.00004
Prompt Student to Explain	0.00000	0.00000	0.02545	0.00008	0.00023	0.00000	0.00017
Encourage Student in Generic Way	0.00007	0.00000	0.00008	0.11845	0.02031	0.00003	0.00033
Affirm Student’s Correct Attempt	0.00027	0.00005	0.00023	0.02031	0.13405	0.01412	0.00034
Give Away Answer/Explanation	0.00076	0.00047	0.00000	0.00003	0.01412	0.05917	0.00008

Appendix

Table 27. Overlap in strategy classification results. Each value represents the proportion of messages with a positive result by the row and column strategy classifiers. The diagonal presents the proportion of messages classified as one specific strategy.

Explore similar work

Jun 16, 2026

FairTutor: Equity-Aware Pedagogical LLM Routing for Budget-Constrained AI Tutoring

Generative AI tutors provide real-time, personalized learning support, but also create a new education inequity: students with access to premium AI services may receive clearer explanations, more personalized guidance, and better scaffolding than students limited to free or low-cost services. To address this challenge, we propose FairTutor, an equity-aware model-routing framework that achieves cost-effective AI tutoring via pedagogically motivated multi-agent orchestration. FairTutor combines query analysis, pedagogical planning, low-cost model generation, evaluator-guided critique and revision, and selective escalation to premium AI models. We introduce access-tier AI Education (AIED) Advantage Gap to measure the quality difference between premium-access and budget-constrained tutoring, and TutorAccessEval, a benchmark spanning math, reading, writing, science, and language learning. Empirical evaluations show that FairTutor achieves 97.1% of premium pedagogical quality (in floor-adjusted Likert scale) while reducing serving cost by 71.6%. Sensitivity analysis reveals a tunable cost--quality Pareto frontier, enabling FairTutor to be tailored to the needs of diverse student populations.

Qingyang Xu

Independent researcher, Shanghai, China

Intelligent Tutoring Systems AI in Education Algorithmic Fairness Cost-Aware Inference Generative AI in Education LLM Routing

Ask PDF

Sep 23, 2026

StudentBench: AI and human tutoring yield equivalent GRE learning gains

Artificial intelligence offers an unprecedented opportunity to augment human capabilities, yet progress at the frontier has focused primarily on advancing model capabilities. We introduce StudentBench, a suite of AI teaching evaluations and a public platform that enables large-scale data collection with over 175,000 student-AI messages to study whether large language models (LLMs) produce learning gains equivalent to human tutoring. Using StudentBench, we measured learning gains on Quantitative and Verbal GRE questions across 2,383 human participants receiving AI tutoring, human tutoring, or no tutoring. We establish that AI tutoring is statistically equivalent to expert human tutoring for GRE learning gains (p = .015), and in five of the seven GRE domains, the best performing AI tutor surpassed the human tutor, on average. In a second study, expert human tutors compared LLM-generated lesson plans and practice problems through 2,028 pairwise rubric evaluations. Together, the two studies clearly separate AI tutors across: (1) lesson planning, (2) practice-problem creation, (3) conversational pedagogy, (4) cost, and (5) engagement. Surprisingly, one AI tutor achieved learning gains equivalent to human tutoring (p = .044) at 918 times lower cost (USD 0.0052 for AI versus USD 4.81 for human, per percentage point gained). For Quantitative GRE sessions, faster AI replies correlated with more student messages, more messages with more correct practice, and more correct practice with larger learning gains (all p < .002). To support future research, we open-source the de-identified data collected in our studies.

Curtis Northcutt, Inaara Hasmani, Kevin Feng +3

Intelligent Tutoring Systems Generative AI in Education

Ask PDF

Jun 17, 2026

AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice

There exist numerous tutor training platforms. However, few provide AI-driven training and evaluation for human tutors based on real-life performance. We present an AI-driven system that assesses both open responses during training and authentic real-life tutoring. Unlike platforms that only assess learning through online training or simulations, our system utilizes Generative AI (Gemini-2.5-pro) to analyze transcriptions of authentic tutoring, measuring the transfer of tutor skills to real-life application. Human tutors instructing students remotely in math (N=86) completed six scenario-based lessons, averaging a significant 7.4% learning gain. Using mixed-effects models across 405 session-to-lesson pairs, we found that training performance significantly predicted real-life transcript scores with an effect size of 0.25 SD. Model comparison (AIC/BIC) indicated averaging open response and multiple choice performance during training predicted real-life tutor performance best, although open responses were comparatively more predictive. Exploratory analysis showed that after training, tutors were significantly more likely to encounter pedagogical opportunities to apply their skills (61.1% to 68.9%) and demonstrated higher execution quality within those opportunities (65.5% to 68.1%). Interrupted time series analysis suggested that these tutor improvements were part of a gradual trend over time rather than an immediate intervention effect of training. We illustrate an AI-driven method to link tutor training with real-life assessment. In doing so, we contribute open datasets, AI prompts, and scoring rubrics to support transparency and reproducibility.

Danielle R. Thomas, Marie Cynthia Abijuru Kamikazi, Clara Brandt +2

Carnegie Mellon University, Pittsburgh, PA, USA · Vanderbilt University, Nashville, TN, USA

Educational Assessment

Ask PDF