StudentBench: AI and human tutoring yield equivalent GRE learning gains

Curtis Northcutt; Inaara Hasmani; Kevin Feng; Trevor Khangi; Andreas Plesner; Jonas Mueller

cs.AISep 23, 2026

StudentBench: AI and human tutoring yield equivalent GRE learning gains

Authors: Curtis Northcutt, Inaara Hasmani, Kevin Feng, Trevor Khangi, Andreas Plesner, Jonas Mueller

Abstract

Artificial intelligence offers an unprecedented opportunity to augment human capabilities, yet progress at the frontier has focused primarily on advancing model capabilities. We introduce StudentBench, a suite of AI teaching evaluations and a public platform that enables large-scale data collection with over 175,000 student-AI messages to study whether large language models (LLMs) produce learning gains equivalent to human tutoring. Using StudentBench, we measured learning gains on Quantitative and Verbal GRE questions across 2,383 human participants receiving AI tutoring, human tutoring, or no tutoring. We establish that AI tutoring is statistically equivalent to expert human tutoring for GRE learning gains (p = .015), and in five of the seven GRE domains, the best performing AI tutor surpassed the human tutor, on average. In a second study, expert human tutors compared LLM-generated lesson plans and practice problems through 2,028 pairwise rubric evaluations. Together, the two studies clearly separate AI tutors across: (1) lesson planning, (2) practice-problem creation, (3) conversational pedagogy, (4) cost, and (5) engagement. Surprisingly, one AI tutor achieved learning gains equivalent to human tutoring (p = .044) at 918 times lower cost (USD 0.0052 for AI versus USD 4.81 for human, per percentage point gained). For Quantitative GRE sessions, faster AI replies correlated with more student messages, more messages with more correct practice, and more correct practice with larger learning gains (all p < .002). To support future research, we open-source the de-identified data collected in our studies.

Figures & tables

Figure 1: Learning across GRE domains and tutors. A, Mean pre-to-post test score change within each GRE domain under different tutoring conditions: human tutoring, AI tutoring (results pooled across all AI tutors), or no tutoring control. Error bars depict 95% Student- $t$ confidence intervals. Diamonds mark the top performing model in each domain; the dotted line traces the best AI performance across domains. Question counts refer to each assessment (pre-test, post-test). B, Learning gains (post $-$ pre test scores) pooled across Quantitative and Verbal, with 95% intervals; $n$ denotes the number of sessions under each tutoring condition. Dashed guides indicate the confidence interval for the gain from human tutoring. The inset shows the adjusted AI $-$ human difference and 90% confidence interval within $\pm 0.25$ pooled-SD equivalence bounds ( $p{=}.015$ ); the interval also lies within the stricter $\pm 0.20$ -SD bounds ( $p{=}.023$ ). C, Learning gains for 12 AI tutors, adjusted for pre-test score and section, with 95% HC3 intervals. See Appendix C for learning gains (Table C.1 ) and AI tutor leaderboards for all seven domains (Section C.2 ).

Figure 2: Experimental design of the learning gain study. 1, Students completed either a 27-question Quantitative or Verbal pre -test, with eligibility and effort screening, and then spent five minutes reviewing incorrect answers. 2, Students were assigned to one of three tutoring conditions: AI tutoring (with one of 13 LLM-based tutors; Table A.3 ), human tutoring, or no tutoring control. 3, After tutoring, students completed a post -test with the same length and concepts as the pre-test.

Figure 3: Three evaluations of AI teaching. Rows are ordered by lesson-planning score. A, B, Expert preferences for lesson planning and practice-problem creation, fitted with separate centered Bradley–Terry models; score differences are log-odds. Ratings marked as ties are excluded. Planning uses 9,577 ratings across 381 pre-tests; practice design uses 5,265 across 380. Whiskers are pointwise 95% intervals clustered by pre-test. C, Mean prevalence of six fixed conversation indicators across 1,971 AI and 135 human transcripts, with 95% Student-

t

intervals. The human reference is spoken tutoring; AI sessions use typed chat. Scores should be compared only within panels; blank cells mean not evaluated. Colors identify model families. See Appendices E and F for evaluation details.

Figure 4: Tutoring cost, AI reply time and student engagement. A, B, Mean learning gain versus (A) total AI inference cost per session and (B) median AI reply time. Blue points and dashed lines mark the AI Pareto frontier. The red star uses a $75 per hour human reference. C, Mean student-authored chat messages per session versus the median reply time. Shading shows the pointwise 95% confidence band around the fitted curve. See Appendix D for methods and section-specific results (Figures D.2 and D.3 ).

Figure 5: Cost per percentage point of learning gain. Cost per percentage point (pp) is the mean session cost divided by the mean learning gain. Whiskers show 95% confidence intervals. AI costs reflect inference; human tutoring uses a

75-per-hour reference. Cost multiples use Gemma 4 31B as the

1\times

reference and are calculated from unrounded values. Blue shading and checkmarks identify all six AI tutors that passed individual equivalence tests against human tutoring (

p{<}.05$ ). See Figure D.1 for section-specific rankings of models.

Figure 6: Explaining learning gains. Dots show averages for ten similarly sized groups of sessions per GRE section, ordered by AI reply time ( A ), student messages ( B ) or correct practice ( C ). Values on the vertical axis and regression lines adjust for pre-test score, AI tutor and assessment form. Lines use individual sessions (1,137 Quantitative and 998 Verbal); shading shows pointwise 95% confidence intervals for the fitted means. Tests account for 276 exploratory comparisons (Table B.1 ). See Section 5.2 for definitions and Appendix D.2 for methods.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Condition	Sessions	Pre-test, mean (SD)	P $\to$ Q	Q $\to$ P
Quantitative	AI	1,139	50.21 (21.16)	580	559
Quantitative	Human	61	45.17 (18.23)	27	34
Quantitative	Control	91	52.99 (21.22)	45	46
Verbal	AI	1,000	46.35 (19.47)	499	501
Verbal	Human	79	41.44 (15.43)	57	22
Verbal	Control	99	47.29 (19.08)	50	49

Appendix

Table A.1: Starting scores and assessment-form order for the learning comparisons in Figure 1 and the study design in Section 2 . Both forms were used as pre-tests and post-tests; the counts show how often each order was used within each condition. Pre-test scores are percentages correct; parentheses give the sample standard deviation. P

\to

Q and Q

\to

P denote the pre-test and post-test form order. The human tutoring condition had lower mean pre-test scores in both sections by chance; assignment was random and independent of pre-test score.

Primary exclusion reason	Quant	Verbal	Total
Eligibility and study-protocol checks
Pre-test score below eligibility range	225	83	308
Pre-test score above eligibility range	266	38	304
Insufficient assessment effort	143	124	267
Incomplete study	139	34	173
Assessment score unavailable	87	51	138

Appendix

Table A.2: Data-quality exclusions for the study in Section 2 and the learning results in Figure 1 . Each record has one primary reason within its screening stage. No record appears in both stages.

Figure A.1: Student tutoring interface. Lesson planning generates the concepts and time allocations across the top and the initial practice problems on the left. The live tutoring conversation appears on the right, with access to the student’s practice answers. See Figure 2 for the study design.

Figure A.2: Quantitative assessment interface. Pre-tests and post-tests use the same interface to measure unaided performance. The notice states that outside assistance is prohibited and tab switching is recorded. See Figure 1 for learning results.

Figure A.3: Human tutor reviewing a student’s pre-test. The tutor and student review a Quantitative question through screen sharing during a video call. See Figure 2 for the study design.

Panel of Figure 12 — Figure A.4: Human tutoring by video call. The tutor shares and annotates a Quantitative Comparison problem, illustrating place values and repeating units-digit patterns. See Figure 2 for the study design.

AI tutor	Endpoint	Quant	Verbal
Gemini 3.1 Pro (high)	gemini-3.1-pro-preview	91	74
Gemini 3.5 Flash (low)	gemini-3.5-flash	97	80
Gemini 3.6 Flash (low)	gemini-3.6-flash	89	79
Gemini 3.7 Flash (med)	gemini-3.7-flash	87	92
Gemma 4 31B (high)	gemma-4-31b-it	87	72
GPT-5.4 mini (off)	gpt-5.4-mini	90	71

Appendix

Table A.3: AI tutors with reasoning effort in parentheses, API endpoints, and Quantitative and Verbal sample sizes for the learning study. See Figure 1 for learning results.

Component	Main-study system templates	Additional guidance in the detailed variants
GRE context	Brief content and format overview	Detailed test structure, pacing, scoring conventions and format-specific strategies
Diagnosis and sequencing	AI tutor identifies needs and orders concepts	Explicit question-by-question misconception diagnosis, response-time interpretations and prerequisite sequencing
Difficulty calibration	AI tutor selects appropriate concepts and practice	Instructions for interpreting easy versus hard mistakes and using supplied expert ratings
Practice progression	Generate practice with worked solutions	Vary surface forms and progress from easier to difficult GRE-level practice
Conversational teaching	Teach transferable methods; choose explanation and practice	Attempt before hints, guide students to identify mistakes, request explanations and predictions, and connect conceptual understanding to efficient GRE methods

Appendix

Table A.4: Guidance intentionally omitted from the study prompts in Section 2 . The more detailed Quantitative variants were evaluated separately. Both included student-specific pre-test context. Shared application messages still invited practice attempts and responses to students’ mistakes.

Question	Reporting rule
Equivalence	TOST at $\alpha{=}.05$ with $\pm 0.25$ -SD bounds; $p$ is the larger of the two one-sided $p$ values. Each AI tutor is tested separately against human tutoring, without correction across tutors; all six passing tutors are highlighted.
Differences and associations	Two-sided tests, or the stated omnibus test. For families of related comparisons, we report the Holm-corrected $p$ value; the families and individual tests are specified below.

Appendix

Table B.1: Two statistical questions. The same

p

notation is used throughout. See Section 2 for the learning-gain definition and Appendix B for model specifications and comparison families.

Section	Contrast	Estimate	95% CI	$p$
Quantitative	AI $-$ control	6.86	[4.02,9.69]	$<.001$
Quantitative	Human $-$ control	6.27	[1.72,10.82]	.007
Verbal	AI $-$ control	5.47	[2.46,8.47]	$<.001$
Verbal	Human $-$ control	7.52	[3.17,11.87]	$<.001$
Combined	AI $-$ control	6.15	[4.08,8.21]	$<.001$
Combined	Human $-$ control	7.04	[3.88,10.20]	$<.001$

Appendix

Table B.2: Learning-gain differences between each tutoring condition and control, after adjustment for starting score. These complement the mean gains in Figure 1 B . Students in the control condition reviewed their graded pre-test; positive estimates favor tutoring. Values are percentage points with 95% HC3 intervals.

	Original data		After exclusion
Result	Estimate	$p$	Estimate	$p$
Combined AI $-$ human gain	$-0.58$	.015	$-0.05$	$<.001$
Quantitative AI $-$ human gain	$0.88$	.028	$1.18$	.034
Student messages per 10 seconds longer reply time	$-5.19$	$<.001$	$-4.83$	$<.001$
Correct practice per 10 more student messages	0.52	.002	0.47	.006
Gain per 10 more correct practice problems	5.62	$<.001$	5.77	$<.001$

Appendix

Table B.3: Main learning results after excluding students who completed both sections. The first two rows report AI

-

human learning gain and equivalence-test

p

values using the same model and fixed margins as the pooled comparison in Section 3 . The final three rows report the Quantitative associations in Section 7 , using 1,137 sessions before and 1,061 after exclusion. Their

p

values use Holm correction across the same 276 tests, applied separately in each cohort. Learning gains are in percentage points.

AI tutor	$R$	$E_{Q}$	$E_{V}$	$E/(R+E)$	$\Delta_{Q}$	$\Delta_{V}$
Gemini 3.1 Pro (high)	165	6	7	7.3%	$+0.94$	$-0.55$
Gemini 3.5 Flash (low)	177	2	4	3.3%	$-0.24$	$+0.15$
Gemini 3.6 Flash (low)	168	5	4	5.1%	$-0.11$	$-0.14$
Gemini 3.7 Flash (med)	179	7	8	7.7%	$+0.39$	$-0.36$
Gemma 4 31B (high)	159	3	7	5.9%	$-0.41$	$+1.07$
GPT-5.4 mini (off)	161	1	8	5.3%	$+0.04$	$+1.15$

Appendix

Table B.4: Participation-only exclusions and learning-gain sensitivity by AI tutor.

R

is the retained count;

E_{Q}

and

E_{V}

are restored Quantitative and Verbal counts, with

E=E_{Q}+E_{V}

. The fraction

E/(R+E)

is conditional on meeting the other recorded criteria and is not overall randomized attrition.

\Delta_{Q}

and

\Delta_{V}

are changes in unadjusted mean learning gain after restoration (expanded minus retained, in percentage points). A dash denotes a section in which that tutor was not studied.

Figure C.1: Observed and adjusted learning gains across GRE sections. Panels A–B show Quantitative and C–D show Verbal, with all 13 AI tutors and both reference conditions in each section, extending Figure 1 C . Observed means have 95% Student-

t

confidence intervals. To account for differences in starting scores (Table A.1 ), adjusted gains use the same pre-test score distribution for every tutor within a section: we average predicted post-test scores and subtract the section’s mean pre-test score. The regression includes a quadratic pre-test term and uses 95% HC3 intervals. Within each section, panels share the same horizontal scale and are ordered by adjusted gain. See Table A.3 for sample sizes.

Panel of Table 20 — Table C.1: Learning gains for the Combined comparison in Figure 1 C . Values are percentage points with 95% HC3 confidence intervals. The regression includes pre-test score, its quadratic term and section. $n$ is the number of study sessions. AI tutors are ordered by learning gain.

Domain	Condition	Gain relative to control [95% CI]
Quantitative domains
Data analysis	AI	5.7 [1.6, 9.9]
	Human	6.8 [0.2, 13.3]
Geometry	AI	3.1 [ $-$ 1.9, 8.2]
	Human	1.3 [ $-$ 7.3, 9.9]
Arithmetic	AI	6.2 [2.1, 10.2]

Appendix

Table C.2: Learning gains relative to control in all seven GRE domains. These estimates complement the mean domain gains in Figure 1 A and the complete AI tutor rankings in Figure C.2 (Section 3 ). Values are percentage-point differences from a regression with linear domain pre-test score, with pointwise 95% HC3 confidence intervals.

Figure C.2: Complete AI tutor rankings across the seven GRE domains. Bars show mean learning gains in percentage points within each domain for the students analyzed in Figure 1 A and Section 3 . The 13 AI tutors per domain are ordered independently within each domain; human and control results appear below the divider. Each horizontal axis starts one percentage point below its smallest mean, so bar lengths should not be compared across panels. See Table A.3 for sample sizes. The rankings are descriptive; see the domain-outcome data for intervals and all 105 estimates.

Figure C.3: AI tutoring gains relative to control across starting proficiency. All four pre-test score groups (Q1–Q4) are shown for Quantitative and Verbal. Points give the difference in learning gains between pooled AI and control from a regression with linear pre-test score within each group; whiskers are 95% HC3 confidence intervals. Q1–Q4 keep students with tied pre-test scores together. See Table B.1 for the reporting rules for intervals and tests. See Figure 1 B for overall learning gains.

Figure C.4: Quantitative differences among AI tutors vary jointly with subject matter and starting proficiency. All 13 AI tutors are shown across the four pre-test score groups in each Quantitative domain. Each cell gives the AI tutor’s learning gain relative to control in the same domain and score group, in percentage points. These estimates come from regressing post-test domain score on pre-test domain score and tutoring condition. Positive values favor AI tutoring; negative values favor control. Each AI cell contains 12–35 students, and the control groups contain 24, 22, 16, and 29 students, respectively. The title reports a separate test of whether differences among AI tutors vary jointly with domain and quartile, using CR1 uncertainty clustered by study session. The accompanying data include all 364 estimates by AI tutor, domain, and quartile, and their intervals, including Verbal. See Figure 1 C for combined tutor rankings and Figure C.3 for the pooled AI comparison.

Figure D.1: Section-specific inference cost per percentage point of learning gain. Panels show Quantitative and Verbal estimates corresponding to the pooled results in Figure 5 . The human tutor reference is

75 per hour. Points are the mean session cost divided by the mean learning gain; whiskers are 95% paired-bootstrap intervals. Panels rank tutors separately. Cost multiples use Gemma 4 31B in the same section as the

1\times$ reference; learning gains are in percentage points (pp).

Figure D.2: Learning gain versus tutoring cost and AI reply time by GRE section. Panels show Quantitative (left) and Verbal (right) frontiers corresponding to Figure 4 A, B . Cost and reply time axes are logarithmic. Blue points and dashed lines mark the AI Pareto frontier; gray points show other AI tutors. Red stars give the human reference at $75 per hour. See Section 5 and Appendix D.1 for cost and AI reply time definitions. Frontiers are descriptive and have no uncertainty intervals.

Figure D.3: Student engagement versus AI reply time. Panels show comparisons across AI tutors for Quantitative, Verbal and the 12 shared AI tutors, corresponding to Figure 4 C and Section 5 . For AI reply time, we first take the median duration of complete provider calls in each session, then the median across sessions for each AI tutor. Reply time axes are logarithmic; solid lines fit mean message counts against

\log_{10}

reply time. These are observational associations.

Figure D.4: The tutoring experience across AI tutors. Combined estimates of pace, dialogue, practice and student ratings for the 12 shared AI tutors. Fractions count how many of the other 11 tutors have significantly longer reply times or lower values on the other measures than the column leader (

p{<}.05

). The practice outcome is the percentage of distinct answered problems receiving final platform credit. See Section 7 for the process comparisons and Appendix D.1 for definitions and adjustment.

Predictor	Scope	$k$	Spearman $\rho$
Cost	Quantitative	13	$0.462$
Cost	Verbal	13	$0.335$
Cost	Combined	12	$0.378$
Reply time	Quantitative	13	$0.198$
Reply time	Verbal	13	$-0.159$
Reply time	Combined	12	$-0.007$

Appendix

Table D.1: Associations of cost, reply time and student engagement with mean learning gain across AI tutors, corresponding to Figures 4 A, B and D.3 . See Appendix D.1 for cost, reply time and engagement definitions.

k

is the number of AI tutors.

Section	Predictor / outcome	$\beta_{10}$	95% CI	$p$
Quant.	Reply time / student messages	$-5.19$	$[-7.09,-3.29]$	$<.001$
Quant.	Student messages / correct practice	$0.52$	$[0.29,0.75]$	$.002$
Quant.	Correct practice / learning gain	$5.62$	$[3.97,7.26]$	$<.001$
Verbal	Reply time / student messages	$-1.31$	$[-2.24,-0.39]$	$.860$
Verbal	Student messages / correct practice	$0.66$	$[0.29,1.04]$	$.101$
Verbal	Correct practice / learning gain	$2.27$	$[1.00,3.54]$	$.081$

Appendix

Table D.2: Student engagement, correct practice and learning. Selected exploratory associations in Figure 6 , using 1,137 Quantitative, 998 Verbal and 2,135 Combined sessions. See Section 5.2 for the definition of correct practice. Coefficients are outcome changes per ten seconds, student messages or credited problems; gain is in percentage points. Intervals are pointwise 95% HC3 intervals;

p

values follow Table B.1 .

Criterion	What the reviewer evaluates
A. Relevant concepts	Choosing concepts that address the student’s pre-test errors.
B. Concept grouping	Teaching related topics together.
C. Concept prioritization	Placing high-impact concepts earlier.
D. Time allocation	Dividing time according to errors, difficulty and likely benefit.
E. Practice alignment	Matching practice to the student’s errors and skill level.
F. Appropriate difficulty	Choosing problems or concepts suitable for what the student knows.

Appendix

Table E.1: Eight shared criteria for expert review of teaching materials. Letters A–H correspond to Figure E.1 . Figure 3 A pools preference ratings for A–D and H (lesson planning); Figure 3 B pools E–G (practice-problem creation and design). See Section 4.1 for evaluation details.

Evaluation	Ratings	Pre-tests	Original	Sensitivity
Lesson planning	9,577	381	9/12	8/12
Practice creation/design	5,265	380	9/12	8/12
All eight shared criteria	14,842	381	9/12	8/12

Appendix

Table E.2: Expert-ranking sensitivity to repeated reviewers. This checks the expert comparisons in Section 4.1 . Counts show how many version-specific AI tutor entries differ significantly from Opus 5 (

p{<}.05

). The original analysis clusters uncertainty by pre-test; the sensitivity clusters by reviewer and pre-test. Fitted scores are held fixed. Ratings marked as ties are excluded.

Figure E.1: Expert preference by teaching criterion and overall. A–H, the eight criteria in Table E.1 ; I–K, all eight fitted jointly for Quantitative, Verbal and both sections combined, respectively. These rankings expand Figure 3 A, B . Points show centered Bradley–Terry estimates; whiskers are pointwise 95% intervals clustered by student pre-test. For each panel, we report the number of student pre-tests and expert ratings. Ratings marked as ties are excluded. A, 371 pre-tests and 1,899 ratings; B, 370 pre-tests and 1,905 ratings; C, 372 pre-tests and 1,916 ratings; D, 376 pre-tests and 1,956 ratings; E, 377 pre-tests and 1,939 ratings; F, 365 pre-tests and 1,653 ratings; G, 366 pre-tests and 1,673 ratings; and H, 378 pre-tests and 1,901 ratings. Overall fits use 246, 135 and 381 pre-tests, with 8,206, 6,636 and 14,842 ratings, respectively. All panels share the combined AI tutor order; blank cells mean not evaluated. Scores are comparable only within panels.

Figure F.1: How tutors interact with students. Cells show the percentage of sessions matching each rule underlying the ranking in Figure 3 C . Dots show mean prevalence across the six indicators, with 95% Student-

t

intervals across sessions;

n

is the transcript count. The human reference uses spoken sessions, whereas AI sessions use typed chat. See Figure F.2 for Quantitative and Verbal results and Table F.1 for indicator definitions.

Indicator	Detection rule
Scaffolding cues	A tutor response following a student message of at least two words contains evaluative language, or shares at least two substantive words with that message and contains a scaffolding cue.
Request for explanation	At least one tutor question contains a request for reasoning or explanation.
Interactive questions	At least four tutor messages contain a question mark or an invitation to respond.
Early attempt request	At least one of the first five tutor messages asks the student to try or attempt a problem.
Long solution after a reply	After a student message of at least two words, a tutor message of at least 140 words includes a solution or calculation cue.
Reasoning checks	At least two tutor questions contain a reasoning-check or strategy cue.

Appendix

Table F.1: Operational definitions for the six transcript indicators in Figures 3 C , F.1 and F.2 .

Figure F.2: Conversational pedagogy by GRE section. A, Quantitative. B, Verbal. Panels expand the Combined ranking in Figure 3 C using the six rules in Table F.1 . Cells give the percentage of sessions satisfying each rule;

n

counts available transcripts. The right column shows the mean of the six indicators with 95% Student-

t

intervals across sessions. Human tutoring is a spoken reference and AI tutoring uses typed chat.

Figure F.3: How often students disputed a practice answer. A–C show flagged problems per 100 distinct answered problems in Quantitative, Verbal and Combined. Whiskers are 95% session-bootstrap intervals across 1,139 Quantitative, 1,000 Verbal and 2,139 Combined sessions. Horizontal scales differ to show the smaller Verbal rates. Rows share the Combined ordering. The Sonnet row pools its section-specific versions in Combined. These student reports complement the expert practice rankings in Figure 3 B ; they record student disagreements with answers, not independently verified errors.

Lesson plan / tutor	Opus 4.8	Gemini 3.5 Flash	GPT-5.5	All
Minimal / Minimal	15.23 (9)	17.78 (10)	13.56 (62)	14.27 (81)
Minimal / Expanded	14.81 (27)	8.50 (27)	12.65 (36)	12.06 (90)
Intermediate / Intermediate	11.62 (29)	2.06 (9)	11.40 (26)	10.19 (64)
Expanded / Minimal	8.38 (19)	2.65 (7)	23.23 (11)	11.71 (37)
Expanded / Expanded	8.02 (18)	11.85 (10)	8.26 (26)	8.85 (54)

Appendix

Table G.1: Prompt-selection evidence for the fixed-instruction design in Section 2 . All five Quantitative pilot settings are shown; these sessions are separate from the main learning results in Figure 1 . Cells give mean percentage-point gain, with the number of sessions in parentheses. The two terms in each row identify the lesson-plan and tutor prompts, respectively. AI tutor columns identify the deployed chat tutor; the lesson-plan generator differed in 41 sessions.

Figure G.1: Learning with minimal and expanded prompts. Supporting the prompt design in Section 2 , points show mean Quantitative learning gain with separate Student-

t

95% confidence intervals; counts on the right show the number of sessions. Both lesson-plan and tutor prompts are minimal for blue circles and expanded for orange diamonds. A, Three AI tutors in the exploratory prompt-selection pilot. This panel shows the two matched prompt settings; see Table G.1 for all five settings, including mixed and intermediate prompts. B, Two later AI tutors, comparing supplementary cohorts that used expanded prompts with main-study cohorts that used minimal prompts. Differences are descriptive: the pilot informed prompt selection, and the later prompt conditions were not concurrently randomized.

Explore similar work

May 27, 2026cs.CL

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors

Evaluating AI tutor responses requires more than factual correctness: tutors must identify mistakes, locate errors, provide guidance, and offer actionable next steps. We present GRADE, a systematic study of open-source models for pedagogical ability assessment in student-tutor dialogues. Building on the BEA 2025 TutorMind setting, we evaluate 120 configurations across five language models, zero-shot inference, LoRA fine-tuning, synthetic augmentation, CoT+Reasoning, and single-task versus multitask formulations. Gemma3-12B performs best for single-task evaluation, while Gemma3-27B in 8-bit precision is more reliable for multitask prediction. We find that augmentation helps models that struggle with the original data, verification adds limited gains despite higher cost, and CoT+Reasoning is more useful for synthetic data generation than direct classification. We further show that LoRA fine-tuning on structured classification objectives interferes with instruction-following behavior under thinking mode, redirecting generation away from the required evaluation format. Carbon analysis shows that model choice and reasoning mode substantially affect emissions. Overall, GRADE shows that carefully selected open-source LoRA pipelines can match or surpass proprietary and ensemble-based systems on key pedagogical dimensions, with code and data available at https://github.com/pvbgeek/GRADE.

Parth Bhalerao, Jeromy Chang, David Chou +1

Santa Clara University

Tutors Grading Pedagogical Frameworks

Ask PDF

Jun 17, 2026cs.CY

AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice

There exist numerous tutor training platforms. However, few provide AI-driven training and evaluation for human tutors based on real-life performance. We present an AI-driven system that assesses both open responses during training and authentic real-life tutoring. Unlike platforms that only assess learning through online training or simulations, our system utilizes Generative AI (Gemini-2.5-pro) to analyze transcriptions of authentic tutoring, measuring the transfer of tutor skills to real-life application. Human tutors instructing students remotely in math (N=86) completed six scenario-based lessons, averaging a significant 7.4% learning gain. Using mixed-effects models across 405 session-to-lesson pairs, we found that training performance significantly predicted real-life transcript scores with an effect size of 0.25 SD. Model comparison (AIC/BIC) indicated averaging open response and multiple choice performance during training predicted real-life tutor performance best, although open responses were comparatively more predictive. Exploratory analysis showed that after training, tutors were significantly more likely to encounter pedagogical opportunities to apply their skills (61.1% to 68.9%) and demonstrated higher execution quality within those opportunities (65.5% to 68.1%). Interrupted time series analysis suggested that these tutor improvements were part of a gradual trend over time rather than an immediate intervention effect of training. We illustrate an AI-driven method to link tutor training with real-life assessment. In doing so, we contribute open datasets, AI prompts, and scoring rubrics to support transparency and reproducibility.

Danielle R. Thomas, Marie Cynthia Abijuru Kamikazi, Clara Brandt +2

Carnegie Mellon University, Pittsburgh, PA, USA · Vanderbilt University, Nashville, TN, USA

Tutors Computerized Adaptive Testing Effect Sizes Open-Ended Responses

Ask PDF

Jul 12, 2026cs.CL

Knowledge Distillation for Automated AI Tutor Evaluation

The rapid integration of Large Language Models (LLMs) into K-12 and higher education has outpaced the development of reliable methods for evaluating their pedagogical quality. As the research community starts to explore the space of automating evaluation of AI tutors, we introduce FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors. Aligned with the four core evaluation tracks from the BEA 2025 Shared Task, our model assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability. Because pedagogical evaluation is a specialized task with limited labeled data, we leverage knowledge distillation from a frontier LLM to generate additional supervision, yielding absolute performance gains up to 22.63 percentage points. Finally, we demonstrate FATE's utility as an automated evaluator by benchmarking instructional responses generated by popular commercial models, including ChatGPT, Claude, Gemini, and DeepSeek. On average, we have found that Gemini 2.5 Flash perfomed best (82.88%), then ChatGPT 5.5 Instant (80.75%), followed by DeepSeek V4 Flash (80.13%) and Claude Sonnet 4.6 (74.00%).

Tahmid Al Hannan, Diego Garcia, Alex Njoroge +2

Folsom Lake College Folsom, CA, USA

Tutors Pedagogical Frameworks Knowledge Distillation

Ask PDF