TutorCaseBench: A Case-Grounded Benchmark for Evaluating LLM Tutors in Supporting Students with Challenging Behaviors

Yeon Su Park; Chanyoung Park; Tae Soo Kim; Seungju Kim; Hyunwoo Kim; Daho Jung; Jinsung Yoon; Juho Kim

cs.CLOct 22, 2026

TutorCaseBench: A Case-Grounded Benchmark for Evaluating LLM Tutors in Supporting Students with Challenging Behaviors

Authors: Yeon Su Park, Chanyoung Park, Tae Soo Kim, Seungju Kim, Hyunwoo Kim, Daho Jung, Jinsung Yoon, Juho Kim

Organizations: KAIST · Seoul National University · Korea Institute for Curriculum and Evaluation · Google Cloud AI Research ·

Abstract

LLMs are increasingly used as AI tutors, yet existing benchmarks overlook the challenging behaviors common among students with learning and behavioral difficulties. We introduce TutorCaseBench, a benchmark that grounds AI-tutor evaluation in qualitative case studies from pedagogy, educational psychology, and special education. Our pipeline converts documented student categories into case-specific scenarios, tutor-student dialogues, and rubrics grounded in source evidence. The benchmark contains 923 dialogues across five challenge types, evaluated through open-ended response generation (1,560 items) and multiple-choice selection (2,537 items). Across 11 state-of-the-art LLMs, even the strongest model satisfies a rubric criterion in only 55.8% of open-ended cases, while models are better at selecting appropriate strategies than generating situated responses. These results reveal a gap between intervention recognition and context-specific adaptation, especially for open-weight models. All data are publicly available at https://github.com/kixlab/TutorCaseBench.

Figures & tables

Figure 1: We introduce TutorCaseBench, a benchmark grounded in qualitative case studies for evaluating LLM tutors’ ability to respond to challenging student behaviors. Example from Park and Pinkelman (2017) .

Figure 2: Overview of the TutorCaseBench construction pipeline. Based on extracted evidence, we compose case-grounded tutor–student dialogues. Examples are adapted from Theodoridou and Koutsoklenis (2013) .

Figure 3: Rubric construction and evaluation process.

Figure 4: 4-option MCQ construction with cross-scenario distractors. Examples from Sander et al. (2010) , Lichtinger and Kaplan (2015) , Newman (2012) .

Figure 5: We use strategies from similar cases as references. Examples from Lack et al. (2014) , Collett (2023) , Alghamdi and Siddiqui (2016) , Cassidy et al. (2018) .

Model	Immediate ( $N{=}824$ )	Long-Term ( $N{=}736$ )	Overall ( $N{=}1{,}560$ )
Claude Opus 4.7	67.0	43.3	55.8
Claude Haiku 4.5	59.0	40.1	50.1
Gemini 3.1 Pro	67.5	35.2	52.2
Gemini 3 Flash	58.7	32.2	46.2
GPT-5.5	67.2	40.6	54.7
GPT-5.4 Mini	49.2	33.3	41.7

Table 1: Success rate of open-ended response (%)

Figure 6: Success rate for open-ended response across models broken down by challenge type.

Figure 7: Correlation between distinct 4-grams ratio and immediate response success rate across models.

Failure Mode	Description
Empathy/Praise	Comforts the student when the response should challenge their belief or behavior.
Generic Scaffolding	Applies a standard support plan without identifying the student’s central difficulty.
Over-Intervention	Steps in to correct or redirect when the tutor should step back instead.
Task Redesign Missed	Helps the student cope with a task that itself should be changed.

Table 2: Recurring failure modes in open-ended response evaluation missed by all models.

Model	Immediate ( $N{=}1{,}428$ )	Long-Term ( $N{=}1{,}109$ )	Overall ( $N{=}2{,}537$ )
Claude Opus 4.7	63.2	62.7	63.0
Claude Haiku 4.5	53.9	55.9	54.8
Gemini 3.1 Pro	61.6	63.8	62.6
Gemini 3 Flash	55.5	56.9	56.1
GPT-5.5	62.7	62.0	62.4
GPT-5.4 Mini	51.9	54.6	53.1

Table 3: Accuracy on 4-option MCQ cases (%).

Figure 8: Success rate across models for immediate response, with and without references.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Descriptors
Student (35 Total)	Behavior Problems
	Behavior Disorders
	Self Destructive Behavior
	Aggression
	Emotional Problems
	Emotional Disturbances

Appendix

Table 4: Descriptors used to retrieve candidate papers.

Figure 9: Binary MCQ accuracy across 11 models.

Group	Category	$N$	Imm.	LT	Overall
Overall	–	518	56.7	33.8	45.9
Region	North America	318	53.5	32.5	43.3
	Europe	106	64.5	33.1	50.3
	Asia	30	57.5	43.3	51.2
	Middle East	28	64.5	38.8	52.9
	Oceania	26	53.7	34.8	45.0

Appendix

Table 5: Open-ended success rates (%) by geographic region, education level, and gender, averaged across 11 LLMs.

N

denotes the number of student categories.

Figure 10: Human validation interface across five pipeline stages. The left panel shows an example evaluation screen, while the right summarizes validation questions. Example from Bruno et al. (2024) .

Output	Yes	Partly	No	Agreement
Student Categories	65.0	31.7	3.3	73.3
Evidence Excerpts	45.6	50.9	3.5	47.4
Challenge Types	76.3	–	23.7	84.2
Scenarios	61.1	34.8	4.0	61.6
Instructional Strategies	50.5	47.5	2.0	43.4
Dialogues	65.7	28.3	6.1	55.6

Appendix

Table 6: Human validation ratings and exact inter-rater agreement (%). Challenge types used a binary (0/1), and other outputs used a three-point scale (1–3). Agreement indicates identical ratings between evaluators.

Model	Immediate ( $N{=}824$ )	Long-Term ( $N{=}736$ )	Overall ( $N{=}1{,}560$ )
Claude Opus 4.7	35.8	20.9	28.8
Claude Haiku 4.5	29.1	18.4	24.1
Gemini 3.1 Pro	35.7	16.0	26.4
Gemini 3 Flash	29.1	14.7	22.3
GPT-5.5	35.7	19.4	28.0
GPT-5.4 Mini	23.4	15.1	19.5

Appendix

Table 7: Average criterion satisfaction rate (%)

Figure 11: Average criterion satisfaction rates across models broken down by challenge type.

Judge Pair	Case-level $\kappa$	Criterion-level $\kappa$	Spearman $\rho$
Gemini vs. GPT	0.75	0.77	0.96
Gemini vs. Claude	0.76	0.77	0.97
GPT vs. Claude	0.75	0.76	0.96

Appendix

Table 8: Pairwise agreement and ranking consistency across judges (

p<0.001

for all Spearman correlations).

Figure 12: Correlation between distinct- $n$ ( $n=3$ ) and immediate response success rate across models.

Figure 13: Correlation between distinct- $n$ ( $n=5$ ) and immediate response success rate across models.

Figure 14: Correlation between case-recovery rate and immediate response success rate across models.

Figure 15: MCQ accuracy across models broken down by the challenge type of students.

Figure 16: Distribution of model agreement on MCQ outcomes across all 2,537 items.

Figure 17: Model-agreement breakdown of MCQ outcomes by challenge type.

Figure 18: Success rate across models for long-term intervention plans, with and without references.

AI in Education

AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.

See all 262 papers

Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors
Jun 15, 2026Junyi Yao, Zihao Zheng, Baichuan LiLLM Evaluation Intelligent Tutoring Systems
Washington University in St. Louis, St. Louis, MO, USA · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX, USA
GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors
May 27, 2026Parth Bhalerao, Jeromy Chang, David Chou +1Intelligent Tutoring Systems Low-Rank Adaptation
Santa Clara University
EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics
Sep 11, 2026Jiaxu Zhao, Bahar Radmehr, Fares Fawzi +2Generative AI in Education Language Model Bias Evaluation
Swiss Federal Institute of Technology in Lausanne (EPFL), Switzerland

Jun 15, 2026J/K move · Enter open · S save

Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors

Junyi Yao, Zihao Zheng, Baichuan Li

Washington University in St. Louis, St. Louis, MO, USA · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX, USA

LLM Evaluation Intelligent Tutoring Systems Educational Assessment Generative AI in Education Education

Open paper AskPDF arXiv

Large language models are increasingly proposed as educational tutors, yet stronger task-solving ability does not necessarily imply stronger learning support. Motivated by recent calls to measure the social impact of NLP systems in practice, we study whether public LLM tutoring benchmarks distinguish learning-supportive behavior from mere answer production. We propose a lightweight diagnostic based on the gap between solving-oriented and pedagogy-oriented benchmark performance. Using public MathTutorBench leaderboard results, we show that these dimensions are only partially aligned: across eight publicly reported models, the correlation between solving and pedagogy composites is 0.421, and several models shift meaningfully in rank when evaluation moves from solving to pedagogy. We then analyze the public TutorBench sample and show that agency-relevant behaviors are explicitly encoded in benchmark rubrics, especially in active-learning settings that reward guiding questions, calibrated hints, and non-disclosive scaffolding. Together, these findings suggest that educational-impact evaluation should not treat task success as a sufficient proxy for learning support. We argue that public tutoring benchmarks can better support positive-impact evaluation by reporting solving-oriented and pedagogy-oriented scores separately and by making disclosure-sensitive, student-agency-preserving criteria more explicit.

Abstract

AI in Education

Explore similar work

Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors

EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics