Incremental Predictive Value of LLM Tutoring Interactions for Student Performance

Zeyu Dong; Mikael Kwame Kuada; Asa Holley Buchakjian-Tweedy; Ibrahem Aljabea

cs.CLOct 24, 2026

Incremental Predictive Value of LLM Tutoring Interactions for Student Performance

Authors: Zeyu Dong, Mikael Kwame Kuada, Asa Holley Buchakjian-Tweedy, Ibrahem Aljabea

Organizations: Crestwood Preparatory College, Toronto, Ontario, Canada · Hammond High Magnet School, Hammond, Louisiana, United States · Spaulding High School, Rochester, New Hampshire, United States · Louisiana State University, Baton Rouge, Louisiana, United States

Abstract

We evaluate whether semester-level interaction features from a large language model tutor improve assignment-score prediction beyond constant baselines. StudyChat provides 16,851 student-assistant exchanges from two offerings of an undergraduate artificial intelligence course, with outcomes matched to 175 of 203 student-semester records. Six fitted estimators and training-fold mean and median baselines share repeated nested cross-validation. After averaging each student's held-out predictions across ten repeats, Gradient Boosting achieves mean absolute error of 3.023 grade points, versus 3.138 for the median baseline. The paired difference is -0.114, with a conditional 95% bootstrap interval of [-0.245,0.013]. Representation changes and a semester-specific median yield similarly small gains. In the lowest observed-score quartile, however, boosting's error is 7.30 points, versus 8.07 for the median and 6.67 for the mean; Ridge has the lowest fitted-model error at 6.37 despite its worse overall error of 3.331. Scores concentrate near the ceiling: 160 students (91.4%) score above 90, and the lowest quartile contributes 62.1% of boosting's total absolute error. This retrospective comparison shows how baseline choice and error distribution affect predictive value, informing evaluation of student-support tools without establishing a learning effect.

Figures & tables

Figure 1. Study workflow. Student-semester aggregation precedes repeated nested prediction; imputation and applicable scaling are learned within training partitions. Baseline contrasts assess predictive value. Clustering and attributions are separate exploratory diagnostics, not validated learning styles or intervention rules. Small inset plots are schematic, not study results. The pipeline separates data preparation, feature construction, retrospective prediction, exploratory clustering, and held-out model diagnostics.

Feature	Operational definition
Prompt count	$P$ exchanges.
Conversation count	$C$ distinct threads.
Dialogue depth	Mean maximum non-system history count per chat.
Prompt length	$W/P$ regex tokens.
Vocabulary diversity	$U/W$ .
Follow-up ratio	Percentage with interactionCount >0.

Table 1. Primary features.

P

: exchanges;

C

: chats;

W

: prompt tokens;

U

: distinct prompt tokens. Tokens match lowercase [a-z0-9’]+ . Percentages use the 0–100 scale.

Estimator	Candidates
Ridge	25 log-spaced $\alpha$ values in $[10^{-4},10^{4}]$ .
Elastic Net	20 log-spaced $\alpha$ values in $[10^{-4},10]$ ; $L_{1}$ ratio $\{0.1,0.25,0.5,0.75,0.9,1\}$ .
Random Forest	Squared/absolute-error criterion; depth $\{2,3,4,5,\mathrm{unlimited}\}$ ; minimum split $\{4,8,12,20\}$ ; leaf $\{2,3,5,8,12\}$ ; feature sampling sqrt or $\{0.5,0.75,1\}$ ; sample fraction $\{0.6,0.8,1\}$ .
Extra Trees	Squared error; same depth, split, leaf, and feature choices as Random Forest; no sample-fraction search.
Gradient Boosting	Squared/Huber/absolute-error loss; trees $\{50,100,200,300\}$ ; rate $\{0.01,0.03,0.05,0.1\}$ ; depth $\{1,2,3\}$ ; leaf $\{3,5,8,12\}$ ; subsample $\{0.6,0.8,1\}$ ; features sqrt or $\{0.5,1\}$ .
MLP	Layers $(8)$ , $(16,8)$ , $(32,16)$ ; dropout $\{0,0.1,0.2\}$ ; $L_{2}$ $\{10^{-4},10^{-3},10^{-2}\}$ ; rate $\{3\times 10^{-4},10^{-3},3\times 10^{-3}\}$ .

Table 2. Available search specification. Random Forest and Extra Trees use 500 trees. Resolved budgets and selected parameters are saved; the full historical sampled-candidate trace and dirty source are not archived. Boosting’s search definition is available in later versioned source and agrees with selected parameters, not a certified original source snapshot.

Model	Fold MAE	Fold RMSE	Fold $R^{2}$	Student MAE	Paired $\Delta$ [95% interval]
Mean dummy	3.446	5.713	$-0.147$	3.440	$+0.303\ [+0.113,+0.485]$
Median dummy	3.141	5.758	$-0.082$	3.138	—
Ridge	3.356	5.578	$-0.113$	3.331	$+0.193\ [-0.052,+0.442]$
Elastic Net	3.383	5.601	$-0.123$	3.344	$+0.206\ [-0.022,+0.440]$
Random Forest	3.071	5.523	$-0.023$	3.038	$-0.100\ [-0.243,+0.043]$
Extra Trees	3.310	5.547	$-0.106$	3.282	$+0.144\ [-0.083,+0.374]$

Table 3. All eight estimators on 50 shared folds and 175 matched records. Student MAE scores ten-repeat-averaged predictions. Paired differences use the median dummy and conditional 95% percentile intervals. All six learner intervals include zero. All displayed metrics are rounded once from their underlying values to three decimals.

Model	Overall MAE	Q1	Q2	Q3	Q4
Mean dummy	3.440	6.669	0.673	2.405	3.965
Median dummy	3.138	8.065	0.799	0.996	2.551
Ridge	3.331	6.368	0.879	1.936	4.107
Elastic Net	3.344	6.445	0.771	2.015	4.108
Random Forest	3.038	7.282	0.832	1.035	2.899
Extra Trees	3.282	6.418	0.762	1.960	3.946

Table 4. Student-averaged MAE by observed-score quartile for all estimators. Counts are 45, 44, 44, and 42; their weighted errors reproduce overall MAE.

Figure 2. Paired Q1 contrasts for all six learners on the same 45 records. Circles compare with the median; squares compare with the mean. Lines are conditional 95% percentile intervals from 5,000 paired within-stratum draws. Negative differences favor the learner. Strata are defined after observing outcomes; intervals are not multiplicity-adjusted. All model-minus-median intervals are negative. Against the mean, three intervals cross zero and three are positive.

Estimator	Fall to Spring	Spring to Fall
Mean dummy	3.385	3.356
Median dummy	3.413	2.775
Ridge	3.331	3.046
Elastic Net	3.377	3.077
Random Forest	3.293	2.895
Extra Trees	3.346	3.101

Table 5. Transfer MAE within the same course. The better constant is the mean for Fall-to-Spring and median for Spring-to-Fall. These are point estimates.

Feature	Pearson $r$	Spearman $\rho$	Adjusted $q$	Interval for $r$
Contextual-question ratio	$-0.329$	$-0.162$	0.0001	$[-0.534,-0.106]$
Active calendar days	0.248	0.220	0.0067	$[0.148,0.335]$
Vocabulary diversity	$-0.199$	$-0.082$	0.0315	$[-0.385,0.003]$
Conversation count	0.197	0.250	0.0315	$[0.101,0.282]$

Table 6. The four associations retained by the 14-test Pearson screen on 175 records. Adjusted

q

values apply only to that screen. Pearson intervals use 2,000 student resamples; Spearman coefficients provide a complementary rank-based summary. Vocabulary diversity’s interval includes zero despite its adjusted parametric test, illustrating the different assumptions.

Figure 3. Held-out attribution magnitudes for 14 predictors. Each bar is the mean absolute ten-repeat-averaged contribution over 175 records; conditional 95% percentile intervals use 2,000 student resamples. The ordering describes the selected boosting function under tree-path-dependent TreeSHAP and correlated inputs. Contextual questions, conversation count, and questions per chat have the largest absolute SHAP contributions.

Educational Data Mining

Educational Data Mining studies computational methods for analyzing learning data to understand learners, improve instruction, and support educational decisions.

See all 74 papers

LLM Pedagogical Behavior in AI Tutoring Interactions
Aug 24, 2026Suhyeon Lee, Juneha Baek, Jaehyeong Park +1Instructional Scaffolding AI in Education
KAIST
Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors
Jun 15, 2026Junyi Yao, Zihao Zheng, Baichuan LiLLM Evaluation Intelligent Tutoring Systems
Washington University in St. Louis, St. Louis, MO, USA · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX, USA
StudentBench: AI and human tutoring yield equivalent GRE learning gains
Sep 23, 2026Curtis Northcutt, Inaara Hasmani, Kevin Feng +3Intelligent Tutoring Systems Generative AI in Education

Aug 24, 2026J/K move · Enter open · S save

LLM Pedagogical Behavior in AI Tutoring Interactions

Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin

KAIST

Instructional Scaffolding AI in Education Intelligent Tutoring Systems

Open paper AskPDF arXiv

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

Abstract

Educational Data Mining

Explore similar work

LLM Pedagogical Behavior in AI Tutoring Interactions

Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors

StudentBench: AI and human tutoring yield equivalent GRE learning gains