Examining Variation in How Guided AI Tutors Resolve Student Impasses

Bakhtawar Ahtisham; Kirk Vanacore; Alessandra Napoli; Josh Arens; Ksenia Ionova; Clayton Cohn; Shima Salehi; Rene Kizilcec

cs.AISep 29, 2026

Examining Variation in How Guided AI Tutors Resolve Student Impasses

Authors: Bakhtawar Ahtisham, Kirk Vanacore, Alessandra Napoli, Josh Arens, Ksenia Ionova, Clayton Cohn, Shima Salehi, Rene Kizilcec

Organizations: Cornell University, Ithaca, NY, United States · Stanford University, Stanford, CA, United States

Abstract

When a student is stuck, a tutor faces the assistance dilemma: help given too early can hinder productive struggle, while help withheld too long leaves the student in a frustrating, persistent impasse (i.e., wheel spinning). Generative AI tutors increasingly use guardrails restricting answer-giving, yet little is known about how such tutors behave once an impasse persists. We analyze 20,462 student turns from 1,260 authentic sessions with a guided LLM chemistry tutor, identifying 6,630 impasse turns of three major types: conceptual errors, expressed uncertainty, or help-seeking. We then used these impasses to simulate three tutoring conditions to study variation in AI tutor guidance through impasses: baseline, no-direct-answer, and guided tutor. For a sample of 150 impasses, prompt specificity changed pedagogy: a baseline tutor provided the answer directly in 50.7% of responses, a no-direct-answer tutor asked a follow-up question every time, and the guided tutor responded in a wide variety of ways depending on the context. We then analyzed impasse trajectories in authentic interactions, finding that each additional impasse turn lowered the odds of next-turn recovery by 12.7% (AOR = 0.873, p < .001), and early dropouts were caught in recursive concept elicitation before reaching execution. The benefit of questioning decayed as impasses persisted (scripted question x depth AOR = 0.78; follow-up x depth AOR = 0.83), whereas addressing the student's error grew more beneficial (AOR = 1.14); after a failed scripted question, repeating it was followed by recovery in 28.1% of cases, compared with 39.8% when the tutor addressed the error instead. For learning analytics, these findings identify impasse depth and type as observable, turn-level dialogue signals that analytics can use to trigger graduated, state-sensitive assistance in real time.

Figures & tables

Category	Code	Definition
Student input	Direct answer or explanation	Response to a question and/or explanation of thinking.
	Answer with uncertainty	Response to a question expressed with doubt or uncertainty.
	Specific question	Question seeking specific information or asking about a specific step.
	General help-seeking	Expresses confusion or general uncertainty without referencing specific information or a step.
Correctness	Completely correct	Fully accurate and complete, with no errors or missing key information (defines recovery).
	Partially incorrect	Some components are correct, but at least one key idea, step, or piece of information is incorrect or missing.

Table 1: Codebook categories and codes used in the analyses. Student-input and correctness codes define impasse types and recovery (Section 3.4 ); tutor-output codes are the moves analyzed in RQ2 and RQ4.

Practice	Turns	%	Error %	Uncert. %	Help %
Relevant concepts	3,465	52.3	27.8	14.1	58.1
Strategy	1,197	18.1	44.8	38.8	16.5
Execution/solution	961	14.5	50.7	48.3	1.0
Information needed	508	7.7	46.3	35.6	18.1
Answer checking	345	5.2	3.8	95.1	1.2
Problem identification	110	1.7	51.8	25.5	22.7

Table 2: Distribution of impasse turns across problem-solving practices and the share of each impasse type within a practice (

N=6{,}630

Figure 1: Next-turn recovery by impasse depth, for all impasses and by impasse type. Error bars show 95% confidence intervals from a session-level cluster bootstrap (2,000 resamples); the number of impasse turns at each depth appears in the bottom panel. Hollow markers indicate fewer than 30 turns.

	% of responses
Tutor move	Baseline	No-Direct-Answer	Guided	$\boldsymbol{\chi^{2}(2)}$	Cramér’s $V$
Direct answer	50.7	0.0	19.3	109.6	.49
Partial answer / hint	43.3	0.0	39.3	86.2	.44
Follow-up question	62.7	100.0	35.3	140.9	.56
Address incorrect answer	37.3	33.3	16.0	18.8	.20
Confirm / elaborate	12.0	7.3	42.0	65.3	.38

Table 3: Prevalence of tutor moves across the three tutoring conditions for the same 150 impasses, with tests of between-conditions differences. Cells give the percentage of each condition’s responses that contained the move; because a single response could contain several moves, columns do not sum to 100%. Response diversity counts the distinct combinations of moves each condition produced and their Shannon entropy (higher values indicate more varied responses).

Figure 2: Pedagogical response profiles across tutor conditions and impasse types for the same 150 impasses ( $n=50$ per type). Cells show the percentage of responses containing each move; responses can contain several moves, so rows need not sum to 100%.

Field	Content
Problem Statement	Naturally occurring cobalt consists of only one isotope, Co-59, whose relative atomic mass is 58.9332 amu. A synthetic radioactive isotope of cobalt, Co-60, has a relative atomic mass of 59.9338 amu and is used in radiation therapy for cancer. A doctor prepares a 1.7516 g sample of cobalt that is a mixture of these two isotopes and has an apparent atomic mass of 58.9901 amu. What number of Co-60 atoms are in this sample?
Preceding tutor turn	“Great! Let’s connect the chemistry knowledge to the problem. How do you think the different atomic masses of Co-59 and Co-60 relate to the reported atomic mass of the cobalt sample?”
Student impasse	“its greater than the both of them?” (conceptual error: the apparent atomic mass is a weighted average, so it must lie between the two isotope masses)
Baseline AI Tutor	“That step is not correct. Here is the correction. Let $x$ be the fractional abundance of Co-60 and $1-x$ the abundance of Co-59. Solve $58.9901=58.9332(1-x)+59.9338x$ . Then find total cobalt atoms from 1.7516 g $\div$ 58.9901 g/mol $\times$ Avogadro’s number and multiply by $x$ to obtain the number of Co-60 atoms.” [Direct answer]
No-Direct-Answer AI Tutor	“Pause and test that idea before using it. If $x$ represents the fraction of Co-60, how would you express the Co-59 fraction? How can those two fractions and isotope masses produce the given apparent atomic mass?” [Follow-up question]
Guided AI Tutor	“Actually, the reported atomic mass of the cobalt sample, 58.9901 amu, is a weighted average of the atomic masses of Co-59 and Co-60. This means it falls between the two values. How do you think we can use this information to find the percentage abundances of Co-59 and Co-60 in the sample?” [Scripted Question; Address incorrect answer; Follow-up question]

Table 4: Responses of the three tutor conditions to the same conceptual-error impasse. Coded tutor moves are shown in brackets and italicized.

Figure 3: First-order practice-transition probabilities for (A) completers, (B) early dropouts, and (C) their difference $\Delta\mathbf{P}$ . Transitions into execution and answer checking are zero for dropouts by definition and are not interpreted.

Sequence (turn $t-1\rightarrow t$ )	$n$	Recovery	AOR	$p$
Onset $\rightarrow$ Scripted Q (ref.)	1,416	55.5%	–	–
Onset $\rightarrow$ Follow-up Q	901	42.1%	$0.71^{**}$	.001
Onset $\rightarrow$ Address incorrect	676	39.5%	$0.72^{**}$	.006
Scripted Q $\rightarrow$ Address incorrect	88	39.8%	0.78	.291
Scripted Q $\rightarrow$ Follow-up Q	259	37.5%	$0.69^{*}$	.018
Address incorrect $\rightarrow$ Address incorrect	202	35.6%	0.81	.245

Table 5: Model 2: Recovery following selected two-turn scaffolding sequences (

M_{t-1}\rightarrow M_{t}

), relative to a first scripted question at onset, adjusted for depth, impasse type, and practice. Bold rows mark the reference and the two sequences compared in the text (repeating vs. switching after a scripted question).

Figure 4: Average predicted next-turn recovery with each tutor move present versus absent, by impasse depth (Model 3). Labels give the difference in percentage points at depths 1 and 4.

Explore similar work

Simulating Disengaged Students to Evaluate LLM-based Tutors
Sep 14, 2026Xianghui Meng, Jionghao LinTutors Student
The University of Hong Kong, Hong Kong SAR, China
Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments
Jun 14, 2026Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer +2Tutors Chatbots
Department of Mechanical Engineering, Imperial College London, UK · Department of Electrical Engineering, Imperial College London, UK · Department of Earth Science and Engineering, Imperial College London, UK
AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective
Jun 22, 2026Boxuan Ma, Huiyong Li, Gen Li +3Ai-Assisted Programming Tasks Artificial Intelligence Education
Faculty of Arts and Science, Kyushu University, Fukuoka, Japan. · Research Institute for Information Technology, Kyushu University, Fukuoka, Japan. · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan. +1

Sep 14, 2026, cs.LGJ/K move · Enter open · S save

Simulating Disengaged Students to Evaluate LLM-based Tutors

Xianghui Meng, Jionghao Lin

The University of Hong Kong, Hong Kong SAR, China

Tutors Student User Simulation Transition Learning

Open paper AskPDF arXiv

Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.

Abstract

Explore similar work

Simulating Disengaged Students to Evaluate LLM-based Tutors

Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective