Examining Variation in How Guided AI Tutors Resolve Student Impasses

Bakhtawar Ahtisham; Kirk Vanacore; Alessandra Napoli; Josh Arens; Ksenia Ionova; Clayton Cohn; Shima Salehi; Rene Kizilcec

cs.AISep 29, 2026

Examining Variation in How Guided AI Tutors Resolve Student Impasses

Authors: Bakhtawar Ahtisham, Kirk Vanacore, Alessandra Napoli, Josh Arens, Ksenia Ionova, Clayton Cohn, Shima Salehi, Rene Kizilcec

Organizations: Cornell University, Ithaca, NY, United States · Stanford University, Stanford, CA, United States

Abstract

When a student is stuck, a tutor faces the assistance dilemma: help given too early can hinder productive struggle, while help withheld too long leaves the student in a frustrating, persistent impasse (i.e., wheel spinning). Generative AI tutors increasingly use guardrails restricting answer-giving, yet little is known about how such tutors behave once an impasse persists. We analyze 20,462 student turns from 1,260 authentic sessions with a guided LLM chemistry tutor, identifying 6,630 impasse turns of three major types: conceptual errors, expressed uncertainty, or help-seeking. We then used these impasses to simulate three tutoring conditions to study variation in AI tutor guidance through impasses: baseline, no-direct-answer, and guided tutor. For a sample of 150 impasses, prompt specificity changed pedagogy: a baseline tutor provided the answer directly in 50.7% of responses, a no-direct-answer tutor asked a follow-up question every time, and the guided tutor responded in a wide variety of ways depending on the context. We then analyzed impasse trajectories in authentic interactions, finding that each additional impasse turn lowered the odds of next-turn recovery by 12.7% (AOR = 0.873, p < .001), and early dropouts were caught in recursive concept elicitation before reaching execution. The benefit of questioning decayed as impasses persisted (scripted question x depth AOR = 0.78; follow-up x depth AOR = 0.83), whereas addressing the student's error grew more beneficial (AOR = 1.14); after a failed scripted question, repeating it was followed by recovery in 28.1% of cases, compared with 39.8% when the tutor addressed the error instead. For learning analytics, these findings identify impasse depth and type as observable, turn-level dialogue signals that analytics can use to trigger graduated, state-sensitive assistance in real time.

Figures & tables

Category	Code	Definition
Student input	Direct answer or explanation	Response to a question and/or explanation of thinking.
	Answer with uncertainty	Response to a question expressed with doubt or uncertainty.
	Specific question	Question seeking specific information or asking about a specific step.
	General help-seeking	Expresses confusion or general uncertainty without referencing specific information or a step.
Correctness	Completely correct	Fully accurate and complete, with no errors or missing key information (defines recovery).
	Partially incorrect	Some components are correct, but at least one key idea, step, or piece of information is incorrect or missing.

Table 1: Codebook categories and codes used in the analyses. Student-input and correctness codes define impasse types and recovery (Section 3.4 ); tutor-output codes are the moves analyzed in RQ2 and RQ4.

Practice	Turns	%	Error %	Uncert. %	Help %
Relevant concepts	3,465	52.3	27.8	14.1	58.1
Strategy	1,197	18.1	44.8	38.8	16.5
Execution/solution	961	14.5	50.7	48.3	1.0
Information needed	508	7.7	46.3	35.6	18.1
Answer checking	345	5.2	3.8	95.1	1.2
Problem identification	110	1.7	51.8	25.5	22.7

Table 2: Distribution of impasse turns across problem-solving practices and the share of each impasse type within a practice (

N=6{,}630

Figure 1: Next-turn recovery by impasse depth, for all impasses and by impasse type. Error bars show 95% confidence intervals from a session-level cluster bootstrap (2,000 resamples); the number of impasse turns at each depth appears in the bottom panel. Hollow markers indicate fewer than 30 turns.

	% of responses
Tutor move	Baseline	No-Direct-Answer	Guided	$\boldsymbol{\chi^{2}(2)}$	Cramér’s $V$
Direct answer	50.7	0.0	19.3	109.6	.49
Partial answer / hint	43.3	0.0	39.3	86.2	.44
Follow-up question	62.7	100.0	35.3	140.9	.56
Address incorrect answer	37.3	33.3	16.0	18.8	.20
Confirm / elaborate	12.0	7.3	42.0	65.3	.38

Table 3: Prevalence of tutor moves across the three tutoring conditions for the same 150 impasses, with tests of between-conditions differences. Cells give the percentage of each condition’s responses that contained the move; because a single response could contain several moves, columns do not sum to 100%. Response diversity counts the distinct combinations of moves each condition produced and their Shannon entropy (higher values indicate more varied responses).

Figure 2: Pedagogical response profiles across tutor conditions and impasse types for the same 150 impasses ( $n=50$ per type). Cells show the percentage of responses containing each move; responses can contain several moves, so rows need not sum to 100%.

Field	Content
Problem Statement	Naturally occurring cobalt consists of only one isotope, Co-59, whose relative atomic mass is 58.9332 amu. A synthetic radioactive isotope of cobalt, Co-60, has a relative atomic mass of 59.9338 amu and is used in radiation therapy for cancer. A doctor prepares a 1.7516 g sample of cobalt that is a mixture of these two isotopes and has an apparent atomic mass of 58.9901 amu. What number of Co-60 atoms are in this sample?
Preceding tutor turn	“Great! Let’s connect the chemistry knowledge to the problem. How do you think the different atomic masses of Co-59 and Co-60 relate to the reported atomic mass of the cobalt sample?”
Student impasse	“its greater than the both of them?” (conceptual error: the apparent atomic mass is a weighted average, so it must lie between the two isotope masses)
Baseline AI Tutor	“That step is not correct. Here is the correction. Let $x$ be the fractional abundance of Co-60 and $1-x$ the abundance of Co-59. Solve $58.9901=58.9332(1-x)+59.9338x$ . Then find total cobalt atoms from 1.7516 g $\div$ 58.9901 g/mol $\times$ Avogadro’s number and multiply by $x$ to obtain the number of Co-60 atoms.” [Direct answer]
No-Direct-Answer AI Tutor	“Pause and test that idea before using it. If $x$ represents the fraction of Co-60, how would you express the Co-59 fraction? How can those two fractions and isotope masses produce the given apparent atomic mass?” [Follow-up question]
Guided AI Tutor	“Actually, the reported atomic mass of the cobalt sample, 58.9901 amu, is a weighted average of the atomic masses of Co-59 and Co-60. This means it falls between the two values. How do you think we can use this information to find the percentage abundances of Co-59 and Co-60 in the sample?” [Scripted Question; Address incorrect answer; Follow-up question]

Table 4: Responses of the three tutor conditions to the same conceptual-error impasse. Coded tutor moves are shown in brackets and italicized.

Figure 3: First-order practice-transition probabilities for (A) completers, (B) early dropouts, and (C) their difference $\Delta\mathbf{P}$ . Transitions into execution and answer checking are zero for dropouts by definition and are not interpreted.

Sequence (turn $t-1\rightarrow t$ )	$n$	Recovery	AOR	$p$
Onset $\rightarrow$ Scripted Q (ref.)	1,416	55.5%	–	–
Onset $\rightarrow$ Follow-up Q	901	42.1%	$0.71^{**}$	.001
Onset $\rightarrow$ Address incorrect	676	39.5%	$0.72^{**}$	.006
Scripted Q $\rightarrow$ Address incorrect	88	39.8%	0.78	.291
Scripted Q $\rightarrow$ Follow-up Q	259	37.5%	$0.69^{*}$	.018
Address incorrect $\rightarrow$ Address incorrect	202	35.6%	0.81	.245

Table 5: Model 2: Recovery following selected two-turn scaffolding sequences (

M_{t-1}\rightarrow M_{t}

), relative to a first scripted question at onset, adjusted for depth, impasse type, and practice. Bold rows mark the reference and the two sequences compared in the text (repeating vs. switching after a scripted question).

Figure 4: Average predicted next-turn recovery with each tutor move present versus absent, by impasse depth (Model 3). Labels give the difference in percentage points at depths 1 and 4.

Explore similar work

Sep 14, 2026cs.LG

Simulating Disengaged Students to Evaluate LLM-based Tutors

Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.

Xianghui Meng, Jionghao Lin

The University of Hong Kong, Hong Kong SAR, China

Tutors Student User Simulation Transition Learning

Ask PDF

Jun 14, 2026cs.AI

Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

A central pedagogical value evaluated in AI tutor benchmarks is scaffolding: guiding students through graduated steps toward a solution. Alignment and evaluation methods for embedding scaffolding behaviour into chatbots, however, rest on an implicit assumption: that students will take up the scaffolding and engage in the conversation. To examine whether this assumption holds, we introduce an evaluation pipeline around two metrics - Chatbot Scaffolding and Student Uptake - and apply them across nine datasets of 9,490 chats, spanning AI tutor benchmarks and real-world deployments of educational chatbots. Our analysis reveals that while benchmarks assume a high-scaffolding, high-student-uptake environment, students in real-world settings exhibit lower levels of uptake overall - frequently bypassing the chatbot's pedagogical framing to drive the interaction toward their own learning goals at little interpersonal cost. We argue that bypassing scaffolding is not necessarily detrimental; rather, it frequently highlights a mismatch between a chatbot's pedagogical framing and the student's learning goals. To meaningfully evaluate the effectiveness of a chatbot's assistance, future benchmarks must move beyond the assumption that students will simply take up the scaffolding, and instead evaluate how these chatbots navigate diverse learning contexts and student-driven interaction patterns.

Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer +2

Department of Mechanical Engineering, Imperial College London, UK · Department of Electrical Engineering, Imperial College London, UK · Department of Earth Science and Engineering, Imperial College London, UK

Tutors Chatbots Student Scaffolds

Ask PDF

Jun 22, 2026cs.AI

AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective

Generative AI tools provide novice programmers with instant, personalized support, but also raise concerns about whether AI use supports or bypasses students' regulation of problem-solving. Existing work has largely focused on correctness, usability, or overall usage frequency, with less attention to how student--AI help-seeking unfolds. This study addresses this gap by analyzing AI-assisted help-seeking trajectories in university-level programming. Using an SRL-informed analytical framework that links prompt-level help-seeking codes to conceptual, implementation, debugging, and reflective forms of support, we analyzed 1,290 task-specific student prompts linked to 17,190 code submissions from 71 students in introductory Python programming courses. Specifically, we examined how help-seeking interactions were structured across turns and attempts, and how trajectory patterns related to task scores and the number of code submissions. Results indicate that many students primarily used AI for reactive troubleshooting rather than for planned, self-regulated problem-solving. Although trajectory patterns were not associated with significant differences in task scores, they differed substantially in the number of code submissions required. These findings suggest that the educational significance of AI support lies not only in whether students use AI, but in how their help-seeking trajectories develop during programming problem-solving.

Boxuan Ma, Huiyong Li, Gen Li +3

Faculty of Arts and Science, Kyushu University, Fukuoka, Japan. · Research Institute for Information Technology, Kyushu University, Fukuoka, Japan. · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan. +1

Ai-Assisted Programming Tasks Artificial Intelligence Education Ai-Based

Ask PDF