cs.HCSep 24, 2026

Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming

Authors: Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor

Organizations: University of Florida Gainesville, FL, United States · University of Auckland Auckland, New Zealand

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects students' learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined students' perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use.

Figures & tables

Figure 1. Student-facing Interface of the Web-based Coding Platform Screenshot of the coding tool showing the problem statement, submission history, AI assistant, code editor, and test results.
 Context Awareness 
Feedback StyleFull ContextNo ContextRow Total
Direct Feedback373269
Socratic Feedback303363
Column Total6765132
Table 1. Participant Distribution across 2×22\times 2 Experimental Conditions ( N=132N=132 )
Experience DimensionMeasuresSurvey Item/Log MetricAnalysis Method
(a) PerceptionsTask completion †The [platform] chatbot helped me complete the tasks successfully.Kruskal-Wallis H test
 Task comprehension †The [platform] chatbot helped me better understand the problem statements.Kruskal-Wallis H test
 Correct information †The [platform] chatbot provided me with correct information.Kruskal-Wallis H test
 Stressful interaction †My experience using the [platform] chatbot was stressful.Kruskal-Wallis H test
(b) BehaviorsUsed another LLM #Did you use another LLM or chatbot?Chi-square test
 Reason for LLM use ‡Why did you use another LLM or chatbot? (if answered ‘Yes’ to previous question)Inductive coding
Table 2. Research Question Operationalization: Measures, Survey Items/Log Metrics, and Analysis Methods
Figure 2. Mean Student Ratings for AI TAs by Interaction of Context Awareness and Pedagogical Guidance Style (N=132)
Figure 3. Decision Flowchart for Task Comprehension Decision flowchart for task comprehension categories
Figure 4. Comprehension Level by Group Assignment Comprehension Level by Group Assignment

Explore similar work

CardsList
Jun 22, 2026cs.AI

AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective

Generative AI tools provide novice programmers with instant, personalized support, but also raise concerns about whether AI use supports or bypasses students' regulation of problem-solving. Existing work has largely focused on correctness, usability, or overall usage frequency, with less attention to how student--AI help-seeking unfolds. This study addresses this gap by analyzing AI-assisted help-seeking trajectories in university-level programming. Using an SRL-informed analytical framework that links prompt-level help-seeking codes to conceptual, implementation, debugging, and reflective forms of support, we analyzed 1,290 task-specific student prompts linked to 17,190 code submissions from 71 students in introductory Python programming courses. Specifically, we examined how help-seeking interactions were structured across turns and attempts, and how trajectory patterns related to task scores and the number of code submissions. Results indicate that many students primarily used AI for reactive troubleshooting rather than for planned, self-regulated problem-solving. Although trajectory patterns were not associated with significant differences in task scores, they differed substantially in the number of code submissions required. These findings suggest that the educational significance of AI support lies not only in whether students use AI, but in how their help-seeking trajectories develop during programming problem-solving.

Boxuan Ma, Huiyong Li, Gen Li +3

Faculty of Arts and Science, Kyushu University, Fukuoka, Japan. · Research Institute for Information Technology, Kyushu University, Fukuoka, Japan. · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan. +1

Ai-Assisted Programming TasksArtificial Intelligence EducationAi-Based
AskPDF
May 12, 2026cs.CY

An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration

Active learning is widely recognized as an effective approach for improving learning outcomes in introductory programming courses. However, insufficient instructional support often limits students' access to timely, personalized feedback, which is crucial for mastering foundational programming concepts. Although recent advances in AI, particularly large language models, offer scalable opportunities for feedback, concerns about explainability and reliability remain. In this paper, we present an AI-driven classroom assistant that leverages an explainable AI model to analyze student code, map logical errors to instructor-identified misconceptions, and deliver instructor-authored feedback, thereby grounding reliability in instructor-defined pedagogical knowledge. To evaluate the effectiveness of our framework, we conducted an expert evaluation to examine its alignment with instructor-verified feedback and deployed the system in a classroom setting to assess students' perceptions of its usability. Results indicate that the assistant can provide accurate, instructor-verified feedback to students while fostering a positive experience.

Muntasir Hoq, Griffin Pitts, Bradford Mott +6

North Carolina State University, Raleigh, NC, USA · University of California, Berkeley, CA, USA

Artificial Intelligence EducationAi-Assisted Programming TasksTutorsPedagogical FrameworksFeedbackActive Learning
AskPDF
May 7, 2026cs.CY

The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

Current Artificial Intelligence (AI)-based tutoring systems (AI tutors) are primarily evaluated based on the pedagogical quality of their feedback messages. While important, pedagogy alone is insufficient because it ignores a critical question: what do students actually do with the feedback they receive? We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data, which complements pedagogical assessment. We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback from an introductory undergraduate programming course to measure whether students act on tutor feedback and whether those actions are applied correctly. Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns that are not captured by pedagogy-only evaluation. Moreover, these engagement-based behavioral signals are more strongly associated with student perception of helpful feedback than pedagogical quality alone, providing a more complete and actionable picture of AI tutor performance.

Rose Niousha, Samantha Boatright Smith, Bita Akram +5

University of California, Berkeley, USA · North Carolina State University, USA · University of Pittsburgh, USA +1

TutorsArtificial Intelligence EvaluationStudentAgent Behavior Modeling
AskPDF