cs.HCSep 24, 2026

Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming

Authors: Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor

Organizations: University of Florida Gainesville, FL, United States · University of Auckland Auckland, New Zealand

Abstract

AI teaching assistants (AI TAs) backed by large language models (LLMs) and pedagogical guardrails are increasingly being integrated into programming courses, providing students with scalable access to hints, conceptual explanations, and code-level feedback. However, guardrails may also create friction. If students feel that the support provided is overly restrictive or poorly contextualized to their current progress, they may bypass approved tools for general-purpose LLMs. To investigate how AI TA design affects students' learning experiences, we conducted a randomized controlled trial with 132 students in an introductory programming course. Students completed three tasks related to code-writing and debugging and were randomly assigned to one of four AI TAs varied across two dimensions: pedagogical guidance style (Socratic vs. Direct instruction) and context awareness (no context vs. full context of the problem and student solution). We examined students' perceptions, interaction behaviors, and evidence of post-task comprehension. Students rated the Socratic AI TA with full context least favorably, reporting significantly lower perceived support for task completion. Descriptively, this condition also showed the highest observed interaction stress, the highest rate of external LLM use, and the lowest proportion of post-task explanations demonstrating full comprehension, though these differences were not statistically significant. These findings suggest that guardrailed AI TAs are not automatically better for learning. Instead, their effectiveness depends on how pedagogical guidance and contextual awareness are balanced in ways that students experience as useful, supportive, and worth continuing to use.

Figures & tables

Explore similar work

CardsList
  1. AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective

    Jun 22, 2026Boxuan Ma, Huiyong Li, Gen Li +3Ai-Assisted Programming TasksArtificial Intelligence Education

  2. An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration

    May 12, 2026Muntasir Hoq, Griffin Pitts, Bradford Mott +6Artificial Intelligence EducationAi-Assisted Programming Tasks

  3. The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

    May 7, 2026Rose Niousha, Samantha Boatright Smith, Bita Akram +5TutorsArtificial Intelligence Evaluation