Organizations: Computer Science, North Carolina State University, Raleigh, NC, USA · School of Education and Human Development, University of Virginia, Charlottesville, VA, USA · Computer Science, Georgia Institute of Technology, Atlanta, GA, USA
AI-based chatbots have the potential to accelerate learning and teaching, but may also have counterproductive consequences without thoughtful design and scaffolding. To better understand teachers' perspectives on large language model (LLM)-based chatbots, we conducted a study with 11 teams of middle school teachers using chatbots for a science and computational thinking activity within a block-based programming environment. Based on a qualitative analysis of audio transcripts and chatbot interactions, we propose three profiles: explorer, frustrated, and mixed, that reflect diverse scaffolding needs. In their discussions, we found that teachers perceived chatbot benefits such as building prompting skills and self-confidence alongside risks including potential declines in learning and critical thinking. Key design recommendations include scaffolding the introduction to chatbots, facilitating teacher control of chatbot features, and suggesting when and how chatbots should be used. Our contribution informs the design of chatbots to support teachers and learners in middle school coding activities.
Figures & tables
Type
Gender
Subjects
Skills
Lead: n=8
8 female
6 Sci, 1 Math 1 CS, 1 ELA
8 Scratch+Snap! 7 code.org
Participants:n=17
12 female 5 male
10 Sci, 3 Math 1 CS, 3 other
11 Scratch 3 code.org
Table 1. Demographics of Teachers in LLM study Table 1: Participant Information This table provides an overview of the participant demographics for the study. It is divided into two sections: Lead and Participants. For the Lead group, there are 8 female participants with a background in various subjects: 6 in science, 1 in math, 1 in computer science (CS), and 1 in English language arts (ELA). They have proficiency in programming tools like Scratch+Snap! (8 leads) and code.org (7 leads). In the Participants group, there are 17 individuals, including 12 females and 5 males. Their subject expertise includes 10 in science, 3 in math, 1 in CS, and 3 in other areas. Their programming skills include 11 participants with experience in Scratch, 3 with experience in code.org, and no additional tools or languages specified.
Figure 1. Stax.fun Platform overview and its chatbot tabs and features a four-tabbed modal window with different LLM options overlayed on the scratch programming environment
Figure 3. Q&A Tab offering real-time answers, explanations, or clarifications Q&A Tab offering real-time answers, explanations, or clarifications
Figure 4. Coach Tab offering hints, tips, and suggestions Coach Tab provides step-by-step guidance, offering hints, tips, and suggestions
Figure 5. Prompt Tab refine and structure the prompt to align with the desired outcome refine and structure the prompt to align with the desired outcome
Figure 6. Process of the study with stax.fun chatbot A four sequence flow chart: set-up, introduction (10 mins), chatbot activity(50 mins), discucssion (15mins)
Positive Emotion
Strategies/Behavior The pair of teachers …
Examples of Quotes
Excited
♠Express enthusiasm about the chatbot’s functionality ♠Exhibit excitement by engaging in multiple trials very quickly • Discuss how the chatbot would be useful to them or students • Show marked interest in using the chatbot to promote student learning • Eagerness in seeing the running code • Show excitement in testing various features • Show interest by comparing multiple waves’ visual behavior
♠“The chatbot functions look very nice” • “I’m really excited that it worked!” • “If we were doing this with kids or teachers, we could use the Coach tab to walk them through the steps” • “I like that it helps focus the learning.” • “We could add a third sprite so that when you click it, it gets absorbed by the third material.”
Confident
→ Demonstrate a strong understanding of the task • See the correct and expected output with confidence ♠Quickly navigate through options; follow prompts without hesitation
→ “What needs to be done here is…” → “I’ve got this..” • “Exactly what I was expecting..” • “Clearly light passes through the glass more easily” ♠“This should be prompted with the coach tab…”
Curious
♠Interacts and prompts the chatbot → Starts project smoothly; showed curiosity in using instructions and other times trying independently • Acts intuitively while exploring the chatbot’s features • Shows enthusiasm and curiosity in materials • Shows curiosity by switching between the tabs • Indicates desire to understand the functionality
♠“Loaded and ready, let’s dive in” ♠“Is it possible to tweak the access?” → “Like to know how my students use the bot with instructions” • “What’s the difference between Coach and Q&A here?” • “Let’s see how this next tab works?”
Exploring
♠Delve into the different functions and buttons ♠Debugs using trial-and-error • Engages with the outcome of replacing different materials • Finds driver-navigator coding interesting • Tests when conditionals are required • Struggles with typing out the message in the prompt box and try to solve the challenges • Compare the effectiveness of different materials • Trying different examples in prompting and observe the result • Show more goal oriented behavior • Debugs by the most experienced person, no help of chatbot (debugging by testing and adjusting the code structure until it worked)
♠“I’ll change the prompt and see how response will change?” ♠“Which tabs have the best response?” • “I swap mirror for glass, does the wave still reflect?” • “If I flip the direction and it reverses, we’ve found the issue” • “Is 180 degrees the reverse direction in bot?”
Table 2. Positive Emotions; Strategies/Behavior and related Quotes. Legend: ♠ L1 = User Interface & system-level interaction; • L2 = Computational Thinking & Scientific reasoning; → L3 = Instructional guidance. This table presents the relationship between positive emotions, the strategies and behaviors of teachers, and related quotes from the study. Emotions include Excited, Confident, Curious, and Exploring.
Neutral Emotion
Strategies/Behavior The pair of teachers …
Examples of Quotes
Satisfied
→ Find their steps in Coding are true ♠Run the code smoothly ♠Code reuse via duplication instead of re-prompting chatbot • Enjoyed watching the outcome of the chatbot
→ “This is cool. Look at me go!” → “We knew what we were doing, and the chatbot made it even more fun.” → “I wanted to figure it out on my own—and I did.
Content
• Understands the interface → Struggles with the codes without any challenge → Reads/ follows the instructions together
• “No issues here, it’s working as expected.” → “Okay, that clicked right away.” → “You scroll, I’ll call out the next instruction.”
Bored
♠low engagement with chatbot ♠Not struggling ♠Long pauses / low interaction ♠Low relevance in UX appearance resizing • Dragging of blocks with no new idea or logic • → Find following the instructions tedious
♠“No need to ask the bot; I’ll just copy.” ♠“I can do it myself.” → “Instructions feels like a chore”
Table 3. Neutral Emotions; Strategies/Behavior and related Quotes. Legend: ♠ L1 = User Interface & system-level interaction; • L2 = Computational Thinking & Scientific reasoning; → L3 = Instructional guidance. This table presents the neutral emotions expressed by teachers and their associated strategies, behaviors, and example quotes during interaction with the chatbot. The table is divided into three sections: Neutral Emotion, Strategies/Behavior, and Examples of Quotes. Satisfied: Teachers felt satisfied when their code ran smoothly and when they successfully reused code without needing to re-prompt the chatbot. They also enjoyed watching the outcome of the chatbot's response. Example quotes include: "This is cool. Look at me go!" and "I wanted to figure it out on my own—and I did." Content: Teachers in this category understood the interface and followed the instructions without difficulty, though they sometimes struggled with the code. Example quotes include: "No issues here, it's working as expected" and "You scroll, I'll call out the next instruction." Bored: Teachers displayed low engagement with the chatbot and showed little struggle, but expressed dissatisfaction with the task. They experienced long pauses or low interaction, and the task felt overly simple or tedious. Example quotes include: "I can do it myself" and "Instructions feel like a chore." The table also includes a legend: Diamond represents user interface and system-level interaction (L1). Bullet list represents computational thinking and scientific reasoning (L2). Arrow list represents instructional guidance (L3).
Negative Emotion
Strategies/Behavior The pair of teachers …
Examples of Quotes
Unsatisfied
→ Trouble finding code blocks → Need a minimize button for chatbot → Chatbot takes too much space; need to hide it → Interface problem: Struggling to resize chat window
→ “Where did the prompt go?, oh it was in other tab” → "It was not a good experience I won’t use it" → “I can’t resize the chatbox height"
Confused
• Not understanding unexpected results • Unclarity in the expected chatbot response ♠Prompt disappears when switching tabs ♠Pressed Enter; message didn’t send (must click Send) ♠Confusion on onboarding and task-framing (Didn’t know the goal (“get to know the ♠chatbot” vs wave activity) ♠Unclarity in finding what the issue is after several attempts Unsure whether costume saved automatically
• “I bet the wave is supposed to stop at the mirror… is that what’s happening?” • “What happens if I pull …does it still work?” ♠“Where did the prompt go?, oh it was in other tab” ♠“Is there more text below? How do I scroll the Coach tab?” ♠“Can I ask the chatbot where to find the block and still figure it out myself?” ♠“Is this pseudocode for non-blocks too, or what am I supposed to follow?”
Frustrated
• Repeated retries without progress multiple time • Left coding activity incomplete? ♠Enter didn’t send the prompt ♠Reliance on researchers, undermining chatbot trust ♠Couldn’t find needed blocks ♠Teachers disliked chatbot code ♠Lowered perceived usefulness of the chatbot/code for classrooms ♠Chatbot UI got in the way, or generated code felt unusable → Following instructions and getting back was tedious
♠"keep dragging this chat box out of the way—why won’t it stay put? I can’t even find the new code” ♠“This is frustrating: the chat takes over the screen, the scripts are all over the place, and I’m hunting for code that should be right there. → “If a sixth grader saw this, they wouldn’t understand it, and honestly, with the chat in .way and no blocks showing, neither do I.”
Table 4. Negative Emotions; Strategies/Behavior and related Quotes. Legend: ♠ L1 = User Interface & system-level interaction; • L2 = Computational Thinking & Scientific reasoning; → L3 = Instructional guidance This table presents the negative emotions expressed by teachers, along with their corresponding strategies, behaviors, and example quotes. The table is divided into three sections: Negative Emotion, Strategies/Behavior, and Examples of Quotes. Unsatisfied: Teachers felt unsatisfied when they had trouble finding code blocks, needed a minimize button for the chatbot, or struggled with the interface (e.g., resizing the chatbot window). Example quotes include: "Where did the prompt go, oh it was in another tab" and "It was not a good experience I won’t use it." Confused: Teachers exhibited confusion when they didn’t understand unexpected results or when the chatbot didn’t respond as expected. Issues included unclear instructions, disappearing prompts, and confusion during onboarding. Example quotes include: "What happens if I pull… does it still work?" and "Where did the prompt go, oh it was in another tab." Frustrated: Teachers experienced frustration due to repeated retries without progress, difficulty finding blocks, and dissatisfaction with the chatbot. They also felt undermined by reliance on researchers and had issues with chatbots in classrooms. Example quotes include: "Keep dragging this chat box out of the way—why won’t it stay put? I can’t even find the new code" and "This is frustrating: the chat takes over the screen, the scripts are all over the place, and I’m hunting for code that should be right there." The table also includes a legend: Diamond represents user interface and system-level interaction (L1). Bullet list represents computational thinking and scientific reasoning (L2). Arrow list represents instructional guidance (L3).
Figure 7. Bubbles encode the number of coded utterances (size); color shows valence (positive/negative/neutral). Large negative bubbles suggest intervention points. This figure shows a bubble chart where each bubble represents the frequency of coded emotional expressions during various activities. The x-axis lists 11 activities, such as getting familiar with the chatbot, coding tasks, and a plastic bag activity. The y-axis categorizes emotions into three sentiment types: Positive, Neutral, and Negative. Each emotion within these categories (e.g., Excited, Confused, Satisfied) is represented by bubbles that vary in size, indicating the number of coded utterances. The color scale ranges from negative emotions to positive emotions, with neutral emotions in between. Larger negative bubbles suggest potential intervention points. The bubble size also encodes the amount of time spent on each activity.
Figure 8. Counts of coded utterances per stage; negative (left), neutral (centered), positive (right), enabling comparison of affect balance across activities. This bar chart presents the frequencies of negative, neutral, and positive emotions across different activities during the study. The horizontal axis represents frequency, with negative values indicating negative emotions, zero representing neutral emotions, and positive values representing positive emotions. The vertical axis lists the activities. Each activity includes bars for negative, neutral, and positive emotion frequencies.
Table 5. Observable Personas, Time Spent, Affect Trajectories, Strategies, and Help Seeking This table presents the characteristics of four observable personas, including the number of groups representing each persona, the amount of time spent on activities, their emotional trajectory, strategies and behaviors, and help-seeking behaviors. The table is divided into the following columns: Persona, # of Groups, Time Spent, Emotion Trajectory, Strategy/Behavior, and Help Seeking. Exploring Persona: Comprising 6 groups, these participants spent a long time on activities and showed mostly positive or neutral emotions, including engagement, curiosity, and satisfaction. They demonstrated an instruction-led approach, explored different tabs, refined prompts, and experienced few stalls. Help seeking was low, typically occurring after several self-retries. Task-Focused Persona: Representing a short time spent on activities, these participants had a clear focus on the task with minimal distractions. Their emotional trajectory was neutral, and their strategy involved following instructions and refining prompts. They did not seek much help during the task. Mixed Persona: Comprising 3 groups, this persona showed a varied emotional trajectory, shifting between positive and negative emotions across different stages. Their strategies shifted within the session, and they combined instruction-following with exploration. Help seeking was occasional and often used to recover from stalls. Frustrated Persona: Representing 2 groups, these participants spent a long time dwelling on failures and had mostly negative emotions, such as confusion, dissatisfaction, and frustration. They experienced repeated retries without progress, often backtracking. Help seeking was early and frequent, with signs of fatigue or abandonment.
Figure 9. Different Personas’ Reactions and Emotion in Each Stage of Activities This flowchart illustrates the emotional reactions and decision-making pathways of three user personas—Exploratory/Task-Focused, Mixed, and Frustrated—across different stages of the activity. The diagram shows how each persona progresses through the steps of getting familiar with the chatbot and coding wave activities, with emotion and behavior displayed at each stage. Exploratory/Task-Focused Persona: This persona starts with curiosity and excitement during the initial task (getting familiar with the chatbot), then moves to confidence and engagement during the coding tasks. If they experience difficulty, they may ask the facilitator or revise their prompts. Positive emotions are indicated by dashed blue arrows, and negative emotions by solid orange arrows. Mixed Persona: This persona shows variable emotions, moving between contentment, curiosity, and frustration. If they encounter issues, they may ask the facilitator or recheck instructions, and if they succeed, they experience relief and happiness. The flow highlights both positive and negative emotional trajectories. Frustrated Persona: This persona begins with confusion and frustration, particularly during the initial phase of using the chatbot. They experience repeated frustration and struggle, reflected by a strong focus on negative emotions. However, if they succeed, they feel relief and are able to proceed. Arrows represent the decision-making process and emotional responses at each stage, with blue dashed arrows indicating positive emotions and orange arrows indicating negative emotions.
Figure 10. Radar chart comparing four personas—Exploratory, Frustrated, Mixed, and Task-Focused—across seven dimensions: three emotion states (negative, neutral, positive), time spent, help-seeking, instruction-following bot queries, and refinement-oriented interactions (prompt editing, tab switching, and trial-and-error exploration). Higher values indicate more frequent or intense presence of that dimension for a given persona. A radar chart comparing the behavioral and emotional profiles of four user personas: Exploratory, Frustrated, Mixed, and Task-Focused. The personas are mapped across axes representing emotions (Positive, Neutral, Negative) and interaction behaviors (Exploratory, Prompt refinement/Trial & Error, Time Spent, Help-seeking, and Interacting with bot). Exploratory users peak in Positive Emotion, Exploratory behavior, and Time Spent. Frustrated users spike in Negative Emotion and Help-seeking, with high Time Spent but low Positive Emotion. Task-Focused users peak in Interacting with the bot (following instructions) and Neutral Emotion. Mixed users show a moderate profile across dimensions between the other groups.
Themes
Categories
Benefits & Risks to students
Chatbot may hurt learning to program
Building student prompting skills
Chatbot can increase student self-efficacy
Chatbot can build student knowledge
Pedagogical Practices Around Chatbot
Build foundational learning first
Scaffold intro to chatbot
Table 6. Themes and categories produced from thematic analysis of teacher discussion groups. This table presents the themes and categories derived from the thematic analysis of teacher discussion groups. The table is divided into two columns: Themes and Categories. Benefits & Risks to Students: Categories include how the chatbot might hurt students’ learning, such as potentially hindering the development of programming skills, and its potential benefits, such as increasing student self-efficacy and building knowledge. Pedagogical Practices Around Chatbot: Categories here explore teaching practices related to the chatbot, such as building foundational learning before introducing the chatbot, scaffolding its use, controlling its features, and using pair programming alongside the chatbot. It also includes teacher requests for better engagement features. Assisting All Learners: Categories focus on how the chatbot can support different learners, including novice students, those needing challenge, and how the chatbot can assist in differentiation. There are also feature requests for improving accessibility. Allow Chatbot in Classroom?: The table outlines teachers' positions on allowing students to use the chatbot in the classroom, with some wanting to allow its use and others not. Benefits for Teachers: Categories include the chatbot’s role in building teacher confidence, supporting teacher instruction, and aiding student independence. Usability: This category identifies usability issues with the chatbot, highlighting problems that need to be addressed for a better user experience.
Figure 11. Integrated TAM (Davis, 1989) and TPB (Ajzen, 1991) model used to interpret teachers’ intention to adopt the AI chatbot. A conceptual path model diagram illustrating the scope of the study. The model flows left-to-right, starting with 'System & Context Factors,' which point to 'Mediators: emotion, help-seeking.' From Mediators, arrows branch out to three destinations: 'Perceived Behavioral Control,' 'Positive Attitudes toward chatbot,' and A bracketed group labeled 'Perception of Chatbot' containing 'Perceived usefulness' and 'Perceived ease of use.' 'Perceived usefulness' and 'Perceived ease of use' also direct arrows to 'Positive Attitudes toward chatbot.' Finally, both 'Positive Attitudes' and 'Perceived Behavioral Control' lead to 'Intention to use chatbot.' A vertical dashed line separates 'Intention to use' from the final box, 'Actual use,' indicating that 'Actual use' is designated as 'Future work,' while the rest of the model represents the 'Scope of this Paper.
Chatbots have long been explored as tools to support learning, and recent advances in large language models have significantly expanded the availability of platforms for educators to author AI tutoring chatbots. Yet effective authorship demands more than writing a system prompt; it requires educators to act as learning designers, AI interaction designers, and QA engineers. In practice, however, teachers rarely fulfill these roles. Our formative study found that virtually none systematically tested their bots before deploying them to students. To address this gap, we present PromptDecipher, a system that restructures the authoring workflow around a direct correction-based interaction rather than writing abstract system prompts, teachers interact with a live chat preview and edit undesirable bot responses. An automated pipeline then analyzes the correction, proposes a targeted system prompt rewrite, and validates the change across pre-defined test scenarios. This enforces QA as a first-class activity and scaffolds teachers in roles they would otherwise skip. PromptDecipher will be deployed in an AI for Educators course enrolling hundreds of higher-education instructors. A live prototype (https://teacher-prompting.vercel.app/), an anonymized codebase (https://anonymous.4open.science/r/teacher-prompting-2EDF/), and anonymized demo (https://tinyurl.com/las-prompt-decipher-demo) are available via links in the footnote.
Miina Koyama, Ruiwei Xiao, John Stamper
Carnegie Mellon University Pittsburgh, Pennsylvania, USA
A central pedagogical value evaluated in AI tutor benchmarks is scaffolding: guiding students through graduated steps toward a solution. Alignment and evaluation methods for embedding scaffolding behaviour into chatbots, however, rest on an implicit assumption: that students will take up the scaffolding and engage in the conversation. To examine whether this assumption holds, we introduce an evaluation pipeline around two metrics - Chatbot Scaffolding and Student Uptake - and apply them across nine datasets of 9,490 chats, spanning AI tutor benchmarks and real-world deployments of educational chatbots. Our analysis reveals that while benchmarks assume a high-scaffolding, high-student-uptake environment, students in real-world settings exhibit lower levels of uptake overall - frequently bypassing the chatbot's pedagogical framing to drive the interaction toward their own learning goals at little interpersonal cost. We argue that bypassing scaffolding is not necessarily detrimental; rather, it frequently highlights a mismatch between a chatbot's pedagogical framing and the student's learning goals. To meaningfully evaluate the effectiveness of a chatbot's assistance, future benchmarks must move beyond the assumption that students will simply take up the scaffolding, and instead evaluate how these chatbots navigate diverse learning contexts and student-driven interaction patterns.
Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer +2
Department of Mechanical Engineering, Imperial College London, UK · Department of Electrical Engineering, Imperial College London, UK · Department of Earth Science and Engineering, Imperial College London, UK
Teachers increasingly configure educational chatbots through natural-language instructions, but how these bots fail in real-world use remains poorly understood. We introduce EduWild, a dataset of 1,726 conversations with 217 teacher-configured chatbots, each paired with the instructions and configuration behind it, including 900 conversations from the intended end users of 35 deployed chatbots. We develop a 29-code taxonomy covering factual, workflow, diagnostic, and pedagogical failures and apply it using a human-calibrated LLM judge. The most frequent failures are lapses in following teacher intent --- bots skip required workflow steps (53.8% of end-user conversations), omit expectation setup (33.4%), and overload learners with information or questions (37.1%). Guided by these findings, we design a prompt-based harness targeting the six most frequent failure modes. In a study with 200 college students across ten bots, the harness reduces the mean number of target failure codes per conversation from 1.406 to 0.489, a 65.2% relative reduction with bots weighted equally. These findings show how real-world failure analysis can guide practical improvements to educational chatbots.