cs.HCFeb 24, 2026
SaveHybrid LLM-Embedded Dialogue Agents for Learner Reflection: Designing Responsive and Theory-Driven Interactions
Organizations: University of Pittsburgh Pittsburgh, Pennsylvania, USA · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Bowie State University Bowie, Maryland, USA
Abstract
Dialogue systems have long supported learner reflections, with theoretically grounded, rule-based designs offering structured scaffolding but often struggling to respond to shifts in engagement. Large Language Models (LLMs), in contrast, can generate context-sensitive responses but are not informed by decades of research on how learning interactions should be structured, raising questions about their alignment with pedagogical theories. This paper presents a hybrid dialogue system that embeds LLM responsiveness within a theory-aligned, rule-based framework to support learner reflections in a culturally responsive robotics summer camp. The rule-based structure grounds dialogue in self-regulated learning theory, while the LLM decides when and how to prompt deeper reflections, responding to evolving conversation context. We analyze themes across dialogues to explore how our hybrid system shaped learner reflections. Our findings indicate that LLM-embedded dialogues supported richer learner reflections on goals and activities, but also introduced challenges due to repetitiveness and misalignment in prompts, reducing engagement.
Figures & tables
Figure 1. This diagram depicts our hybrid dialogue system architecture modeled as a Finite State Machine and illustrates how logic flows across the rule-based prompts, learner responses, and conditional transitions at each system-learner turn. The architecture of the hybrid dialogue system is modeled as a finite state machine. The figure describes the architecture of our hybrid dialogue system, which is modeled as a Finite State Machine. The logic flow starts with the system displaying a rule-based prompt followed by a learner response. If it is a reflection prompt, then the LLM gets triggered and checks for the response relevance. If the response is relevant, the system moves to the next rule-based response. If the response needs more elaboration, the system triggers LLM for contextual response generation, which is then presented to the user. This cycle continues until the user response is relevant to the input prompt or a counter of three is reached, after which the system defaults to the rule-based prompt.
Figure 2. The two stages of LLM in our dialogue system: (1) Relevance Check, where a dynamic prompt with prompt-specific information and few-shot examples determines binary relevance; and (2) Contextual Generation, where, upon a NO in relevance check, a new prompt incorporating prompt-specific information, dialogue history, and few-shot examples guides the LLM in producing a follow-up. The two stage architecture for LLM check and generation in our dialogue system. The figure shows the two stages of LLM in our dialogue system (1) Relevance Check, where a dynamic prompt with prompt-specific information and few-shot examples determines binary relevance; and (2) Contextual Generation, where, upon a NO in relevance check, a new prompt incorporating prompt-specific information, dialogue history, and few-shot examples guides the LLM in producing a follow-up. The figure further shows an example where the learner initially responded
by coding'' to the system's prompt, which failed the relevance check and triggered the LLM to reprompt with That's a great starting point! What specific coding skills or tools do you think you'll use to make your robot walk and talk?''Figure 3. The figure shows the chat widget embedded in our system’s frontend programming interface. Learners interact with the system through this widget using text or speech-to-text input, while system responses are delivered as both text and audio. The example displayed shows a dialogue snippet from our camp where the system prompts a learner to reflect on their goals and plans for the robot showcase. The learner responses “Yes” and “No” are part of a menu-based selection, and “walk and ctalk” and “by coding” are free-text responses. The chat widget in our frontend interaction interface. The figure shows the chat widget embedded in our system's frontend programming interface. Learners interact with the system through this widget using text or speech-to-text input, while system responses are delivered as both text and audio. There is an example displayed in the figure that shows a dialogue snippet from our camp where the system prompts a learner to reflect on their goals and plans for the robot showcase.
| Category | Description | Examples |
|---|---|---|
| LLM Triggers | Under this category, we specifically looked for instances where LLM was triggered for learner goals, plans, activities, feelings. This also includes changes in these components. We also noted instances where LLM failed to trigger. | Robot: No worries! What would you like to do instead? (Goal) Robot: That’s a great starting point! What specific coding skills or tools do you think you’ll use to make your robot walk and talk? (Plan) |
| LLM Reprompts | Under this category, we coded for if the LLM reprompts were either in-context with the conversation and learner inputs or were they out of context. We also looked for instances where the response contained hallucination or bias. | Robot: That’s alright, not every day is about finishing. Sometimes just trying counts too! Can you think of one thing you started, explored, or even just thought about today? (in-context) Robot: Her hair is an interesting choice. Can you think of a way that working on her hair might be connected to what we talked about earlier, like building a robot that can walk and talk? (out-of-context) |
| Learner Reflection | Under this category, we coded for if the learner reflected on their design goals, plans, actions, or changes to these components or their feelings. We also code for if any of these reflections were elicited due to an LLM reprompt in the previous turn. | Learner: I think I’ll do it by putting fur on you (Action) Learner: I coded my robot to move and say word problems (Goal) (LLM) |
| Emotion | Under this category, we coded for explicit mentions of positive sentiments (or logical reasoning steps) and negative sentiments. The default code under this category was neutral. | Learner: because i love robots (positive) Learner: leaveeeeeee meeeeee alone (negative) Learner: make you move (neutral) |
| Engagement | We this category, we coded for explicit moments which defined high-engagements (providing elaborate response) or low-engagements (mentioning the intention not to talk, or a continuous sequence of “no” in response) | Learner: ok but i have to go bye (low-engagement) Learner: move right for 3 secs and then you’ll spin and dance just move your arms (high-engagement) |
Table 1. The table shows categories devised (with description and examples) after qualitative coding of the learner dialogue interactions with the system. The categories that emerged were: LLM Triggers, LLM Reprompts, Learner Reflection, Emotion, and Engagement.
| Category | Metric | Total | Mean (SD) |
|---|---|---|---|
| 1. Session Overview | |||
| Dialogue Turns | 357 | 39.66 (6.46) | |
| Session Duration (min) | – | 6.00 (2.00) | |
| 2. Turn-Level Metrics | |||
| System Turns | 213 | 23.67 (3.64) | |
| Learner Turns | 144 | 16.00 (2.91) | |
Table 2. Descriptive statistics of dialogue sessions ( participants). The table details session duration, the breakdown of learner inputs (open-ended vs. option-based), turns, and word counts, and LLM trigger frequency.
| Evaluation | Metric | Count / Mean (SD) |
| 1. Response Verbosity | ||
| Learner response word count before LLM prompt or re-prompt | 2.59 (1.13) | |
| Immediate learner response word count post LLM prompt or re-prompt | 3.55 (3.07) | |
| Expansion Factor b/w post prompt (or re-prompt) and pre prompt lengths | 1.75 (3.27) | |
| 2. Relevance Check Quality | ||
| LLM correctly triggered (TP) | 23 | |
Table 3. Analysis of system performance for response verbosity, relevance check quality, prompt content, and quality.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4. The rule-based prompts for the dialogue system. A large flow-chart with the rule-based system prompts. The figure displays how the dialogues flow across different prompts in the rule-based design. These prompts were defined to know more about learners' goals, plans, actions, and feelings throughout the camp. There is a broad sequence of states containing prompts for building rapport, revisiting goals, plans, and current activities, noting goal or plan changes, explaining causes for changes, feelings about designs for the upcoming showcase, and reflecting on identity as technology creators, each implemented as a node in the state machine.