Structuring versus Problematizing: How LLM-based Agents Scaffold Learning in Diagnostic Reasoning
Organizations: ETH Zürich, EPFL Lausanne, Switzerland · EPFL Lausanne, Switzerland
Abstract
Supporting students in developing diagnostic reasoning is a key challenge across educational domains. Novices often face cognitive biases such as premature closure and over-reliance on heuristics, and they struggle to transfer diagnostic strategies to new cases. Scenario-based learning (SBL) enhanced by Learning Analytics (LA) and large language models (LLM) offers a promising approach by combining realistic case experiences with personalized scaffolding. Yet, how different scaffolding approaches shape reasoning processes remains insufficiently explored. This study introduces PharmaSim Switch, an SBL environment for pharmacy technician training, extended with an LA- and LLM-powered pharmacist agent that implements pedagogical conversations rooted in two theory-driven scaffolding approaches: structuring and problematizing, as well as a student learning trajectory. In a between-groups experiment, 63 vocational students completed a learning scenario, a near-transfer scenario, and a far-transfer scenario under one of the two scaffolding conditions. Results indicate that both scaffolding approaches were effective in supporting the use of diagnostic strategies. Performance outcomes were primarily influenced by scenario complexity rather than students' prior knowledge or the scaffolding approach used. The structuring approach was associated with more accurate Active and Interactive participation, whereas problematizing elicited more Constructive engagement. These findings underscore the value of combining scaffolding approaches when designing LA- and LLM-based systems to effectively foster diagnostic reasoning.
Figures & tables
| Strategies | Checklist | Interpersonal Relationships | Possible Causes |
| Structuring Scaffolding | |||
| Decomposing complex tasks (e.g., structuring the task solution into the plan) | "Systematically recording symptoms is one of the most important steps for an effective diagnosis. " "Have you already used LINDAFF to collect data?" | "We’ve focused on the baby so far — but is that the whole picture?" | "Have you thought about possible causes that you could investigate further?" "Now let’s think about possible causes." "What could be the cause of the baby’s diarrhea?" |
| Focusing effort (e.g., specific directions on what to do on a level) | "Great that you’re considering localization! This helps to narrow down the problem even more precisely." "Good thing you asked about the nature of the diarrhea! This helps to better understand the problem." | "Could the baby’s condition also play a role? " "If something from the mother’s side were relevant, what would you pay attention to?" | "You asked about the intensity of the symptoms — good decision." "You’ve brought up the baby’s diet—good observation. " |
| Monitoring progress (e.g., tracking progress, acknowledging what was already done) | "So far, you’ve covered about 66% of the checklist—well done! Already asked: - Symptoms - Intensity - Location - Medical history Still open: - Duration - Allergies" | "We’ve focused on the baby so far…" | "You have narrowed down the possible causes. Now you are ready to make a probable diagnosis." |
| Problematizing Scaffolding | |||
| Eliciting articulation (e.g., prompting for explanations) | "Do you think you’ve covered all the areas of the LINDAFF checklist, or is there still information you might be missing?" "Would it help to use a checklist or framework to see what’s still missing?" | "Why do you think her condition might be relevant?" "If something she takes affects the baby, how would we know?" | "What do you think of the father’s answer? Could this help rule something out or confirm something?" "You have considered different perspectives. Does one of them now feel more plausible than the others? Is there anything you would now question or rule out?" |
| Correct | Incorrect | |
|---|---|---|
| Active | Short correct answer without any elaboration (e.g., "yes"; "no") Repetition of factual knowledge given in the simulation (e.g., "The baby is five months old.") | Short incorrect answer without any elaboration (e.g., "Might be"; "I believe so") Attempt to reveal the correct solution (e.g., “What are the possible causes?”) Non-progressing reply (e.g., “leave me alone…”) Expressing confusion (e.g., “I don’t know”) |
| Constructive | Introducing correct new ideas (e.g., a reasonable possible cause), not related to the previous conversation (e.g., “A viral infection could explain the sudden diarrhea.”, “Could it be that the baby doesn’t tolerate puree?”) | Introducing non-productive incorrect new ideas (e.g., "I will send him to the doctors", Maybe the baby has intolerance to breast milk) |
| Interactive | Introducing correct new ideas (e.g., a reasonable possible cause), reasoning with the help of scaffolding, built on ideas of the pharmacist (e.g., “Yes, it could be due to the mother’s medication as this also affects breast milk”) | Introducing incorrect new ideas, built on ideas of the pharmacist (e.g., pharmacist: “could it be diet change?”, student: “yes, I think it is due to bimbosan”) |
| Scaffolding Subcategory | Structuring-heavy implementation | Problematizing-heavy implementation |
|---|---|---|
| Structuring Scaffolding | ||
| Decomposing | “Let’s split this into symptoms, duration, and severity to cover the checklist efficiently.” | “What smaller questions would help you untangle this case?” |
| Focusing Effort | “Focus next on clarifying intensity to advance the checklist.” | “Which single question would most reduce uncertainty now?” |
| Monitoring | “I see that you are already applying Checklist Strategy principles. So far, you’ve covered about 33% of the checklist! Already asked: Symptoms, Intensity. Still open: Duration, Allergies, Location, Medical history.” | “You have already covered many aspects of the Checklist Strategy checklist, such as symptoms, intensity, and duration. Is there anything else you would like to investigate further to better narrow down possible causes of the diarrhea?” |
| Problematizing Scaffolding | ||
| Elicit Articulation | “How likely is the new diet the cause at this point?” | “Explain why the new diet might or might not account for these symptoms.” |