SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training

Zihan Wang; Anita Marie Slominska; Rennie Bimman; Elizabeth Di Flumeri; Amanda Mayappo-Neeposh; Conall Francoeur; Tamara Ellen Carver; Xiao-Wen Chang

cs.CLAug 30, 2026

SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training

Authors: Zihan Wang, Anita Marie Slominska, Rennie Bimman, Elizabeth Di Flumeri, Amanda Mayappo-Neeposh, Conall Francoeur, Tamara Ellen Carver, Xiao-Wen Chang,

Organizations: School of Computer Science, McGill University · Mila – Quebec AI Institute · Institute of Health Sciences Education, McGill University · School of Social Work, McGill University ·

Abstract

Pediatric serious illness communication (SIC) is critically important, yet scalable communication training for clinicians remains limited. Compared with other dialogue simulation settings, pediatric SIC poses additional challenges, including multi-party interactions, response to parental distress and strong dependence on feedback dynamics. Existing LLM-based simulators optimize generic dialogue quality rather than curriculum-contingent behavior required for effective SIC training. In collaboration with educators and pediatric clinicians, we introduce the first benchmark suite and simulation framework tailored to pediatric SIC training. Our benchmarks, PitfallBench and DialogueBench, evaluate simulators both at the turn-level and across full dialogues. We further propose SIC-Agents, a self-improving framework that generates a clinician-editable skill document to guide simulator behavior. Our experiments show that SIC-Agents outperforms static expert prompting. To support future research, we release our benchmarks for parent simulation in pediatric SIC at https://github.com/Beikewzh/sic-benchmarks

Figures & tables

Figure 1: SIC-Agents self-improvement loop. A parent agent generates candidate responses conditioned on the case, dialogue history, and editable skill document. LLM judges select responses and provide feedback, which a Skill Editor uses to revise the skill document under curriculum guidance from a Coach agent. Clinicians and human instructors can audit and edit both the skill document and revisions.

Benchmark	Domain	Gold	Expert
Patient- $\Psi$ ( Wang et al., 2024 )	Psychotherapy	106	designed
PatientSim ( Kyung et al., 2025 )	General	37	checked
AMIE ( Tu et al., 2024 )	Diagnostic	149	rated
PitfallBench (ours)	Pediatric SIC	1,000	curriculum
DialogueBench (ours)	Pediatric SIC	152	curriculum

Table 1: Comparison with existing clinician-anchored dialogue benchmarks. Gold denotes the number of released evaluation items. Our benchmarks are grounded in an external pediatric SIC curriculum and refined under expert oversight.

Figure 2: PitfallBench uses a paired within-case design. The same case and dialogue history are instantiated twice at the trigger turn: once with a curriculum-defined pitfall move and once with its skilled alternative. The simulator must generate differentially appropriate parent responses under otherwise identical conversational conditions.

Phase 1: share and support information
ID	Pitfall	Core SIC skill
P1	Missed early emotion cue	Ask for permission
P2	Forgets to assess understanding	Assess understanding
P3	Information too complex	Headline framing
P4	Missed or combined empathy cues	Respond to emotion
P5	Stuck in validation	Move on after empathy

Table 2: The ten learner pitfalls covered by PitfallBench, grouped by curriculum phase, with the core SIC skill each pitfall compromises.

No loop
Condition	Primary	Secondary
	PitfallB. $\uparrow$	DialogueB. $\uparrow$	Comp. $\uparrow$
Minimal seed	$0.66$	$2.52$	$0.71$
Expert prompt	$0.73$	$2.28$	$0.69$
Full-taxonomy	$0.75$	$2.32$	$0.71$
Self-improvement

Table 3: Main results on PitfallBench and DialogueBench. All self-improvement conditions share the same minimal seed. PitfallBench accuracy and the composite are reported on a

0

–

1

scale; DialogueBench reports Grounding on the original

1

–

3

rubric. The composite averages PitfallBench accuracy with Grounding rescaled to

0

–

1

Figure 4: Self-improvement trajectories from the same minimal seed. Top: PitfallBench sub-metrics. Curriculum guidance preserves or improves pitfall-resistance and skilled-reward; without it, both decline. Bottom: DialogueBench Grounding and Topic Consistency (rescaled to

0

–

1

) remain high in both conditions. The divergence between the two panels is the empirical signature of clinical drift.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Phase 1: share and support information
ID	Curriculum rule (trigger $\rightarrow$ pitfall vs. skilled)
P1	Parent gives an early emotional cue; pitfall proceeds to content, skilled asks permission and names the cue.
P2	New information just delivered; pitfall moves on, skilled asks the parent to restate it.
P3	Complex information to convey; pitfall front-loads detail, skilled leads with a headline.
P4	Parent shows empathy cues; pitfall misses or collapses them, skilled responds to each emotion cue distinctly.
P5	Parent just validated; pitfall keeps re-validating, skilled briefly acknowledges and shifts forward.

Appendix

Table 4: Per-pitfall curriculum rules for PitfallBench, authored by our clinician co-authors (Appendix E ). Expected parent reaction is challenge_signal for the pitfall move and reward_signal for the skilled move in every row; the pitfall name and core SIC skill are in main-text Table 2 .

Figure 5: PitfallBench diversity. UMAP of the

1{,}000

items in Gemini embedding space; the items spread broadly across the space, covering the

100

cases and their twelve care settings.

Figure 6: DialogueBench diversity. UMAP of the parent-response embeddings from the

8{,}391

-dialogue corpus (text-embedding space, cosine metric). The responses spread broadly across the space.

No loop
	PitfallBench			DialogueBench
Condition	skill $\uparrow$	pit $\uparrow$	all $\uparrow$	Grnd $\uparrow$	Topic $\uparrow$	Comp. $\uparrow$
Minimal seed (start)	$0.57$	$0.75$	$0.66$	$2.52$	$2.82$	$0.71$
Expert prompt	$0.61$	$0.85$	$0.73$	$2.28$	$2.78$	$0.69$
Self-improvement
No curriculum	$0.42$	$0.63$	$0.52$	$2.79$	$2.82$	$0.71$

Appendix

Table 5: Full per-condition results. PitfallBench reports skilled-reward (skill), pitfall-resistance (pit), and their mean (all), all on

0

–

1

; DialogueBench reports Grounding and Topic on the

1

–

3

rubric; the composite is on

0

–

1

Pitfall	pit $\uparrow$	skill $\uparrow$	pit $\uparrow$	skill $\uparrow$
	Full system		No curriculum
P1 misses emotion / permission	0.94	0.52	0.58	0.14
P2 forgets to assess	0.94	0.88	0.56	0.88
P3 info too complex	0.96	0.76	0.72	0.26
P4 misses / combines empathy	0.94	0.60	0.44	0.30
P5 stuck in validation	0.80	0.84	0.46	0.68

Appendix

Table 6: Per-pitfall PitfallBench accuracy at cycle

5

, for the full system and the no-curriculum ablation. pit

=

pitfall-resistance, skill

=

skilled-reward.

Condition	Claude	GPT-5	DS R1	Qwen3
Expert prompt	$0.73$	$0.71$	$0.70$	$0.74$
No curriculum	$0.52$	$0.54$	$0.49$	$0.55$
SIC-Agents (full)	$\mathbf{0.78}$	$\mathbf{0.79}$	$\mathbf{0.76}$	$\mathbf{0.80}$

Appendix

Table 7: PitfallBench accuracy re-scored with non-Claude judges (DS R1

=

DeepSeek R1). The Claude column is the paper judge from Table 3 . The condition ordering holds under all judges.

Large Language Model-Based Role-Play Simulation

Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.

See all 111 papers

Synthetic or Authentic? Building Mental Patient Simulators from Longitudinal Evidence
Mar 24, 2026Baihan Li, Bingrui Jin, Kunyao Lan +2Large Language Model-Based Role-Play Simulation Memory-Augmented Language Model Agents
SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · School of Computer Science and Engineering, Northeastern University, China +1
Training language models in simulation for psychiatric intake
Oct 29, 2026Samuel Schmidgall, Tim Strother, Alex Bijamov +53Mental Health Conversational Agents LLM Agent Training
VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise
Apr 12, 2026Sina Mansouri, Mohit Marvania, Vibhavari Ashok Shihorkar +5Healthcare LLM Reliability
Department of Computer Science, George Mason University · Department of Health Administration and Policy, College of Public Health, George Mason University · Department of Science, George Mason University

Mar 24, 2026J/K move · Enter open · S save

Synthetic or Authentic? Building Mental Patient Simulators from Longitudinal Evidence

Baihan Li, Bingrui Jin, Kunyao Lan, Ming Wang+1

SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · School of Computer Science and Engineering, Northeastern University, China+1

Large Language Model-Based Role-Play Simulation Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Patient simulation is essential for developing and evaluating mental health dialogue systems. As most existing approaches rely on snapshot-style prompts with limited profile information, homogeneous behaviors and incoherent disease progression in multi-turn interactions have become key chellenges. In this work, we propose DEPROFILE, a data-grounded patient simulation framework that constructs unified, multi-source patient profiles by integrating demographic attributes, standardized clinical symptoms, counseling dialogues, and longitudinal life-event histories from real-world data. We further introduce a Chain-of-Change agent to transform noisy longitudinal records into structured, temporally grounded memory representations for simulation. Experiments across multiple large language model (LLM) backbones show that with more comprehensive profile constructed by DEPROFILE, the dialogue realism, behavioral diversity, and event richness have consistently improved and exceed state-of-the-art baselines, highlighting the importance of grounding patient simulation in verifiable longitudinal evidence.

Abstract

Large Language Model-Based Role-Play Simulation

Explore similar work

Synthetic or Authentic? Building Mental Patient Simulators from Longitudinal Evidence

Training language models in simulation for psychiatric intake

VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise