cs.CLJun 29, 2026

SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation

Authors: Zhuhan Bao, Rui Yang, Bohao Yang, Zhiyi Liu, Sicheng Shu, Ruio Heerschap, Le Li, Doris Yang, +11 more

Organizations: Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC, USA · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Centre for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Department of Statistical Science, Duke University, Durham, NC, USA · Leiden University Medical Centre, Leiden, The Netherlands · Department of Mathematics, University of Texas at Austin, Austin, USA · Department of Internal Medicine, Yale School of Medicine, New Haven, CT, USA · Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA · The Graduate Group in Applied Mathematics and Computational Science, School of Arts and Sciences, University of Pennsylvania, Philadelphia, PA, USA · Medstar Health National Center for Human Factors in Healthcare, Washington, DC, USA · Department of Biomedical Informatics, Columbia University, New York, NY, USA · Cancer Prevention and Control, Duke Cancer Institute, Durham, NC, USA · Department of Population Health Sciences, Duke University School of Medicine, Durham, NC, USA · Pre-hospital and Emergency Research Centre, Health Services Research and Population Health, Duke-NUS Medical School, Singapore, Singapore · NUS Artificial Intelligence Institute, National University of Singapore, Singapore, Singapore · Division of Pulmonary, Allergy and Critical Care Medicine, Duke University School of Medicine, Durham, NC, USA · Division of Rheumatology and Immunology, Duke University School of Medicine, Durham, NC, USA · Duke Center for Health Informatics, Duke University, Durham, NC, USA · Duke Clinical Research Institute, Durham, NC, USA

Abstract

Background. The widespread deployment of ambient digital scribes is driving large-scale capture of clinician-patient dialogues. Human coding of clinical communication data remains costly, inconsistent, and difficult to scale, motivating AI-driven communication coding systems. However, evaluating these systems requires real-world dialogues and human-coded labels, both hard to obtain at scale. Methods. We developed SIMAX (Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation), a framework for generating controlled clinical dialogue data with reference behavioral annotations. SIMAX generates clinician-patient dialogues from predefined clinical scenarios, personas and voice conditions, and target communication behaviors. Behaviors are controlled using two codebooks: the Global Codebook for overall communication quality and the WISER Codebook for specific countable behaviors. We evaluated SIMAX using automated and human quality assessments and an example communication coding system. Results. SIMAX generated 3,388 simulated dialogues across three specialties, multiple visit stages, persona characteristics, and accent conditions. Automated assessment showed mean UTMOS and WV-MOS scores of 3.03 and 2.61, WER and CER of 0.07 and 0.05, and CLAP cosine similarity of 0.41, suggesting reasonable speech naturalness, high transcription fidelity, and positive text-audio correspondence. Human evaluation showed a median MOS of 4.67 and a median clinical realism score of 3.00. Downstream evaluation suggests that SIMAX can assess how a communication coding system responds to behavioral targets and reveal insufficient sensitivity in some dimensions. Conclusions. SIMAX generates controlled and reproducible simulated clinician-patient dialogues, providing a data foundation for developing, validating, and refining communication coding systems.

Explore similar work

CardsList
Date pendingcs.AI

VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise

Medical large language models are typically evaluated on idealized patient cases that do not reflect how real patients communicate. We introduce VeriSim, a patient simulation framework that injects controllable noise along six clinically grounded communication dimensions while substantially preserving each patient's medical record. Truth adherence is supported by a verifier that extracts atomic claims from each candidate utterance and judges them against a UMLS-grounded vector index built with BioLORD embeddings, using the retrieved atoms' structured clinical metadata (e.g., drug class, anatomical site, treats-condition relations) rather than surface-text similarity alone. Across seven open-weight LLMs, realistic noise reduces diagnostic accuracy by 15-25 percentage points and increases conversation length by 34-55%; the 7-8B models degrade 1.4x more than 70B+ models. A board-certified physician and a licensed nurse rate VeriSim's conversations highly on truth, realism, clinical utility, and noise fidelity (inter-annotator agreement >= 0.80 across all dimensions), and an LLM-as-judge closely tracks their ratings, enabling scalable evaluation. We release VeriSim as an open-source framework for rigorous stress-testing of clinical AI: https://github.com/mohitmarvania/VeriSim
Sina Mansouri, Mohit Marvania, Vibhavari Ashok Shihorkar +5
Simulated PatientsDiagnostic BenchmarkConversational BenchmarksCommunication Efficiency
AskPDF
Apr 25, 2026cs.HC

Scalable LLM-based Coding of Dialogue in Healthcare Simulation: Balancing Coding Performance, Processing Time, and Environmental Impact

Research shows that dialogue, the interactive process through which participants articulate their thinking, plays a central role in constructing shared understanding, coordinating action, and shaping learning outcomes in teams. Analysing dialogue content has been central to advancing team learning theory and informing the design of computer-supported collaborative learning environments, yet this progress has depended on labour-intensive qualitative coding. LLMs offer new possibilities for automating and enhancing the dialogue layer within emerging multimodal learning analytics approaches, with recent studies showing that they can approximate human coding through few-shot prompting. However, prior work has focused on replicating human coding accuracy for research purposes, rather than addressing a more educationally consequential question: how can we design prompts that allow an LLM to label team dialogue accurately and fast enough to be useful in real settings, such as in-person healthcare simulations, where results must be returned quickly and computational cost and sustainability also matter? This paper investigates how prompt design and batching strategies can be optimised to balance coding accuracy, processing time, and environmental impact in team-based healthcare simulation debriefing. Using a dataset of 11,647 utterances coded across 6 dialogue constructs, we compared 4 prompt designs across varying batch sizes, evaluating coding performance, processing time, and energy consumption, as well as the trade-offs between these metrics. Results indicate that increasing batch size improves speed and reduces energy use, but negatively impacts coding performance. Beyond demonstrating the feasibility of LLM-based qualitative analysis, this study offers practical guidance for scaling dialogue analytics in contexts where timeliness, privacy, and sustainability are critical.
Kiyoshige Garces, Gloria Milena Fernandez-Nieto, Linxuan Zhao +4
Simulated PatientsConversational BenchmarksDialogueCode Generation
AskPDF
Aug 6, 2026cs.CL

Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies

Much clinical value is conveyed not through structured records but through communication: exchanges in which patients describe symptoms, clinicians reason and give instructions, ambulances hand over to emergency departments, and nurses pass on a shift. Such language differs from tabular data because meaning depends on speaker role, intent, causality, uncertainty, omission, and channel noise. Healthcare natural language processing must therefore interpret information as conveyed rather than coded. This requires well-annotated corpora, which are scarce because authentic exchanges are private, fragmented, and costly to annotate. Large language models offer a way forward by transforming clinical sources, such as records, diagnostic labels, symptom lists, or care plans, into written and transcribed communication for downstream models. We present a structured narrative survey organized by source representation, communication form and participants, generation method, and downstream task, complemented by thirteen novel case studies. These build clinical NLP systems for communication channels and languages without labeled real-world data, including EMS pre-arrival reports, field-radio casualty documentation, nurse handoffs, patient-portal triage, and low-resource discharge communication. They show that synthetic communication can bootstrap such systems. Findings include the competitiveness of fine-tuned encoder models over evaluated zero-shot baselines and the value of deliberately degraded communication for robustness. The main limitation is that most studies evaluate on held-out synthetic communication, while train-on-synthetic, test-on-authentic evidence remains limited. We conclude that syn-thetic clinical communication is becoming a practical research resource; establishing it as reusable clinical infrastructure will require authentic-data transfer, safety and external validation.
Alexander Apartsin, Yehudit Aperstein
Clinical NotesNatural Language ProcessingSynthetic DataCase StudySurvey
AskPDF