EMSDialog: Synthetic Multi-person Emergency Medical Service Dialogue Generation from Electronic Patient Care Reports via Multi-LLM Agents

Xueren Ge; Sahil Murtaza; Anthony Cortez; Homa Alemzadeh

cs.CLApr 8, 2026

EMSDialog: Synthetic Multi-person Emergency Medical Service Dialogue Generation from Electronic Patient Care Reports via Multi-LLM Agents

Authors: Xueren Ge, Sahil Murtaza, Anthony Cortez, Homa Alemzadeh

Organizations: School of Engineering and Applied Sciences, University of Virginia, Charlottesville, VA, USA · School of Medicine, University of Virginia, Charlottesville, VA, USA

Abstract

Conversational diagnosis prediction requires models to track evolving evidence in streaming clinical conversations and decide when to commit to a diagnosis. Existing medical dialogue corpora are largely dyadic or lack the multi-party workflow and annotations needed for this setting. We introduce an ePCR-grounded, topic-flow-based multi-agent generation pipeline that iteratively plans, generates, and self-refines dialogues with rule-based factual and topic flow checks. The pipeline yields EMSDialog, a dataset of 4,414 synthetic multi-speaker EMS conversations based on a real-world ePCR dataset, annotated with 43 diagnoses, speaker roles, and turn-level topics. Human and LLM evaluations confirm high quality and realism of EMSDialog using both utterance- and conversation-level metrics. Results show that EMSDialog-augmented training improves accuracy, timeliness, and stability of EMS conversational diagnosis prediction. Our datasets and code are publicly available at https://uva-dsa.github.io/EMSDialog

Figures & tables

Figure 1: Qwen3-series model performance on conversational diagnosis prediction for cardiac arrest scenario

Data characteristics

Generation method

Data Use

Dataset

Setting

Lang.

#Roles

#Dx

Topic

#U/D

Gen.

Src.

Checker

Refiner

Tasks

Real

Dxy Xu et al. (2019)

Telehealth

✗

–

Human

Web

–

Diagnosis Pred

MedDialog Zeng et al. (2020)

Telehealth

cn,en

✗

3.3

Human

Web

–

Generation

MedDG Liu et al. (2022)

Telehealth

✗

21.6

Human

Web

–

NER

MidMed Shi et al. (2023)

Telehealth

✓

11.8

Human

Web

–

Generation

MTS-Dialogue Abacha et al. (2023)

Clinical

–

✗

9.0

Human

Roleplay

–

Note2Dialog

Table 1: Comparison of real vs. synthetic medical multi-turn dialogue datasets. #Roles = Average number of speaker roles per dialogue. #Dx = Number of diagnosis classes. #U/D = Average number of utterances per dialogue.

Figure 2: a) ePCR; b) EMS Topic Flow; c) Synthetic Dialogue Generation Pipeline; d) Synthetic Dialogue Example. Color-highlighted text across (a) and (d) demonstrates the factual grounding of generated dialogue concepts to the original ePCR data.

	Conversation-level					Utterance-level
Method	Logic ( $\uparrow$ )	MRR ( $\uparrow$ )	Fac NER ( $\uparrow$ )	Fac H* ( $\uparrow$ )	Diversity ( $\downarrow$ )	Realism ( $\uparrow$ )	Safety ( $\uparrow$ )	Role Acc ( $\uparrow$ )	Groundedness ( $\uparrow$ )
	H* / LLM (1–5)	H* / LLM (%)	P / R (%)	P / R (%)	Self-BLEU (%)	H* / LLM (%)	H* / LLM (%)	H* / LLM (%)	H* / LLM (%)
0-shot	$1.85$ / $2.45$	$31.14$ / $32.83$	$71.32\,/\ 55.88$	$67.69\,/\ 64.71$	$\mathbf{40.36}$	$44.96$ / $52.77$	$96.77$ / $95.76$	$96.03$ / $87.39$	$80.76$ / $75.85$
0-shot+Rules	$2.91$ / $3.55$	$48.07$ / $50.58$	$73.58\,/\ 54.13$	$69.23\,/\ 66.88$	$65.73$	$55.68$ / $63.91$	$96.91$ / $94.88$	$96.74$ / $95.34$	$82.17$ / $77.12$
CoT	$3.05$ / $3.55$	$37.50$ / $36.67$	$75.76\,/\ 58.79$	$68.15\,/\ 66.09$	$\underline{40.43}$	$78.23$ / $70.38$	$99.85$ / $95.46$	$98.51$ / $93.55$	$87.08$ / $85.79$
CoT+Rules	$\underline{3.63}$ / $\underline{3.80}$	$\underline{58.38}$ / $\underline{55.33}$	$78.23\,/\ 61.77$	$74.91\,/\ 68.56$	$61.59$	$\underline{85.59}$ / $76.25$	$99.85$ / $98.21$	$98.87$ / $95.77$	$92.36$ / $86.33$

Table 2: Intrinsic Evaluation: Conversation- and Utterance-level performance of synthetic dialogues generation methods. H*: human evaluation on a 43-scenario subset. The best and runner-up results are in bold and underlined .

Mode	Size	Train Data	Prompt	First Acc ( $\uparrow$ ) / Conf	Last Acc ( $\uparrow$ ) / Conf	Earliness ( $\uparrow$ ) (1st / 1st-correct)	Edit Overheads ( $\downarrow$ )
No Train	4B	–	0-shot	$37.78_{\pm 0.79}/88.57_{\pm 0.44}$	$60.22_{\pm 0.79}/93.78_{\pm 0.46}$	$93.02_{\pm 0.06}/80.20_{\pm 1.63}$	$83.51_{\pm 0.34}$
	4B	–	CoT	$30.00_{\pm 1.36}/88.37_{\pm 1.11}$	$61.67_{\pm 2.72}/95.98_{\pm 0.00}$	$95.18_{\pm 0.08}/81.73_{\pm 0.44}$	$73.26_{\pm 0.29}$
	32B	–	0-shot	$\mathbf{63.89}_{\pm 0.79}/88.07_{\pm 0.15}$	$\mathbf{80.56}_{\pm 3.14}/94.20_{\pm 0.20}$	$92.41_{\pm 0.00}/84.93_{\pm 0.93}$	$\mathbf{57.11}_{\pm 0.71}$
	32B	–	CoT	$51.11_{\pm 2.08}/81.60_{\pm 0.79}$	$76.67_{\pm 1.36}/93.07_{\pm 0.37}$	$94.44_{\pm 0.08}/\mathbf{88.73}_{\pm 1.05}$	$60.32_{\pm 1.20}$
Static Train	4B	Real	–	$58.23_{\pm 4.57}/67.38_{\pm 1.26}$	$63.98_{\pm 1.34}/78.53_{\pm 1.26}$	$90.10_{\pm 2.17}/80.81_{\pm 2.13}$	$69.07_{\pm 1.32}$
		0-shot	–	$68.33_{\pm 2.08}/69.27_{\pm 2.47}$	$70.78_{\pm 0.00}/85.18_{\pm 3.33}$	$81.32_{\pm 1.75}/82.47_{\pm 2.96}$	$61.91_{\pm 2.92}$

Table 3: Conversational diagnosis prediction results of Qwen3-4B/32B models.

Figure 3: Ablation results. (a-b) Downstream forecasting performance: last accuracy and edit overheads. (c-d) Conversation-level and utterance-level evaluation.

Component	Precision $\uparrow$	Recall $\uparrow$
NER extractor ( $P_{\text{src}/\text{tgt}}$ vs. $G_{\text{src}/\text{tgt}}$ )	74.26	62.29
LLM generator ( $G^{\prime}_{\text{tgt}}$ vs. $G_{\text{src}}$ )	98.41	71.68
Concept Checker: hallucination ( $FP$ )	81.52	86.00
Concept Checker: missing ( $FN$ )	83.74	85.23
Style Checker	77.02	–

Table 4: Error source decomposition on 43 manually annotated scenarios.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Data	Dataset	#Dialogues	#Diagnoses	#Utterances	#Tokens	Vocab	#U/D	#T/U
Synthetic	0-shot	4,411	43	255,145	3,875,249	30,185	57.84	15.19
	CoT	4,411	43	264,499	3,333,248	36,674	59.96	12.60
	NoteChat	4,411	43	804,645	14,741,540	12,564	182.42	18.32
	EMSDialog	4,411	43	503,948	4,297,678	18,291	114.25	8.53
Real-world	Total	149	9	18,410	113,192	2,836	123.56	6.15
	Train	89	5	11,393	68,747	2,310	128.01	6.03

Appendix

Table 5: Statistics of synthetic and real-world dialogue datasets. #U/D = average utterances per dialogue. #T/U = average tokens per utterance.

	Conversation ( $\uparrow$ )		Utterance ( $\uparrow$ )
LLM Judge	Logic	Ranking	Realism	Safety	Role Acc	Grounded
Qwen3-235B	0.634	0.685	0.580	0.784	0.747	0.671
Llama-3.3-70B	0.598	0.630	0.534	0.751	0.691	0.644

Appendix

Table 6: Human-LLM Agreement. We report Spearman Correlation for conversation-level metrics and Krippendorff’s

\alpha

for utterance-level metrics on the 43-scenario human evaluation subset.

Mode	Size	Train Data	Prompt	First Acc ( $\uparrow$ ) / Conf	Last Acc ( $\uparrow$ ) / Conf	Earliness ( $\uparrow$ ) (1st / 1st-correct)	Edit Overheads ( $\downarrow$ )
No Train	0.6B	–	0-shot	$0.00_{\pm 0.00}/74.85_{\pm 0.00}$	$12.22_{\pm 0.79}/76.17_{\pm 0.00}$	$93.01_{\pm 0.41}/73.76_{\pm 5.19}$	$96.41_{\pm 0.01}$
	0.6B	–	CoT	$8.89_{\pm 2.83}/77.75_{\pm 0.64}$	$19.44_{\pm 1.57}/80.72_{\pm 0.40}$	$\mathbf{96.89}_{\pm 0.23}/76.31_{\pm 1.09}$	$97.89_{\pm 0.17}$
	4B	–	0-shot	$37.78_{\pm 0.79}/88.57_{\pm 0.44}$	$60.22_{\pm 0.79}/93.78_{\pm 0.46}$	$93.02_{\pm 0.06}/80.20_{\pm 1.63}$	$83.51_{\pm 0.34}$
	4B	–	CoT	$30.00_{\pm 1.36}/88.37_{\pm 1.11}$	$61.67_{\pm 2.72}/95.98_{\pm 0.00}$	$95.18_{\pm 0.08}/81.73_{\pm 0.44}$	$73.26_{\pm 0.29}$
	32B	–	0-shot	$\mathbf{63.89}_{\pm 0.79}/88.07_{\pm 0.15}$	$\mathbf{80.56}_{\pm 3.14}/94.20_{\pm 0.20}$	$92.41_{\pm 0.00}/84.93_{\pm 0.93}$	$\mathbf{57.11}_{\pm 0.71}$
	32B	–	CoT	$51.11_{\pm 2.08}/81.60_{\pm 0.79}$	$76.67_{\pm 1.36}/93.07_{\pm 0.37}$	$94.44_{\pm 0.08}/\mathbf{88.73}_{\pm 1.05}$	$60.32_{\pm 1.20}$

Appendix

Table 7: Conversational diagnosis prediction results of Qwen3-0.6B models.

Figure 4: Ablation Study: Conversational Diagnosis Prediction Performance

Figure 5: Ablation Study on Checker: Extrinsic Evaluation. Con: Concept Checker Only. TF: Topic Flow Checker Only. Style: Style Checker only.

Figure 6: Ablation Study on Checker: Intrinsic Evaluation. None: w/o Checkers. Con: Concept Checker Only. TF: Topic Flow Checker Only. Style: Style Chekcer Only. Logic (scale 1-5) is transform to scale 20-100

	Conversation-level				Utterance-level
Method	Logic ( $\uparrow$ )	MRR ( $\uparrow$ )	Fac NER ( $\uparrow$ )	Diversity ( $\downarrow$ )	Realism ( $\uparrow$ )	Safety ( $\uparrow$ )	Role Acc ( $\uparrow$ )	Groundedness ( $\uparrow$ )
	LLM (1–5)	LLM (%)	P / R (%)	Self-BLEU (%)	LLM (%)	LLM (%)	LLM (%)	LLM (%)
Qwen3-32B	$\mathbf{2.63}$	$\mathbf{86.25}$	$\mathbf{71.32}\,/\ \mathbf{55.88}$	$\mathbf{40.36}$	$\mathbf{54.10}$	$\mathbf{93.30}$	$\mathbf{86.23}$	$\mathbf{76.88}$
LLama-3.3-70B	$2.33$	$63.75$	$68.24\,/\ 55.14$	$62.83$	$45.70$	$90.85$	$83.35$	$62.56$

Appendix

Table 8: Qwen3-32B vs. LLama-3.3-70B performance in synthetic dialogue generation. Metrics are evaluated by Qwen3-235B model.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare
Mar 25, 2026Shubham Kumar Nigam, Suparnojit Sarkar, Piyush PatelMedical LLMs Text Generation
University of Birmingham, Dubai, United Arab Emirates · Heritage Institute of Technology, Kolkata, India · Madan Mohan Malaviya University of Technology, India
Decoupling Patient Personas and Clinical Workflows: A Dual-Agent Approach to Realistic Medical Dialogue Generation
Oct 23, 2026Qiyuan Li, Xiaomei Guo, Chao Gao +4Synthetic Clinical Data Generation Text Generation
School of Computer Science and Technology, Wuhan University of Science and Technology, China · Department of Cardiology, Tongji Hospital, Huazhong University of Science and Technology, China · Department of Computer Science, Vrije Universiteit Amsterdam, Netherlands +1
MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments
May 8, 2026Yicheng Gao, Xiaolin Zhou, Yahan Li +2Healthcare Medical Diagnosis
University of Southern California · Arizona State University

Mar 25, 2026J/K move · Enter open · S save

MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare

Shubham Kumar Nigam, Suparnojit Sarkar, Piyush Patel

University of Birmingham, Dubai, United Arab Emirates · Heritage Institute of Technology, Kolkata, India · Madan Mohan Malaviya University of Technology, India

Medical LLMs Text Generation Multilingual Language Models Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Conversational artificial intelligence has the potential to assist users in preliminary medical consultations, particularly in settings where access to healthcare professionals is limited. However, many existing medical dialogue systems operate in a single-turn question--answering paradigm or rely on template-based datasets, limiting conversational realism and multilingual applicability. In this work, we introduce MedAidDialog, a multilingual multi-turn medical dialogue dataset designed to simulate realistic physician--patient consultations. The dataset extends the MDDial corpus by generating synthetic consultations using large language models and further expands them into a parallel multilingual corpus covering seven languages: English, Hindi, Telugu, Tamil, Bengali, Marathi, and Arabic. Building on this dataset, we develop MedAidLM, a conversational medical model trained using parameter-efficient fine-tuning on quantized small language models, enabling deployment without high-end computational infrastructure. Our framework additionally incorporates optional patient pre-context information (e.g., age, gender, allergies) to personalize the consultation process. Experimental results demonstrate that the proposed system can effectively perform symptom elicitation through multi-turn dialogue and generate diagnostic recommendations. We further conduct medical expert evaluation to assess the plausibility and coherence of the generated consultations.

Abstract

Synthetic Data Generation for Language Models

Explore similar work

MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare

Decoupling Patient Personas and Clinical Workflows: A Dual-Agent Approach to Realistic Medical Dialogue Generation

MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments