MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

Pia Chouayfati; Alexander M. Fichtl; Miriam Anschütz; George Doumat; Georg Groh

cs.CLAug 25, 2026

MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

Authors: Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz, George Doumat, Georg Groh

Organizations: Technical University of Munich · Department of Internal Medicine, UT Southwestern

Abstract

Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.

Figures & tables

Figure 1: The UMLS (Unified Medical Language System) Bodenreider (2004) is a metathesaurus of international biological and medical knowledge bases, ontologies and vocabularies built for healthcare and medical system interoperability. It indexes unique concepts by CUI (unique concept identifier), and instantiates a semantic network (Fig. 2 ) which links a single concept across all participating source vocabularies.

Figure 2: Clinically-relevant subset of the UMLS Semantic Network, showing Semantic Type nodes and typed relation edges. Nodes are color-coded by semantic category: $\bullet$ Events (actions occurring in time), e.g. Procedure $\bullet$ Entities (physical matter and anatomy) e.g. Clinical Drug $\bullet$ Conceptual Entities (abstract classifications and states), e.g. Disease or Syndrome Solid edges denote hierarchical is a relations; dashed edges denote typed associative relations.

Figure 3: ICD-10-CM code structure illustrated with S52.501A (unspecified fracture of the lower end of the right radius, initial encounter for closed fracture). Each position encodes a specific clinical dimension: category, anatomic site/etiology, severity, and encounter extension. The WHO ICD-10 equivalent without the CM extension would be S52.5 .

Figure 4: (Condensed) example of ontology-enabled dialogue evaluation - MTDiag AJCR case. Dx: Giant Cell Arteritis (GCA, M31.6) Szydełko-Paśko et al. (2022) . Left (of dashed line): the PatientVector * for this case, comprising the ground-truth diagnosis (masked from examiner) and UMLS CUI -mapped symptoms (subset only). Utterances are pre-generated offline by UserLM-8B for each element of the PatientVector , constituting the CUI -mapped utterance pool. At dialogue runtime, a Patient Orchestrator LLM answers examiner questions from the utterance pool. Right: Dialogue runtime: Two dialogues conducted from the same chief complaint utterance. Dialogue A : examiner probes for scalp tenderness, jaw claudication, and visual field loss and reaches the correct diagnosis. Dialogue B : examiner anchors prematurely on migraine/tension headache, asks only about stress, screen time, hydration, and family history, and diagnoses tension-type headache. A Harm Tier 2 error (delay of care) occurs. *simplified

Figure 5: Construction of MTDiag A. Data Collection & Pre-Processing : the three source datasets (DDXPlus, MIMIC-IV, AJCR) and their heterogeneous sparse feature spaces, preprocessed to satisfy the minimum anchor requirements and patient realism. B. Anchor Resolution : clinical strings from all three sources are segmented by medspaCy NER and matched against a local UMLS installation via QuickUMLS Soldaini and Goharian (2016) , yielding a CUI and mapping_confidence score per span. Diagnoses follow a separate ICD-10 crosswalk path. The pipeline runs fully locally against a fixed UMLS release, producing reproducible, verifiable mappings with no external API calls. C. Utterance generation: For each item in CasePresentation , UserLM-8B is conditioned on PatientVector context and generates candidate utterances spanning expression styles (precise, colloquial, third-party); masking and persona parameters or noise injections can be injected at generation time. Elements of the PatientVector context can be selectively masked to produce presentation variants. D. Dataset Artefact: The dataset proper consists of the PatientVector , and the dataset artefact , is the result of utterance generation over the PatientVector . This artefact is structured, reproducible, and fully decoupled from the dialogue runtime described in Section 4.4 .

Source	Type	Scale (cases)
DDXPlus	Synthetic	1.3M
MIMIC-IV	Real EHR	68K
AJCR	Case reports	415

Table 1: Data sources used in MTDiag after filtering and deduplication (detailed in Appendix B.2.2 ).

Figure 6: PatientVector schema. Left : the four top-level blocks: Provenance, GroundTruth (masked), PatientFacts, and CasePresentation, with their fields. Right : the full class hierarchy, showing how SymptomPresentation decomposes into Symptom ( CUI + string) and Experience (Temporality, Locality, Characterization), and how Diagnosis carries ICD-10 code, CUI , and provenance_tier . All fields are serialized to JSON; sparsity varies by source. *ICD-10 is also indexed in the UMLS, so every ICD-10 code resolves to a CUI .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Directory	Source	Docs	Key Columns
ed/ MIMIC-IV-ED v2.2
triage.csv.gz		Docs	subject_id, stay_id, chiefcomplaint, temperature, heartrate, resprate, o2sat, sbp, dbp, pain, acuity
edstays.csv.gz		Docs	subject_id, hadm_id, stay_id, intime, outtime, arrival_transport, disposition
diagnosis.csv.gz		Docs	stay_id, seq_num, icd_code, icd_version
medrecon.csv.gz		Docs	stay_id, name, gsn, ndc, etc_rn
vitalsign.csv.gz		Docs	stay_id, charttime, temperature, heartrate, resprate, o2sat, sbp, dbp, pain

Appendix

Table 2: MIMIC-IV data file organization, sources, and extracted columns.

Figure 7: MIMIC-IV: Two Tracks/Cohorts

Figure 8: Dialogue runtime : The Patient Orchestrator receives the Examiner’s turn and selects which pre-generated utterances to surface, drawing exclusively from the utterance pool for the current case. Content (what the patient says) is fixed at dataset creation time; selection logic (which utterance to return) is handled at runtime. The Examiner LM receives only the chief complaint at Turn 0 and issues follow-up questions until a terminal diagnosis or turn limit is reached.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 395 papers

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs
May 8, 2026Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang +9Healthcare Medical Diagnosis
1National Chengchi University · 2Georgetown University · University of Michigan +3
MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Sep 11, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Medical Diagnosis Multi-Turn Dialogue Evaluation
MBZUAI · Cairo University · Ain Shams University +2
Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction
Apr 6, 2026Jinrui Fang, Runhan Chen, Xu Yang +9Clinical Language Model Evaluation Medical Diagnosis
The University of Texas at Austin · New York University · The University of Texas Southwestern Medical Center +2

May 8, 2026J/K move · Enter open · S save

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang, Nai-Chia Chen+8

1National Chengchi University · 2Georgetown University · University of Michigan+3

Healthcare Medical Diagnosis Clinical Reasoning LLM Agent Training Differential Diagnosis Medical LLMs

Open paper AskPDF arXiv

Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplification of real clinical practice. We study active diagnosis: the real-life clinical process of starting from initial observation, ordering tests, interpreting results, and updating a differential diagnosis across multiple turns. Through systematic analysis, we identify three recurring failure modes in current LLMs: ungrounded test ordering, unreliable diagnostic update, and degraded multi-turn coherence. Together, these failures reveal a core deficit: existing medical training data teaches models to reason from complete information but not to act under evolving, partial evidence. To address this gap, we introduce MedAction, a tree-structured distillation pipeline that synthesizes diverse and high-quality multi-turn diagnostic trajectories via LLM-environment interaction. We propose two knowledge-graph-grounded metrics to filter trajectory quality: Disease Trajectory Consistency (DTC), which tracks whether the model's hypothesis converges toward the correct diagnosis, and Reasoning-Action Consistency (RAC), which verifies that belief updates are driven by gathered evidence. Using this pipeline, we construct MedAction-32K, a dataset of 32,681 trajectories from 2,896 PMC cases. Fine-tuning an 8B model on MedAction-32K achieves state-of-the-art performance among open-source models on both MedR-Bench and our curated MedAction-300-Hard benchmark, pushing the edge for open-source medical LLMs.

Abstract

Clinical Language Model Evaluation

Explore similar work

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction