Large language models (LLMs) have shown strong performance on medical question answering and diagnostic reasoning benchmarks. However, existing evaluations typically provide all patient information upfront and assess only the final diagnosis or rationale. This setup ignores the need to acquire diagnostic evidence incrementally from incomplete information in real-world diagnosis. To bridge this gap, we introduce DiagTrace, a source-grounded benchmark for evaluating whether LLMs can reconstruct the documented diagnostic trajectory through interactive information seeking. Each benchmark instance is built from a PubMed Central case report and includes an initial presentation, an event bank, a clinical course, and diagnostic references. DiagTrace contains 510 cases across eight specialties, with 225 used to evaluate four LLMs on clinical-course alignment, interaction quality, and efficiency. Results reveal trade-offs in coverage, source-consistent ordering, and turn use.
Figures & tables
Figure 1: Overview of DiagTrace . (A) PMC case reports are converted into validated reference resources. (B) A diagnosis-aware clinician agent interacts with a source-grounded report agent under a fixed turn budget to reconstruct the documented diagnostic trajectory. (C) Interactions are evaluated using clinical course alignment, interactive quality, turn efficiency and availability.
Specialty
Cases
Events
Figure-linked rate
Dermatology
49
541
0.325
Emergency Medicine
25
307
0.218
Internal Medicine
24
293
0.270
Nephrology
29
366
0.235
Neurology
95
1,128
0.239
Oncology
112
1,284
0.321
Table 1: Characteristics of DiagTrace .
Category
Weight
Distribution
Pathology
2.00
7.2%
Imaging
1.50
22.7%
Laboratory tests
1.00
14.9%
Microbiology
1.00
2.7%
Procedure/intervention
1.00
5.8%
Treatment/medication
0.75
10.8%
Table 2: Request/response taxonomy
Metric
GPT
Claude
Gemini
Kimi
Request-category alignment
Tversky alignment
0.360
0.334
0.359
0.399
LCS recall
0.465
0.510
0.500
0.561
Pairwise order accuracy
0.646
0.690
0.654
0.601
Coverage decomposition and request behavior
Precision
0.626
0.560
0.607
0.623
Table 3: Interactive reconstruction results.
Figure 2: Category-wise recall. Each cell is the fraction of cases in which a source-derived category was requested by the clinician agent.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Model
Checklist
Δ
95% CI
GPT
0.546
0.524
+0.022
[−0.055,0.104]
Claude
0.638
0.540
+0.097
[−0.019,0.216]
Gemini
0.689
0.531
+0.159
[0.047,0.277]
Kimi
0.656
0.449
+0.207
[0.108,0.327]
Appendix
Table 4: Event-level recall on the 16-case fixed-checklist evaluation. Checklist scores are matched to each model’s case-specific acquisition-turn budget. Differences and confidence intervals are based on paired case-level comparisons.
Model ( n )
Tversky
LCS
Pair.
F1
Event R
GPT (225)
.360
.465
.646
.592
.577
GPT (502)
.393
.502
.669
.618
.575
Claude (225)
.334
.510
.690
.589
.665
Claude (498)
.362
.537
.679
.610
.666
Gemini (225)
.359
.500
.654
.603
.655
Gemini (502)
.380
.535
.670
.622
.654
Appendix
Table 5: Matched-subset and full completed-case sensitivity results. Pair. denotes pairwise order accuracy; Event R denotes event-level recall. Full-case estimates are not strictly paired across models.
Figure 3: Example diagnosis-aware interactive reconstruction episode. The clinician agent is initialized with the initial presentation and known diagnosis, actively requests source-backed information, and reconstructs the diagnostic trajectory under a fixed turn budget.
Medical diagnosis is not a single prediction from a fully specified vignette. It is a sequential workup: clinicians decide what evidence to obtain, revise a differential diagnosis, and stop when the diagnosis is sufficiently supported. Most medical AI benchmarks instead reveal the relevant context upfront and score only the final answer, making unsupported correct guesses, premature closure, inefficient workups, and poor uncertainty updating invisible. We introduce DDX-TRACE, a physician-adjudicated benchmark for multimodal neuroradiology that evaluates diagnostic trajectories under hidden evidence over 211 challenging cases. Each case begins with limited clinical history; models request imaging studies in free form, receive matched image bundles when available, update a probabilistic differential diagnosis after each turn, and stop with a localized final diagnosis. Evaluating state-of-the-art VLMs, we find that final diagnosis scores can substantially misrepresent workup quality: models may guess plausible diagnoses without essential evidence, request useful studies but misinterpret raw images, or acquire evidence inefficiently while updating uncertainty poorly. Controlled evidence variants isolate bottlenecks in planning, visual evidence extraction, and downstream differential reasoning. DDX-TRACE shifts medical AI evaluation from final answers to evidence-supported diagnostic trajectories.
Jiazhen Pan, Weixiang Shen, Jun Li +7
Technical University of Munich (TUM) · TUM University Hospital · Munich Center for Machine Learning (MCML) +3
Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.
Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz +2
Technical University of Munich · Department of Internal Medicine, UT Southwestern
Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplification of real clinical practice. We study active diagnosis: the real-life clinical process of starting from initial observation, ordering tests, interpreting results, and updating a differential diagnosis across multiple turns. Through systematic analysis, we identify three recurring failure modes in current LLMs: ungrounded test ordering, unreliable diagnostic update, and degraded multi-turn coherence. Together, these failures reveal a core deficit: existing medical training data teaches models to reason from complete information but not to act under evolving, partial evidence. To address this gap, we introduce MedAction, a tree-structured distillation pipeline that synthesizes diverse and high-quality multi-turn diagnostic trajectories via LLM-environment interaction. We propose two knowledge-graph-grounded metrics to filter trajectory quality: Disease Trajectory Consistency (DTC), which tracks whether the model's hypothesis converges toward the correct diagnosis, and Reasoning-Action Consistency (RAC), which verifies that belief updates are driven by gathered evidence. Using this pipeline, we construct MedAction-32K, a dataset of 32,681 trajectories from 2,896 PMC cases. Fine-tuning an 8B model on MedAction-32K achieves state-of-the-art performance among open-source models on both MedR-Bench and our curated MedAction-300-Hard benchmark, pushing the edge for open-source medical LLMs.
Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang +9
National Chengchi University · Georgetown University · University of Michigan +3