Large language models (LLMs) have shown strong performance on medical question answering and diagnostic reasoning benchmarks. However, existing evaluations typically provide all patient information upfront and assess only the final diagnosis or rationale. This setup ignores the need to acquire diagnostic evidence incrementally from incomplete information in real-world diagnosis. To bridge this gap, we introduce DiagTrace, a source-grounded benchmark for evaluating whether LLMs can reconstruct the documented diagnostic trajectory through interactive information seeking. Each benchmark instance is built from a PubMed Central case report and includes an initial presentation, an event bank, a clinical course, and diagnostic references. DiagTrace contains 510 cases across eight specialties, with 225 used to evaluate four LLMs on clinical-course alignment, interaction quality, and efficiency. Results reveal trade-offs in coverage, source-consistent ordering, and turn use.
Figures & tables
Figure 1: Overview of DiagTrace . (A) PMC case reports are converted into validated reference resources. (B) A diagnosis-aware clinician agent interacts with a source-grounded report agent under a fixed turn budget to reconstruct the documented diagnostic trajectory. (C) Interactions are evaluated using clinical course alignment, interactive quality, turn efficiency and availability.
Specialty
Cases
Events
Figure-linked rate
Dermatology
49
541
0.325
Emergency Medicine
25
307
0.218
Internal Medicine
24
293
0.270
Nephrology
29
366
0.235
Neurology
95
1,128
0.239
Oncology
112
1,284
0.321
Table 1: Characteristics of DiagTrace .
Category
Weight
Distribution
Pathology
2.00
7.2%
Imaging
1.50
22.7%
Laboratory tests
1.00
14.9%
Microbiology
1.00
2.7%
Procedure/intervention
1.00
5.8%
Treatment/medication
0.75
10.8%
Table 2: Request/response taxonomy
Metric
GPT
Claude
Gemini
Kimi
Request-category alignment
Tversky alignment
0.360
0.334
0.359
0.399
LCS recall
0.465
0.510
0.500
0.561
Pairwise order accuracy
0.646
0.690
0.654
0.601
Coverage decomposition and request behavior
Precision
0.626
0.560
0.607
0.623
Table 3: Interactive reconstruction results.
Figure 2: Category-wise recall. Each cell is the fraction of cases in which a source-derived category was requested by the clinician agent.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Model
Checklist
Δ
95% CI
GPT
0.546
0.524
+0.022
[−0.055,0.104]
Claude
0.638
0.540
+0.097
[−0.019,0.216]
Gemini
0.689
0.531
+0.159
[0.047,0.277]
Kimi
0.656
0.449
+0.207
[0.108,0.327]
Appendix
Table 4: Event-level recall on the 16-case fixed-checklist evaluation. Checklist scores are matched to each model’s case-specific acquisition-turn budget. Differences and confidence intervals are based on paired case-level comparisons.
Model ( n )
Tversky
LCS
Pair.
F1
Event R
GPT (225)
.360
.465
.646
.592
.577
GPT (502)
.393
.502
.669
.618
.575
Claude (225)
.334
.510
.690
.589
.665
Claude (498)
.362
.537
.679
.610
.666
Gemini (225)
.359
.500
.654
.603
.655
Gemini (502)
.380
.535
.670
.622
.654
Appendix
Table 5: Matched-subset and full completed-case sensitivity results. Pair. denotes pairwise order accuracy; Event R denotes event-level recall. Full-case estimates are not strictly paired across models.
Figure 3: Example diagnosis-aware interactive reconstruction episode. The clinician agent is initialized with the initial presentation and known diagnosis, actively requests source-backed information, and reconstructs the diagnostic trajectory under a fixed turn budget.