cs.CLNov 16, 2026

DiagTrace: An Interactive Benchmark for Clinical Decision Making

Authors: Yufan Wang, Rui Yang, Yi Lin, Chunhua Weng, Yifan Peng

Organizations: Weill Cornell Medicine, New York, NY, USA · Cornell University, Ithaca, NY, USA · Columbia University, New York, NY, USA

Abstract

Large language models (LLMs) have shown strong performance on medical question answering and diagnostic reasoning benchmarks. However, existing evaluations typically provide all patient information upfront and assess only the final diagnosis or rationale. This setup ignores the need to acquire diagnostic evidence incrementally from incomplete information in real-world diagnosis. To bridge this gap, we introduce DiagTrace, a source-grounded benchmark for evaluating whether LLMs can reconstruct the documented diagnostic trajectory through interactive information seeking. Each benchmark instance is built from a PubMed Central case report and includes an initial presentation, an event bank, a clinical course, and diagnostic references. DiagTrace contains 510 cases across eight specialties, with 225 used to evaluate four LLMs on clinical-course alignment, interaction quality, and efficiency. Results reveal trade-offs in coverage, source-consistent ordering, and turn use.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

    May 22, 2026Jiazhen Pan, Weixiang Shen, Jun Li +7Medical ImagingRadiology VLMs

  2. MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

    Aug 25, 2026Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz +2Clinical Language Model EvaluationDifferential Diagnosis

  3. MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

    May 8, 2026Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang +9HealthcareMedical Diagnosis