Organizations: REMEDI Lab, Universit`a della Svizzera italiana, Lugano, Switzerland. · Department of Statistics and Quantitative Methods, University of Milano-Bicocca, Milan, Italy. · Department of Mathematics, Pavia University, Pavia, Italy. · Department of Science and High Technology, Insubria University, Como, Italy. · Cantonal Sociopsychiatric Organisation, Public Health Division, Department of Health and Social Care, Repubblica e Cantone Ticino, Mendrisio, Switzerland.
Large language models (LLMs) achieve expert-level diagnostic accuracy in static evaluations, where the model receives a curated vignette or case report that already condenses the diagnostically relevant findings. Such evaluations measure inference over already selected evidence; the evaluations do not account for the inquiry used to obtain it. In clinical practice, however, diagnosis is an iterative process of inquiry. Clinicians take the patient's history, or anamnesis (Hx), through targeted questioning, examine the patient, and request and interpret tests, progressively narrowing the differential until they reach a final diagnosis (Dx). We introduce Hx2Dx ARENA, an interactive benchmark of 1'000 clinical cases from the New England Journal of Medicine (NEJM). Each case is a simulated encounter that proceeds from initial presentation through history taking and physical examination to diagnostic workup. We evaluate open- and closed-source LLMs under multiple configurations. Performance varies considerably across models in the interactive setting, and every model loses diagnostic accuracy relative to the static setting, where the full case is presented upfront. The ability to ask the right questions accounts for much of both the gap between models and the loss of accuracy: diagnosis improves as questions become more targeted and build a more complete clinical picture. LLMs thus reason well once the relevant data are available but remain limited in hypothesis-driven information gathering, in knowing what to ask in order to know. Closing this gap is a prerequisite for their safe integration into real-world diagnostic workflows.
Figures & tables
Figure 1: (A) Overview of Hx2Dx ARENA. Each NEJM case is decomposed into a reference diagnosis, an investigation registry, and a simulated-patient script. Starting from a one-sentence referral, the Doctor questions the Patient and requests investigations from Data , which returns only documented results. The Examiner grades the final diagnosis as correct, partial or, incorrect. (B) Visual summary of the three main Research Questions.
Model
Access
Parameters
Multimodal
Built-in thinking
GPT-6 Luna
Proprietary
–
✓
Yes
Gemini 3.5 Flash Lite
Proprietary
–
✓
Yes
DeepSeek V4.1 Flash
Open-weight
763B
✓
Yes
GLM 5.3 Flash
Open-weight
321B
✓
Yes
Llama 4 Scout
Open-weight
109B
✓
No
Gemma 3 27B IT
Open-weight
27B
✓
No
Table 1: Models evaluated as the physician agent. Parameter counts are reported as total parameters; “-” denotes undisclosed values.
Figure 2: Diagnostic performance of seven LLMs on 1,000 NEJM clinical cases in the static and conversational settings. (A) Distribution of judge verdicts (Correct, Partially correct, Incorrect) as a percentage of cases. For each model, the upper (light) bar shows the static setting, where a description of case report is given, and the lower (dark) bar shows the conversational setting, where the model has to gather information by questioning a simulated patient. (B) Diagnostic accuracy, defined as Correct + Partially correct, with 95% bootstrap confidence intervals (10’000 resamples), for the static ( gray ) and conversational (black) settings. Brackets mark pairwise comparisons between models within the static setting (left) and within the conversational setting (right). Each bracket group compares one reference model, marked with a gold star ( ★ ), against each of the other models. We used exact McNemar tests with Benjamini–Hochberg FDR correction ( ∗∗∗p<0.001,∗∗p<0.01,∗p<0.05 ); only significant comparisons are shown. (C) Accuracy loss when moving from the static to the conversational setting (Static - Conversation, in percentage points), with 95% bootstrap CIs. Stars give the significance of the difference (McNemar tests with Benjamini–Hochberg FDR correction). Models are ordered by accuracy loss.
Figure 3: Diagnostic accuracy when all models receive the same conversation transcripts from GLM 5.3 Flash, the best performing model. For each model, the left stacked bar shows the number of cases judged Correct or Partially correct, with the total above the bar. The right bar shows Incorrect cases. Error bars are 95% bootstrap confidence intervals (10’000 resamples). Brackets mark significant pairwise differences in accuracy (Correct + Partially correct), from exact McNemar tests with Benjamini–Hochberg FDR correction ( ∗∗∗p<0.001,∗∗p<0.01,∗p<0.05 ).
Figure 4: Diagnostic accuracy in the conversational setting, stratified by main clinical specialty. Each cell gives the percentage of cases judged Correct or Partially correct for one model (rows) and one specialty (columns). Only specialties with at least 10 cases are shown; the number of cases (n) is given below each specialty name. Specialties are ordered by number of cases. The bottom row is the unweighted mean across the seven models.
Figure 5: Process metrics and diagnostic accuracy in the conversational setting. (A) Turns used and (B) investigations requested per case, for correctly (Correct or Partially correct) and incorrectly diagnosed cases. Boxes show the median and IQR, and dots are outliers. Thick bars mark distributions collapsed at the 50-turn limit. Within each model, the two groups were compared with Mann–Whitney U tests, with Benjamini–Hochberg FDR correction across all 14 tests ( ∗∗∗p<0.001,∗∗p<0.01,∗p<0.05 ; ns, not significant).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Correct
Partially correct
Incorrect
GLM 5.3 Flash
366
178
456
DeepSeek V4.1 Flash
327
176
497
GPT Luna 6
276
198
526
Gemini 3.5 Flash Lite
192
193
615
Gemma 3 27B IT
62
195
743
Ministral 8B
63
156
781
Appendix
Table 2: Judge verdicts in the conversational setting (correct, partially correct, and incorrect). Cell shading is proportional to the percentage within each column; the highest value per column is in bold.
Large language models (LLMs) are increasingly used for health-related decision support. Yet most evaluations treat diagnosis as a single-shot task with complete information provided upfront, often as a multiple-choice selection. This diverges from clinical practice, where diagnosis is interactive and open-ended, involving sequential hypothesis refinement through targeted questioning. We address this gap. We build MeDxBench, a large-scale benchmark of 4,421 clinical cases across 20 specialties. We further propose MeDxAgent, a multi-agent consultation system for interactive diagnosis, and systematically study its prompt-, flow- and agent-level design choices. MeDxAgent achieves a 10.3% accuracy gain over the baseline on MeDxBench, closing 52.3% of the gap to a full-information oracle. We find that specific design choices: collecting demographics first, passing summarized dialogue for diagnosis, and feeding candidate diagnoses for targeted questioning, improve accuracy, mirroring how physicians reason, though their effect emerges fully only in combination. Code and dataset will be released upon publication.
Akshat Sanghvi, Naren Akash, Raza Imam +2
Microsoft Research India · Institute of AI and Innovation, AIG Hospitals · Mohamed bin Zayed University of Artificial Intelligence
Large language models perform well on static medical examinations, yet clinical diagnosis often requires iterative evidence gathering under uncertainty. Building on prior interactive evaluation efforts, we introduce an OSCE-inspired standardized patient simulator and a controlled, reproducible benchmark for active diagnostic inquiry. Across 468 cases and 15 models in our protocol, we observe that multi-turn evidence seeking reduces diagnostic accuracy by 12.75% and lowers supporting-evidence quality by 24.36% relative to full-context evaluation; error analyses associate these drops with premature diagnostic closure and inefficient questioning. Together, these results suggest that static full-context benchmarks may overestimate performance in interactive evidence-seeking settings, motivating complementary interactive assessment for safer clinical decision support.
Chen Zhan, Xihe Qiu, Xiaoyu Tan +8
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, China. · Department, Tencent Youtu Lab, Shanghai, 200233, China. · Department of Oncology, Jinshan Hospital, Fudan University, Shanghai, 201508, China. +4
The application of large language models (LLMs) in clinical decision support faces significant challenges of "tunnel vision" and diagnostic hallucinations present in their processing unstructured electronic health records (EHRs). To address these challenges, we propose a novel chain-based clinical reasoning framework, called DxChain, which transforms the diagnostic workflow into an iterative process by mirroring a clinician's cognitive trajectory that consists of "Memory Anchoring", "Navigation" and "Verification" phases. DxChain introduces three key methodological innovations to elicit the potential of LLM: (i) a Profile-Then-Plan paradigm to mitigate cold-start hallucinations by establishing a panoramic patient baseline, (ii) a Medical Tree-of-Thoughts (Med-ToT) algorithm for strategic look ahead planning and resource aware navigation, and (iii) a Dialectical Diagnostic Verification procedure utilizing "Angel-Devil" adversarial debates to resolve complex evidence conflicts. Evaluated on two real world benchmarks, MIMIC-IV-Ext Cardiac Disease and MIMIC-IV-Ext CDM, DxChain achieves state-of-the-art performances in both diagnostic accuracy and logical consistency, offering a modular and reliable architecture for next-generation clinical AI. The code is at https://anonymous.4open.science/r/Dx-Chain.
Zhiqi Lv, Duofan Tu, Jun Li +4
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, P.R. China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, 215123, P.R. China · Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing, PR China +2