MedClarify: An information-seeking AI agent for medical diagnosis with case-specific follow-up questions
Authors: Hui Min Wong, Philip Heesen, Pascal Janetzky, Martin Bendszus, Stefan Feuerriegel
Organizations: LMU Munich, Munich, Germany · Munich Center for Machine Learning, Germany · Department of Neuroradiology, Heidelberg University, Heidelberg, Germany
Large language models (LLMs) are increasingly used for diagnostic tasks in medicine. In clinical practice, the correct diagnosis can rarely be immediately inferred from the initial patient presentation alone. Rather, reaching a diagnosis often involves systematic history taking, during which clinicians reason over multiple potential conditions through iterative questioning to resolve uncertainty. This process requires considering differential diagnoses and actively excluding emergencies that demand immediate intervention. Yet, the ability of medical LLMs to generate informative follow-up questions and thus reason over differential diagnoses remains underexplored. Here, we introduce MedClarify, an AI agent for information-seeking that can generate follow-up questions for iterative reasoning to support diagnostic decision-making. Specifically, MedClarify computes a list of candidate diagnoses analogous to a differential diagnosis, and then proactively generates follow-up questions aimed at reducing diagnostic uncertainty. By selecting the question with the highest expected information gain, MedClarify enables targeted, uncertainty-aware reasoning to improve diagnostic performance. In our experiments, we first demonstrate the limitations of current LLMs in medical reasoning, which often yield multiple, similarly likely diagnoses, especially when patient cases are incomplete or relevant information for diagnosis is missing. We then show that our information-theoretic reasoning approach can generate effective follow-up questioning and thereby reduces diagnostic errors by ~27 percentage points (p.p.) compared to a standard single-shot LLM baseline. Altogether, MedClarify offers a path to improve medical LLMs through agentic information-seeking and to thus promote effective dialogues with medical LLMs that reflect the iterative and uncertain nature of real-world clinical reasoning.
Figures & tables
Figure 1: Overview of MedClarify, an AI agent for medical diagnosis that optimizes the question selection strategy. a , The system generates a set of differential diagnoses with corresponding confidence scores, which are then adjusted using a Bayesian update that combines knowledge from old and new evidence. Based on this improved differential diagnosis, MedClarify selects the optimal follow-up question through optimizing information gain, thus reducing uncertainty. This question is then used to interview the patient, and the patient’s response is integrated into the patient’s case. This iterative diagnostic refinement process is repeated until specific thresholds are met, and eventually outputs a final diagnosis. b , The question selection strategy and technical overview of MedClarify.
Figure 2: Example with proactive, case-specific follow-up questions for medical diagnosis. The figure presents an example patient case with clinical information (top), as well an the diagnostic dialogue generated by the naïve multi-turn baseline system (middle) and by MedClarify (bottom) for the same patient case. Additional dialogues are in Extended Figure 2 and 3 .
Figure 3: Overview of Bayesian update and question selection in MedClarify. a , Question selection strategy in MedClarify can shift LLM focus to alternative condition groups, encouraging to rule out a wider range of diseases for a more reliable final diagnosis. The process begins by generating targeted questions for each predicted disease to refute the highest-probability condition and confirm low-ranked alternatives, plus an additional explorative question. Afterwards, the system selects the question with the highest information gain. In this example, although the initial hypothesis was related to ICD-11 Chapter 13 (disease of the digestive system), MedClarify selects a question that effectively confirms Chapter 18 (pregnancy, childbirth, or the puerperium), leading to the correct diagnostic pathway, which the baseline system would have missed. b , Bayesian update prevents static entropy by incorporating historical diagnoses and temperature scaling.
Figure 4: Decrease in LLM diagnostic accuracy under incomplete patient cases. a , Diagnostic accuracy of Llama-3.3-70B in single-shot without inquiries on complete cases (green) and on cases with a single clinical feature masked (striped), evaluated across all datasets (NEJM, MediQ, and MedQA) for each feature category (laboratory findings, imaging results, symptoms, physical examinations, social history, and past medical history). b , Decrease in diagnostic accuracy (in p.p.) for masked cases relative to the corresponding complete cases across different feature categories (This shows the subtraction of striped bars from solid green bars across feature categories in a ).
Figure 5: MedClarify improves diagnostic accuracy under incomplete patient cases. Top-1 diagnostic accuracy is shown for three LLM-based diagnostic systems on the NEJM dataset: a single-shot baseline (striped), a naïve multi-turn baseline with heuristic follow-up questions (orange), and MedClarify with information-theoretic question selection (purple). All LLM-based systems are instantiated with Llama-3.3-70B. Patient cases are evaluated under single-feature masking, where one clinical feature group (laboratory tests, imaging results, symptoms, physical examination, social history, or past medical history) is removed at a time. All systems are evaluated on the same masked patient cases using an identical multi-agent architecture and differ only in their question selection strategy.
Figure 6: Diagnostic accuracy under joint masking of all feature categories. The experiments are based on a more challenging setting in which all clinical feature categories are masked concurrently, which provides a stricter numerical benchmark to assess when and where information-seeking through follow-up questioning is most effective. a , Comparison of top- k accuracy of MedClarify (instantiated with Llama-3.3-70B as the LLM backbone) and the baseline across the NEJM, MediQ, and MedQA datasets. Statistical significance was tested using 95% confidence intervals ( ∗ : p≤0.05 ; ∗∗ : p≤0.01 ; ∗∗∗ : p≤0.001 ). b , Top-1 diagnostic accuracy of MedClarify (instantiated with Llama-3.3-70B) stratified by medical specialty under joint feature masking, comparing MedClarify with the naïve multi-turn baseline. Overall, MedClarify has a strong performance gain across all medical specialties, but with particularly large relative gains for endocrinology. c , Top-1 diagnostic accuracy of MedClarify (instantiated with Llama-3.3-70B) under joint feature masking to demonstrate the overall difficulty of patient cases across datasets.
Figure 7: Overview of the diagnostic accuracy of MedClarify across different LLM backbones. Here, MedClarify is compared when instantiated with different LLM backbones, namely, GPT-5.1, Deepseek-R1-0528, and Llama-3.3-70B. Reported is the top-1 accuracy of MedClarify (purple) in comparison the naïve multi-turn system across different datasets: a , NEJM; b , MediQ; and c , MedQA.
Figure 8: Analysis of diagnostic belief updating in MedClarify quantified through entropy reduction. We evaluate Shannon entropy (in bits) of the diagnostic probability distribution across successive turns in the diagnostic dialogue. Entropy measures the concentration of probability mass across candidate diagnoses, with lower values indicating stronger belief concentration. Entropy reduction is computed relative to the initial state (round 0), where all candidate diagnoses are initialized with equal probability. Comparison of the average entropy stratified by dataset (NEJM, MediQ, MedQA) for a , MedClarify vs. b , the naïve multi-turn LLM as a baseline. c , Average reduction in entropy per iteration over turns. MedClarify is more effective due to Bayesian belief updating, reflected by a faster concentration of diagnostic beliefs as evidence accumulates.
Chapter ( ci )
c1
c2
c3
c4
c5
c1
1.00
0.35
0.35
0.65
0.20
c2
0.35
1.00
0.50
0.40
0.25
c3
0.35
0.50
1.00
0.60
0.30
c4
0.65
0.40
0.60
1.00
0.40
c5
0.20
0.25
0.30
0.40
1.00
Table 1: LLM-generated similarity matrix for example of ICD chapters 1–5 which enter MedClarify as domain knowledge.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Extended Figure 2: Example with proactive, case-specific follow-up questions for medical diagnosis. The figure presents an example patient case with clinical information (top), as well an the diagnostic dialogue generated by the naïve multi-turn baseline system (middle) and by MedClarify (bottom) for the same patient case.
Extended Figure 3: Example with proactive, case-specific follow-up questions for medical diagnosis. The figure presents an example patient case with clinical information (top), as well an the diagnostic dialogue generated by the naïve multi-turn baseline system (middle) and by MedClarify (bottom) for the same patient case.
Extended Figure 4: Dataset analysis. a , Medical specialties across three datasets: NEJM, MediQ, and MedQA. b , Relative size of the datasets during the experiments.
Extended Figure 5: Overview of the diagnostic accuracy of MedClarify and baseline systems across datasets The comparison of top- k accuracy for MedClarify and baseline across NEJM, MediQ, and MedQA datasets in stacked bar charts.
Extended Figure 6: Results for top-3 accuracy showing how MedClarify improves diagnostic accuracy under incomplete patient cases. Top-3 diagnostic accuracy on NEJM cases with one clinical feature category masked at a time (laboratory findings, imaging results, symptoms, physical examinations, social history, and past medical history), comparing a standalone single-shot LLM baseline (orange) with the MedClarify system (purple). The green bar indicates single-shot performance on complete cases and serves as an upper reference bound.
Extended Figure 7: Efficiency of diagnostic information-seeking in MedClarify under joint masking of all feature categories. Violin plot of the number of questions asked over iterations for MedClarify versus the naïve multi-turn baseline to reach the correct diagnosis. Here, μ^ denotes the mean number of questions per case. Fewer questions indicate more efficient acquisition of diagnostically relevant information through targeted follow-up.
Extended Figure 8: Efficiency of diagnostic information-seeking in MedClarify under joint masking of all feature categories. Shown is the Mean Reciprocal Rank, which measures how early the correct diagnosis appears in the ranked differential diagnosis list. Higher MRR values indicate that MedClarify prioritizes the correct diagnosis earlier than the naïve baseline system.
Extended Figure 9: Confidence scores in MedClarify are clinically informative. a , Evolution of confidence scores across diagnostic iterations comparing MedClarify with the naïve multi-turn baseline. In the case of MedClarify, we see that the belief updates are reflected in increasing confidence scores as evidence accumulates. b , The calibration plot compares actual diagnostic accuracy with predicted confidence for each confidence bin. The dashed line represents perfect calibration. Together, these results show that MedClarify produces confidence estimates that are better aligned with diagnostic correctness, whereas the baseline exhibits higher but poorly calibrated confidence scores.
Extended Figure 10: Overview of the diagnostic accuracy of MedClarify for up to five interaction questions across different LLM backbones. Here, MedClarify is compared when instantiated with different LLM backbones, namely, GPT-5.1, Deepseek-R1-0528, and Llama-3.3-70B. Reported is the top-1 accuracy of MedClarify (purple) in comparison the naïve multi-turn system across different datasets: a , NEJM; b , MediQ; and c , MedQA.
Dataset
mean
s.e.
margin
95% CI
NEJM (single-shot complete)
0.368
0.005
0.007
[0.362; 0.375]
MediQ (single-shot complete)
0.504
0.012
0.015
[0.489; 0.519]
MedQA (single-shot complete)
0.499
0.013
0.016
[0.483; 0.515]
NEJM (single-shot incomplete)
0.087
0.005
0.006
[0.081; 0.093]
MediQ (single-shot incomplete)
0.260
0.009
0.011
[0.250; 0.271]
MedQA (single-shot incomplete)
0.152
0.011
0.013
[0.139; 0.165]
Appendix
Extended Table 1: Performance comparison between baseline and MedClarify in terms of diagnostic accuracy in the all-features masking setting. A single-shot (incomplete case) indicates a direct diagnosis from the masked patient cases without follow-up questioning. A single-shot (complete case) indicates a direct diagnosis based on complete cases. The baseline is a naïve multi-turn LLM system analogous to MedClarify but where questions are selected through a heuristic rather than an information-theoretic approach. The results are reported using 95% confidence intervals, calculated from the sample mean and standard error as the mean plus or minus the margin derived from the Student’s t -distribution. Abbreviation: s.e., standard error; CI, confidence interval.
Feature category
NEJM
MediQ
MedQA
Symptoms
170
129
170
Past medical history
95
88
170
Social history
32
71
170
Physical examination
135
118
170
Laboratory findings
82
58
170
Imaging results
66
34
170
Appendix
Extended Table 2: Dataset size across different feature categories. Values represent the number of cases containing each medical feature after preprocessing. We included only cases that focus on diagnostic reasoning with sufficient clinical details, excluding management, treatment, and prognosis content.
Clinical diagnosis is time-consuming, requiring intensive interactions between patients and medical professionals. While large language models (LLMs) could ease the pre-diagnostic workload, their limited domain knowledge hinders effective medical question generation. We introduce a Knowledge Graph-augmented LLM with active in-context learning to generate relevant and important follow-up questions, KG-Followup, serving as a critical module for the pre-diagnostic assessment. The structured medical domain knowledge graph serves as a seamless patch-up to provide professional domain expertise upon which the LLM can reason. Experiments demonstrate that KG-Followup outperforms state-of-the-art methods by 5% - 8% on relevant benchmarks in recall.
Large language models (LLMs) are increasingly used for health-related decision support. Yet most evaluations treat diagnosis as a single-shot task with complete information provided upfront, often as a multiple-choice selection. This diverges from clinical practice, where diagnosis is interactive and open-ended, involving sequential hypothesis refinement through targeted questioning. We address this gap. We build MeDxBench, a large-scale benchmark of 4,421 clinical cases across 20 specialties. We further propose MeDxAgent, a multi-agent consultation system for interactive diagnosis, and systematically study its prompt-, flow- and agent-level design choices. MeDxAgent achieves a 10.3% accuracy gain over the baseline on MeDxBench, closing 52.3% of the gap to a full-information oracle. We find that specific design choices: collecting demographics first, passing summarized dialogue for diagnosis, and feeding candidate diagnoses for targeted questioning, improve accuracy, mirroring how physicians reason, though their effect emerges fully only in combination. Code and dataset will be released upon publication.
Akshat Sanghvi, Naren Akash, Raza Imam +2
Microsoft Research India · Institute of AI and Innovation, AIG Hospitals · Mohamed bin Zayed University of Artificial Intelligence
In this paper, we propose a modified version of the MedQA-USMLE dataset, named MEDQA-OPEN, which contains open-ended medical questions without options to mimic clinical scenarios, along with clinician-approved reasoned answers. Additionally, we implement a prompt driven by Chain of Thought (CoT) reasoning, CLINICR, to mirror the prospective process of incremental reasoning, reaching a correct response to medical questions. We empirically demonstrate how CLINICR outperforms the state-of-the-art 5-shot CoT-based prompt (Liévin et al., 2022). We also present an approach that mirrors real-life clinical practice by first exploring multiple differential diagnoses through MCQ-CLINICR and subsequently narrowing down to a final diagnosis using MCQ-ELIMINATIVE. Finally, emphasizing the importance of response verification in medical settings, we utilize a reward model mechanism, replacing the elimination process performed by MCQ-ELIMINATIVE.