Dialogue to Question Generation for Evidence-based Medical Guideline Agent Development

Zongliang Ji; Ziyang Zhang; Xincheng Tan; Matthew Thompson; Anna Goldenberg; Carl Yang; Rahul G. Krishnan; Fan Zhang

cs.CLMar 25, 2026

Dialogue to Question Generation for Evidence-based Medical Guideline Agent Development

Authors: Zongliang Ji, Ziyang Zhang, Xincheng Tan, Matthew Thompson, Anna Goldenberg, Carl Yang, Rahul G. Krishnan, Fan Zhang

Organizations: Google Research · University of Toronto, Canada · Vector Institute, Canada · Emory University, USA

Abstract

Evidence-based medicine (EBM) is central to high-quality care, but remains difficult to implement in fast-paced primary care settings. Physicians face short consultations, increasing patient loads, and lengthy guideline documents that are impractical to consult in real time. To address this gap, we investigate the feasibility of using large language models (LLMs) as ambient assistants that surface targeted, evidence-based questions during physician-patient encounters. Our study focuses on question generation rather than question answering, with the aim of scaffolding physician reasoning and integrating guideline-based practice into brief consultations. We implemented two prompting strategies, a zero-shot baseline and a multi-stage reasoning variant, using Gemini 2.5 as the backbone model. We evaluated on a benchmark of 80 de-identified transcripts from real clinical encounters, with six experienced physicians contributing over 90 hours of structured review. Results indicate that while general-purpose LLMs are not yet fully reliable, they can produce clinically meaningful and guideline-relevant questions, suggesting significant potential to reduce cognitive burden and make EBM more actionable at the point of care.

Figures & tables

Figure 1: Illustration of an outpatient visit. A 40-year-old female with a high BMI presents with headache and fatigue. (a) In a routine encounter, the PCP simultaneously juggles the patient’s narrative, EHR, prior comorbidities, and guidelines, while considering differential diagnoses such as vestibular migraine. (b) With intelligent support, the PCP receives context-aware, evidence-based information that help structure reasoning without interrupting patient communication.

Figure 2: Overview of our work. (a) We sample 80 primary care cases from 2,000 real-world physician-patient dialogue transcripts. (b) We develop two methods to generate evidence-based medical questions from dialogue using Gemini 2.5. (c) We conduct a pilot study with more than 10 internal clinical experts regarding the evaluation metrics, pilot labeling, and rationale of designing the experiments. (d) We perform auto evaluation with LLM and human evaluation on 80 cases with 6 experienced clinicians.

Metric	Statement
Relevance	The questions are relevant and highlight insightful aspects of the case.
Guideline Navigation	The questions guide me toward which specific guidelines for evidence I should consult.
Thought Alignment	The questions align with my own clinical reasoning or thought process, without challenging my judgment.
Non-Redundancy	The questions are not redundant and will not impose a cognitive burden during the visit.
Usefulness	I would find these questions genuinely helpful for saving time and improving my daily workflow.

Table 1: Five metrics for evaluation.

Panel of Figure 3 — Figure 3: Stacked bar plots of (a) the human evaluation by six clinicians and (b) the automated evaluation by Gemini on 80 dialogue transcripts. M1 refers to the multi-stage reasoning method and M2 refers to the zero-shot baseline. Each method is rated on a Likert-scale from 1 – Strongly Disagree to 7 – Strongly Agree.

Figure 4: Averaged scores of the proposed framework rated by PCPs at 30%, 70%, and 100% dialogue context.

Figure 5: Proportional trend of question types preferred by PCPs at 30%, 70%, and 100% dialogue context. Only questions generated by the proposed framework are included. The proportions may be biased by an uneven initial distribution of question types.

Figure 6: Clinicians exhibit different rating styles. On the X-axis, P1 to P6 means participated clinicians. The Y-axis, shows the range of Likert score for the evaluation result. Each group of box plots shows a clinician’s ratings of the multi-stage reasoning method across 40 annotated cases, evaluated under 30%, 70%, and 100% dialogue context.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 2 — Table 2: Prompt for the summarization agent.

Panel of Table 3 — Table 3: Prompt for the question raising agent.

Panel of Table 4 — Table 4: Prompt for the question evaluation agent.

Panel of Table 5 — Table 5: Prompt for the zero-shot baseline setting.

Panel of Figure 7 — Figure 7: Progress list with one row per case, a completion checkbox, and a link to the questionnaire.

Panel of Figure 8 — Figure 8: Sample questionnaire page at a given context level with three candidate questions, Likert items, and a best question choice.

Metric	Zero-shot	Multi-stage
Average	6.36	6.08
Relevance	6.6	6.2
Guideline Navigation	6.8	5.8
Thought Alignment	6.4	6.2
Non-Redundancy	5.6	6.2
Usefulness	6.4	6.0

Appendix

Table 6: LLM-as-judge evaluation results across five dimensions by MedGemma-27B (text-only).

Metric	$\rho$	$p$
Relevance-M1	0.0718	0.2677
Relevance-M2	0.0519	0.4237
Guideline Navigation-M1	0.1190	0.0657
Guideline Navigation-M2	0.0212	0.7439
Thought Alignment-M1	0.0804	0.2145
Thought Alignment-M2	-0.0335	0.6055

Appendix

Table 7: Spearman correlation between LLM-as-judge scores and human physician ratings across five evaluation metrics, reported separately for the zero-shot baseline (M2) and the proposed multi-stage pipeline (M1).

Metric	Average Spearman Correlation
Relevance	-0.013329
Guideline Navigation	-0.025142
Thought Alignment	0.042234
Non-Redundancy	-0.048712
Usefulness	0.014104

Appendix

Table 8: Average Spearman correlation between LLM-as-judge scores and physician ratings across five evaluation metrics, computed by aggregating correlations over all dialogue stages and both generation methods.

Clinical Decision Support

Clinical Decision Support uses data, medical knowledge, and AI to help clinicians assess patients and choose tests, diagnoses, and treatments.

See all 242 papers

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
May 20, 2026Doeun Lee, Muge Zhang, Yi Yu +11LLM Grounding Healthcare
The Ohio State University · The Ohio State University Wexner Medical Center · University of Chicago Medical Center
MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline
Apr 20, 2026Jiyao Liu, Jianghan Shen, Sida Song +19Healthcare AI Agent Benchmarks
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
MED-COPILOT: A Medical Assistant Powered by GraphRAG and Similar Patient Case Retrieval
Feb 28, 2026Shuheng Chen, Namratha Patil, Haonan Pan +4Case-Based Reasoning Clinical Decision Support
University of Southern California

May 20, 2026J/K move · Enter open · S save

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

Doeun Lee, Muge Zhang, Yi Yu, Ashish Manne+10

The Ohio State University · The Ohio State University Wexner Medical Center · University of Chicago Medical Center

LLM Grounding Healthcare Clinical QA RAG Evaluation Medical Reasoning

Open paper AskPDF arXiv

Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts.

Abstract

Clinical Decision Support

Explore similar work

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline

MED-COPILOT: A Medical Assistant Powered by GraphRAG and Similar Patient Case Retrieval