Benchmarking Clinical Decision Pathway Adherence in Large Language Models

Nuo Chen; Xinyang Jiang; Zilong Wang; Zhifei Zhang; Xiaoye Qu; Jiajun Deng; Yulan Guo; Cairong Zhao

cs.CLAug 27, 2026

Benchmarking Clinical Decision Pathway Adherence in Large Language Models

Authors: Nuo Chen, Xinyang Jiang, Zilong Wang, Zhifei Zhang, Xiaoye Qu, Jiajun Deng, Yulan Guo, Cairong Zhao

Organizations: Tongji University · Shenzhen Loop Area Institute · Independent Researcher · Shanghai Artificial Intelligence Laboratory · Tongji University School of Medicine · Sun Yat-sen University · Fuzhou University

Abstract

Following clinical decision pathways (CDPs) defined by clinical practice guidelines is essential for safe and reliable medical decision-making. However, existing medical large language model (LLM) benchmarks mainly evaluate final-answer accuracy, providing limited evaluation of models' ability to adhere to guidelines. To address this gap, we introduce MEGA-CDP, a benchmark for evaluating whether medical LLMs can generate guideline-adherent CDPs using provided guidelines as references. MEGA-CDP is constructed from 2,274 English and Chinese clinical practice guidelines through a guideline-to-case pipeline, yielding 42,353 clinical cases with explicit reference CDPs. It supports both single-turn vignette and multi-turn interactive settings, and introduces a CDP-oriented evaluation framework for measuring pathway consistency. Experiments on 16 representative LLMs show that reliable clinical decision support remains challenging for current models, demonstrating the need for CDP-oriented evaluation and the value of MEGA-CDP for advancing guideline adherence in medical LLMs.

Figures & tables

Figure 1: Overview of MEGA-CDP. (a) Comparison between existing benchmarks and MEGA-CDP. (b) Feature comparison between MEGA-CDP and representative clinical decision-making benchmarks. (c) Representative model performance under the two settings.

Figure 2: Framework of MEGA-CDP. The framework consists of guideline collection, guideline-to-case construction, downstream task settings, and CDP-oriented evaluation. Clinical guidelines are transformed into reasoning trees, guideline-defined CDPs, and case vignettes to support both single-turn vignette and multi-turn interactive settings. Model-generated CDPs are then evaluated through a two-stage procedure for measuring pathway consistency.

Model	Single-Turn		Multi-Turn
	PCC (%) $\downarrow$	Acc (%) $\uparrow$	PCC (%) $\downarrow$	Acc (%) $\uparrow$	SR (%) $\uparrow$
Proprietary Models
Gemini-3.1-Pro-Preview	44.61	65.14	65.32	28.73	78.17
GPT-5.4	28.41	63.06	67.43	23.73	74.77
Claude-Sonnet-4.6	32.42	75.38	55.92	45.40	99.22
Open-source Models

Table 1: Performance of evaluated LLMs on single-turn vignette and multi-turn interactive settings. PCC denotes the pathway consistency cost, Acc denotes the outcome accuracy, and SR denotes the success rate. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values are better.

Figure 3: Example of pathway-level inconsistency in guideline-adherent reasoning. Although the model reaches the correct final decision, its predicted CDP still deviates from the reference CDP through reversal, skipping, omission, and redundancy.

Figure 4: Threshold experiment results for step-level matching. The seven models labeled A through G in the figure are, respectively, Qwen3-Next-80B-A3B-Instruct, Qwen3-Next-80B-A3B-Thinking, Qwen3-8B, Qwen3-235B-A22B, HuatuoGPT-o1-7B, HuatuoGPT-o1-72B, and Baichuan-M3-235B.

Setting	Pair	$r$	$\rho$	$\tau$	$W$
DTW	GPT vs Qwen	0.79	0.79	0.60	0.70
	BERT vs GPT	0.48	0.46	0.32
	Qwen vs BERT	0.39	0.38	0.26
OT	GPT vs Qwen	0.80	0.79	0.60	0.71
	BERT vs GPT	0.50	0.49	0.34
	Qwen vs BERT	0.43	0.43	0.29

Table 2: Correlation analysis of metrics and judge models. Pearson

r

, Spearman

\rho

, Kendall

\tau

, and Kendall’s

W

are reported for pairwise comparisons. The table is divided into groups fixing either the metric or the judge model.

Component	Specification
Hardware Infrastructure
GPU	8 $\times$ NVIDIA H100 80GB
CPU	Intel Xeon Platinum 8468V
Memory	1920 GB
Software Environmen
Operating System	Ubuntu 20.04 LTS

Table 3: Computing infrastructure and software environment used for experiments.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of reference CDP lengths. The inset shows a magnified view of the long-tail region.

Statistic	Full	Test	Non-Test
Mean	6.51	6.12	6.59
Std. Dev.	3.74	3.18	3.84
Min	1	1	1
P5	2	2	2
P25	4	4	4
Median	6	6	6

Appendix

Table 4: Summary statistics of guideline-defined CDP lengths across dataset splits.

Model	DTW-PCC (%) $\downarrow$
Model	GPT-5.4 Mini	Qwen3-8B	BERTScore
Single-Turn Vignette Setting
Gemini-3.1-Pro-Preview	44.61 (44.07–45.16)	33.67 (33.16–34.19)	71.37 (71.13–71.62)
GPT-5.4	28.41 (27.94–28.86)	18.49 (18.08–18.88)	62.88 (62.61–63.14)
Claude-Sonnet-4.6	32.42 (31.90–32.93)	24.17 (23.72–24.64)	68.35 (68.09–68.61)
Qwen3-8B	47.88 (47.41–48.35)	33.66 (33.24–34.09)	74.07 (73.84–74.29)

Appendix

Table 5: DTW-based PCC results with bootstrap confidence intervals. Results are reported across different judge models under both single-turn vignette and multi-turn interactive settings. Each entry shows the mean PCC value with its 95% bootstrap confidence interval. Lower values indicate better pathway consistency.

Model	OT-PCC (%) $\downarrow$
Model	GPT-5.4 Mini	Qwen3-8B	BERTScore
Single-Turn Vignette Setting
Gemini-3.1-Pro-Preview	39.37 (38.88–39.86)	31.41 (30.96–31.87)	73.10 (72.86–73.35)
GPT-5.4	22.02 (21.62–22.40)	15.35 (15.02–15.66)	64.44 (64.21–64.69)
Claude-Sonnet-4.6	27.11 (26.68–27.55)	21.28 (20.90–21.67)	69.70 (69.45–69.95)
Qwen3-8B	36.94 (36.56–37.33)	25.72 (25.38–26.06)	74.22 (74.00–74.44)

Appendix

Table 6: OT-based PCC results with bootstrap confidence intervals. Results are reported across different judge models under both single-turn vignette and multi-turn interactive settings. Each entry shows the mean PCC value with its 95% bootstrap confidence interval. Lower values indicate better pathway consistency.

Model	LNDS-PCS (%) $\uparrow$
Model	GPT-5.4 Mini	Qwen3-8B	BERTScore
Single-Turn Vignette Setting
Gemini-3.1-Pro-Preview	53.01 (52.37–53.65)	50.97 (50.35–51.58)	67.09 (66.48–67.68)
GPT-5.4	74.09 (73.55–74.65)	75.86 (75.32–76.40)	81.73 (81.28–82.17)
Claude-Sonnet-4.6	67.92 (67.32–68.52)	64.87 (64.30–65.45)	74.17 (73.64–74.71)
Qwen3-8B	56.20 (55.65–56.76)	57.58 (57.05–58.11)	67.22 (66.66–67.77)

Appendix

Table 7: LNDS-based PCS results with bootstrap confidence intervals. Results are reported across different judge models under both single-turn vignette and multi-turn interactive settings. Each entry shows the mean PCS value with its 95% bootstrap confidence interval. Higher values indicate better pathway consistency.

Prompt 1: Reasoning Tree Extraction
You are a medical expert. You will receive a clinical guideline as input. Your task is to read the input, understand each paragraph, extract all content related to clinical decision-making, and convert it into a decision tree represented as a JSON object. Each non-leaf node must branch into two paths, "yes" and "no", and represent a criteria-check question about patient-specific clinical data or findings (e.g., “Is symptom A present?”, “Is lab value B $\geq$ threshold?”, “Did event C occur within time window D?”); each leaf node should represent a specific conclusion.

Before constructing the decision tree, analyze the entire guideline and determine whether the primary clinical focus for the selected disease is diagnosis, classification, screening, or management. Report this explicitly in the final JSON under "primary_clinical_focus". Use this determination to guide node construction, but do not represent it as a decision node. The decision tree must represent a complete and clinically meaningful pathway. It should reflect the main logical architecture of the selected domain, typically with multiple hierarchical levels, unless the guideline itself is inherently simple.

When constructing the decision tree:
- Replace vague or implicit conditions with explicit, objective, and clinically actionable criteria (e.g., symptoms, laboratory thresholds, imaging findings, or time-based conditions), strictly based on the guideline text.

Appendix

Table 8: Prompt template for reasoning tree extraction.

Prompt 2: Case Vignette Synthesis
You are a medical expert. I will provide you with a clinical decision path consisting of multiple steps. Each step includes a "condition" representing the clinical condition or decision criterion, and a "decision_outcome" indicating whether the condition is satisfied. In addition, I will specify the primary clinical focus of this path (e.g., diagnosis, classification, screening, or management). Your task is to convert this clinical decision path into a single, coherent, and natural-sounding clinical case description using medically styled language, appropriate to the specified primary clinical focus. The output should be written as a realistic clinical case vignette, resembling the narrative style commonly used in medical textbooks or case reports, rather than a summary or explanation of a decision pathway.

Note the following constraints:
1. The tone of each step’s rewritten content must reflect its decision_outcome value. Use an affirmative sentence if the decision_outcome is “yes”, and use a negative sentence if the decision_outcome is “no”.
2. For any medical indicator involving a numerical range, you must provide a specific numeric value. The value must fall within the range if the decision_outcome indicates the condition is satisfied, and outside the range if the decision_outcome indicates the condition is not satisfied.
3. If the same medical indicator appears in multiple steps, ensure that the chosen numeric value is consistent and falls within the intersection of all applicable ranges.

Appendix

Table 9: Prompt template for case vignette synthesis.

Prompt 3: Reasoning Narrative Conversion
You are a medical expert. You will be provided with a single JSON object containing information about a clinical decision pathway and a corresponding patient case under a specific clinical focus.

The input JSON of the pathway contains:
- "primary_clinical_focus": the clinical purpose of the pathway,
- "decision_path": an ordered list of decision steps along a single pathway,
- "case_vignette": a narrative description of the patient’s clinical presentation.

Appendix

Table 10: Prompt template for reasoning narrative conversion.

Prompt 4: Single-Turn Inference
You are a medical expert. I will provide you with a clinical guideline, a case vignette, and an example of the required output format. You must strictly follow the content of the guideline to make clinical decisions for the case, according to the specified primary clinical focus. Your output must be concise and formal, and must include both a structured clinical decision process and a final clinical outcome.

In the clinical decision process:
- The process must be divided into clear, sequential steps.
- Each decision step must explicitly state the core reasoning logic that links the key patient symptoms or findings to that clinical decision step.
- Each decision step must assess only one independent clinical variable or threshold. Do not merge multiple distinct criteria into a single reasoning step.

Appendix

Table 11: Prompt template for single-turn inference.

Prompt 5: Single-Turn Step-Level Consistency Scoring
You are a medical expert.

You will receive two clinical decision pathways:

- Reference Clinical Pathway: a reference sequence of clinical reasoning steps
- Candidate Clinical Pathway: a pathway to be evaluated against the reference

Appendix

Table 12: Prompt template for single-turn step-level consistency scoring.

Prompt 6: Multi-Turn Inference
You are a doctor. Your task is to determine the correct clinical conclusion strictly according to the provided guideline.

You will receive the patient’s clinical inquiry goal. Based on this goal, you must follow the explicit decision-making process in the guideline and ask the patient targeted questions step by step until either:
- exactly one clinical conclusion is determined, or
- it is impossible to determine a single conclusion based on the guideline and the patient’s answers.

Appendix

Table 13: Prompt template for multi-turn inference.

Prompt 7: Multi-Turn Patient Environment
You will role-play as a patient. You must answer the doctor’s questions as if you are a real person with a coherent and consistent underlying health state.

Follow these rules:
1. Keep your answer to exactly one word: "Yes" or "No".
2. Do not add explanations or any additional information.
3. Use the Case Vignette as the foundation of your health state. All answers must be consistent with the information explicitly stated in the vignette.

Appendix

Table 14: Prompt template for multi-turn patient environment.

Prompt 8: Multi-Turn Step-Level Consistency Scoring
You are a medical expert.

You will receive two clinical questioning pathways:

- Reference Clinical Questioning Pathway: a reference sequence of clinician questions used to gather patient information
- Candidate Clinical Questioning Pathway: a questioning pathway to be evaluated against the reference

Appendix

Table 15: Prompt template for multi-turn step-level consistency scoring.

Prompt 9: Outcome Accuracy Scoring
You are a medical expert.

You will receive two clinical conclusions:
- Reference Clinical Conclusion: a reference statement representing a final clinical judgment
- Candidate Clinical Conclusion: a conclusion to be evaluated against the reference

Appendix

Table 16: Prompt template for outcome accuracy scoring.

Explore similar work

A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations
Mar 26, 2026Andong Tan, Shuyu Dai, Jinglu Wang +7Clinical Language Model Evaluation Medical LLMs
Media Computing Group, Microsoft Research Asia, Beijing, China · Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science, Peking University, Beijing, China +6
MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning
May 26, 2026Yuhao Shen, Lang Cao, Simo Du +4Healthcare Clinical Decision Support
University of Illinois Urbana-Champaign · 2The Chinese University of Hong Kong, Shenzhen · 3Albert Einstein College of Medicine
More Reasoning Is Not Always Better: Executing Clinical Guidelines with LLMs and Rules
Oct 11, 2026Xiaoqian Lu, Guangfu Guo, Long Cheng +1Clinical Decision Support Medical LLMs
University of Ottawa · Clemson University

Mar 26, 2026J/K move · Enter open · S save

A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations

Andong Tan, Shuyu Dai, Jinglu Wang, Fengtao Zhou+6

Media Computing Group, Microsoft Research Asia, Beijing, China · Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science, Peking University, Beijing, China+6

Clinical Language Model Evaluation Medical LLMs LLM Safety Benchmarks

Open paper AskPDF arXiv

Clinical practice guidelines (CPGs) play a pivotal role in ensuring evidence-based decision-making and improving patient outcomes. While Large Language Models (LLMs) are increasingly deployed in healthcare scenarios, it is unclear to which extend LLMs could identify and adhere to CPGs during conversations. To address this gap, we introduce CPGBench, an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs in multi-turn conversations. We collect 3,418 CPG documents from 9 countries/regions and 2 international organizations published in the last decade spanning across 24 specialties. From these documents, we extract 32,155 clinical recommendations with corresponding publication institute, date, country, specialty, recommendation strength, evidence level, etc. One multi-turn conversation is generated for each recommendation accordingly to evaluate the detection and adherence capabilities of 8 leading LLMs. We find that the 71.1%-89.6% recommendations can be correctly detected, while only 3.6%-29.7% corresponding titles can be correctly referenced, revealing the gap between knowing the guideline contents and where they come from. The adherence rates range from 21.8% to 63.2% in different models, indicating a large gap between knowing the guidelines and being able to apply them. To confirm the validity of our automatic analysis, we further conduct a comprehensive human evaluation involving 56 clinicians from different specialties. To our knowledge, CPGBench is the first benchmark systematically revealing which clinical recommendations LLMs fail to detect or adhere to during conversations. Given that each clinical recommendation may affect a large population and that clinical applications are inherently safety critical, addressing these gaps is crucial for the safe and responsible deployment of LLMs in real world clinical practice.

Abstract

Explore similar work

A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

More Reasoning Is Not Always Better: Executing Clinical Guidelines with LLMs and Rules