A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations
Authors: Andong Tan, Shuyu Dai, Jinglu Wang, Fengtao Zhou, Yan Lu, Xi Wang, Yingcong Chen, Can Yang, +2 more
Organizations: Media Computing Group, Microsoft Research Asia, Beijing, China · Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science, Peking University, Beijing, China · AI Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Department of Mathematics, Hong Kong University of Science and Technology, Hong Kong, China · Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong, China · Division of Life Science, Hong Kong University of Science and Technology, Hong Kong, China · HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Futian, Shenzhen, China. · State Key Laboratory of Nervous System Disorders, Hong Kong University of Science and Technology, Hong Kong, China
Clinical practice guidelines (CPGs) play a pivotal role in ensuring evidence-based decision-making and improving patient outcomes. While Large Language Models (LLMs) are increasingly deployed in healthcare scenarios, it is unclear to which extend LLMs could identify and adhere to CPGs during conversations. To address this gap, we introduce CPGBench, an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs in multi-turn conversations. We collect 3,418 CPG documents from 9 countries/regions and 2 international organizations published in the last decade spanning across 24 specialties. From these documents, we extract 32,155 clinical recommendations with corresponding publication institute, date, country, specialty, recommendation strength, evidence level, etc. One multi-turn conversation is generated for each recommendation accordingly to evaluate the detection and adherence capabilities of 8 leading LLMs. We find that the 71.1%-89.6% recommendations can be correctly detected, while only 3.6%-29.7% corresponding titles can be correctly referenced, revealing the gap between knowing the guideline contents and where they come from. The adherence rates range from 21.8% to 63.2% in different models, indicating a large gap between knowing the guidelines and being able to apply them. To confirm the validity of our automatic analysis, we further conduct a comprehensive human evaluation involving 56 clinicians from different specialties. To our knowledge, CPGBench is the first benchmark systematically revealing which clinical recommendations LLMs fail to detect or adhere to during conversations. Given that each clinical recommendation may affect a large population and that clinical applications are inherently safety critical, addressing these gaps is crucial for the safe and responsible deployment of LLMs in real world clinical practice.
Figures & tables
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 1: Overview of CPGBench for evaluating LLM’s detection and adherence to CPGs. a Geographical distribution of our collected clinical practice guidelines across the world. b Our benchmark covers all 24 medical specialties defined in American Board on Medical Specialties. c We leverage the original guideline information to synthesize conversations and convert these conversations to proper forms for the detection and adherence capability evaluations. d Recommendation distribution over 10 years across different regions. e An overview of the evaluated models and their content detection, title grounding as well as adherence rates.
Detection
Models
Size
Content Detection
Title Grounding
Adherence
Average
GPT5
unknown
79.47%
29.68%
63.18%
57.44%
Deepseek-R1
671B
89.62%
13.24%
50.01%
50.96%
Qwen3-32B
32B
84.77%
7.96%
45.89%
46.21%
GPT4o
unknown
82.65%
7.91%
41.75%
44.10%
Baichuan-m2-32B
32B
71.13%
4.80%
49.32%
41.75%
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmTable 1: Overall content detection, title grounding and adherence rates in all 32,155 extracted clinical practice recommendations.
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 2: Specialty statistics of each country/region/international organization in our collected database. Percentage numbers of specialties less than 1% are not displayed for better readability.
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 3: a The content detection rates in all regions/international organizations of all models are above 50%. b Title grounding rates in the detection task. GPT5 is leading in this sub-task in most regions except the Chinese Mainland. c The adherence rates of all models across all regions or international organizations are consistently lower then their corresponding content detection rates. All plots share the same legend.
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 4: Content detection and adherence rates across medical specialties. The average performance across models in each specialty is shown in the upper subplot. The average performance across specialties in each model is shown in the right subplot. The subplot layout is consistent across all plots. a The content detection rates are generally high in all medical specialties. b The adherence rates in different models are generally higher in specialties such as allergy and immunology, pediatrics, preventive medicine and family medicine comapred to other specialties. c The content detection rates in the safety critical subset are higher than the rates of the full set. d Adherence rates in the safety critical subset are mostly lower than that of the full set shown in b across specialties.
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 5: Capability difference of models in detection and adherence measured by detection rates minus adherence rates across specialties.
Detection
Models
Content Detection
Title Grounding
Adherence
Average
GPT5
86.87%
27.67%
61.25%
58.60%
Deepseek-R1
93.60%
11.41%
48.67%
51.23%
Qwen3-32B
90.37%
7.22%
44.29%
47.29%
GPT4o
89.56%
6.98%
39.37%
45.30%
Baichuan-m2-32B
78.04%
4.5%
47.57%
43.37%
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmTable 3: Overall detection, title grounding and adherence rates on the safety-critical subset of 6,632 extracted CPG recommendations.
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 6: Human evaluation results. a Overview of the human evaluation processes in information extraction, conversation generation, and Judge-LLM agreement. b Information extraction precision evaluated by humans. c Conversation qualities evaluated by humans. d Judge-LLM’s agreement level with humans. The numbers indicate the percentage of different scenarios. e. Human-human agreement level in different scores. f Number of clinicians from different medical specialties participating the conversation quality evaluation. g Number of annotators participating evaluations of information extraction precision as well as the agreement evaluations in adherence score, content detection score and title grounding scores.
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmFigure 7: Automated framework design: a Experts’ discussion results based on newest evidences are often published in the form of consensus statements, position papers or guideline documents. We automatically filter them and only keep the CPG documents. b Key information from the guideline documents are extracted and formed into a structured database. We filter out only the information from guidelines published on or after 2015. c We use large language models to generate synthetic conversations based on extracted information and check the format constraints. d Annotation symbols are removed for the prompts used in the detection task and background conversation before the guideline appeared is kept to be used as the prompt in the adherence evaluation. e Tested LLMs are required to detect the guideline in the conversation (in detection capability evaluation) or generate a response according to the background conversation (in adherence capability evaluation). The responses are compared to the ground-truth used to generate the raw conversations and scored accordingly.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Country/Region
Institute/platform
Website
Chinese Mainland
Chinese Medical Association
https://videodata.cma-cmc.com.cn
Hong Kong
Health Department of Hong Kong SAR
https://www.chp.gov.hk
Taiwan
Taiwan Society of Cardiology
https://www.tsoc.org.tw
Japan
Japanese Circulation Society
https://www.j-circ.or.jp
Japan
The Japan Diabetes Society
https://www.jds.or.jp
Japan
Japanese Society of Otorhinolaryngology-Head and Neck Surgery
https://www.jibika.or.jp
Appendix
rm \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishrmTable 4: Institutes/platforms and corresponding links for guideline document collection.
Following clinical decision pathways (CDPs) defined by clinical practice guidelines is essential for safe and reliable medical decision-making. However, existing medical large language model (LLM) benchmarks mainly evaluate final-answer accuracy, providing limited evaluation of models' ability to adhere to guidelines. To address this gap, we introduce MEGA-CDP, a benchmark for evaluating whether medical LLMs can generate guideline-adherent CDPs using provided guidelines as references. MEGA-CDP is constructed from 2,274 English and Chinese clinical practice guidelines through a guideline-to-case pipeline, yielding 42,353 clinical cases with explicit reference CDPs. It supports both single-turn vignette and multi-turn interactive settings, and introduces a CDP-oriented evaluation framework for measuring pathway consistency. Experiments on 16 representative LLMs show that reliable clinical decision support remains challenging for current models, demonstrating the need for CDP-oriented evaluation and the value of MEGA-CDP for advancing guideline adherence in medical LLMs.
Nuo Chen, Xinyang Jiang, Zilong Wang +5
Tongji University · Shenzhen Loop Area Institute · Independent Researcher +4
As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.
Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science
Clinical practice guidelines (CPGs) encode evidence-based decision logic that clinicians apply by evaluating patient variables, conditional criteria, and recommendation rules. However, existing methods often use CPGs as free-text training data or retrieval sources, underutilizing their procedural decision structure. To better exploit this structure, we introduce a guideline-derived training pipeline that transforms CPG recommendations into executable clinical decision logic and uses it to generate factual and counterfactual question-answering data. Theses data teach models both guideline-supported decisions and how decisions change under different patient conditions. Post-training a medical LLM on the generated data yields MedGuideX. Across four clinical reasoning benchmarks, MedGuideX achieves a 10.28% relative improvement in average accuracy. Physician evaluation further shows that MedGuideX better recovers clinician authored reasoning steps and produces physician-preferred rationales in faithfulness, validity, completeness, and clarity. Overall, our results show that executable decision logic from CPGs can be transformed into scalable supervision for building reliable medical LLMs.
Yuhao Shen, Lang Cao, Simo Du +4
University of Illinois Urbana-Champaign · 2The Chinese University of Hong Kong, Shenzhen · 3Albert Einstein College of Medicine