More Reasoning Is Not Always Better: Executing Clinical Guidelines with LLMs and Rules

Xiaoqian Lu; Guangfu Guo; Long Cheng; Kunpeng Liu

cs.CLOct 11, 2026

More Reasoning Is Not Always Better: Executing Clinical Guidelines with LLMs and Rules

Authors: Xiaoqian Lu, Guangfu Guo, Long Cheng, Kunpeng Liu

Organizations: University of Ottawa · Clemson University

Abstract

Large language models (LLMs) have shown considerable promise in clinical decision support. However, existing approaches often rely on multi-step reasoning or multi-agent collaboration to execute clinical guidelines, introducing substantial computational overhead without reliably preventing decision errors. To address these limitations, we propose a framework for Medical Decision Execution (MedDEx) that integrates structured patient states with deterministic rule execution to support guideline-based treatment planning. MedDEx separates clinical record interpretation from guideline execution, leveraging LLMs to extract patient information and generate treatment plans while assigning guideline-defined decision logic to a rule executor. Experiments across multiple clinical guideline benchmarks demonstrate that MedDEx improves decision accuracy while reducing inference costs. Further analyses highlight the critical roles of structured patient representations and deterministic rule execution, suggesting that additional LLM reasoning does not necessarily translate into better guideline adherence.

Figures & tables

Figure 1: The importance of guideline execution and the division of responsibilities. Top: Deterministic rule execution is more reliable than LLM-based guideline execution given matched inputs. Bottom: MedDEx improves decision performance by refining routing inputs and allocating responsibilities for TNM computation.

Figure 2: Overview of MedDEx. Top: LLM-executed agents assign scenario selection to the model. Middle: MedDEx assigns record interpretation and plan generation to the LLM, and TNM computation and scenario routing to rules. Bottom (dashed): coding assistants draft rule packs, audits and tests guide revision, and the authors review the code before evaluation. The graph is simplified.

	Qwen3-8B			Llama-3.1-8B-Instruct
	MedGUIDE	MEGA-CDP		MedGUIDE	MEGA-CDP
Method	Acc.	Concl.	Path OT	Acc.	Concl.	Path OT
Single LLM	68.77	83.17	77.60	66.65	74.31	71.70
Guideline-RAG	51.68	76.00	76.95	54.78	67.41	72.89
MedAgents ( Tang et al., 2024 )	69.36	77.15	73.72	56.09	67.34	64.92
MDAgents ( Kim et al., 2024 )	66.03	78.64	72.84	50.29	65.38	69.00

Table 1: LLM execution vs. rule execution of the same guideline. MedGUIDE reports option accuracy (%); MEGA-CDP reports conclusion agreement (%) and path similarity (%). Best in bold, second underlined; the shaded row is ours. ⋆ : Rules execute the guideline.

	TNM Staging				Treatment Recommendation				End-to-End Decision Support
	Acc(%) $\uparrow$		RQ $\uparrow$		Precision(%) $\uparrow$		F1 $\uparrow$		Precision(%) $\uparrow$		F1 $\uparrow$
Method	ZH	EN	ZH	EN	ZH	EN	ZH	EN	ZH	EN	ZH	EN
Base model: Qwen2.5-7B-Instruct
Direct prompting	0.10	2.20	33.24	36.56	38.19	57.05	41.38	52.86	40.98	55.51	39.80	45.48
Guideline-RAG	2.70	2.60	39.77	38.91	36.77	48.32	43.88	49.90	40.03	48.64	42.60	46.28
MedAgents ( Tang et al., 2024 )	5.40	6.30	46.04	51.28	44.09	55.99	51.38	51.42	40.58	45.76	47.70	49.16

Table 2: Results on LungCURE. Staging, treatment planning with the reference stage, and the end-to-end task from the record, in Chinese (ZH) and English (EN). F1 is the LungCURE plan agreement score and RQ the staging reasoning score. Best in bold, second underlined; shaded rows are ours. ⋆ : rules execute the guideline.

Figure 3: Downstream effects of the execution boundary. Qwen2.5-7B, English. (a) Agreement with reference T, N, and M categories and exact TNM. (b) LungCURE plan agreement by reference stage group. (c) End-to-end plan agreement against measured model tokens per case. Scores measure benchmark agreement; the cost protocol is in Appendix A.2 .

	TNM	Treatment		End-to-end
Variant	Acc $\uparrow$	P $\uparrow$	F1 $\uparrow$	P $\uparrow$	F1 $\uparrow$
MedDEx (full)	26.90	73.90	65.50	70.58	57.62
w/o rule selection	26.90 0.0	65.17 $-$ 8.7 ∗	60.81 $-$ 4.7 ∗	64.15 $-$ 6.4 ∗	54.39 $-$ 3.2 ∗
w/o stage calculator	8.08 $-$ 18.8 ∗	73.90 0.0	65.50 0.0	70.69 +0.1	56.32 $-$ 1.3 ∗
w all nine guidelines	26.90 0.0	60.27 $-$ 13.6 ∗	49.82 $-$ 15.7 ∗	61.24 $-$ 9.3 ∗	44.89 $-$ 12.7 ∗

Table 3: Ablation results of MedDEx. Values below each score indicate changes from the full model. ∗ denotes a 95% bootstrap confidence interval excluding zero.

	Plan agreement		Stage
Routing input	Exact	Group	consistency
MedDEx (full)	67.0	78.9	71.6
$-$ stage calculator	65.3	77.9	68.5
$-$ staging call	54.5	65.5	54.3
$-$ typed state	24.7	59.6	46.4

Table 4: Effect of routing inputs on rule execution. Results on Qwen2.5-7B, English (%). The rule executor remains unchanged while input components are progressively removed. Plan agreement is measured against scenarios extracted from clinician plans. Stage consistency is evaluated against reference TNM categories.

	Stage computed by
Patient state	the model	the rules
free text	46.2 ∗	48.9 ∗
structured	55.9 ∗	57.6

Table 5: Effect of patient state representation and staging responsibility. End-to-end plan agreement (F1) on Qwen2.5-7B, English. The rule router remains unchanged across all settings. ∗ indicates a 95% bootstrap confidence interval excluding zero relative to the full system.

	MedGUIDE		MEGA-CDP
Method	Calls	Sec	Calls	Sec
Single LLM	1.0	0.9	1.0	49.7
Guideline-RAG	1.0	2.3	1.0	8.0
MedAgents	20.7	152.3	20.5	211.1
MDAgents	19.4	303.9	17.1	399.3
MA-RAG	1.1	1.5	5.9	164.9

Table 6: Computational cost per case across clinical guideline benchmarks. Results on Qwen3-8B using the same GPU and concurrency settings. We report the average number of model calls and execution time (seconds) per case. Best in bold, second underlined. ⋆ indicates rule-based guideline execution.

	All		Non-met.	Metastatic
Scenario chosen by	Exact	Grp.	Exact	Exact
scenario list only (§ 5.1 , Q1)	38.3	54.4	69.4	4.9
+ guideline entry rules, verbatim	59.5	75.2	69.2	49.1
+ chain of thought	58.3	94.2	64.2	52.1
+ 9 worked examples	35.4	93.3	58.2	11.0
+ self-consistency, 5 samples	36.1	93.3	58.2	12.5

Table 7: Improving LLM-based scenario routing does not close the gap. All methods receive identical patient states and reference stages. We report overall scenario agreement and exact agreement across non-metastatic and metastatic cases (%).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Guideline	Size	Task
MedGUIDE	53 author trees	7,747	multiple-choice decision
MEGA-CDP	440 guideline trees	7,458	decision path and conclusion
LungCURE	9 NCCN/CSCO scenarios	1,000	staging, plan, end to end

Appendix

Table 8: Guideline benchmarks. LungCURE cases are available in Chinese and English.

Scenario	Entry condition (checked in order)
Adv. driver $+$ , later line	M1, driver $+$ , prior therapy and progression
Adv. driver $+$ , first line	M1, driver $+$
Oligometastatic	M1, oligometastatic, $\leq$ 5 lesions, first line
Adv. driver $-$ , later line	M1, prior therapy and progression
Adv. driver $-$ , first line	M1
Postoperative adjuvant	M0, curative resection done

Appendix

Table 9: Guideline scenarios and the rules that select them.

	Treatment (T2)				End-to-end (T3)
Method	Calls	In	Out	s	Calls	In	Out	s
Guideline-RAG	1.0	2.9	0.59	0.8	1.0	2.9	0.70	0.8
DermAgent	4.6	12.0	1.80	2.3	4.6	12.1	1.82	2.1
LCAgent ⋆	1.8	3.5	0.47	2.4	7.8	22.3	4.18	4.5
MedAgents	9.1	19.2	3.95	4.9	9.0	19.0	4.23	5.2
MDAgents	10.7	27.8	7.35	8.8	10.7	28.0	7.63	9.1

Appendix

Table 10: Cost per case. All methods use the same model, hardware, and concurrency. Calls, input and output tokens (thousands), and average time cover the full pipeline. ⋆ : rules execute the guideline.

Stage	Expl.	T $\uparrow$	N $\uparrow$	M $\uparrow$	TNM $\uparrow$	RQ $\uparrow$
		Accuracy (%)
Calculator	Rule (ours)	52.49	70.79	67.43	26.90	70.78
Calculator	Model’s reply	52.49	70.79	67.43	26.90	46.49
Model	Model’s reply	27.05	36.47	44.15	8.08	44.60

Appendix

Table 11: Staging and its explanation. The stage comes from the stage calculator or the model’s own answer; the explanation comes from the rule that set each category or the model’s own answer. RQ is the judge’s reasoning score for T, N, and M. Best in bold, second underlined; the shaded row is the full system.

	Treatment (Task 2)				End-to-end (Task 3)
Method	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	#drugs	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	#drugs
Direct prompting	48.8	38.1	39.7	3.8	47.3	43.0	42.3	4.3
Guideline-RAG	42.2	40.1	38.2	3.9	42.2	41.8	39.0	4.1
MedAgents	49.1	36.4	38.3	3.6	41.1	44.7	38.9	5.6
MDAgents	46.9	35.6	36.8	3.5	44.1	38.8	37.6	4.1
MA-RAG	52.2	39.4	41.2	3.5	52.2	38.8	41.1	3.5

Appendix

Table 12: Drug metrics without an LLM. We compare the drug names in each plan with the clinician’s plan and report precision, recall, F1, and drugs per plan; no language model is used for scoring. Best in bold, second underlined; shaded rows are ours. ⋆ : rules execute the guideline.

Record: 1.8 cm lung adenocarcinoma, already resected by lobectomy; 1 of 2 hilar nodes positive ; EGFR L858R mutation.
Clinician: stage T1bN1M0 $\rightarrow$ postoperative adjuvant therapy $\rightarrow$ osimertinib for $\geq$ 3 years.
Method	How the patient was read	Plan
MedAgents	No setting chosen	Chemotherapy or “any EGFR-TKI”	✗
LCAgent	Positive node missed (T1bN0)	Osimertinib	$\sim$
MedDEx	Resected: yes; driver: EGFR L858R; stage T1bN1M0 (exact) $\Rightarrow$ postoperative adjuvant	Adjuvant osimertinib for $\geq$ 3 years	✓

Appendix

Table 13: Case study. Each row shows how a system read the patient and the plan it wrote; key facts in the record are in blue. ✓ : matches the clinician;

\sim

: partly matches, with a staging error; ✗ : differs from the reference setting. MedDEx records the key facts, computes the reference stage, and selects postoperative adjuvant treatment.

Explore similar work

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning
May 26, 2026Yuhao Shen, Lang Cao, Simo Du +4Healthcare Clinical Decision Support
University of Illinois Urbana-Champaign · 2The Chinese University of Hong Kong, Shenzhen · 3Albert Einstein College of Medicine
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Aug 27, 2026Nuo Chen, Xinyang Jiang, Zilong Wang +5Medical LLMs Clinical Language Model Evaluation
Tongji University · Shenzhen Loop Area Institute · Independent Researcher +4
GuideSkill: Evolving Executable LLM Agent Skills for Guideline-Grounded Clinical Reasoning
Jul 28, 2026Lang Cao, Yuhao Shen, Tianyang Luo +3Clinical Decision Support LLM Agent Skill Learning
University of Illinois Urbana-Champaign · ♦The Chinese University of Hong Kong, Shenzhen · ⋂Albert Einstein College of Medicine

May 26, 2026J/K move · Enter open · S save

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

Yuhao Shen, Lang Cao, Simo Du, Yuqing Wang+3

University of Illinois Urbana-Champaign · 2The Chinese University of Hong Kong, Shenzhen · 3Albert Einstein College of Medicine

Healthcare Clinical Decision Support Clinical Reasoning Medical LLMs

Open paper AskPDF arXiv

Clinical practice guidelines (CPGs) encode evidence-based decision logic that clinicians apply by evaluating patient variables, conditional criteria, and recommendation rules. However, existing methods often use CPGs as free-text training data or retrieval sources, underutilizing their procedural decision structure. To better exploit this structure, we introduce a guideline-derived training pipeline that transforms CPG recommendations into executable clinical decision logic and uses it to generate factual and counterfactual question-answering data. Theses data teach models both guideline-supported decisions and how decisions change under different patient conditions. Post-training a medical LLM on the generated data yields MedGuideX. Across four clinical reasoning benchmarks, MedGuideX achieves a 10.28% relative improvement in average accuracy. Physician evaluation further shows that MedGuideX better recovers clinician authored reasoning steps and produces physician-preferred rationales in faithfulness, validity, completeness, and clarity. Overall, our results show that executable decision logic from CPGs can be transformed into scalable supervision for building reliable medical LLMs.

Abstract

Explore similar work

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

Benchmarking Clinical Decision Pathway Adherence in Large Language Models

GuideSkill: Evolving Executable LLM Agent Skills for Guideline-Grounded Clinical Reasoning