cs.CLDec 3, 2025

Guideline-Based Medical Reasoning in LLMs

Authors: Michael Staniek, Artem Sokolov, Stefan Riezler

Organizations: Computational Linguistics, Heidelberg University, Germany · Google DeepMind, Berlin, Germany · IWR, Heidelberg University, Germany

Abstract

Large Language Models (LLMs) excel at medical question-answering, yet progress toward medical reasoning models is limited by the lack of scalable, trustworthy reasoning supervision. We propose tapping into the so far underused resource of medical consensus guidelines. These map numerical clinical measurements to disease scores and can optionally be curated for site-specific exceptions. Automatically generating gold-standard reasoning chains from consensus rules enables both large-scale supervised fine-tuning and automatic evaluation. Applying SOFA, SAPS-II, and Sepsis-3 guidelines to MIMIC-III, we show that small fine-tuned models significantly outperform larger one-shot prompted models in derivation correctness and F1 score, achieving near-perfect deductive inference and reliable exception handling. Trustworthy step-by-step reasoning does not reliably emerge from one-shot prompting, even when models receive the consensus definition and an example. Performance across all models declines when forecasting future clinical variables, indicating that, on the evaluated structured ICU tasks, the central challenge is not rule application, but the orthogonal problem of generalizing into the future for prognostic reasoning.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

    May 26, 2026Yuhao Shen, Lang Cao, Simo Du +4HealthcareClinical Decision Support

  2. Medical Reasoning with Large Language Models: A Survey and MR-Bench

    Mar 17, 2026Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4Medical ReasoningClinical Reasoning

  3. More Reasoning Is Not Always Better: Executing Clinical Guidelines with LLMs and Rules

    Oct 11, 2026Xiaoqian Lu, Guangfu Guo, Long Cheng +1Clinical Decision SupportMedical LLMs