Guideline-Based Medical Reasoning in LLMs
Organizations: Computational Linguistics, Heidelberg University, Germany · Google DeepMind, Berlin, Germany · IWR, Heidelberg University, Germany
Abstract
Large Language Models (LLMs) excel at medical question-answering, yet progress toward medical reasoning models is limited by the lack of scalable, trustworthy reasoning supervision. We propose tapping into the so far underused resource of medical consensus guidelines. These map numerical clinical measurements to disease scores and can optionally be curated for site-specific exceptions. Automatically generating gold-standard reasoning chains from consensus rules enables both large-scale supervised fine-tuning and automatic evaluation. Applying SOFA, SAPS-II, and Sepsis-3 guidelines to MIMIC-III, we show that small fine-tuned models significantly outperform larger one-shot prompted models in derivation correctness and F1 score, achieving near-perfect deductive inference and reliable exception handling. Trustworthy step-by-step reasoning does not reliably emerge from one-shot prompting, even when models receive the consensus definition and an example. Performance across all models declines when forecasting future clinical variables, indicating that, on the evaluated structured ICU tasks, the central challenge is not rule application, but the orthogonal problem of generalizing into the future for prognostic reasoning.
Figures & tables
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned | pipeline | multimodal | |
| SAPS-II current | S | 0.138 | 0.200 | 0.133 | 0.094 | 0.171 | 1.000 | ||
| S | 0.322 | 0.548 | 0.629 | 0.551 | 0.845 | 1.000 | |||
| S | 0.270 | 0.487 | 0.298 | 0.059 | 0.862 | 1.000 | |||
| S | 0.797 | 0.761 | 0.942 | 0.750 | 0.930 | 0.997 | |||
| S | 0.533 | 0.795 | 0.955 | 0.477 | 0.878 | 1.000 | |||
| S | 0.434 | 0.286 | 0.622 | 0.326 | 0.287 | 0.999 |
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned | |
|---|---|---|---|---|---|---|---|
| SOFA current | S | 0.810 | 0.788 | 0.652 | 0.450 | 0.987 | 0.989 |
| S | 0.416 | 0.627 | 0.301 | 0.361 | 0.767 | 0.999 | |
| S | 0.577 | 0.342 | 0.542 | 0.323 | 0.636 | 0.994 | |
| S | 0.562 | 0.649 | 0.555 | 0.244 | 0.784 | 0.945 | |
| S | 0.626 | 0.855 | 0.847 | 0.555 | 0.942 | 0.996 | |
| S | 0.153 | 0.202 | 0.133 | 0.384 | 0.444 | 0.998 |
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned | pipeline | multimodal | |
| SAPS-II current | Age | 1.000 | 0.972 | 1.000 | 0.903 | 1.000 | 1.000 | ||
| HR | 0.618 | 0.600 | 0.342 | 0.188 | 0.435 | 0.998 | |||
| SBP | 0.549 | 0.631 | 0.316 | 0.074 | 0.912 | 0.999 | |||
| Temperature | 0.909 | 0.925 | 0.926 | 0.728 | 0.950 | 1.000 | |||
| GCS | 0.491 | 0.695 | 0.387 | 0.302 | 0.748 | 0.998 | |||
| PaO2/FiO2 | 0.413 | 0.258 | 0.439 | 0.301 | 0.271 | 0.998 |
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned | pipeline | multimodal | |
| SAPS-II current | S | 0.138 | 0.200 | 0.133 | 0.090 | 0.171 | 1.000 | ||
| S | 0.270 | 0.506 | 0.393 | 0.333 | 0.675 | 0.999 | |||
| S | 0.244 | 0.482 | 0.299 | 0.361 | 0.853 | 0.999 | |||
| S | 0.779 | 0.759 | 0.928 | 0.739 | 0.927 | 0.999 | |||
| S | 0.487 | 0.710 | 0.486 | 0.405 | 0.775 | 0.999 | |||
| S | 0.432 | 0.251 | 0.573 | 0.507 | 0.203 | 0.999 |
| variable | % changes ID | % changes OOD | ID score | OOD score |
|---|---|---|---|---|
| 0.404 | 0.424 | 1.000 | 1.000 | |
| 0.433 | 0.452 | 1.000 | 1.000 |
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned |
|---|---|---|---|---|---|---|
| S | 0.727 | 0.773 | 0.603 | 0.385 | 0.974 | 0.998 |
| S | 0.380 | 0.531 | 0.311 | 0.306 | 0.653 | 0.984 |
| S | 0.605 | 0.377 | 0.505 | 0.395 | 0.700 | 0.995 |
| S | 0.573 | 0.658 | 0.554 | 0.280 | 0.799 | 0.999 |
| S | 0.625 | 0.843 | 0.858 | 0.573 | 0.923 | 1.000 |
| S | 0.463 | 0.651 | 0.577 | 0.274 | 0.827 | 0.998 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| CNS | Cardiovascular | Respiratory | Coagulation | Liver | Renal | |
|---|---|---|---|---|---|---|
| Score | GCS | MAP or vasopressors | PaO2/FiO2 (mmHg) | Platelets ( / l) | Bilirubin (mg/dl) | Creatinine (mg/dl) or UO |
| +0 | 15 | MAP 70 mmHg | 400 | 150 | 1.2 | 1.2 |
| +1 | 13-14 | MAP 70 mmHg | 400 | 150 | 1.2-1.9 | 1.2-1.9 |
| +2 | 10-12 | dopamine 5 g/kg/min OR dobutamine (any dose) | 300 | 100 | 2.0-5.9 | 2.0-3.4 |
| +3 | 6-9 | dopamine 5 g/kg/min OR epinephrine 0.1 g/kg/min OR norepinephrine 0.1 g/kg/min | 200 AND MV | 50 | 6.0-11.9 | 3.5-4.9 OR 500 ml/day |
| +4 | 6 | dopamine 15 g/kg/min OR epinephrine 0.1 g/kg/min OR norepinephrine 0.1 g/kg/min | 100 AND MV | 20 | 12.0 | 5.0 OR 200 ml/day |
| ALP | Epinephrine* | LDH | Packed RBC |
| ALT | Famotidine | Lactate | Pantoprazole |
| AST | Fentanyl | Lactated Ringers | Phosphate |
| Albumin | FiO2* | Levofloxacin | Piggyback |
| Albumin 25% | Fiber | Lorazepam | Piperacillin |
| Albumin 5% | Free Water | Lymphocytes | Platelet Count* |
| Amiodarone | Fresh Frozen Plasma | Lymphocytes (Absolute) | Potassium |
| Category | Premise | Conclusion |
|---|---|---|
| Chronic Kidney Disease Stage 3 (CKD3) | 585.3, 585.9 | Renal SOFA = 2 |
| CKD4 | 585.4 | Renal SOFA = 3 |
| CKD5 | 585.5, 585.6, 403.01, 403.11, 403.91, 404.02, 404.03, 404.13, 404.93 | Renal SOFA = 4 |
| Leukemia or Lymphoma | 201.0, 201.1, 201.6, 201.7, 201.2, 201.9, 202.0, 202.8, 200.7, 200.1, 200.2, 202.1, 202.2, 202.7, 200.6, 200.3, 203.8, 203.0, 203.1, 238.6, 204.0, 204.1, 204.8, 202.4, 204.9, 205.0, 205.1, 205.3, 205.9, 206.0, 206.1, 206.2, 206.9, 207.0, 207.2, 207.8, 208.0, 208.1, 208.9, 202.5, 202.6, 202.9, 200.0, 200.9, 201.4, 201.5, 203.00, 203.01, 203.02, 203.10, 203.11, 203.12, 204.00, 204.01, 204.02, 204.10, 204.11, 204.12, 205.00, 205.01, 205.02, 205.10, 205.11, 205.12, 206.00, 206.01, 206.10, 206.11, 207.00, 207.01, 207.20, 207.21, 208.00, 208.01, 208.10, 208.11, 208.90, 208.91 | Coag. SOFA = 4 |
| Moderate-Severe Liver Disease | 456.0, 456.1, 456.21, 572.8, 572.3, 572.4 | Liver SOFA = 2 |
| Dementia | 290.11, 290.0, 290.41, 290.40, 290.43, 294.10, 294.11, 290.3, 331.2, 290.42, 290.8, 290.9, 290.10, 290.12, 290.13, 290.20, 290.21 | CNS SOFA = 2 |
| lung | kidney | coagulation | liver | cardiovascular | |
|---|---|---|---|---|---|
| ID | J40, J41, J42 | N18.9, N28 | D68.4, D68.5 | K70.0, K70.41 | I50.0, I50.9 |
| OOD | J44.9 | N19 | D68.6 | K70.3 | I50.1 |
| ICD-10 Code | # ID Test | # OOD Test |
|---|---|---|
| J40 | 263 | 194 |
| J41 | 239 | 173 |
| J42 | 253 | 194 |
| J44.9 | - | 166 |
| N18.9 | 266 | 172 |
| N19 | - | 204 |
| Model | Sepsis-3 Accuracy | Sepsis-3 F1 |
|---|---|---|
| one-shot LLaMA-3-8B (baseline) | 0.834 | 0.231 |
| HuatuoGPT-o1-8B | 0.765 | 0.204 |
| Olmo-3-7B-Instruct | output incoherent; metrics undefined | |
| Hyperparameter | value |
|---|---|
| Embedding Size | 512 |
| Hidden size encoder | 512 |
| Hidden size decoder | 512 |
| # Encoder layers | 2 |
| # Decoder layers | 1 |
| learning rate | 0.0005 |
| Hyperparameter | value |
|---|---|
| optimizer | adam |
| learning rate | 0.000002 |
| max epochs | 10 |
| connector first layer | 131x2096 |
| connector second layer | 2096x4096 |
| lora-r | 16 |
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned | |
|---|---|---|---|---|---|---|---|
| current | GCS-eye | 0.849 | 0.981 | 0.932 | 0.807 | 0.985 | 0.998 |
| GCS-motor | 0.913 | 0.976 | 0.936 | 0.829 | 0.984 | 0.998 | |
| GCS-verbal | 0.950 | 0.984 | 0.939 | 0.874 | 0.984 | 0.999 | |
| MAP | 0.031 | 0.143 | 0.021 | 0.055 | 0.310 | 0.994 | |
| Dopamine | 0.950 | 0.951 | 0.949 | 0.062 | 0.953 | 0.972 | |
| Dobutamine | 0.981 | 0.978 | 0.985 | 0.065 | 0.979 | 0.993 |
| variable | one-shot | one-shot-70B | me-llama | deepseek | qwen3.6-27B | finetuned | |
|---|---|---|---|---|---|---|---|
| current | S | 0.727 | 0.773 | 0.603 | 0.385 | 0.974 | 0.998 |
| S | 0.380 | 0.531 | 0.311 | 0.306 | 0.653 | 0.984 | |
| S | 0.605 | 0.377 | 0.505 | 0.395 | 0.700 | 0.995 | |
| S | 0.573 | 0.658 | 0.554 | 0.280 | 0.799 | 0.999 | |
| S | 0.625 | 0.843 | 0.858 | 0.573 | 0.923 | 1.000 | |
| S | 0.463 | 0.651 | 0.577 | 0.274 | 0.827 | 0.998 |
Medical LLMs
Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.
See all 428 papers