Medical Reasoning with Large Language Models: A Survey and MR-Bench

Xiaohan Ren; Chenxiao Fan; Wenyin Ma; Hongliang He; Chongming Gao; Xiaoyan Zhao; Fuli Feng

cs.CLMar 17, 2026

Medical Reasoning with Large Language Models: A Survey and MR-Bench

Authors: Xiaohan Ren, Chenxiao Fan, Wenyin Ma, Hongliang He, Chongming Gao, Xiaoyan Zhao, Fuli Feng

Organizations: School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China · School of Information Science and Technology, University of Science and Technology of China, Anhui, China · School of Computing, National University of Singapore, Singapore

Abstract

Large language models (LLMs) have achieved strong performance on medical exam-style tasks, motivating growing interest in their deployment in real-world clinical settings. However, clinical decision-making is inherently safety-critical, context-dependent, and conducted under evolving evidence. In such situations, reliable LLM performance depends not on factual recall alone, but on robust medical reasoning. In this work, we present a comprehensive review of medical reasoning with LLMs. Grounded in cognitive theories of clinical reasoning, we conceptualize medical reasoning as an iterative process of abduction, deduction, and induction, and organize existing methods into seven major technical routes spanning training-based and training-free approaches. We further conduct a unified cross-benchmark evaluation of representative medical reasoning models under a consistent experimental setting, enabling a more systematic and comparable assessment of the empirical impact of existing methods. To better assess clinically grounded reasoning, we introduce MR-Bench, a benchmark derived from real-world hospital data. Evaluations on MR-Bench expose a pronounced gap between exam-level performance and accuracy on authentic clinical decision tasks. Overall, this survey provides a unified view of existing medical reasoning methods, benchmarks, and evaluation practices, and highlights key gaps between current model performance and the requirements of real-world clinical reasoning.

Figures & tables

Category	Approach	Research
Training-based approaches	Continue pretraining	Baichuan-M1 [ 42 ] , Baichuan-M2 [ 43 ] , MedFound [ 44 ] , Citrus [ 45 ] , EHR-R1 [ 46 ]
	Supervised fine-tuning	Med-PALM [ 4 ] , Med-PALM 2 [ 5 ] , Baichuan-M1 [ 42 ] , Baichuan-M2 [ 43 ] , Lingshu [ 47 ] , HuatuoGPT-o1 [ 23 ] , UltraMedical [ 48 ] , m1 [ 25 ] , MedReason [ 24 ] , BioMed-R1 [ 26 ] , MedS 3 [ 49 ] , Med-Gemma [ 50 ] , DrugGPT [ 51 ] , MedFound [ 44 ] , Med-Gemini [ 52 ] , LongMonolog [ 53 ] , MedResearcher-R1 [ 54 ] , Citrus [ 45 ] , ClinicalGPT-R1 [ 55 ] , AMIE [ 56 , 57 ] , MedAgentGym [ 58 ] , DiagnoseGPT [ 59 ] , DDO [ 60 ] , Meerkat [ 61 ] , Self-BioRAG [ 62 ] , Med42-v2 [ 63 ] , MedAdapter [ 64 ] , QuarkMed [ 65 ] , EHR-R1 [ 46 ]
	Reinforcement learning	Baichuan-M1 [ 42 ] , Baichuan-M2 [ 43 ] , Lingshu [ 47 ] , HuatuoGPT-o1 [ 23 ] , UltraMedical [ 48 ] , m1 [ 25 ] , BioMed-R1 [ 26 ] , AlphaMed [ 66 ] , MedS 3 [ 49 ] , Med-Gemma [ 50 ] , MedFound [ 44 ] , MedResearcher-R1 [ 54 ] , Citrus [ 45 ] , ClinicalGPT-R1 [ 55 ] , DDO [ 60 ] , Deep-DxSearch [ 67 ] , AdaThink-Med [ 68 ] , Med42-v2 [ 63 ] , QuarkMed [ 65 ] , EHR-R1 [ 46 ]
Training-free approaches	Prompt engineering	Med-PALM [ 4 ] , Med-PALM 2 [ 5 ] , HuatuoGPT-o1 [ 23 ] , UltraMedical [ 48 ] , m1 [ 25 ] , MedReason [ 24 ] , BioMed-R1 [ 26 ] , MedS 3 [ 49 ] , Med-Gemma [ 50 ] , DrugGPT [ 51 ] , MedFound [ 44 ] , Med-Gemini [ 52 ] , CLINICR [ 69 ] , EnsReas [ 70 ] , AMIE [ 56 , 57 ] , Medprompt [ 71 ] , Clinical CoT [ 72 ] , DR-CoT [ 73 ] , Diagnostic reasoning prompts [ 74 ] , DiagnoseGPT [ 59 ] , MEDIQ [ 75 ] , Meerkat [ 61 ] , MedRAG [ 76 ] , MedAdapter [ 64 ]
	Test-time strategies	Med-PALM [ 4 ] , Med-PALM 2 [ 5 ] , m1 [ 25 ] , LongMonolog [ 53 ] , Med-Gemini [ 52 ] , CLINICR [ 69 ] , EnsReas [ 70 ] , Medprompt [ 71 ] , MEDIQ [ 75 ] , Meerkat [ 61 ] , MedAdapter [ 64 ]
	Retrieval-augmented generation	Med-PALM 2 [ 5 ] , DrugGPT [ 51 ] , Med-Gemini [ 52 ] , MedResearcher-R1 [ 54 ] , Medprompt [ 71 ] , DiagnoseGPT [ 59 ] , Deep-DxSearch [ 67 ] , MedRAG [ 76 ] , Self-BioRAG [ 62 ] , QuarkMed [ 65 ]

Table 1: Approaches for Medical Reasoning with LLMs. Underlined : Open-source. Yellow : Research focused on model training. Green : Research utilizing LLMs as tools (e.g., agents).

Benchmark	Type	Year	Samples
PubMedQA [ 100 ]	Multiple Choice QA	2019	1,000
MedQA (USMLE) [ 38 ]	Multiple Choice QA	2020	1,273
MMLU [ 101 ]	Multiple Choice QA	2020	15,908
MedMCQA (Validation) [ 102 ]	Multiple Choice QA	2022	4,183
CMBExam [ 103 ]	Multiple Choice QA	2023	11,200
GPQA [ 104 ]	Multiple Choice QA	2023	448

Table 2: Summary of Existing Benchmarks. Underlined benchmarks represent general-domain datasets containing specific medical subsets.

Open-source General LLMs (Base Models)
Model	MedQA	MedMCQA	PubMedQA	GPQA	JMED	ReDis-QA	MedXpertQA	MMLU-Pro	Average	R. I. (%)
gemma3-4B-it	0.462	0.464	0.708	0.423	0.353	0.501	0.114	0.496	0.440	-
Qwen3-4B-Thinking-2507	0.736	0.642	0.763	0.610	0.572	0.727	0.177	0.772	0.625	-
Qwen-2.5-7B-Instruct	0.582	0.562	0.748	0.474	0.580	0.667	0.131	0.623	0.546	-
Meta-Llama-3-8B-Instruct	0.530	0.543	0.773	0.433	0.422	0.732	0.122	0.550	0.513	-
Llama-3.1-8B-Instruct	0.518	0.554	0.776	0.438	0.425	0.735	0.129	0.542	0.515	-

Table 3: Model performance on existing benchmarks. Background colors denote base LLM families, with each medical LLM inheriting the background color of its corresponding base model. Bold indicates the best overall score; underlining marks the best medical LLM within each base model group.

Open-source General LLMs (Base Models)
Model	Medication		Procedure		Average
Model	Acc.	R.I. (%)	Acc.	R.I. (%)	Acc.	R.I. (%)
gemma3-4B-it	0.237	-	0.504	-	0.371	-
Qwen3-4B-Thinking-2507	0.272	-	0.588	-	0.430	-
Qwen-2.5-7B-Instruct	0.213	-	0.595	-	0.404	-
Meta-Llama-3-8B-Instruct	0.166	-	0.431	-	0.299	-

Table 4: Model performance on MR-Bench. Background colors denote base LLM families, with each medical LLM inheriting the background color of its corresponding base model. Bold indicates the best overall score (excluding advanced fundation models); underlining marks the best medical LLM within each base model group. R.I. denotes Relative Improvement compared to the corresponding base model.

Explore similar work

Jul 8, 2026

Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning

Large language models (LLMs) have emerged as important tools in healthcare, showing growing potential for clinical reasoning and patient care. This survey examines recent progress in medical LLMs, focusing on reasoning applications and requirements. We present a dual-view approach that connects clinical practice with computational methods. On the clinical side, we establish a five-level competency scheme following Miller's Pyramid, progressing from knowledge recall to dynamic case management. On the computational side, we link deductive, inductive, and abductive reasoning patterns to common medical goals and tasks. We also introduce a benchmark dataset spanning five levels of medical reasoning capability and report results on 18 state-of-the-art models, revealing that medical specialist models excel in diagnosis-centric tasks while general models lead in decision support and dialogue. We conclude by discussing current progress and open challenges, including data limitations, hallucination, and grounding issues, and outline directions toward safer, more reliable, and workflow-ready systems.

Qi Peng, Jiatong Li, Sirui Huang +10

1The Hong Kong Polytechnic University, Hong Kong, China. · 2Hong Kong University, Hong Kong, China. · 3South China University of Technology, Guangzhou, China. +3

Clinical Reasoning Clinical Language Model Evaluation Medical LLMs

Ask PDF

Apr 19, 2026

MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

Medical reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing PRM benchmarks largely target mathematics rather than patient-specific clinical errors. We introduce MedPRMBench, a step-level medical reasoning benchmark with 14 clinical error types, including five new and two clinically re-scoped types, and rule-defined severity grades. Built from seven medical QA sources via structured error construction and review by 86 physicians from Grade-A tertiary hospitals, it provides 6{,}500 evaluation questions, 13{,}000 reasoning chains, and 113{,}910 step labels, plus a leakage-clean 6{,}870-question training split. The best of 20 prompted critics reaches only 75.4% PRMScore, and critics differ sharply in error-type profiles and operating points. A Qwen3-8B PRM trained on the split reaches 86.6% PRMScore and 89.0% strict first-error localization; in matched ablations, physician review raises error recall by 6.55 points and cuts clean-chain false alarms from 5.6% to 2.1%. The training signal replicates across model families; without further training, the PRM transfers to ProcessBench and PRMBench, exceeding six of nine math-trained PRMs on each.

Xiang Zheng, Lingyan Wu, Weiqi Zhai +4

Alibaba Group, China

Healthcare Process Reward Models Medical Reasoning Clinical Language Model Evaluation

Ask PDF

May 2, 2026

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks

Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32

Sophont · MedARC · Prime Intellect

LLM Evaluation Healthcare Clinical Language Model Evaluation Medical LLMs

Ask PDF