An evidence-traceable benchmark of clinical reasoning in large language models
Authors: Zhiqun Zhao, Xiaodong Yang, Bruce Gu, Ming Li, Yi Fu, Song Li, Qi Mei, Zhi Liu, Kai Zhang, Kexue Fu, Jing Wu, Xiaofei Lu, Yule Li, Zheng Wang, Haiquan Lu, Xiaoming Wu
Organizations: Advanced Medical Research Institute and Key Laboratory for Experimental Teratology of the Ministry of Education, Cheeloo College of Medicine, Shandong University, Jinan, China · Department of Urology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China · Shandong Computer Science Center, National Supercomputer Center in Jinan, Jinan, China · Qilu University of Technology, Shandong Academy of Sciences, Jinan, China · Department of Rheumatology and Immunology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China · Information Network Management Office, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China · Department of Medical Oncology, Qilu Hospital, Shandong University, Jinan, China · Department of Oncology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China · Department of Clinical Laboratory, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China · Department of Breast Surgery, General Surgery, Qilu Hospital, Shandong University, Jinan, China · Department of Central Laboratory, Public Health Clinical Center Affiliated to Shandong University, Shandong University, Jinan, China · Department of Breast Surgery, The First Affiliated Hospital of Shandong First Medical University, Jinan, China
Benchmark scores for medical large language models (LLMs) are difficult to interpret when scoring criteria cannot be traced to clinical evidence. Here we developed Med Evidence Bench (MEB; https://www.medevidencebench.org.cn/), comprising 1,536 physician-constructed items across three levels: guideline knowledge, standardized clinical scenarios and real-world laboratory scenarios. Each criterion stating a medical claim is linked to a versioned source-text span. Across ten LLMs, the Evidence Trace Score (ETS) was 0.975-0.997 for guideline knowledge, 0.292-0.658 for standardized scenarios and 0.182-0.447 for real-world scenarios, and rankings varied across levels. For real-world scenarios, scores for identifying clinical information (mean, 0.770) exceeded those for matching evidence (0.165) or formulating recommendations (0.160). On 1,296 criterion-level judgments, automated grading agreed with the majority vote of three physicians (macro-F1, 0.930; 95% confidence interval, 0.915-0.945), although agreement was lower for evidence application. These findings from Chinese-language materials indicate that guideline recall does not ensure patient-specific evidence use and do not establish clinical effectiveness.
Figures & tables
Figure 1: Construction and evaluation framework of MEB. a , Evaluation levels. MEB comprises 319 guideline-knowledge items (single-best-answer questions), 852 standardized clinical scenarios (3,100 criteria) and 365 real-world clinical scenarios built from laboratory records (2,992 criteria), classified by eight clinical intents. b , Construction of the benchmark. Physicians defined the target construct and its four capability domains, wrote item-specific rubrics, and verified each criterion through the physician-in-the-loop MEB-QC workflow, which links each criterion to a medical claim, an evidence record and a source-text span. c , Scoring. The CEC Agent adjudicated each criterion independently; criterion-level judgments were aggregated into the Evidence Trace Score (ETS), which was examined for sensitivity to response length, and the grader was compared with three blinded physician reviewers.
Figure 2: Quality control of item-specific rubrics and an example of evidence traceability. a , The MEB-QC workflow. Physicians submitted a case, its rubric and the initial supporting evidence. The MEB-QC Agent audited each criterion against the controlled evidence corpus and returned a structured report that classified the criterion as verified, insufficient evidence or applicability mismatch. Physicians reviewed each finding, and revised rubrics re-entered the audit until they were accepted for inclusion. b , Example of a real-world scenario concerning risk stratification of liver fibrosis in metabolic dysfunction-associated fatty liver disease (MAFLD), showing the prompt in English translation and an excerpt of the laboratory report, the criteria organized by CCEECC stage with their weights, and the chain from criterion to medical claim, evidence and source-text span for two criteria (shaded). ALT, alanine aminotransferase; AST, aspartate aminotransferase; CAP/UAP, controlled attenuation parameter/ultrasound attenuation parameter; FIB-4, fibrosis-4 index; LSM, liver stiffness measurement; SLEDAI, systemic lupus erythematosus disease activity index; TBIL, total bilirubin.
Figure 3: Performance of ten LLMs on MEB. a , Overall ETS across all 1,536 items. Bars show point estimates and error bars show 95% CIs from 5,000 case-level bootstrap resamples stratified by evaluation level. b , ETS (values above the markers) and rank (numbers within the markers) of each model for guideline knowledge ( n=319 items), standardized clinical scenarios ( n=852 ) and real-world clinical scenarios ( n=365 ). Diamonds denote medical systems and circles denote general-purpose models; models with equal scores share a rank. c , Scores in the four capability domains across all items; error bars are defined as in a . In a and c , model names in green denote medical systems and those in blue denote general-purpose models.
Figure 4: Performance on real-world clinical scenarios. a , Scores of each model at the six CCEECC stages ( n=365 items; 2,992 criteria). The score for critical-error detection is one minus the proportion of available penalty points that were triggered. b , ETS by laboratory category. In the left panel, each marker represents one model in one category; color indicates the ETS and marker size indicates ETS density (ETS × 2,000 / chars). In the right panel, points and horizontal lines show the median and interquartile range of the ETS across the ten models. c , ETS for each of the 53 diseases or conditions. Each dot represents one model, and shaded bands group the scenarios by laboratory category.
Figure 5: Agreement between automated and physician grading. a , Macro-F1 (blue) and accuracy (green) of the CEC Agent against the majority vote of three physicians, and of each physician against a reference derived from the other two, with disagreements adjudicated by a fourth physician, on 159 real-world items (1,296 criterion-level judgments). The dashed red line shows the mean macro-F1 of the three physicians (0.904). b , Macro-F1 by CCEECC stage. Filled circles show the CEC Agent, open diamonds show the mean macro-F1 of the three physicians and vertical ticks show individual physicians; horizontal lines show 95% CIs for the CEC Agent from 5,000 item-level cluster bootstrap resamples. The dashed blue line shows the overall macro-F1 of the CEC Agent (0.930). n , number of criterion-level judgments; Δ , macro-F1 of the CEC Agent minus the mean macro-F1 of the physicians. Values are given in Extended Data Table 3.
Figure 6: Sensitivity of the ETS to response length. a , Response length of DeepSeek-V4-Pro and Qwen3.8-Max under prompts that requested a concise, default or detailed response. Each dot represents one response; boxes show the median and interquartile range, whiskers 1.5× that range and shaded curves the distribution. The median is given beside each box. b , Mean ETS against mean response length under the three prompts, before (left) and after (right) adjustment for length. Triangles, squares and circles denote the concise, default and detailed prompts, respectively. c , ETS before (circles) and after (squares) adjustment for length for each of the ten models across the 1,217 open-ended tasks (standardized and real-world scenarios combined); bars at the right show the mean response length in characters.
Extended Data Fig. 1 ∣ ETS by clinical intent. ETS of each model across all 1,536 items, grouped by the eight clinical intents. Values above the bars are point estimates; error bars show 95% CIs obtained from 5,000 case-level bootstrap resamples conducted within each clinical-intent group. The number of items for each intent is given in Extended Data Table 2. Model names in green denote medical systems and those in blue denote general-purpose models.
Extended Data Fig. 2 ∣ ETS by clinical department in standardized clinical scenarios. ETS of each model across the 852 standardized clinical scenarios, grouped by the 12 clinical departments. Values above the bars are point estimates; error bars show 95% CIs obtained from 5,000 case-level bootstrap resamples conducted within each clinical department. Model names in green denote medical systems and those in blue denote general-purpose models.
Large language models (LLMs) have achieved strong performance on medical exam-style tasks, motivating growing interest in their deployment in real-world clinical settings. However, clinical decision-making is inherently safety-critical, context-dependent, and conducted under evolving evidence. In such situations, reliable LLM performance depends not on factual recall alone, but on robust medical reasoning. In this work, we present a comprehensive review of medical reasoning with LLMs. Grounded in cognitive theories of clinical reasoning, we conceptualize medical reasoning as an iterative process of abduction, deduction, and induction, and organize existing methods into seven major technical routes spanning training-based and training-free approaches. We further conduct a unified cross-benchmark evaluation of representative medical reasoning models under a consistent experimental setting, enabling a more systematic and comparable assessment of the empirical impact of existing methods. To better assess clinically grounded reasoning, we introduce MR-Bench, a benchmark derived from real-world hospital data. Evaluations on MR-Bench expose a pronounced gap between exam-level performance and accuracy on authentic clinical decision tasks. Overall, this survey provides a unified view of existing medical reasoning methods, benchmarks, and evaluation practices, and highlights key gaps between current model performance and the requirements of real-world clinical reasoning.
Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4
School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2
Recent advances in deep research systems enable large language models to retrieve, synthesize, and reason over large-scale external knowledge. In medicine, developing clinical guidelines critically depends on such deep evidence integration. However, existing benchmarks fail to evaluate this capability in realistic workflows requiring multi-step evidence integration and expert-level judgment. To address this gap, we introduce MedProbeBench, the first benchmark leveraging high-quality clinical guidelines as expert-level references. Medical guidelines, with their rigorous standards in neutrality and verifiability, represent the pinnacle of medical expertise and pose substantial challenges for deep research agents. For evaluation, we propose MedProbe-Eval, a comprehensive evaluation framework featuring: (1) Holistic Rubrics with 1,200+ task-adaptive rubric criteria for comprehensive quality assessment, and (2) Fine-grained Evidence Verification for rigorous validation of evidence precision, grounded in 5,130+ atomic claims. Evaluation of 17 LLMs and deep research agents reveals critical gaps in evidence integration and guideline generation, underscoring the substantial distance between current capabilities and expert-level clinical guideline development. Project: https://github.com/uni-medical/MedProbeBench
Jiyao Liu, Jianghan Shen, Sida Song +19
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts.
Doeun Lee, Muge Zhang, Yi Yu +11
The Ohio State University · The Ohio State University Wexner Medical Center · University of Chicago Medical Center