cs.CLNov 10, 2026

An evidence-traceable benchmark of clinical reasoning in large language models

Authors: Zhiqun Zhao, Xiaodong Yang, Bruce Gu, Ming Li, Yi Fu, Song Li, Qi Mei, Zhi Liu, 

Organizations: Advanced Medical Research Institute and Key Laboratory for Experimental Teratology of the Ministry of Education, Cheeloo College of Medicine, Shandong University, Jinan, China · Department of Urology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China · Shandong Computer Science Center, National Supercomputer Center in Jinan, Jinan, China · Qilu University of Technology, Shandong Academy of Sciences, Jinan, China · 

Abstract

Benchmark scores for medical large language models (LLMs) are difficult to interpret when scoring criteria cannot be traced to clinical evidence. Here we developed Med Evidence Bench (MEB; https://www.medevidencebench.org.cn/), comprising 1,536 physician-constructed items across three levels: guideline knowledge, standardized clinical scenarios and real-world laboratory scenarios. Each criterion stating a medical claim is linked to a versioned source-text span. Across ten LLMs, the Evidence Trace Score (ETS) was 0.975-0.997 for guideline knowledge, 0.292-0.658 for standardized scenarios and 0.182-0.447 for real-world scenarios, and rankings varied across levels. For real-world scenarios, scores for identifying clinical information (mean, 0.770) exceeded those for matching evidence (0.165) or formulating recommendations (0.160). On 1,296 criterion-level judgments, automated grading agreed with the majority vote of three physicians (macro-F1, 0.930; 95% confidence interval, 0.915-0.945), although agreement was lower for evidence application. These findings from Chinese-language materials indicate that guideline recall does not ensure patient-specific evidence use and do not establish clinical effectiveness.

Figures & tables

Explore similar work

CardsList
  1. Medical Reasoning with Large Language Models: A Survey and MR-Bench

    Mar 17, 2026Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4Medical ReasoningClinical Reasoning

  2. MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline

    Apr 20, 2026Jiyao Liu, Jianghan Shen, Sida Song +19HealthcareAI Agent Benchmarks

  3. When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

    May 20, 2026Doeun Lee, Muge Zhang, Yi Yu +11LLM GroundingHealthcare