cs.CLApr 5, 2026

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

Authors: Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra, Robert E. Tillman

Organizations: Optum AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.

Figures & tables

Figure 1: Medical concept extraction F1 across medical experts and LLMs on our MedicalBench.
Figure 2: Performance of LLMs on the medical concept extraction task (y-axis) and evidence retrieval task (x-axis).
Figure 3: Precision–recall comparison of LLMs and a pretrained language model (PLM-ICD), which is an encoder with ≈125M\approx 125M parameters, on the medical concept extraction task.
ConditionPrecisionRecallF1TPTNFPFN
GPT-5 Without Reasoning Cues0.63170.49710.5564175369102177
GPT-5 With Reasoning Cues0.75780.68450.719424139477111
Table 1: GPT-5 Performance metrics comparison with and without reasoning cues
ConditionPrecisionRecallF1TPTNFPFN
GPT-5 With Implicit Evidence0.63170.49710.5564175369102177
GPT-5 Without Implicit Evidence0.49700.24140.32508538586267
Table 2: GPT-5 Performance metrics comparison with and without extraction of implicit evidence
Figure 4: GPT-5 performance across different note lengths
More in this topic

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 103 papers

Explore similar work

CardsList
Mar 17, 2026

Medical Reasoning with Large Language Models: A Survey and MR-Bench

Large language models (LLMs) have achieved strong performance on medical exam-style tasks, motivating growing interest in their deployment in real-world clinical settings. However, clinical decision-making is inherently safety-critical, context-dependent, and conducted under evolving evidence. In such situations, reliable LLM performance depends not on factual recall alone, but on robust medical reasoning. In this work, we present a comprehensive review of medical reasoning with LLMs. Grounded in cognitive theories of clinical reasoning, we conceptualize medical reasoning as an iterative process of abduction, deduction, and induction, and organize existing methods into seven major technical routes spanning training-based and training-free approaches. We further conduct a unified cross-benchmark evaluation of representative medical reasoning models under a consistent experimental setting, enabling a more systematic and comparable assessment of the empirical impact of existing methods. To better assess clinically grounded reasoning, we introduce MR-Bench, a benchmark derived from real-world hospital data. Evaluations on MR-Bench expose a pronounced gap between exam-level performance and accuracy on authentic clinical decision tasks. Overall, this survey provides a unified view of existing medical reasoning methods, benchmarks, and evaluation practices, and highlights key gaps between current model performance and the requirements of real-world clinical reasoning.

Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4

School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2

Medical ReasoningClinical ReasoningClinical Language Model EvaluationClinical Decision-Making
AskPDF
Mar 6, 2026

MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?

Large Language Models (LLMs) have demonstrated impressive capabilities across various specialist domains and have been integrated into high-stakes areas such as medicine. However, as existing medical-related benchmarks rarely stress-test the long-context memory, interference robustness, and safety defense required in practice. To bridge this gap, we introduce MedMT-Bench, a challenging medical multi-turn instruction following benchmark that simulates the entire diagnosis and treatment process. We construct the benchmark via scene-by-scene data synthesis refined by manual expert editing, yielding 400 test cases that are highly consistent with real-world application scenarios. Each test case has an average of 22 rounds (maximum of 52 rounds), covering 5 types of difficult instruction following issues. For evaluation, we propose an LLM-as-judge protocol with instance-level rubrics and atomic test points, validated against expert annotations with a human-LLM agreement of 91.94%. We test 17 frontier models, all of which underperform on MedMT-Bench (overall accuracy below 60.00%), with the best model reaching 59.75%. MedMT-Bench can be an essential tool for driving future research towards safer and more reliable medical AI. The benchmark is available in https://openreview.net/attachment?id=aKyBCsPOHB&name=supplementary_material

Lin Yang, Yuancheng Yang, Xu Wang +2

ByteDance

Long-Context Language Model EvaluationClinical Language Model EvaluationMulti-Turn Dialogue EvaluationLLM Safety Benchmarks
AskPDF
May 2, 2026

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks

Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32

Sophont · MedARC · Prime Intellect

LLM EvaluationHealthcareClinical Language Model EvaluationMedical LLMs
AskPDF