Clinical Language Model Evaluation

Follow this topic Ask about this topic

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

419 papers49 papers in the last four weeks, up 2% on the four weeks before. 0.3% of all new papers.

TCM-DiagCommit: A TCM-SD-Derived Benchmark for Evidence-Conditioned Syndrome Selection and Commitment
Oct 27, 2026Erwei Hou, Yue HuClinical Language Model Evaluation Clinical Decision-Making
Beijing University of Posts and Telecommunications, Beijing, China
Patient-specific rubric-based evaluation of large language models across longitudinal cancer care
Oct 26, 2026Wei-Wei Zhang, Pengkai Wang, Yan Li +13Clinical Language Model Evaluation Medical LLMs
Department of Radiation Oncology, Sun Yat-sen University Cancer Center, Guangzhou, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Pathology, Sun Yat-sen University Cancer Center, Guangzhou, China. +2
Can JEV Support Clinical Trial Criteria Assessment?
Oct 26, 2026Mengxian He, Ziyue Gao, Chun Li +4Clinical Decision Support Clinical Language Model Evaluation
The Chinese University of Hong Kong · Capital Medical University
MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine
Oct 26, 2026Simon A. Lee, Jiahang Sha, Moonseong Jeong +2Clinical Language Model Evaluation Zero-Shot Learning
UCLA Computational Medicine · UCLA Neurosurgery
EXCISE: Locating and Excising Distracting Sentences in Medical Question Answering
Oct 26, 2026Yerim Hong, Sohyun Son, Heeseung Chung +1Language Model Robustness Clinical QA
Kyung Hee University
DocTalk: Enhancing Medical Knowledge Retrieval via Fine-tuned Large Language Models with Chain-of-Thought Reasoning for Robust Clinical Decision Support
Oct 26, 2026Asad Gohar, Md Ariful Islam Mozumder, Moon-Il Joo +1Clinical QA EHR QA
Institute of Digital Anti-Aging HealthCare, Inje University, Gimhae 50834, Republic of Korea
The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs
Oct 25, 2026Viswonathan Manoranjan, Andrea Mock, Myra Cheng +4LLM Sycophancy Clinical Language Model Evaluation
University of North Carolina at Chapel Hill · Vals AI · Stanford University +2
Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis
Oct 24, 2026Jian Yang, Peilin Liu, Xiaohan Li +4Clinical Language Model Evaluation LLM Auditing
The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China. · The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, China.
MedCaseAgent: From Multimodal Clinical Evidence to Rare Disease Case Reports
Oct 23, 2026Hongkang Chu, Yifan Shen, Tianjiao Yu +4Medical Report Generation Medical LLMs
PediaMed AI · University of Chinese Academy of Sciences · University of Illinois Urbana-Champaign +3
AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy
Oct 22, 2026Daniel S. EpsteinClinical Language Model Evaluation Clinical QA
Department of General Practice, Monash University, Melbourne, Victoria, Australia · The Long Game Project, Melbourne, Victoria, Australia
Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination
Oct 22, 2026Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera +2Clinical Language Model Evaluation Medical QA
Faculty of Engineering, University of Deusto (Bilbao, Spain)
From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries
Oct 22, 2026Bayram Ayadi, Florian Geissler, Johanna SchmidhuberText Summarization Healthcare
University of Passau, Passau, Germany · Fraunhofer Institute for Cognitive Systems IKS, Munich, Germany
The Value of System 1 Models in Medicine
Oct 22, 2026Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben +17Clinical Language Model Evaluation Multimodal Model Evaluation
Department of Diagnostic and Interventional Radiology, School of Medicine and Health, Technical University of Munich, Munich, Germany · Artificial Intelligence and Translational Imaging Lab, Department of Radiology, School of Medicine, University of Crete, Heraklion, Greece · Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece +6
IndoNutriQA: An Indonesian Clinical Nutrition Question Answering Benchmark
Oct 22, 2026Tirana Noor Fatyanosa, Ummu Ditya Erliana, Mahendra Data +3Clinical QA Healthcare
Universitas Brawijaya · Universitas Ma Chung · MBZUAI
Large language models show systematic failures in incorporating evolving medical knowledge
Oct 22, 2026Min Zeng, Zaifu Zhan, Kai Yu +1Medical LLMs LLM Reliability
Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, 55455, MN, USA.
SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots
Oct 21, 2026Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova +3Clinical Language Model Evaluation User Simulation
UserTrace · Cliniva · hearX Foundation +2
Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI
Oct 20, 2026Rotem Gershon, Seffi Cohen, Dean Ariel +11LLM-as-a-Judge Clinical Language Model Evaluation
Innovation Division, Clalit Research Institute, Clalit Health Services, Ramat-Gan, Israel. · Stein Faculty of Computer and Information Science, Ben Gurion University of the Negev, Be’er Sheva, Israel. · Department of Family Medicine, Clalit Health Services, Tel Aviv, Israel. +4
Scaling Therapy Supervision: Assessing Psychotherapy Fidelity Using Large Language Models
Oct 20, 2026Arvind Pillai, Alexis Roach, Jason Holden +11Clinical Language Model Evaluation
Dartmouth College · University of California, San Diego · University of Virginia
Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing
Oct 20, 2026Xi Yang, Guodong Liu, Cheng Peng +7Recurrent Language Models Clinical NLP
Advanced Intelligence and Research Eli Lilly and Company Indianapolis, IN, USA · Dept. of Health Outcomes and Biomedical Informatics University of Florida Gainesville, FL, USA · Higher Education and Research NVIDIA Santa Clara, CA, USA +2
EviDial: When Correct Depression Predictions Lack Evidence
Oct 20, 2026Junjie Xu, Jin Zhu, Yuzhe Yang +6Clinical Language Model Evaluation Depression Severity Estimation
East China Normal University · Fudan University
When Health Assistants Agree Too Readily: A Benchmark of Time-Critical Self-Diagnosis
Oct 20, 2026Marilyn Zhang, Tianfeng Chen, Patricia M Pechter +1Clinical Language Model Evaluation LLM Safety Benchmarks
Pareto Inc · NYU Langone
Does Multi-Agent Collaboration Reduce Hallucination in Clinical Tasks?
Oct 19, 2026Won Seok Jang, Harry Bhaskar Joshi, Hong YuHallucination in Language Models Multi-Agent Collaboration
University of Massachusetts Lowell
When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine
Oct 19, 2026Yusuke Watanabe, Yohei Kobashi, Taisei Tosaki +4LLM Reliability Language Model Bias Evaluation
Kyoto University, Department of Biomedical Data Intelligence · Kyoto University, Department of Cardiovascular Medicine · The University of Tokyo
Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
Oct 17, 2026Liang You, Zhiyin Zhou, Dawei Tang +2Clinical Language Model Evaluation Clinical Triage
University of Pittsburgh · Pratt Institute · Icahn School of Medicine at Mount Sinai +2
Recognizing Study Lineage Is Not Enough: Over-Crediting Same-Trial Evidence in LLM-Based Medical Verification
Oct 16, 2026Mukand Krishna, Anoushka Harit, Chenghao Xiao +1LLM Reliability LLM Answer Verification
School of Computing, University of Kent, UK · Cancer Research UK Cambridge Institute, University of Cambridge, UK · Shanghai University of Finance and Economics, China
MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
Oct 15, 2026John Mendonça, Daniel Tschernutter, José Pombal +6Clinical Language Model Evaluation Multi-Turn Dialogue Evaluation
Sword Health · Instituto Superior Técnico · Yale University +1
AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions
Oct 15, 2026Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev +10Clinical Language Model Evaluation Clinical Decision Support
AIRI, Moscow, Russian Federation · Northern State Medical University, Arkhangelsk, Russian Federation · Arkhangelsk Clinical Oncological Dispensary, Arkhangelsk, Russian Federation +5
Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures
Oct 15, 2026Zihao Zheng, Junyi Yao, Baichuan Li +1LLM Agent Reliability AI Agent Safety Benchmarks
Washington University in St. Louis · Southern Methodist University
From Hindsight to Foresight: Dissecting Clinical Evidence Acquisition in Large Language Models
Oct 13, 2026Ziyang LingMedical LLMs Clinical Language Model Evaluation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, China
How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs
Oct 13, 2026Yanrui Du, Danyang Zhao, Sendong Zhao +6Clinical Decision-Making LLM Reliability
Harbin Institute of Technology · Ant Group

Oct 27, 2026J/K move · Enter open · S save

TCM-DiagCommit: A TCM-SD-Derived Benchmark for Evidence-Conditioned Syndrome Selection and Commitment

Erwei Hou, Yue Hu

Beijing University of Posts and Telecommunications, Beijing, China

Clinical Language Model Evaluation Clinical Decision-Making Selective Prediction Clinical Reasoning

Open paper AskPDF arXiv

Traditional Chinese Medicine (TCM) syndrome differentiation is usually evaluated as static final-label selection over complete clinical records. This setting hides three diagnostic behaviors: whether a model distinguishes semantic-neighbor syndrome labels from easy random distractors, whether it updates predictions as evidence fields are revealed, and whether it should commit under uncertainty. We introduce TCM-DiagCommit, a benchmark derived from the public TCM-SD/ZY-BERT resource while retaining its source syndrome labels; it converts 1,834 source cases into 23,842 task rows over 148 syndrome labels. The benchmark combines random, profile-neighbor, and full-vocabulary candidate ladders with staged t1/t2/t3 evidence. It also includes fixed-label tongue/pulse perturbations and insufficient-evidence variants. A benchmark-defined action contract scores commit, acquire, defer, and abstain. Across five zero-shot DirectLLM calibrations, random10 top-1 accuracy is 63.2% to 66.4%, but hard20 falls to 24.7% to 30.3% and full148 to 14.8% to 22.3%; on hard20, WT(all) wrong-terminal commits remain 62.2% to 74.3%. These results show that final-answer accuracy alone is insufficient for diagnostic benchmark evaluation: answer selection, evidence use, and benchmark commitment scoring should be measured separately.