Clinical Language Model Evaluation

Follow this topic Ask about this topic

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

419 papers49 papers in the last four weeks, up 2% on the four weeks before. 0.3% of all new papers.

Clinician input steers AI toward accurate and harmful recommendations
Mar 14, 2026Ivan Lopez, Selin S. Everett, Bryan J. Bunning +10Medical LLMs Clinical Language Model Evaluation
Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Stanford University School of Medicine, Stanford, CA, USA · Quantitative Sciences Unit, Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA +14
Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Mar 14, 2026Halimat Afolabi, Zainab Afolabi, Elizabeth Friel +13Faithfulness of Language Model Explanations Medical LLMs
Stratified Precision, United Kingdom · Harvard Medical School, United States · Imperial College London, United Kingdom +3
QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models
Mar 14, 2026Yao Wu, Kangping Yin, Liang Dong +13LLM Evaluation Clinical Language Model Evaluation
Quark Medical Team, Alibaba Group
MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences
Mar 13, 2026Eric Wu, Kevin Wu, Jason Hom +11Clinical Language Model Evaluation Medical LLMs
Department of Electrical Engineering, Stanford University, Stanford, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Division of Hospital Medicine, Department of Medicine, Stanford School of Medicine, Stanford, CA, USA +8
Developing and evaluating a chatbot to support maternal health care
Mar 13, 2026Smriti Jha, Vidhi Jain, Jianyu Xu +8Healthcare Medical LLMs
Carnegie Mellon University · Population Council Institute · Sitaram Bhartia Institute of Science and Research +1
Shattering the Shortcut: A Topology-Regularized Benchmark for Multi-hop Medical Reasoning in LLMs
Mar 12, 2026Xing Zi, Xinying Zhou, Jinghao Xiao +2Shortcut Learning in Language Models Multi-Hop QA
University of Technology Sydney, Australia
Performance Evaluation of Open-Source Large Language Models for Assisting Pathology Report Writing in Japanese
Mar 12, 2026Masataka Kawai, Singo Sakashita, Shumpei Ishikawa +8Medical Report Generation Medical LLMs
Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan · Department of Pathology, University of Yamanashi, Chuo, Japan · Division of Pathology, Exploratory Oncology Research & Clinical Trial Center, National Cancer Center, Kashiwa, Japan +3
Evaluation format, not only model capability, shapes measured triage failure in consumer health AI
Mar 12, 2026David Fraile Navarro, Jialei Sheng, Farah Magrabi +1Language Model Safety Evaluation LLM Safety Benchmarks
Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia · NSW Health, Sydney, NSW, Australia
Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability
Mar 12, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5LLM Reliability Multi-Turn Dialogue Evaluation
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Human-AI Co-reasoning for Clinical Diagnosis with Evidence-Integrated Language Agent
Mar 11, 2026Zhongzhen Huang, Yan Ling, Hong Chen +7Medical Diagnosis Clinical Decision Support
Shanghai Jiao Tong University, Shanghai, China · Department of Endocrinology, Zhongshan Hospital, Fudan University, Shanghai, China · Department of Endocrinology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China +3
WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics
Mar 11, 2026Sneha Maurya, Spandana Govindgari, Girish Kumar +1Clinical Language Model Evaluation LLM Safety Benchmarks
Columbia University · Akhara AI
Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses
Mar 10, 2026Congning Ni, Sarvech Qadir, Bryan Steitz +14Hallucination in Language Models LLM Safety Evaluation
Vanderbilt University Medical Center, Nashville, TN, USA · Vanderbilt University, Nashville, TN, USA · Lirio, Knoxville, TN, USA +1
A prospective clinical feasibility study of a conversational diagnostic AI in an ambulatory primary care clinic
Mar 9, 2026Peter Brodeur, Jacob M. Koshy, Anil Palepu +45Healthcare Medical LLMs
COACH meets QUORUM: A Framework and Pipeline for Aligning User, Expert and Developer Perspectives in LLM-generated Health Counselling
Mar 9, 2026Yee Man Ng, Bram van Dijk, Pieter Beynen +5Clinical Language Model Evaluation Medical LLMs
Leiden University · Leiden University Medical Center · Healthy Chronos
Language Shapes Mental Health Evaluations in Large Language Models
Mar 6, 2026Jiayi Xu, Xiyang HuCross-Lingual Consistency in Language Models Multilingual Language Model Evaluation
University of North Carolina at Chapel Hill · Arizona State University
MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?
Mar 6, 2026Lin Yang, Yuancheng Yang, Xu Wang +2Long-Context Language Model Evaluation Clinical Language Model Evaluation
ByteDance
Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
Mar 4, 2026Ikram Belmadani, Oumaima El Khettari, Pacôme Constant dit Beaufils +2LLM-as-a-Judge Bias in LLM-as-a-Judge
Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N, UMR 6004, 44000 Nantes, France · Nantes Université, CHU Nantes, PHU 11: Santé Publique, Clinique des données, INSERM, CIC 1413, 44000 Nantes, France +2
Assessing the Effectiveness of LLMs in Delivering Cognitive Behavioral Therapy
Mar 4, 2026Navdeep Singh Bedi, Ana-Maria Bucur, Noriko Kando +1Clinical Language Model Evaluation Medical LLMs
Università della Svizzera italiana, Switzerland · National Institute of Informatics, Japan
Benchmarking Motivational Interviewing Competence of Large Language Models
Mar 4, 2026Aishwariya Jha, Prakrithi Shivaprakash, Lekhansh Shukla +3Clinical Language Model Evaluation Medical LLMs
Centre for Addiction Medicine, Department of Psychiatry, National Institute of Mental Health and Neuro Sciences (NIMHANS), Bengaluru, India. · Department of Computer Science and Engineering, Indian Institute of Technology (IIT) Kharagpur, India. · Department of Psychiatry, National Institute of Mental Health and Neuro Sciences (NIMHANS), Bengaluru, India.
TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
Mar 3, 2026Zixin Xiong, Ziteng Wang, Haotian Fan +2LLM Safety Benchmarks LLM Reliability
Renmin University of China, China · Beijing University of Posts and Telecommunications, China · Hefei University of Technology, China
ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs
Mar 2, 2026Xiang Zheng, Han Li, Wenjie Luo +15Clinical Language Model Evaluation Medical LLMs
Alibaba Group, China
Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring
Mar 2, 2026Aditya Shukla, Yining Yuan, Ben Tamo +7Summarization Evaluation Clinical Language Model Evaluation
Georgia Institute of Technology · Shepherd Center
PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology
Mar 2, 2026Yimin Zhao, Sheela R. Damle, Simone E. Dekker +13Clinical Language Model Evaluation Medical QA
Department of Biostatistics, University of Washington · Clinical Research Division, Fred Hutch Cancer Center · Division of Hematology and Oncology, Department of Medicine, University of Washington +5
Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare
Mar 1, 2026Shravani HariprasadClinical Language Model Evaluation Prompt Sensitivity
Independent Researcher
Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking
Feb 28, 2026Tianpeng Zheng, Zhehan Jiang, Jiayi Liu +1Computerized Adaptive Testing LLM Evaluation
Institute of Medical Education, Health Science Center, Peking University, Beijing, China · School of Public Health, Peking University, Haidian District, Beijing, China · Peking University Health Science Center-Chaoxing Joint Laboratory for Digital and Smart Medical, Peking University, Beijing, China +1
CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
Feb 28, 2026Yannian Gu, Zhongzhen Huang, Linjie Mu +3Clinical Language Model Evaluation Multimodal Model Evaluation
Shanghai Jiao Tong University, Shanghai, China · SenseTime Research, China · Shanghai Innovation Institute, Shanghai, China
When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation
Feb 27, 2026Bian Sun, Zhenjian Wang, Orvill de la Torre +1Clinical Language Model Evaluation Generative AI Evaluation
Heinz College, Carnegie Mellon University · School of Computer Science, Carnegie Mellon University
MediX-R1: Open Ended Medical Reinforcement Learning
Feb 26, 2026Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed +5RL for VLMs Medical VLMs
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College
Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department
Feb 26, 2026Gabriela Anna Kaczmarek, Pietro Ferrazzi, Lorenzo Porta +2Clinical NLP Clinical Information Extraction
Fondazione Bruno Kessler, Povo, Trento, Italy · University of Padova, Padova, Italy · Emergency Medicine Unit, Fatebenefratelli Hospital, Milan, Italy +1
ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making
Feb 26, 2026Yusuke Watanabe, Yohei Kobashi, Takeshi Kojima +3Clinical Language Model Evaluation LLM Safety Benchmarks
Kyoto University, Department of Biomedical Data Intelligence · Kyoto University, Department of Cardiovascular Medicine · The University of Tokyo

Mar 14, 2026J/K move · Enter open · S save

Clinician input steers AI toward accurate and harmful recommendations

Ivan Lopez, Selin S. Everett, Bryan J. Bunning, April S. Liang+9

Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Stanford University School of Medicine, Stanford, CA, USA · Quantitative Sciences Unit, Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA+14

Medical LLMs Clinical Language Model Evaluation LLM Sycophancy Adversarial Robustness of Language Models LLM Inference Clinical Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) are entering clinical workflows, yet evaluations rarely assess how clinician reasoning shapes model behavior during clinical interactions. Using 61 curated NEJM Case Records, we tested how expert or misleading clinician reasoning influenced AI-generated differential diagnoses and next step recommendations across 21 reasoning variants from 8 proprietary and open-source models. After clinician exposure, LLM-clinician concordance increased: simulations with >=3 overlapping differential diagnoses rose from 65.8% to 93.5%, and those with >=3 overlapping next step recommendations from 20.3% to 53.8%. Expert context significantly improved correct final-diagnosis inclusion in all 21 models (mean +20.4 pp), reflecting both improved reasoning and passive content echoing, while adversarial context significantly degraded performance in 14 models (mean -5.4 pp). Expert context also significantly increased leading-diagnosis accuracy in all 21 models, whereas adversarial context significantly reduced it in 13. Multi-turn disagreement challenges revealed distinct model phenotypes, from highly conformist to dogmatic, with adversarial arguments remaining a vulnerability even in otherwise resilient models. Inference-time scaling reduced harmful echoing of clinician-introduced recommendations across WHO harm-severity tiers by 62.7% for mild, 57.9% for moderate, 76.3% for severe, and 83.5% for death-tier recommendations. Inference-time prompting recovered diagnostic accuracy lost to adversarial context while preserving expert-context benefits across GPT-5, Claude Sonnet 4.5, and Gemini 3 Flash, and sharply reduced highly consistent harmful echoing across severity tiers. These findings provide a foundation for evaluating clinician-AI collaboration and introduce interactive metrics and mitigation strategies essential to safety and robustness.