Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9LLM Safety Benchmarks Mental Health Counseling
School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom +4
KnowBench: Effort Reduction as a Unified, Deployment-Grounded Benchmark for Clinical AI
Sep 14, 2026Jocelyn Kang, Caroline ZhangBenchmark Design Workflow Automation
Knowtex Inc.
Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML
Sep 14, 2026Sidi Chang, Peiying ZhuLLM-as-a-Judge LLM Auditing
Blossom AI San Francisco, CA, USA
When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation
Sep 14, 2026Griffin Farrow, Lily Sijia Li, Jack Johnson +4LLM Hallucination Detection Rubric-Based Evaluation
University of Oxford
MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Sep 14, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Medical Diagnosis Multi-Turn Dialogue Evaluation
MBZUAI · Cairo University · Ain Shams University +2
A primer on evaluation methods for large language models in healthcare
Sep 13, 2026Suzannah E McKinney, Phuc Vu, Samuel A Justice +7LLM Evaluation Clinical Language Model Evaluation
Mass General Brigham AI, Mass General Brigham, Boston, MA, USA · Generative AI Laboratory, School of Informatics, University of Edinburgh, Edinburgh, UK · University of Edinburgh Centre for Cardiovascular Science, Edinburgh, UK +5
Towards a Deterministic Math Solver for Clinical Language Models
Sep 12, 2026Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange +5Numerical Reasoning in Language Models LLM Reliability
MIT Critical Data · UNC Chapel-Hil · University of Pavia +1
Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment
Sep 12, 2026Jiayu Huang, Zichen Tang, Qianhui Ling +2Reasoning Consistency in Language Models Clinical Language Model Evaluation
Beijing University of Posts and Telecommunications · Hypertension Center, Beijing Anzhen Hospital, Capital Medical University
Scaling Clinical Judgment to Evaluate Medical AI
Sep 11, 2026Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur +15LLM-as-a-Judge LLM Fine-Tuning
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA · Stanford Division of Hospital Medicine, Stanford University, Stanford, CA +11
The widening evaluation gap in medical large language model research 2023 to 2026
Sep 11, 2026Raad Bin Tareaf, Murad Al-Rajab, Samia LoucifLLM Evaluation Clinical Language Model Evaluation
XU Exponential University of Applied Sciences, Data Science and Artificial Intelligence Cluster, Marlene-Dietrich-Allee 12B, 14482 Potsdam, Germany · College of Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates · College of Technological Innovation, Zayed University, Abu Dhabi, United Arab Emirates
Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support
Sep 9, 2026Jonathan A. Handler, Marlene I. Robles-Granda, Jacob E. Mefford +5Clinical Decision Support Clinical Decision-Making
Clinical Intelligence and Advanced Data Lab, OSF HealthCare, Peoria, IL · Digital Innovation Development, OSF HealthCare, Peoria, IL · OnCall, OSF HealthCare, Peoria, IL +7
Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics
Sep 8, 2026Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei +6Healthcare Medical Diagnosis
1A.I. Doctor Medical Assist LTD, Limassol, Cyprus. · 2Family Medicine Specialist, Kraków, Poland.
LLM4CKD: Large Language Models for Early Stage Chronic Kidney Disease Screening
Sep 3, 2026Muhammad Ashad Kabir, Sirajam MuniraMedical Diagnosis In-Context Learning
School of Computing, Mathematics and Engineering Charles Sturt University NSW, Australia · Department of Computer Science Rensselaer Polytechnic Institute NY, USA
Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor
Sep 2, 2026Rohith Reddy Bellibaltu, Manpreet Singh, Deepak Parashar +1Counterfactual Evaluation Algorithmic Fairness
Florida International University · Boston University, Boston, MA 02215 USA · Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, India +1
Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA
Sep 1, 2026Nishant Mishra, Ameen Abu-Hanna, Iacer CalixtoClinical Language Model Evaluation Language Model Robustness
Department of Medical Informatics, Amsterdam UMC, University of Amsterdam · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands
ClinTraceBench: Source-Verifiable Longitudinal Clinical Reasoning over EHR-Derived Dialogues
Sep 1, 2026Huimin Wang, Zhengyi Zhao, Yutian ZhaoClinical QA Clinical Reasoning
Shenzhen University · The Chinese University of Hong Kong · Dealism
Toward Workflow-Aware Benchmarking for Healthcare NLP Agents
Aug 31, 2026Junyi Yao, Baichuan Li, Zihao Zheng +1Benchmark Design LLM Agent Evaluation
Washington University in St. Louis, USA · Southern Methodist University, USA
One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
Aug 31, 2026Sebastian Fox, Luke Markham, Ryan Lail +1Medical Report Generation Clinical Language Model Evaluation
Composo
LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
Aug 31, 2026Sebastian Fox, Luke Markham, Ryan Lail +1LLM-as-a-Judge Clinical Language Model Evaluation
Composo
ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography
Aug 31, 2026Mohammadsina Hassannia, Matthew A. Reyna, Reza SameniElectrocardiography LLM Fine-Tuning
Department of Biomedical Informatics Emory University Atlanta, GA 30322, USA · Department of Biomedical Engineering Georgia Institute of Technology and Department of Biomedical Informatics Emory University Atlanta, GA 30332, USA
GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space
Aug 31, 2026Boqi Chen, Xudong Liu, Yunke Ao +2Healthcare Constrained RL
ETH Zurich · MBZUAI · University of Toronto
Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models
Aug 30, 2026Hermione Warr, Harry Anthony, Lilli J Freischem +3LLM Evaluation PET/CT Imaging
Department of Engineering Science, University of Oxford, UK · Department of Physics, University of Oxford, UK · Department of Oncology, University of Oxford, UK +1
Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions
Aug 13, 2026Qingfang Liu, Qiao Jin, Joe D. Menke +2Clinical QA Information Retrieval
National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, MD, USA · National Library of Medicine, National Institutes of Health, Bethesda, MD, USA · School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA
A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench
Aug 12, 2026Praveen Reddy, Charuta Mandke, Suvrankar Datta +4Retrieval-Augmented Generation LLM Grounding
Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMs
Aug 12, 2026Alireza A. Safaei, Laura M. Vowels, Matthew J. Vowels +2Energy-Efficient ML LLM Safety Evaluation
University of Isfahan Isfahan, Iran · University of Roehampton London, UK · Kivira Health New York City, New York, U.S.A.
Locally Deployable Small Language Models for Emergency Department Decision Support: A Systematic Benchmark of Fine-Tuning Strategies
Aug 10, 2026Qingfeng Zhang, Yuanxiong Guo, Yanmin GongLLM Fine-Tuning Clinical Decision Support
The University of Texas at San Antonio, San Antonio, TX · Texas A&M University, Houston, TX
TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent
Aug 10, 2026Waleed Jamil, Raphael SchmittLLM Safety Benchmarks LLM Safety Evaluation
1Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg, Germany
An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer
Aug 10, 2026Mengxian Lyu, Cheng Peng, Tim Jang +18Clinical Decision-Making Domain Adaptation for LLMs
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA. · Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · University of Florida Health Cancer Institute, Gainesville, FL, USA +8
When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
Aug 10, 2026Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam +1Hallucination in Language Models LLM Uncertainty Estimation
School of Computing, Macquarie University, Sydney, NSW 2109, Australia
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Aug 5, 2026Mouxiao Bian, Zhi Chen, Ruiyao Chen +8LLM Safety Benchmarks Healthcare
Shanghai Artificial Intelligence Laboratory, Shanhai, China · Macau University of Science and Technology, Macau, China · First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China +1

Sep 14, 2026, cs.CLJ/K move · Enter open · S save

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

Laura M. Vowels, Matthew J. Vowels, Shivali Sharma, Apoorv Jha+8

School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom+4

LLM Safety Benchmarks Mental Health Counseling Language Model Safety Evaluation LLM Safety Clinical Language Model Evaluation

Open paper AskPDF arXiv

People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk conversations remains poorly characterised. We developed K-Bench, a clinician-calibrated, protected benchmark evaluating 125 model configurations representing 33 base models from 14 providers across a fixed cohort of 200 multi-turn vignettes involving suicide, self-harm, domestic violence, substance misuse, and no-risk presentations. Synthetic patient conversations showed substantial distributional overlap with real human-AI conversations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible item comparisons from 151 clinician-rated transcripts. Leading models combined strong supportive conversation with combined-risk scores above 95, whereas risk exploration exposed substantial variation among lower-performing configurations. Therapeutic prompting produced configuration-specific gains concentrated among weaker models, while elevated reasoning produced no average improvement. K-Bench combines broader clinical coverage and configuration-scale comparison with a continuously updated public leaderboard whose operational test materials are protected from direct optimisation. The leaderboard is available at www.k-bench.ai.

Momentum

Related topics

Latest papers 259