Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text
Jun 16, 2026Hongbo Du, Zixin Lu, Jiaming QuLLM Reliability Clinical Language Model Evaluation
Trine University · University of Michigan · Amazon
RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills
Jun 16, 2026Weizhi Zhang, Zechen Li, Hamid Palangi +16AI Agent Evaluation Human-in-the-Loop Evaluation
1Google Research, 2University of Illinois Chicago, *Work done during an internship at Google, †Corresponding Author
Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour
Jun 16, 2026Abeer Badawi, Moyosoreoluwa Olatosi, Negin Baghbanzadeh +5Emotional Support Conversation Mental Health
York University, Canada · Vector Institute, Canada · Dalhousie University, Canada +3
Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews
Jun 16, 2026Franziska Braun, Alea Rüggeberg, Thomas Ranzenberger +4Depression Detection Mental Health
1TH Nürnberg · 2FAU Erlangen · 3PMU Klinikum Nürnberg, Germany
AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
Jun 16, 2026Jiahui Niu, Huizi Yu, Wenkong Wang +11Clinical Reasoning Clinical Language Model Evaluation
School of Control Science and Engineering, Shandong University, Ji’nan, Shandong, China · Key Laboratory of Machine Intelligence and System Control, Shandong University, Ji’nan, Shandong, China · Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China +5
Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering
Jun 15, 2026Sanjay BasuMulti-Hop QA Compositional Reasoning
Department of Medicine, University of California San Francisco, San Francisco, CA · Waymark, San Francisco, CA
EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries
Jun 14, 2026Jiyoun Kim, Muhan Yeo, Eunhye Jang +14Clinical QA Evidence-Grounded Generation
KAIST · Seoul National University · Seoul National University Bundang Hospital +9
Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering
Jun 13, 2026Zaifu Zhan, Shuang Zhou, Rui ZhangMulti-Agent LLM Systems LLM Answer Verification
Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States
Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System
Jun 10, 2026Alyssa Unell, Miguel Fuentes, Brenna Li +4LLM Guardrails Clinical Language Model Evaluation
Department of Computer Science Stanford University Stanford, CA 95305 · Department of Medicine Stanford University Stanford, CA 95305 · Department of Biomedical Data Science Stanford University Stanford, CA 95305
Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
Jun 10, 2026Hongjian Zhou, Xinyu Zou, Jinge Wu +19LLM Reliability Medical QA
University of Oxford · University of Washington · University College London +1
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Jun 10, 2026Guido Freire, Agustín Martínez-Suñé, Viviana CotikHealthcare AI Control
Universidad de Buenos Aires, Facultad de Ciencias Exactas y Naturales, Departamento de Computación, Argentina. · AI Safety Argentina (AISAR). · Department of Computer Science, University of Oxford, United Kingdom. +1
CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries
Jun 8, 2026Vasudha Varadarajan, Akhila Yerukola, Mona T. Diab +1LLM Evaluation Cultural Bias in Language Models
Carnegie Mellon University Pittsburgh, PA 15213, USA
Clinically Grounded Privacy Evaluation of Medical LMs
Jun 8, 2026Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle +6Data Leakage Memorization in Language Models
1Stanford University · 2Massachusetts Institute of Technology · 3American Board of Family Medicine
SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models
Jun 6, 2026Ayah Al-Naji, Edoardo Fazzari, Saif Alkindi +3LLM Evaluation Synthetic Benchmark Generation
Mohamed bin Zayed University of Artificial Intelligence, UAE
From 'May' to 'Is': Certainty Distortion in Language Model Rewriting
Jun 6, 2026Catarina G Belem, Shang Wu, Hongyu Yao +3Scientific Communication Language Model Calibration
Department of Computer Science, University of California Irvine · Massachusetts Institute of Technology · Department of Cognitive Science, University of California Irvine
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
Jun 6, 2026Yuan Shen, Xiaojun Wu, Linghua YuLLM Evaluation Language Model Safety Evaluation
College of Computer Science and Technology, Zhejiang University, PR China · The First Hospital of Jiaxing, Zhejiang Province, PR China
Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
Jun 5, 2026Giordano de Pinho Souza, Glaucia Melo, Josefino Cabral Melo Lima +1Multilingual Language Model Evaluation Clinical Decision-Making
Federal University of Rio de Janeiro · Toronto Metropolitan University Toronto, CA
When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
Jun 5, 2026Mahdi AlkaeedPrompt Sensitivity Adversarial Attacks on LLMs
Department of Computer Science and Engineering, Doha, Qatar
Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries
Jun 4, 2026Martin MurinPrompt Sensitivity Clinical Information Extraction
DryLabz GmbH, Basel, Switzerland
Beyond WER: A Paired Acoustic Stress Test for Ambient Clinical Scribes
Jun 4, 2026Xiao-Hang Jiang, Han-Jie Guo, Ying-Si Liang +4ASR Evaluation Language Model Safety Evaluation
University of Science and Technology of China, China · iFLYTEK Co., Ltd., China
PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage
Jun 3, 2026Keqi Han, Ryan Young, Annabel Strauss +7Healthcare AI Agent Benchmarks
1Emory University · 2Scale AI · 3Mayo Clinic +1
Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison
Jun 3, 2026Alejandro Lozano, Keiko Ihara, Ping-Hao Yang +13Text Summarization RAG Evaluation
Stanford University, Palo Alto, CA, USA · Department of Neurology, Mayo Clinic, Rochester, MN, USA · Department of Neurology, Dalhousie University, Halifax, Canada +8
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Jun 3, 2026Cheng Liang, Pengcheng Qiu, Ya Zhang +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
When Clients Stop Following: A Cognitive Conceptualization Diagram-driven Framework for Strategic Counseling
Jun 3, 2026Yihao Qin, Junyi Zhao, Changsheng Ma +4Mental Health Counseling Cognitive Modeling
School of Information Science and Engineering, Lanzhou University
Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same Symptoms, Unequal Urgency
Jun 2, 2026Qi Han WongGender Bias in Language Models Clinical Triage
AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making
Jun 2, 2026Sangwon Baek, Kyu Yeon Hur, Kyunga KimLLM-as-a-Judge Clinical Decision Support
Asclep Korea Inc., Changwon, Republic of Korea · Center for Data Science, New York University, New York, USA · Division of Endocrinology and Metabolism, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea +1
ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models
Jun 2, 2026Ruihui Hou, Siyi Zhu, Ziyue Huai +4Clinical Decision-Making Clinical Language Model Evaluation
East China University of Science and Technology, Shanghai, China · Renji Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.
Food Noise & False Safety: A Systematic Evaluation of How LLMs Fail to Adapt to Eating Disorder Queries with Clinician Feedback
Jun 1, 2026Giulia Pucci, Emily Hemendinger, Ruizhe Li +3Mental Health LLM Safety
University of Aberdeen · University of Colorado Anschutz · Heriot-Watt University +1
KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts
Jun 1, 2026Christian Autenried, Cosimo PersiaLanguage Model Pretraining Domain-Adaptive Pretraining
Helse Vest ICT

Jun 16, 2026, cs.CLJ/K move · Enter open · S save

Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

Hongbo Du, Zixin Lu, Jiaming Qu

Trine University · University of Michigan · Amazon

LLM Reliability Clinical Language Model Evaluation Medical LLMs

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for clinical text tasks such as summarization and revision. While most studies evaluate the fluency and coherence of LLM-generated text, whether LLMs correctly preserve diagnostic uncertainty remains underexplored. In clinical practice, phrases such as ``possible pneumonia'' communicate the strength of available evidence and directly guide decisions about follow-up testing and treatment. Altering these uncertainty expressions can change the clinical meaning entirely. In this paper, we systematically evaluated this problem in two steps. First, we constructed a benchmark of 1,200 clinical documents with 9,184 uncertainty annotations across five levels. Second, we evaluated three LLMs on this benchmark. Our results show that (1) LLMs preserve the original uncertainty cues poorly, often less than half the time; (2) LLMs struggle with nuanced distinctions between adjacent levels. This work reveals a failure mode not captured by standard evaluation metrics and provides implications for the safe deployment of LLMs in clinical workflows.

Momentum

Related topics

Latest papers 259