Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 261

All topics

Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
Jun 17, 2026Qingyu Lu, Ruochen Li, Liang Ding +3Clinical Language Model Evaluation Language Model Robustness
Nanyang Technological University · Technical University of Munich · Alibaba +2
Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Jun 17, 2026Tianming Du, Peijie Yu, Sihan Shang +12AI Agent Benchmarks Clinical Language Model Evaluation
1ELLIS Institute Finland · 2Aalto University · 3Tencent +7
Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text
Jun 16, 2026Hongbo Du, Zixin Lu, Jiaming QuLLM Reliability Clinical Language Model Evaluation
Trine University · University of Michigan · Amazon
RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills
Jun 16, 2026Weizhi Zhang, Zechen Li, Hamid Palangi +16AI Agent Evaluation Human-in-the-Loop Evaluation
1Google Research, 2University of Illinois Chicago, *Work done during an internship at Google, †Corresponding Author
Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour
Jun 16, 2026Abeer Badawi, Moyosoreoluwa Olatosi, Negin Baghbanzadeh +5Emotional Support Conversation Mental Health
York University, Canada · Vector Institute, Canada · Dalhousie University, Canada +3
Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews
Jun 16, 2026Franziska Braun, Alea Rüggeberg, Thomas Ranzenberger +4Depression Detection Mental Health
1TH Nürnberg · 2FAU Erlangen · 3PMU Klinikum Nürnberg, Germany
AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
Jun 16, 2026Jiahui Niu, Huizi Yu, Wenkong Wang +11Clinical Reasoning Clinical Language Model Evaluation
School of Control Science and Engineering, Shandong University, Ji’nan, Shandong, China · Key Laboratory of Machine Intelligence and System Control, Shandong University, Ji’nan, Shandong, China · Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China +5
Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering
Jun 15, 2026Sanjay BasuMulti-Hop QA Compositional Reasoning
Department of Medicine, University of California San Francisco, San Francisco, CA · Waymark, San Francisco, CA
EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries
Jun 14, 2026Jiyoun Kim, Muhan Yeo, Eunhye Jang +14Clinical QA Evidence-Grounded Generation
KAIST · Seoul National University · Seoul National University Bundang Hospital +9
Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering
Jun 13, 2026Zaifu Zhan, Shuang Zhou, Rui ZhangMulti-Agent LLM Systems LLM Answer Verification
Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States
Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System
Jun 10, 2026Alyssa Unell, Miguel Fuentes, Brenna Li +4LLM Guardrails Clinical Language Model Evaluation
Department of Computer Science Stanford University Stanford, CA 95305 · Department of Medicine Stanford University Stanford, CA 95305 · Department of Biomedical Data Science Stanford University Stanford, CA 95305
Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
Jun 10, 2026Hongjian Zhou, Xinyu Zou, Jinge Wu +19LLM Reliability Medical QA
University of Oxford · University of Washington · University College London +1
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Jun 10, 2026Guido Freire, Agustín Martínez-Suñé, Viviana CotikHealthcare AI Control
Universidad de Buenos Aires, Facultad de Ciencias Exactas y Naturales, Departamento de Computación, Argentina. · AI Safety Argentina (AISAR). · Department of Computer Science, University of Oxford, United Kingdom. +1
CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries
Jun 8, 2026Vasudha Varadarajan, Akhila Yerukola, Mona T. Diab +1LLM Evaluation Cultural Bias in Language Models
Carnegie Mellon University Pittsburgh, PA 15213, USA
Clinically Grounded Privacy Evaluation of Medical LMs
Jun 8, 2026Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle +6Data Leakage Memorization in Language Models
1Stanford University · 2Massachusetts Institute of Technology · 3American Board of Family Medicine
SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models
Jun 6, 2026Ayah Al-Naji, Edoardo Fazzari, Saif Alkindi +3LLM Evaluation Synthetic Benchmark Generation
Mohamed bin Zayed University of Artificial Intelligence, UAE
From 'May' to 'Is': Certainty Distortion in Language Model Rewriting
Jun 6, 2026Catarina G Belem, Shang Wu, Hongyu Yao +3Scientific Communication Language Model Calibration
Department of Computer Science, University of California Irvine · Massachusetts Institute of Technology · Department of Cognitive Science, University of California Irvine
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
Jun 6, 2026Yuan Shen, Xiaojun Wu, Linghua YuLLM Evaluation Language Model Safety Evaluation
College of Computer Science and Technology, Zhejiang University, PR China · The First Hospital of Jiaxing, Zhejiang Province, PR China
Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
Jun 5, 2026Giordano de Pinho Souza, Glaucia Melo, Josefino Cabral Melo Lima +1Multilingual Language Model Evaluation Clinical Decision-Making
Federal University of Rio de Janeiro · Toronto Metropolitan University Toronto, CA
When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
Jun 5, 2026Mahdi AlkaeedPrompt Sensitivity Adversarial Attacks on LLMs
Department of Computer Science and Engineering, Doha, Qatar
Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries
Jun 4, 2026Martin MurinPrompt Sensitivity Clinical Information Extraction
DryLabz GmbH, Basel, Switzerland
Beyond WER: A Paired Acoustic Stress Test for Ambient Clinical Scribes
Jun 4, 2026Xiao-Hang Jiang, Han-Jie Guo, Ying-Si Liang +4ASR Evaluation Language Model Safety Evaluation
University of Science and Technology of China, China · iFLYTEK Co., Ltd., China
PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage
Jun 3, 2026Keqi Han, Ryan Young, Annabel Strauss +7Healthcare AI Agent Benchmarks
1Emory University · 2Scale AI · 3Mayo Clinic +1
Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison
Jun 3, 2026Alejandro Lozano, Keiko Ihara, Ping-Hao Yang +13Text Summarization RAG Evaluation
Stanford University, Palo Alto, CA, USA · Department of Neurology, Mayo Clinic, Rochester, MN, USA · Department of Neurology, Dalhousie University, Halifax, Canada +8
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Jun 3, 2026Cheng Liang, Pengcheng Qiu, Ya Zhang +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
When Clients Stop Following: A Cognitive Conceptualization Diagram-driven Framework for Strategic Counseling
Jun 3, 2026Yihao Qin, Junyi Zhao, Changsheng Ma +4Mental Health Counseling Cognitive Modeling
School of Information Science and Engineering, Lanzhou University
Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same Symptoms, Unequal Urgency
Jun 2, 2026Qi Han WongGender Bias in Language Models Clinical Triage
AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making
Jun 2, 2026Sangwon Baek, Kyu Yeon Hur, Kyunga KimLLM-as-a-Judge Clinical Decision Support
Asclep Korea Inc., Changwon, Republic of Korea · Center for Data Science, New York University, New York, USA · Division of Endocrinology and Metabolism, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea +1
ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models
Jun 2, 2026Ruihui Hou, Siyi Zhu, Ziyue Huai +4Clinical Decision-Making Clinical Language Model Evaluation
East China University of Science and Technology, Shanghai, China · Renji Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.

Jun 17, 2026, cs.CLJ/K move · Enter open · S save

Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

Qingyu Lu, Ruochen Li, Liang Ding, Yufei Xia+2

Nanyang Technological University · Technical University of Munich · Alibaba+2

Clinical Language Model Evaluation Language Model Robustness Medical LLMs Radiology Report Evaluation

Open paper AskPDF arXiv

Reliable evaluation of generated radiology reports requires strict clinical accuracy, as omitted critical findings or mischaracterized radiographic observations can directly affect patient care. Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar. Although Large Language Models (LLMs) possess rich medical knowledge, they likewise struggle to draw a reliable boundary between clinically significant errors and harmless variation. We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance from detecting true clinical errors ("Discrimination") and tolerating insignificant variations ("Robustness"). Across 8 LLM evaluators under one-pass and two-pass settings, we identify a widespread discrimination bias: models effectively detect errors but also over-penalize harmless rephrasings. To mitigate this, we synthesize 4k report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B. Our trained metric sharpens the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models. Crucially, the more costly two-pass setting fails to consistently improve overall performance and mainly trades discrimination for robustness. These findings suggest one-pass trained metrics as the practical choice for cost-sensitive deployment, with two-pass inference reserved for settings where D-R balance is critical. We will release the dataset and metric.

Momentum

Related topics

Latest papers 261