Clinical Language Model Evaluation

Ask about this topic

Overview

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

CardsList

419 papers49 papers in the last four weeks, up 2% on the four weeks before. 0.3% of all new papers.

  1. IatroBench: A Pre-Registered Benchmark of Clinical Omission in Language Models

    Apr 9, 2026David GringrasLLM Safety BenchmarksHealthcare

    Harvard T.H. Chan School of Public Health

  2. To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models

    Apr 8, 2026Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela +2Medical LLMsDomain Adaptation for LLMs

  3. MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

    Apr 8, 2026Xiaotian Luo, Xun Jiang, Jiangcheng WuAdversarial Robustness of Language ModelsMedical LLMs

    Shanda Group

  4. Multilingual Cognitive Impairment Detection in the Era of Foundation Models

    Apr 8, 2026Damar Hoogland, Boshko Koloski, Jaya Caporusso +5Speech-Based Dementia DetectionClinical Language Model Evaluation

    Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · Institute of Synergetics and Psychotherapy Research, Paracelsus Medical University, Salzburg, Austria +2

  5. FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts

    Apr 7, 2026Sylvia Vassileva, Ivan Koychev, Svetla BoytchevaClinical Information ExtractionNamed Entity Recognition

    Faculty of Mathematics and Informatics, Sofia University St. Kliment Ohridski, Sofia, Bulgaria · Graphwise, Sofia, Bulgaria

  6. A Multi-Stage Validation Framework for Trustworthy Large-scale Clinical Information Extraction using Large Language Models

    Apr 7, 2026Maria Mahbub, Gregory M. Dams, Josh Arnold +9Clinical Information ExtractionClinical Language Model Evaluation

    Oak Ridge National Laboratory, Oak Ridge, TN, USA · Program Evaluation and Resource Center, Office of Mental Health and Office of Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA · Vanderbilt University Medical Center, Nashville, TN, USA +4

  7. Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health Conversation

    Apr 7, 2026Abdullah Mazhar, Het Riteshkumar Shah, Aseem Srivastava +2Clinical Language Model EvaluationLLM Evaluation

    IIIT Delhi · MBZUAI · Wysa

  8. CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

    Apr 7, 2026Akshat Dasula, Prasanna Desikan, Jaideep Srivastava +2LLM AuditingClinical Language Model Evaluation

    Centific AI Research, Redmond, USA · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA

  9. Retrieval-Augmented LLMs for Evidence Localization in Clinical Trial Recruitment from Longitudinal EHR Narratives

    Apr 6, 2026Ziyi Chen, Mengxian Lyu, Cheng Peng +1Medical LLMsClinical Language Model Evaluation

    Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL

  10. This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA

    Apr 6, 2026Hye Sun Yun, Geetika Kapoor, Michael Mackert +4Framing Effects in Language ModelsMedical LLMs

    Northeastern University · UC Berkeley · UT Austin +2

  11. Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction

    Apr 6, 2026Jinrui Fang, Runhan Chen, Xu Yang +9Clinical Language Model EvaluationMedical Diagnosis

    The University of Texas at Austin · New York University · The University of Texas Southwestern Medical Center +2

  12. MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

    Apr 5, 2026Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra +1Clinical Information ExtractionClinical Language Model Evaluation

    Optum AI

  13. A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework

    Apr 3, 2026Chenyu Li, Zohaib Akhtar, Mingu Kwak +13LLM-as-a-JudgeClinical Language Model Evaluation

    Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA · Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, Pittsburgh, PA, USA · OpenCura, Health Innovation Consortium +6

  14. Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients

    Apr 2, 2026Oumaima El Khettari, Virgile Barthet, Guillaume Hocquet +3Mortality PredictionMultimodal Fusion

    Nantes Univ., École Centrale Nantes, CNRS, LS2N, UMR 6004, 44000 Nantes, France · Université Paris-Saclay, CNRS, LISN, Orsay, France · Hôpital Saint Joseph, DIMID, et Service de Cardiologie, Paris, France

  15. Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study

    Mar 31, 2026Zaifu Zhan, Mengyuan Cui, Rui ZhangLLM Self-CorrectionLLM Reliability

    Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, USA · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, USA · Department of Software Engineering, University of St. Thomas, Saint Paul,USA

  16. Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations

    Mar 31, 2026Yahan Li, Xinyi Jie, Wanjia Ruan +5HealthcareClinical Reasoning

    University of Southern California

  17. AI-Generated Prior Authorization Letters: Strong Clinical Content, Weak Administrative Scaffolding

    Mar 31, 2026Moiz Sadiq Awan, Maryam RazaMedical LLMsMedical Report Generation

    Independent Researcher

  18. Development and Preliminary Evaluation of a Domain-Specific Large Language Model for Tuberculosis Care in South Africa

    Mar 28, 2026Thokozile Khosa, Olawande DaramolaMedical LLMsHealthcare

    Department of Computer Science, University of Pretoria, South Africa · Department of Informatics, University of Pretoria, South Africa

  19. Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs

    Mar 27, 2026Vinicius Anjos de Almeida, Sandro Saorin da Silva, Josimar Chire +8Named Entity RecognitionClinical NLP

    Spesia, Curitiba - PR, Brazil · Faculdade de Medicina, Universidade de São Paulo, São Paulo - SP, Brazil · Pontifícia Universidade Católica do Paraná (PUCPR), Curitiba - PR, Brazil +2

  20. Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

    Mar 27, 2026Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten +2EHR QAClinical QA

    Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.

  21. When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

    Mar 26, 2026Binesh Sadanandan, Vahid BehzadanPrompt SensitivityLLM Prompting

    SAIL Lab, University of New Haven, West Haven, CT 06516, USA

  22. Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness

    Mar 26, 2026Alexandra DeLucia, Heyuan Huang, Sonal Joshi +3LLM-as-a-JudgeLLM Reliability

    Data Science and AI Institute, Johns Hopkins University

  23. Doctorina MedBench: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI

    Mar 26, 2026Anna Kozlova, Stanislau Salavei, Pavel Satalkin +3Medical AI AgentsAI Agent Benchmarks

    A.I. Doctor Medical Assist LTD, 13 Myrtiotissis Street, Aqua Mansions, Office 1, Germasogeia, 4041 Limassol, Cyprus

  24. A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations

    Mar 26, 2026Andong Tan, Shuyu Dai, Jinglu Wang +7Clinical Language Model EvaluationMedical LLMs

    Media Computing Group, Microsoft Research Asia, Beijing, China · Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science, Peking University, Beijing, China +6

  25. OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs

    Mar 26, 2026Suraj Racha, Prashant Harish Joshi, Utkarsh Maurya +6Medical LLMsLLM Fine-Tuning

    Indian Institute of Technology Bombay

  26. Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset

    Mar 25, 2026Mohammed Nowshad Ruhani Chowdhury, Mohammed Nowaz Rabbani Chowdhury, Sakari LukkarinenLLM Fine-TuningMedical LLMs

    School of ICT and Industrial Management, Metropolia University of Applied Sciences, Helsinki, Finland · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA

  27. Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis

    Mar 20, 2026May Lynn Reese, Markela Zeneli, Mindy Ng +3LLM SafetyLLM Safety Evaluation

    Apart Research · Odyssean Institute · London School of Economics and Political Science +1

  28. The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition

    Mar 18, 2026Alvin Rajkomar, Pavan Sudarshan, Angela Lai +1Benchmark ValidityClinical Language Model Evaluation

    Apple

  29. Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

    Mar 17, 2026Khizar Hussain, Bradley A. Malin, Zhijun Yin +2Language Model Error DetectionHallucination Detection

    Virginia Tech, Blacksburg, VA, USA · Vanderbilt University Medical Center, Nashville, TN, USA

  30. Medical Reasoning with Large Language Models: A Survey and MR-Bench

    Mar 17, 2026Xiaohan Ren, Chenxiao Fan, Wenyin Ma +4Medical ReasoningClinical Reasoning

    School of Cyber Science and Technology, University of Science and Technology of China, Anhui, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Anhui, China · Department of Infectious Disease, The First Affiliated Hospital of USTC, Anhui, China +2