Clinical Language Model Evaluation

Ask about this topic

Overview

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

CardsList

419 papers49 papers in the last four weeks, up 2% on the four weeks before. 0.3% of all new papers.

  1. Decomposing Physician Disagreement in HealthBench

    Feb 26, 2026Satya Borgohain, Roy MariathasAnnotator DisagreementClinical Language Model Evaluation

  2. Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs

    Feb 25, 2026Anqi Li, Ruihan Wang, Zhaoming Chen +5Mental Health CounselingClinical Language Model Evaluation

    Zhejiang University · Westlake University, Hangzhou, China

  3. Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages

    Feb 24, 2026Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani +4Clinical Information ExtractionSmall Language Models

    Student Research Committee, Isfahan University of Medical Sciences, Isfahan, Iran · Department of Emergency Medicine, Isfahan University of Medical Sciences, Isfahan, Iran

  4. Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

    Feb 23, 2026Ian Steenstra, Paola Pedrelli, Weiyan Shi +2LLM Safety EvaluationClinical Language Model Evaluation

    Northeastern University Boston, MA, USA · Harvard Medical School Boston, MA, USA

  5. TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

    Feb 23, 2026Fangrui Huang, Souhad Chbeir, Arpandeep Khatua +8Mental Health Conversational AgentsClinical Language Model Evaluation

    Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA · The University of Hong Kong, Hong Kong +2

  6. LiveClin: A Live Clinical Benchmark without Leakage

    Feb 18, 2026Xidong Wang, Shuqi Guo, Yue Shen +6Clinical Language Model EvaluationMedical LLMs

    The Chinese University of Hong Kong, ShenZhen · Ant Healthcare, Ant Group · The Ohio State University +1

  7. Multi-Objective Alignment of Language Models for Personalized Psychotherapy

    Feb 17, 2026Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna +4Personalized Language Model AlignmentMulti-Objective Language Model Alignment

    Department of Computer Science, University of California, Los Angeles, CA, USA · Department of Psychology, The University of Texas at Austin, Austin, TX, USA · Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA

  8. Evidence-Grounded Subspecialty Reasoning: Evaluating a Curated Clinical Intelligence Layer on the 2025 Endocrinology Board-Style Examination

    Feb 17, 2026Amir Hosseinian, MohammadReza Zare Shahneh, Umer Mansoor +3Clinical Language Model EvaluationEvidence-Grounded Reasoning

    January AI · Stanford University

  9. Fine-Tune, Don't Prompt, Your Language Model to Identify Biased Language in Clinical Notes

    Feb 16, 2026Isotta Landi, Eugenia Alleva, Nicole Bussola +5Clinical NLPClinical Language Model Evaluation

    Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, NY. · Icahn School of Medicine at Mount Sinai, NY. · Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, NY. +2

  10. Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

    Feb 16, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2Medical QAMedical LLMs

    Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh

  11. LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

    Feb 2, 2026Rui Hua, Yu Wei, Zixin Shu +17Traditional Chinese MedicineMedical LLMs

    Department of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd., Tianjin, China · Tasly Biopharmaceuticals Co., Ltd., Tianjin, China +11

  12. Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

    Jan 25, 2026Zhihao Zhang, Liting Huang, Guanghao Wu +3LLM Safety BenchmarksLanguage Model Safety Evaluation

    Macquarie University1 · UTS2 · MBZUAI3 +1

  13. SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care

    Jan 23, 2026Dongshen Peng, Yi Wang, Austin Schoeffler +5LLM SycophancyLanguage Model Safety Evaluation

    UNC Chapel Hill · University of Waterloo · Stanford University

  14. Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation

    Jan 22, 2026Zhiyao Ren, Yibing Zhan, Siyuan Liang +3HealthcareClinical Reasoning

    Nanyang Technological University, Singapore, Singapore. · Wuhan University, Wuhan, China.

  15. MAPLE: Medical Aspect-Based Summarization with Phrase-Level Evidence

    Jan 6, 2026Bohao Chu, Hendrik Damm, Tabea M. G. Pakull +5HealthcareText Summarization

    University of Duisburg-Essen · University of Applied Sciences and Arts Dortmund · University Hospital Essen +1

  16. Context-Aware Classification and Grading of Sensitive Information in Online Conversational Health Data

    Dec 25, 2025Yiwei Yan, Guanfeng LiuHealthcareText Classification

    School of Computing, Macquarie University, Australia

  17. First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultations

    Dec 1, 2025David Wu, Fateme Nateghi Haredasht, Saloni Kumar Maharaj +54HealthcareClinical Decision Support

  18. Algorithmic Reflectivity in Large Language Models: Methodological Study of Linguistic Structure in Mentalization

    Nov 20, 2025Stefano Epifani, Giuliano Castigliego, Giuliano Razzicchia +2Self-Reflection in Language Model AgentsTheory of Mind

    Digital Transformation Institute, Rome, Italy · Department of Communication and Social Research, Sapienza University of Rome, Italy · Psychoanalytic Academy of Italian-Speaking Switzerland, Chur, Switzerland +1

  19. MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

    Nov 1, 2025Naoto Iwase, Hiroki Okuyama, Junichiro IwasawaMultilingual Language Model EvaluationHealthcare

    Preferred Networks, Inc., Tokyo, Japan · School of Medicine, Nagoya University, Nagoya, Japan

  20. Zoom In Disparities in Healthcare LLM Q&A

    Oct 20, 2025Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao +5Multilingual Language Model EvaluationKnowledge Augmentation for Language Models

    Universitat Politècnica de València, Valencia, Spain · University of Trento, Trento, Italy · University of Sheffield, Sheffield, UK +5

  21. "Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated

    Aug 7, 2025Tong Li, Rasiq Hussain, Mehak Gupta +1Depression DetectionMental Health

    Division of Computational & Data Sciences (DCDS), Washington University in St. Louis, St. Louis, MO · Department of Computer Science, SMU · Department of Psychological & Brain Sciences, Washington University in St. Louis

  22. Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs

    Aug 5, 2025Shintaro Sakai, Jisun An, Migyeong Kang +1Mental HealthCultural Bias in Language Models

    Indiana University Bloomington, USA · Sungkyunkwan University, Republic of Korea

  23. Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications

    Aug 1, 2025Zizhan Ma, Wenxuan Wang, Meidan Ding +7LLM PromptingRL for Language Model Reasoning

    The Chinese University of Hong Kong · Renmin University of China · Shenzhen University +3

  24. PRiME: Pragmatic Reasoning In MEntal health

    Jul 31, 2025Sneha Oram, Tejomay Kishor Padole, Pushpak Bhattacharyya +1Language Model Reasoning EvaluationPragmatic Reasoning in Language Models

    Indian Institute of Technology, Bombay

  25. HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

    Jul 29, 2025Kaixuan Wang, Chenxin Diao, Jason T. Jacques +2HealthcareQuestion Answering

    University of Oxford, United Kingdom · University of Edinburgh, United Kingdom · University of St Andrews, United Kingdom +1

  26. Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues

    Jun 28, 2025Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto +5Mental Health CounselingLLM-as-a-Judge

    Japan National Institute of Occupational Safety and Health, Kanagawa, Japan · Kaze To Taiyo, Tokyo, Japan · Saga Occupational Health Association, Saga, Japan +5

  27. Open Foundation Models in Healthcare: Challenges, Paradoxes, and Opportunities with GenAI Driven Personalized Prescription

    Feb 4, 2025Mahdi Alkaeed, Sofiat Abioye, Adnan Qayyum +6Medical LLMsOpen-Weight Language Models

    Department of Computer Science and Engineering, College of Engineering, Qatar University, Doha, Qatar · Birmingham City University, Birmingham, United Kingdom · College of Science and Engineering, Hamad Bin Khalifa University (HBKU), Doha, Qatar +2

  28. Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges

    Sep 25, 2024Konstantinos Skianis, John Pavlopoulos, A. Seza DoğruözMultilingual Language Model EvaluationClinical Language Model Evaluation

    Department of Computer Science and Engineering, University of Ioannina, Greece · Department of Informatics, Athens University of Economics and Business, Greece · Archimedes/AthenaRC, Greece +1

  29. Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

    Sep 3, 2024Zainab Iftikhar, Sean Ransom, Amy Xiao +2Mental Health CounselingMental Health Conversational Agents

    Brown University, USA · Department of Psychiatry, Louisiana State University Health Sciences, USA · Alpert Medical School of Brown University, Brown University, USA