Clinical Language Model Evaluation

Overview

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

CardsList

461 papers56 papers in the last four weeks, up 33% on the four weeks before. 0.4% of all new papers.

  1. PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

    Mar 2, 2026Yimin Zhao, Sheela R. Damle, Simone E. Dekker +13Clinical Language Model EvaluationMedical QA

  2. Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking

    Feb 28, 2026Tianpeng Zheng, Zhehan Jiang, Jiayi Liu +1Computerized Adaptive TestingLLM Evaluation

  3. CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation

    Feb 28, 2026Yannian Gu, Zhongzhen Huang, Linjie Mu +3Clinical Language Model EvaluationMultimodal Model Evaluation

  4. When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation

    Feb 27, 2026Bian Sun, Zhenjian Wang, Orvill de la Torre +1Clinical Language Model EvaluationGenerative AI Evaluation

  5. MediX-R1: Open Ended Medical Reinforcement Learning

    Feb 26, 2026Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Jaseel Muhammad +6RL for VLMsMedical VLMs

  6. Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department

    Feb 26, 2026Gabriela Anna Kaczmarek, Pietro Ferrazzi, Lorenzo Porta +2Clinical NLPClinical Information Extraction

  7. ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making

    Feb 26, 2026Yusuke Watanabe, Yohei Kobashi, Takeshi Kojima +3Clinical Language Model EvaluationLLM Safety Benchmarks

  8. Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages

    Feb 24, 2026Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani +4Clinical Information ExtractionSmall Language Models

  9. Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

    Feb 23, 2026Ian Steenstra, Paola Pedrelli, Weiyan Shi +2LLM Safety EvaluationClinical Language Model Evaluation

  10. TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

    Feb 23, 2026Fangrui Huang, Souhad Chbeir, Arpandeep Khatua +8Mental Health Conversational AgentsClinical Language Model Evaluation

  11. LiveClin: A Live Clinical Benchmark without Leakage

    Feb 18, 2026Xidong Wang, Shuqi Guo, Yue Shen +6Clinical Language Model EvaluationMedical LLMs

  12. Multi-Objective Alignment of Language Models for Personalized Psychotherapy

    Feb 17, 2026Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna +4Personalized Language Model AlignmentMulti-Objective Language Model Alignment

  13. Fine-Tune, Don't Prompt, Your Language Model to Identify Biased Language in Clinical Notes

    Feb 16, 2026Isotta Landi, Eugenia Alleva, Nicole Bussola +5Clinical NLPClinical Language Model Evaluation

  14. Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

    Feb 16, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2Medical QAMedical LLMs

  15. LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

    Feb 2, 2026Rui Hua, Yu Wei, Zixin Shu +17Traditional Chinese MedicineMedical LLMs

  16. SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care

    Jan 23, 2026Dongshen Peng, Yi Wang, Austin Schoeffler +5LLM SycophancyLanguage Model Safety Evaluation

  17. Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation

    Jan 22, 2026Zhiyao Ren, Yibing Zhan, Siyuan Liang +3HealthcareClinical Reasoning

  18. MAPLE: Medical Aspect-Based Summarization with Phrase-Level Evidence

    Jan 6, 2026Bohao Chu, Hendrik Damm, Tabea M. G. Pakull +5HealthcareText Summarization

  19. Context-Aware Classification and Grading of Sensitive Information in Online Conversational Health Data

    Dec 25, 2025Yiwei Yan, Guanfeng LiuHealthcareText Classification

  20. Algorithmic Reflectivity in Large Language Models: Methodological Study of Linguistic Structure in Mentalization

    Nov 20, 2025Stefano Epifani, Giuliano Castigliego, Giuliano Razzicchia +2Self-Reflection in Language Model AgentsTheory of Mind

  21. Knowledge and Reasoning Augmented Learning for Language Model Assisted Antimicrobial Decision Support

    Nov 20, 2025Zhe Li, Yehan Qiu, Yuntian Huang +15Clinical Decision SupportKnowledge Augmentation for Language Models

  22. MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

    Nov 1, 2025Naoto Iwase, Hiroki Okuyama, Junichiro IwasawaMultilingual Language Model EvaluationHealthcare

  23. Zoom In Disparities in Healthcare LLM Q&A

    Oct 20, 2025Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao +5Multilingual Language Model EvaluationKnowledge Augmentation for Language Models