Clinical Language Model Evaluation

Overview

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

CardsList

461 papers56 papers in the last four weeks, up 33% on the four weeks before. 0.4% of all new papers.

  1. Computational Phenomenology of Borderline Personality Disorder: A Comparative Evaluation of LLM-Simulated Expert Personas and Human Clinical Experts

    Aug 26, 2025Marcin Moskalewicz, Anna Sterna, Karolina Drożdż +3Clinical Language Model EvaluationMental Health

  2. "Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated

    Aug 7, 2025Tong Li, Rasiq Hussain, Mehak Gupta +1Depression DetectionMental Health

  3. Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs

    Aug 5, 2025Shintaro Sakai, Jisun An, Migyeong Kang +1Mental HealthCultural Bias in Language Models

  4. Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications

    Aug 1, 2025Zizhan Ma, Wenxuan Wang, Meidan Ding +7LLM PromptingRL for Language Model Reasoning

  5. PRiME: Pragmatic Reasoning In MEntal health

    Jul 31, 2025Sneha Oram, Tejomay Kishor Padole, Pushpak Bhattacharyya +1Language Model Reasoning EvaluationPragmatic Reasoning in Language Models

  6. HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

    Jul 29, 2025Kaixuan Wang, Chenxin Diao, Jason T. Jacques +2HealthcareQuestion Answering

  7. Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues

    Jun 28, 2025Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto +5Mental Health CounselingLLM-as-a-Judge

  8. Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling

    May 21, 2025He Hu, Juzheng Si, Qianning Wang +5Mental Health CounselingStructured Reasoning in Language Models

  9. Open Foundation Models in Healthcare: Challenges, Paradoxes, and Opportunities with GenAI Driven Personalized Prescription

    Feb 4, 2025Mahdi Alkaeed, Sofiat Abioye, Adnan Qayyum +6Medical LLMsOpen-Weight Language Models

  10. Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges

    Sep 25, 2024Konstantinos Skianis, John Pavlopoulos, A. Seza DoğruözMultilingual Language Model EvaluationClinical Language Model Evaluation

  11. Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

    Sep 3, 2024Zainab Iftikhar, Sean Ransom, Amy Xiao +2Mental Health CounselingMental Health Conversational Agents