Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models
May 15, 2026Weixin Liu, Congning Ni, Shelagh A. Mulvaney +4LLM Evaluation Healthcare
Vanderbilt University, Nashville, TN, USA · Vanderbilt University Medical Center, Nashville, TN, USA · Virginia Tech, Blacksburg, VA, USA
Quantifying and Mitigating Premature Closure in Frontier LLMs
May 14, 2026Rebecca Handler, Suhana Bedi, Nigam ShahHealthcare LLM Prompting
Department of Medicine, Stanford University · Department of Biomedical Data Science, Stanford University
RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation
May 14, 2026Shuhao Chen, Weisen Jiang, Changmiao Wang +4Healthcare Clinical Decision Support
The Hong Kong University of Science and Technology · Southern University of Science and Technology · The Chinese University of Hong Kong +2
Patients With Personality: Realistic Patient Simulation through Controlled Diversity and Selective Disclosure
May 13, 2026Moritz Schlager, Friederike Jungmann, Samuel Schmidgall +12Personality Modeling in Language Models Persona-Conditioned Generation
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · TUM University Hospital +3
RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation
May 13, 2026Chengzhi Shen, Weixiang Shen, Tobias Susetzky +8Healthcare Clinical Decision-Making
Cherise · Technical University of Munich (TUM) · TUM University Hospital +8
VERA-MH: Validation of Ethical and Responsible AI in Mental Health
May 13, 2026Luca Belli, Kate H. Bentley, Josh Gieringer +6Healthcare Mental Health
Spring Health · Yale University
Large Language Models Lack Temporal Awareness of Medical Knowledge
May 13, 2026Zihan Guan, Qiao Jin, Guangzhi Xiong +6Healthcare Temporal Reasoning in Language Models
University of Virginia · National Institutes of Health · Dana-Farber Cancer Institute +2
Simulating Eating Disorder Patients with LLMs: Evaluating Psychological Persona Stability in Multi-Turn Conversations
May 12, 2026Jennifer Haase, Jana Gonnermann-Müller, See Heng Yim +3Persona Consistency Personality Modeling in Language Models
Weizenbaum Institute, Berlin, Germany · HU Berlin, Berlin, Germany · Zuse Institute Berlin, Berlin, Germany +1
AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment
May 12, 2026Robin Linzmayer, Georgianna Lin, Di Coneybeare +22Healthcare Clinical Triage
Department of Computer Science, Columbia University · Department of Biomedical Informatics, Columbia University · Department of Computer Science, Columbia University, New York, NY, USA +3
CPEMH: An Agentic Framework for Prompt-Driven Behavior Evaluation and Assurance in Foundation-Model Systems for Mental Health Screening
May 11, 2026Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar +1LLM Auditing Prompt Optimization
University of Waterloo · University of Waterloo Waterloo, ON, Canada
Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
May 11, 2026Jiaqing Zhang, Sandeep Elluri, Bhanu Cherukuvada +7LLM Auditing Multimodal Large Language Models
Department of Electrical & Computer Engineering · Department of Computer and Information Science and Engineering · Department of Clinical and Health Psychology +1
MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies
May 10, 2026Huy Hoang Ha, Benoit Favre, Francois PortetHealthcare RAG Evaluation
Can We Trust LLMs for Mental Health Screening? Consistency, ASR Robustness, and Evidence Faithfulness
May 10, 2026Erfan Loweimi, Sofia de la Fuente Garcia, Samira Loveymi +2Healthcare Mental Health
Usher Institute, Edinburgh Medical School, University of Edinburgh, Edinburgh, UK · Department of Computer Engineering, Ahvaz Campus, Islamic Azad University, Ahvaz, Iran · School of Health and Social Care, Edinburgh Napier University, Edinburgh, UK
CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics
May 10, 2026Aishik Nagar, Arun-Kumar Kaliya-Perumal, Yu-Hsuan Han +5Healthcare Clinical Decision Support
ASUS Intelligent Cloud Services (AICS) · Rehabilitation Research Institute of Singapore, Nanyang Technological University · Department of Family Medicine,2026 Taipei Veterans General Hospital +2
Can Revealed Preferences Clarify LLM Alignment and Steering?
May 8, 2026Khurram Yamin, Jingjing Tang, Eric Horvitz +1Language Model Steering LLM Decision-Making
1Carnegie Mellon University · 2Microsoft Research
Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?
May 8, 2026Natalie Seah, Danielle S. Bitterman, Daphna Spiegel +1Breast Cancer Healthcare
University of Virginia · Mass General Brigham, Dana-Farber Cancer Institute, Harvard Medical School
Automated Clinical Report Generation for Remote Cognitive Remediation: Comparing Knowledge-Engineered Templates and LLMs in Low-Resource Settings
May 7, 2026Yongxin Zhou, Fabien Ringeval, François PortetMedical Report Generation Clinical Language Model Evaluation
Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France
Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction
May 7, 2026Shivali Dalmia, Ananya Mantravadi, Prasanna DesikanDocument Information Extraction Clinical Information Extraction
Centific AI Research · Centific Global Solutions, Inc.
TheraAgent: Self-Improving Therapeutic Agent for Precise and Comprehensive Treatment Planning
May 7, 2026Junkai Li, Yunghwei Lai, Tianyi Zhu +3LLM Self-Refinement Healthcare
1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · School of Computing, National University of Singapore, Singapore
Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry
May 6, 2026Oriana Presacan, Andreea Grama, Larisa Irimină +6Selective Prediction Healthcare
National University of Science and Technology Politehnica Bucharest · Psychiatric Hospital Doctor Gheorghe Preda · Oslo Metropolitan University +3
Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction
May 5, 2026Yao-Shun Chuang, Tushti Mody, Uday Pratap Singh +6Named Entity Recognition Healthcare
McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States · School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States · School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX 77054, United States +1
Safety and accuracy follow different scaling laws in clinical large language models
May 5, 2026Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa +9LLM Safety Benchmarks Retrieval-Augmented Generation
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Institute of Radiology, University Hospital Erlangen, Erlangen, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +3
EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage
May 5, 2026Richard J. Young, Alice M. MatthewsGender Bias in Language Models Clinical Triage
University of Nevada, Las Vegas, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA · Concorde Career Colleges, Dept. of Cardiovascular and Medical Diagnostic Sonography
Atomic Fact-Checking Increases Clinician Trust in Large Language Model Recommendations for Oncology Decision Support: A Randomized Controlled Trial
May 5, 2026Lisa C. Adams, Linus Marx, Erik Thiele Orberg +8Healthcare Clinical Decision Support
Department of Diagnostic and Interventional Radiology, TUM University Hospital Rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany · Department of Radiation Oncology, TUM University Hospital Rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany · Department of Internal Medicine III: Hematology and Clinical Oncology, University Hospital Regensburg, University of Regensburg, Regensburg, Germany +4
BIT.UA-AAUBS at ArchEHR-QA 2026: Evaluating Open-Source and Proprietary LLMs via Prompting in Low-Resource QA
May 5, 2026Richard A. A. Jonker, Alexander Christiansen, Alexandros Maniatis +4LLM Grounding Healthcare
Aalborg University Business School Fibigerstræde 2, Aalborg East 9220, Denmark · IEETA, DETI, LASI, University of Aveiro Campus Universitário de Santiago, Aveiro 3810-193, Portugal
Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks
May 5, 2026Tianze Han, Beining Xu, Hanbo Zhang +1Mental Health Counseling LLM Sycophancy
Shenzhen MSU-BIT University, Shenzhen, China
ADAPTS: Agentic Decomposition for Automated Protocol-agnostic Tracking of Symptoms
May 4, 2026Alexandria K. Vail, Marcelo Cicconet, Katie Aafjes-van Doorn +2Depression Detection Mental Health
Deliberate Solutions, Inc. New York, New York, USA
Reliability-Oriented Multilingual Orthopedic Diagnosis: A Domain-Adaptive Modeling and a Conceptual Validation Framework
May 4, 2026Danish Ali, Li Xiaojian, Sundas Iqbal +1Multilingual Language Model Evaluation Healthcare
School of Computer Science, Wuhan University, China · School of Software, Nanjing University of Information Science and Technology (NUIST), China · Department of Orthopedic, Bahawal Victoria Hospital, Pakistan
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10Healthcare Long-Horizon Agent Evaluation
Stanford University
Artificial intelligence language technologies in multilingual healthcare: Grand challenges ahead
May 2, 2026Vicent Briva-IglesiasMultilingual Language Model Evaluation Language Model Safety Evaluation
School of Applied Languages and Intercultural Studies (SALIS) · CTTS, ADAPT Centre · Dublin City University

May 15, 2026, cs.CLJ/K move · Enter open · S save

MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models

Weixin Liu, Congning Ni, Shelagh A. Mulvaney, Susannah L. Rose+3

Vanderbilt University, Nashville, TN, USA · Vanderbilt University Medical Center, Nashville, TN, USA · Virginia Tech, Blacksburg, VA, USA

LLM Evaluation Healthcare Mental Health Clinical Language Model Evaluation Knowledge Graphs LLM Reasoning+1

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in the mental health domain, yet it remains unclear how well they capture related biomedical knowledge and how reliably they apply it to clinically salient structured judgments. Here, we present a knowledge-graph (KG)-grounded benchmark for assessing LLMs on mental-health entity recognition, relation judgment, and two-hop reasoning. The benchmark is derived from PrimeKG and comprises nine task families with KG-supported answers and controlled negative options. Experiments across 15 closed- and open-source LLMs reveal a persistent recognition-to-judgment gap: leading models achieve near-ceiling performance on entity typing and on the small relation-typing subset, yet they still struggle with relation prediction and two-hop reasoning. Additionally, short KG-derived snippets benefit some models but degrade performance for others. Moreover, output-format reliability can substantially influence measured performance under constrained multiple-choice settings, highlighting the critical role of response validity in benchmark-based evaluation. MHGraphBench should therefore be interpreted as evaluating agreement with a curated mental-health slice of PrimeKG under a constrained multiple-choice interface, rather than as a direct assessment of real-world clinical safety.

Momentum

Related topics

Latest papers 259