Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
May 2, 2026Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32LLM Evaluation Healthcare
Sophont · MedARC · Prime Intellect
Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines
May 1, 2026Hugo Abonizio, Filipe Rocha Lopes, Roberto Lotufo +1Knowledge Augmentation for Language Models Healthcare
FEEC, UNICAMP, Campinas, Brazil · NoHarm.ai, Brazil · Maritaca AI, Brazil
CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine
May 1, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5Healthcare LLM Reliability
Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA · Intuit AI Research, Mountain View, CA, USA · Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA +2
HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats
Apr 30, 2026Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim +13Healthcare Clinical Decision Support
OpenAI
Domain-Adapted Small Language Models for Reliable Clinical Triage
Apr 29, 2026Manar Aljohani, Brandon Ho, Kenneth McKinley +2Clinical Triage LLM Fine-Tuning
Department of Computer Science, Virginia Tech, Blacksburg, VA, US · Children’s National Hospital, Washington DC, USA
PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators
Apr 28, 2026Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu +4Benchmark Design Clinical Language Model Evaluation
University of Illinois Urbana-Champaign · VinUniversity
Single-turn emergency psychiatric triage across 15 frontier AI chatbots
Apr 28, 2026Veith Weilnhammer, Lennart Luettgau, Christopher Summerfield +5Healthcare Clinical Triage
Max Planck UCL Centre for Computational Psychiatry and Ageing Research, London, UK · UK AI Security Institute · Department of Experimental Psychology, University of Oxford, Oxford, UK +2
Analyzing LLM Reasoning to Uncover Mental Health Stigma
Apr 27, 2026Sreehari Sankar, Aliakbar Nafar, Mona Barman +8LLM Evaluation Social Bias in Language Models
BetterHelp
Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters
Apr 27, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6LLM-as-a-Judge Human-AI Agreement
1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2
Green Shielding: A User-Centric Approach Towards Trustworthy AI
Apr 27, 2026Aaron J. Li, Nicolas Sanchez, Hao Huang +8Prompt Sensitivity Clinical Language Model Evaluation
University of California, Berkeley · University of Melbourne · University of California, San Francisco +2
Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus
Apr 27, 2026Johannes Moll, Jannik Lübberstedt, Christoph Nuernbergk +21Healthcare LLM Agent Evaluation
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, School of Medicine and Health, Technical University of Munich, Munich, Germany · Department of Cardiovascular Radiology and Nuclear Medicine, German Heart Center, TUM University Hospital, School of Medicine and Health, Technical University of Munich, Munich, Germany +11
AI Safety Training Can be Clinically Harmful
Apr 25, 2026Suhas BN, Andrew M. Sherrill, Rosa I. Arriaga +2Mental Health Counseling LLM Alignment
College of Information Sciences and Technology, Penn State University, USA · Department of Psychiatry and Behavioral Sciences, Emory University, USA · School of Interactive Computing, Georgia Tech, USA +1
Reliability Auditing for Downstream LLM tasks in Psychiatry: LLM-Generated Hospitalization Risk Scores
Apr 23, 2026Shevya Panda, Shinjini Bose, Ananya JoshiPrompt Sensitivity LLM Auditing
Johns Hopkins University Baltimore, USA
Evaluating Patient Safety Risks in Generative AI: Development and Validation of a FMECA Framework for Generated Clinical Content
Apr 23, 2026Lydie Bednarczyk, Jamil Zaghir, Julien Ehrsam +11Clinical Language Model Evaluation Medical LLMs
Division of Medical Information Sciences, Diagnostic Department, Geneva University Hospitals (HUG), Geneva, Switzerland. · Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, Switzerland. · Pharmacy, Geneva University Hospital (HUG), and Institute of Pharmaceutical Sciences of Western Switzerland, School of Pharmaceutical Sciences, University of Geneva, Geneva, Switzerland. +3
When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure
Apr 23, 2026Boyu Xiao, Xiuqi Tian, Xuwen Song +4LLM Sycophancy Healthcare
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
Serialisation Strategy Matters: How FHIR Data Format Affects LLM Medication Reconciliation
Apr 22, 2026Sanjoy PatorHealthcare LLM Information Extraction
Independent Researcher
Can "AI" Be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs
Apr 22, 2026Mariano Barone, Francesco Di Serio, Roberto Moio +4Healthcare LLM Prompting
Department of Electrical Engineering and Information Technology, University of Naples Federico II, Via Claudio 21, Naples, 80125, Italy. · Department of Translational Medical Sciences, University of Campania ”Luigi Vanvitelli”, Via Leonardo Bianchi, Naples, 80131, Italy. · Department of Computer Science, McCormick School of Engineering and Applied Science, Northwestern University, 2309 Sheridan Rd, Evanston, 60201, IL, United States.
Graph2Counsel: Clinically Grounded Synthetic Counseling Dialogue Generation from Client Psychological Graphs
Apr 22, 2026Aishik Mandal, Hiba Arnaout, Clarissa W. Ong +5Mental Health Counseling Mental Health
UKP Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt · Zuse School · National Research Center for Applied Cybersecurity ATHENE +4
Cross-Model Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Across Three Large Language Models
Apr 21, 2026Kihyuk LeeLLM Evaluation Clinical Language Model Evaluation
Data Convergence Team, Office of Hospital Information, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline
Apr 20, 2026Jiyao Liu, Jianghan Shen, Sida Song +19Healthcare AI Agent Benchmarks
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
First, Do No Harm (With LLMs): Mitigating Racial Bias via Agentic Workflows
Apr 20, 2026Sihao Xing, Zaur GoulievHealthcare Social Bias in Language Models
School of Enterprise Computing and Digital Transformation Technological University Dublin · School of Information and Communication Studies University College Dublin
MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models
Apr 20, 2026Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav +2Mental Health Counseling LLM Evaluation
Department of Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · School of Computing and Information Systems, Singapore Management University, Singapore
PsychBench: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations
Apr 19, 2026Patrick KeoughHealthcare LLM Auditing
Independent Researcher Bremerton, Washington, USA
Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA
Apr 19, 2026Alberto Testoni, Iacer CalixtoHealthcare Social Bias in Language Models
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands. · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands.
MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning
Apr 19, 2026Lingyan Wu, Xiang Zheng, Weiqi Zhai +5Healthcare Process Reward Models
Alibaba Group China
Evaluating Multimodal LLMs for Inpatient Diagnosis: Real-World Performance, Safety, and Cost Across Ten Frontier Models
Apr 18, 2026Bruce A. Bassett, Amy Rouillard, Sitwala Mundia +8Language Model Safety Evaluation Healthcare
School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · Wits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · Grai Labs, Cape Town, South Africa +3
HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine
Apr 18, 2026Brandon DentHealthcare AI Agent Benchmarks
GOATnote Inc.
Blinded Multi-Rater Comparative Evaluation of a Large Language Model and Clinician-Authored Responses in CGM-Informed Diabetes Counseling
Apr 16, 2026Zhijun Guo, Alvina Lai, Emmanouil Korakas +6Clinical Decision Support RAG Evaluation
Institute of Health Informatics, University College London, London, United Kingdom · University College London Hospitals NHS Foundation Trust, London, United Kingdom · Dartford and Gravesham NHS Foundation Trust, Dartford, United Kingdom +1
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-Judge Healthcare
aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa +3
Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluation
Apr 16, 2026Bhavik Vachhani, Kush Shrisvastava, Pranshu Nema +1Medical Report Generation Clinical Reasoning
Augnito Research

May 2, 2026, cs.CLJ/K move · Enter open · S save

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer, Aymane Ouraq+31

Sophont · MedARC · Prime Intellect

LLM Evaluation Healthcare Clinical Language Model Evaluation Medical LLMs

Open paper AskPDF arXiv

Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks

Momentum

Related topics

Latest papers 259