Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

CARE-Bench: Benchmarking Patient-Facing LLM Triage
Aug 4, 2026Yining Hua, Hongbin Na, Cyrus AyubchaLLM Safety Benchmarks Clinical Triage
1Harvard University · University of Technology Sydney
Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning
Aug 4, 2026Zhitian Hou, Yuhang Liu, Pengkai Wang +8Counterfactual Evaluation Healthcare
The Hong Kong Polytechnic University · Sun Yat-sen University
MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs
Aug 3, 2026Saman Sarker Joy, Niloy FarhanLLM Safety Benchmarks LLM Sycophancy
Universiti Malaya, Malaysia · BRAC University, Bangladesh
High-Stakes Decisions with Language Models: Insights from Emergency Triage
Aug 2, 2026Khurram Yamin, Christopher Kelly, Bryan Wilder +1Clinical Triage Clinical Decision-Making
Microsoft · Carnegie Mellon University
Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy
Aug 2, 2026Kaike Ping, Buse Çarık, Caleb Wohn +3LLM Sycophancy Language Modeling
Virginia Tech · Shanghai Tongren Hospital, Shanghai Jiao Tong University School of Medicine · Emory University
Large language models improve physician accuracy but lead to false reliance
Aug 1, 2026Tirtha Chanda, Christoph Wies, Franziska Schramm +10Retrieval-Augmented Generation LLM Grounding
Division of Digital Prevention, Diagnostics and Therapy Guidance, German Cancer Research Center (DKFZ), Heidelberg, Germany · Medical Faculty, University Heidelberg, Heidelberg, Germany · Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany +6
Sixteen models, fewer than two voices: measuring ensemble dispersion where no answer is uniquely correct
Jul 31, 2026Mario Vega-Barbas, Lidia Mora-Valenciano, Iván Pau +2Language Model Ensembles Clinical Language Model Evaluation
ETSIS de Telecomunicación, Universidad Politécnica de Madrid, Madrid, Madrid, Spain · Independent Researcher, Guadalajara, Spain · Department of Clinical Science, Intervention and Technology, Karolinska Institutet, Huddinge, Stockholm, Sweden +4
Performance of large language models in the optical diagnosis of colorectal polyps
Jul 31, 2026Joshua C. Vences, William T. Tran, Nikko Gimpaya +15Endoscopy Multimodal Large Language Models
1. Scarborough Health Network Research Institute, Toronto, Ontario, Canada · 2. Division of Gastroenterology, Hepatology and Nutrition, and The SickKids Learning and Research Institutes, The Hospital for Sick Children, Toronto, Ontario, Canada · 3. Department of Paediatrics, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada +11
EarlyDx: An Admission-Anchored Benchmark for Open-Ended Generation of Evidence-Supported ED-Encounter Diagnoses
Jul 30, 2026Jiahui Li, Ruili Fang, Zishuai Liu +5Medical Diagnosis Clinical Prediction
University of Georgia Athens, USA · Beijing Luhe Hospital Beijing, China
Same Facts, Different Diagnosis: Measuring and Mitigating Narrative Anchoring in Clinical Language Models
Jul 29, 2026Prabhjot Singh, Pritam Deka, Vijay ChennareddySocial Bias in Language Models Clinical Reasoning
University of Texas at Austin Austin, Texas, USA · Queen’s University Belfast Belfast, United Kingdom · Middlesex University London, United Kingdom
Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support
Jul 29, 2026Shayndhan Sivanathan, Shravan Nageswaran, Mehdi Zadem +10Clinical Triage Clinical Decision Support
Atman Labs · Oxford University Hospitals · Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford +11
Cognivia: A Cognitive Behavioral Therapy Copilot for Evidence-Based Mental Healthcare
Jul 28, 2026Qi Chen, Siria Xiyueyao Luo, Jian Wang +3Mental Health Counseling Conversational Agents
Sichuan University Sichuan, China · Southwest Petroleum University Sichuan, China · University of Groningen Netherland +3
CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios
Jul 28, 2026Xiao Li, Mouxiao Bian, Zhaodi Wu +8Healthcare Clinical Language Model Evaluation
Department of Cardiology, Zhongshan Hospital Fudan University , Shanhai, China · Shanghai Institute of Cardiovascular Diseases, National Clinical , Shanghai, China · Research Center for Interventional Medicine , Shanghai, China +2
Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs
Jul 27, 2026Jianru ShenLLM Self-Correction Healthcare
University of Montana, Missoula, MT 59812, USA
Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration
Jul 26, 2026Fatema Tuj Johora Faria, Mukaffi Bin Moin, Md. Mahfuzur Rahman +3Mental Health Counseling LLM Prompting
Ahsanullah University of Science and Technology · Bangladesh University of Business and Technology · The University of New South Wales +2
Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records
Jul 24, 2026Jian Lu, Panyu Chen, Miriam Treggiari +5LLM Reliability Clinical Language Model Evaluation
Duke University, 2424 Erwin Road, Durham, 27705, North Carolina, USA. · Columbia University Medical Center, 622 West 168 Street, New York, 10032, NY, USA. · Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA.
MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education
Jul 23, 2026Qian Wu, Xinrong Zhou, Zizhan Ma +8Educational Technology Clinical Language Model Evaluation
CUHK · Tencent · Southern Medical University +1
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety
Jul 21, 2026Koyar AfrasyabLLM-as-a-Judge Language Model Safety Evaluation
Kinvectum AB, Sweden
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
Jul 20, 2026Koyar AfrasyabLLM-as-a-Judge LLM Prompting
Kinvectum AB, Sweden
Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Models
Jul 19, 2026Anthonio Oladimeji Gabriel, Dimeji Olawuyi, Toba Ajayi +1Multilingual Language Model Evaluation Language Model Safety Evaluation
AI SafetyX
Risk Governance for Generative AI Mental Health Support: A Multi-Turn Safety Architecture
Jul 17, 2026Anabela C. Areias, Catarina Botelho, António Farinhas +7Emotional Support Conversation Mental Health
Sword Health, New York, NY, United States · Yale Child Study Center, Yale School of Medicine, New Haven, CT, United States · Department of Psychology, University of Virginia, Charlottesville, VA, USA
MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation
Jul 15, 2026Thanni Adewuyi, Anuoluwa Sotome, Samuel Okoko +7Retrieval-Augmented Generation Healthcare
Helpmum Africa · University of Ibadan
Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations
Jul 14, 2026Monica Munnangi, Saiph SavageMulti-Turn Dialogue Evaluation Clinical Language Model Evaluation
Khoury College of Computer Sciences Northeastern University
Agentic systems for breast cancer treatment recommendations
Jul 13, 2026Vinicius Anjos de Almeida, Nícolas Henrique Borges, Leonardo Vicenzi +5Multi-Agent LLM Systems Clinical Decision Support
Faculdade de Medicina, Universidade de S˜ao Paulo, Av. Dr. Arnaldo, 455, 01246-903, S˜ao Paulo, S˜ao Paulo, Brazil · Spesia · Laboratory of Artificial Intelligence Applied to Bioinformatics, SEPT, Universidade Federal do Paran´a (UFPR), Rua Alcides Vieira Arcoverde, 1225, 81520-260, Curitiba, Paran´a, Brazil +2
Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns
Jul 13, 2026Yong Yang, Xiang Guan, Sophie Arheix-Parras +7Multimodal Large Language Models Language Modeling
Department of Computer Science and Engineering, University of South Carolina · Department of Psychology, University of South Carolina · Department of Communication Sciences and Disorders, University of South Carolina +2
Toward Contemplative LLM: A Modular Framework for Evaluating and Enhancing LLM Alignment in Mental Health
Jul 12, 2026Asher Sprigler, Yang-Yang Feng, Iftach Amir +5LLM Alignment Moral Reasoning in Language Models
Electrical and Computer Engineering Purdue University West Lafayette, IN, USA · Biomedical Engineering Washington University in St. Louis St. Louis, MO, USA · Psychological & Brain Sciences Washington University in St. Louis St. Louis, MO, USA +3
Capabilities of Claude Fable 5 on Biomedical Challenge Problems
Jul 12, 2026Dominic Okonkwo, Magnus Hodgson, Temitope I. David +1Biomedical QA LLM Refusal Behavior
School of Computing, University of Georgia · Department of Chemistry, University of Illinois · Institute of Bioinformatics, University of Georgia
Information-seeking failures of large language models in agentic clinical reasoning
Jul 11, 2026Krischan Braitsch, Laura K. Schmalbrock, Theresa Weltermann +11Healthcare Agentic Reasoning
Technical University of Munich, School of Medicine and Health, TUM University Hospital, Klinikum rechts der Isar, Department of Medicine III, Munich, Germany · Department of Hematology, Oncology and Cancer Immunology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany. · German Cancer Research Center (DKFZ), Heidelberg, Germany +14
Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences
Jul 10, 2026Robert WilliamsHallucination in Language Models Text Summarization
University of Texas at Austin Computer & Data Science Online Austin, Texas, USA

Aug 4, 2026, cs.AIJ/K move · Enter open · S save

CARE-Bench: Benchmarking Patient-Facing LLM Triage

Yining Hua, Hongbin Na, Cyrus Ayubcha

1Harvard University · University of Technology Sydney

LLM Safety Benchmarks Clinical Triage Clinical Language Model Evaluation

Open paper AskPDF arXiv

Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed GPT-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.

Momentum

Related topics

Latest papers 259