Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

Implicit Geographic Inference in LLM Medical Triage: Language-Driven Disparities in Emergency Recommendations
May 31, 2026Qi Han WongMultilingual Language Model Evaluation Clinical Triage
Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs
May 28, 2026Mahdi Alkaeed, Adnan Qayyum, Nabeel Abo Kashreef +2Prompt Sensitivity Clinical Language Model Evaluation
Department of Computer Science and Engineering, College of Engineering, Qatar University, Doha, Qatar · College of Science and Engineering, Hamad Bin Khalifa University (HBKU), Doha, Qatar · Primary Health Care Corporation (PHCC), Doha, Qatar +1
EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs
May 28, 2026Yuzhang Xie, Keqi Han, Yunpeng Xiao +7Healthcare Clinical Decision-Making
Emory University Atlanta, GA, USA · Stanford University Stanford, CA, USA
AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis
May 28, 2026Saeedeh Davoudi, Reihaneh Iranmanesh, Ophir Frieder +1Healthcare Privacy Leakage in Language Models
1IR Lab, Computer Science Department, Georgetown University, Washington D.C.
Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents
May 28, 2026Matt TurkCounterfactual Evaluation Healthcare
Protege Data Lab New York City, USA
MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
May 28, 2026Valentina Bui Muti, Eugénie Dulout, Ziquan FuHealthcare Clinical Decision Support
System Inc., New York, NY, USA.
Internal Representation, Not Clinical Knowledge: Where Apparent LLM Triage Failures Originate
May 28, 2026David Fraile Navarro, Berardino Como, Jialei Sheng +2Healthcare Clinical Triage
Macquarie University, Sydney, Australia · Politecnico di Bari, Bari, Italy · NSW Health, Sydney, Australia +1
SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models
May 27, 2026Chao Ding, Mouxiao Bian, Tianbin Li +12Healthcare LLM Auditing
Shanghai Artificial Intelligence Laboratory, Shanghai, China · Joint Laboratory of Biomedical Artificial Intelligence, Shanghai East Hospital, Tongji University School of Medicine, Shanghai, China · School of Computer Science and Technology, Tongji University, Shanghai, China +5
Do Clinical Models Change Treatment Decisions?
May 27, 2026Dongkyu Cho, Miao Zhang, Rumi ChunaraHealthcare Clinical Decision-Making
New York University
MIRA: A Bilingual Benchmark for Medical Information Response Audit
May 27, 2026Mengyu Xu, Qiaoxin Yang, Qianqian Wang +3Multilingual Language Model Evaluation Healthcare
The University of Chicago · The Chinese University of Hong Kong, Shenzhen · Jinzhou Medical University +3
A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks
May 26, 2026Heriberto Cuayahuitl, Grace JangHealthcare Speech Processing
School of Engineering and Physical Sciences University of Lincoln, UK · Lincoln Medical School Universities of Lincoln and Nothingham, UK
Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety
May 25, 2026Fay Elhassan, David Sasu, Alexandra Kulinkina +2LLM Safety Benchmarks Human Preference Evaluation
LiGHT Laboratory, EPFL, Lausanne, Switzerland
AI Content Moderation in Therapy Conversations
May 25, 2026Jiwon Kim, Claire Wang, Taeung Yoon +2LLM Auditing LLM Guardrails
University of Illinois Urbana-Champaign, U.S.
Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients
May 24, 2026Abdurrahim Yilmaz, Ayşe Esra Koku Aksu, Duygu Yamen +5Privacy-Preserving Language Models Healthcare
Division of Systems Medicine, Department of Metabolism, Digestion and Reproduction, Imperial College London · Department of Dermatology and Venereology, Istanbul Research and Training Hospital · Department of Dermatology and Venereology, Istanbul Medeniyet University +1
When Reasoning Hurts: Source-Aware Evaluation of Frontier LLMs for Clinical SOAP Note Generation
May 24, 2026Faizan FaisalRetrieval-Augmented Generation Medical Report Generation
University of California, Davis
MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional
May 23, 2026Roberto Cruz, David Rey-BlancoMulti-Agent LLM Systems Healthcare
TietAI
GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration
May 23, 2026Junjie Zhao, Jingyi Liang, Zhenyang Cai +22Language Model Safety Evaluation Clinical Reasoning
When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening
May 22, 2026Jianfeng Zhu, Megan Korhummel, Ruoming Jin +1Medical Diagnosis Mental Health
Departments of Computer Science1 and Psychological Science2 Kent State University Kent, OH USA
Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions
May 21, 2026Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio +2Healthcare Benchmark Design
New York University
Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support
May 21, 2026Chen Zhan, Xihe Qiu, Xiaoyu Tan +8Healthcare Clinical Decision Support
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, China. · Department, Tencent Youtu Lab, Shanghai, 200233, China. · Department of Oncology, Jinshan Hospital, Fudan University, Shanghai, 201508, China. +4
GraphRAG on Consumer Hardware: Benchmarking Local LLMs for Healthcare EHR Schema Retrieval
May 20, 2026Peter Fernandes, Ria KanjilalLLM Inference Efficiency LLM Grounding
Department of Computer Engineering California Polytechnic State University San Luis Obispo, CA, USA
Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models
May 20, 2026Sunday Oyinlola Ogundoyin, Muhammad Ikram, Rahat MasoodLanguage Model Safety Evaluation Healthcare
Macquarie University, Sydney, Australia · The University of New South Wales, Sydney, Australia
ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning
May 19, 2026Juncheng Wu, Letian Zhang, Yuhan Wang +5Healthcare Clinical Decision Support
UC Santa Cruz
Prompting language influences diagnostic reasoning and accuracy of large language models
May 18, 2026Adrien Bazoge, Josselin Corvellec, Sofiane Djillali Sid-Ahmed +1Multilingual Language Model Evaluation LLM Prompting
Data Clinic, Nantes University Hospital, France
What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models
May 18, 2026Payal Chandak, Victoria Alkin, David Wu +11Moral Reasoning in Language Models Healthcare
Evaluating the Utility of Personal Health Records in Personalized Health AI
May 18, 2026Rory Sayres, Kejia Chen, Ayush Jain +19LLM Personalization Healthcare
1Google Research · †Work done while at Google Research
Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale
May 18, 2026Jinghui Liu, Sarvesh Soni, Anthony NguyenSynthetic Data Augmentation Automated Medical Coding
Australian e-Health Research Centre, CSIRO, Australia · National Library of Medicine, National Institutes of Health, USA
Artificial Intolerance: Stigmatizing Language in Clinical Documentation Skews Large Language Model Decision-Making
May 17, 2026Jen-tse Huang, Didi Zhou, Faith Kamau +5Social Bias in Language Models Clinical Language Model Evaluation
1Johns Hopkins University.
Adversarial Fragility and Language Vulnerability in Clinical AI: A Systematic Audit of Diagnostic Collapse Under Imperceptible Perturbations and Cross-Lingual Drift in Low-Resource Healthcare Settings
May 16, 2026Anthonio Oladimeji Gabriel, Ahmad Rufai YusufMultilingual Language Model Evaluation Healthcare
Centre for Clinical Intelligence & Safety, Ìyàwó, Nigeria · Tomorrow University of Applied Sciences, Germany
Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries
May 15, 2026Liqi Zhou, Jiafu LiHealthcare Clinical Triage

May 31, 2026, cs.CLJ/K move · Enter open · S save

Implicit Geographic Inference in LLM Medical Triage: Language-Driven Disparities in Emergency Recommendations

Qi Han Wong

Multilingual Language Model Evaluation Clinical Triage Clinical Decision-Making Language Model Bias Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

We investigate whether large language models produce different medical triage recommendations for identical symptoms based solely on the language of the patient prompt. Using Gemini 3.5 Flash, we evaluate a neurological symptom profile (persistent headache, blurred vision, nausea) across six languages (English, Spanish, Chinese, Hindi, Japanese, Arabic) with 30 runs per condition (n=450 total API calls). We find that the model recommends emergency room visits at rates ranging from 0% (Japanese, Hindi) to 30% (English, Arabic), despite assigning nearly identical severity scores (7.7-8.0/10) across all languages. Adding a single sentence specifying the patient's US location increases ER recommendations by up to 76.7 percentage points for non-English prompts, while the reverse anchor (English prompt with a Tokyo location) reduces the ER rate from 30% to 6.7%. A back-translation control (Japanese to English) produces ER rates comparable to the English baseline, confirming that the disparity is not caused by translation quality but by implicit geographic inference from the input language. We release the complete dataset, experiment code, and results.

Momentum

Related topics

Latest papers 259