Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation
Jul 10, 2026Cedric Caruzzo, Donggeun Yoo, Tae Soo KimRetrieval-Augmented Generation RAG Evaluation
Lunit
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Jul 10, 2026Runhan Shi, Quan Zhou, Yuqian Xu +14Healthcare Clinical Language Model Evaluation
JD Health International Inc. · Shanghai Jiao Tong University · National University of Singapore +3
The complexities of patient-centred conversational artificial intelligence
Jul 9, 2026João Matos, Olivia Buege, Donny Cheung +8Clinical Triage Conversational Agents
Verily Health, Dallas, TX, USA · Centre for Statistics in Medicine, University of Oxford, Oxford, UK · Department of Applied Health Sciences, School of Health Sciences, University of Birmingham, Birmingham, UK +2
MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
Jul 9, 2026Yuming Yang, Xiao Sun, Yuanwei Zou +6Mental Health Clinical Reasoning
School of Computer Science, Chongqing University, Chongqing, China · School of Computer Science, Hunan University, Changsha, China
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
Jul 8, 2026Qi Peng, Jiatong Li, Sirui Huang +10Clinical Reasoning Clinical Language Model Evaluation
1The Hong Kong Polytechnic University, Hong Kong, China. · 2Hong Kong University, Hong Kong, China. · 3South China University of Technology, Guangzhou, China. +3
Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability
Jul 7, 2026Tamara Wit, Lifeng Han, Carly Heipon +3Privacy-Preserving Language Models Small Language Models
BDS, Leiden University Medical Centre, NL · The Leiden Institute of Advanced Computer Science, LU, NL
Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization
Jul 4, 2026Nicole Sonne Heckmann, Arnault-Quentin Vermillet, Søren Norlin Mølgaard +4Bayesian Optimization Large Language Model-Guided Optimization
Department of Drug Design and Pharmacology, University of Copenhagen, Copenhagen, Denmark. · Safety Operations, Novo Nordisk, Søborg, Denmark. · Clinical Development Centre Denmark, Novo Nordisk, Søborg, Denmark. +1
Where do LLMs Fall Short in CBT-Guided Affective Reasoning?
Jul 3, 2026Vaishnavi Sinha, Pooja Guttal, Pranay Deep Reddy Katike +5Mental Health Clinical Language Model Evaluation
Department of Computer Science and Electrical Engineering, UMBC · Department of Electrical Engineering and Computer Science, UC Irvine · Department of Psychology, UMBC +1
A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
Jul 2, 2026Samiha A. Ismail, Fan X. Chen, Ali MeraliClinical Decision-Making Clinical Reasoning
Prime Analytics Consulting Limited · Talbert House; Thomas More University · MAKZ
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
Jul 1, 2026William Philipp, Finn Fassbender, Daniel Fister +12Multilingual Language Model Evaluation LLM-as-a-Judge
Section for Clinical Research-IT, Institute of Medical Biometry and Statistics, University of Luebeck, Luebeck, Germany · Medical Faculty, University of Tübingen, Tübingen, Germany · German Medical Students’ Association (bvmd), Berlin, Germany +9
CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning
Jun 30, 2026Ajmal M., Abin Roy, Afthab Salam Kanniyan +4LLM-as-a-Judge Human-in-the-Loop Evaluation
1MBZUAI · 2IIT Madras · 3Calicut Medical College
Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support
Jun 29, 2026Mizanur Rahman, Abeer Badawi, Elahe Rahimi +4Mental Health Counseling Multi-Agent LLM Systems
1York University, Ontario, Canada · 2Vector Institute, Ontario, Canada · 3Connected Minds, Ontario, Canada +1
Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency
Jun 29, 2026Nisarg A. PatelLLM Evaluation Reasoning Consistency in Language Models
University of California, San Francisco · Department of Oral and Maxillofacial Surgery, University of California, San Francisco, CA, USA · Department of Medicine, Division of Clinical Informatics and Digital Transformation, University of California, San Francisco, CA, USA
The strength of clinical evidence is recoverable from language model representations but not from their stated grades
Jun 27, 2026Soroosh Tayebi ArastehLLM Interpretability Language Model Calibration
Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen · Department of Urology, Stanford University, Stanford, CA, USA +1
Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries
Jun 27, 2026Jean Feng, Vishal Patel, Patrick Heagerty +5Clinical Decision Support Clinical QA
Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USA. · Harvard Medical School, Boston, Massachusetts, USA. · Brigham and Women’s Hospital, Boston, Massachusetts, USA. +4
IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
Jun 26, 2026Maria Xenochristou, Ashutosh Joshi, Korosh Vatanparvar +10Multimodal Large Language Models Clinical Language Model Evaluation
Amazon Health AI
MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction
Jun 24, 2026Congbo Ma, Hu Wang, Yichun Zhang +1LLM Guardrails LLM Safety
New York University Abu Dhabi · Khalifa University · New York University
MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models
Jun 23, 2026Jinru Ding, Chuchu Jiang, Lu Lu +12AI Agent Benchmarks Medical VLMs
1Shanghai Artificial Intelligence Laboratory
T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph
Jun 23, 2026Saba A. Farahani, Hung Cao, Ramesh Jain +1Type 1 Diabetes Clinical Decision Support
University of California, Irvine, Irvine, CA, USA
One Year Later...The Harms Persist, But So Do We!
Jun 22, 2026Annika Marie Schoene, Cansu Canca, Gautham Vijay Kumar +1Mental Health Adversarial Attacks on LLMs
Northeastern University Boston, MA, USA · Florida International University Miami, FL, USA
Self-Stigma Is Not a Monolith, but Generic Empathy Is: Persona-Conditioned LLM Support for People Who Use Drugs
Jun 22, 2026Layla Bouzoubaa, Rezvaneh RezapourHealthcare User Modeling
Department of Information Science Drexel University Philadelphia, PA
Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering
Jun 21, 2026Madhulatha Mandarapu, Sandeep KunkunuruLLM Grounding LLM Reasoning with Graphs
samyama
Old Fictions, New Skins: Evaluating the Manipulative Capabilities of LLMs in Healthcare
Jun 20, 2026Gathoni Ireri, Roger D. OdipoHealthcare Deception in Language Models
1ILINA Program · 2Haki AI
MedHal-Loc: Are "Explainable-by-Architecture" Medical Hallucination Detectors Faithful Localizers? A Localization Benchmark
Jun 19, 2026Minmin Chen, Daojian Lu, Yining Dai +2Hallucination Detection Clinical Language Model Evaluation
Shanwei Institute of Technology, Shanwei, China · Guangdong Provincial People’s Hospital, Guangzhou, China
A Multi-Agent Audit Framework for High-Stakes Reasoning: Evaluation and Interpretability in Clinical Mental Health Screening
Jun 19, 2026Jingchen Ye, Yanpei Yu, Luyao ZhangLLM Auditing LLM Interpretability
Duke Kunshan University Suzhou, China · Social Science Division and Digital Innovation Research Center, Duke Kunshan University, No.8 Duke Ave., Kunshan, Jiangsu 215316, China.
Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language
Jun 17, 2026Yunkai Xu, Saeed AbdullahMultilingual Language Model Evaluation Depression Detection
Pennsylvania State University, United States
LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data
Jun 17, 2026Akshat Dasula, Prasanna Desikan, Jaideep SrivastavaLanguage Model Calibration LLM Uncertainty Estimation
Centific AI Research · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA.
Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA
Jun 17, 2026Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3LLM Fine-Tuning Domain-Adaptive Pretraining
Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004, 44000 Nantes, France · Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217, 38000 Grenoble, France
Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
Jun 17, 2026Qingyu Lu, Ruochen Li, Liang Ding +3Clinical Language Model Evaluation Language Model Robustness
Nanyang Technological University · Technical University of Munich · Alibaba +2
Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Jun 17, 2026Tianming Du, Peijie Yu, Sihan Shang +12AI Agent Benchmarks Clinical Language Model Evaluation
1ELLIS Institute Finland · 2Aalto University · 3Tencent +7

Jul 10, 2026, cs.CLJ/K move · Enter open · S save

Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation

Cedric Caruzzo, Donggeun Yoo, Tae Soo Kim

Lunit

Retrieval-Augmented Generation RAG Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Retrieval-augmented generation evaluation checks whether model claims are factually grounded in retrieved documents. It does not check whether retrieved evidence is attributed to the correct entity. A clinical RAG response can pass every automated check (zero hallucinations, near-perfect faithfulness, real citations) while presenting drug Y's clinical evidence as evidence about queried drug X. We term this deceptive grounding (DG): a failure invisible to faithfulness, hallucination, and citation checks because every claim is sourced from a real document, about the wrong entity. Using a controlled factorial benchmark across 13 models, we find DG rates spanning 8-87% at peak adversarial conditions. Medical and biomedical fine-tuned models reach up to 86.7%; domain specialization amplifies the failure rather than mitigating it. A controlled ablation identifies the mechanism: removing entity-specific clinical evidence from retrieved documents eliminates entity-attribution failure entirely, shifting all failures to confabulation. The two failure modes respond to the same trigger, taking different paths. Production measurement across 740 drug-disease pairs finds 7.8% overall DG in a deployed RAG system, rising to 13.6% for recently approved drugs. Entity-attribution verification (checking that cited evidence applies to the queried entity) detects DG at 97.0% precision and 98.7% DG recall (IPW-adjusted human gold standard); no existing framework implements it.

Momentum

Related topics

Latest papers 259