Clinical Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 259

All topics

Large language models are vulnerable to incidental information in clinical documentation and reasoning
Oct 6, 2026Krithik Vishwanath, Brandon Ye, Anton Alyakin +4Medical Report Generation Clinical Language Model Evaluation
Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Johns Hopkins University School of Medicine, Baltimore, MD, USA · Washington University School of Medicine in St. Louis, St. Louis, MO, USA +7
Language-model ratings of depression reflect the rater more than the patient
Oct 6, 2026Baihan LinInter-Rater Reliability Annotator Disagreement
Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2
Symphony for Text Generation: Benchmarking Clinical Note Generation
Oct 6, 2026Daniel Varab, Victor Petrén Bach Hansen, Asbjørn W. Helge +8Medical Report Generation Clinical Language Model Evaluation
Corti
COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
Oct 5, 2026Baihan LinRepresentational Similarity Analysis Psychometric Validation
Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2
MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks
Oct 5, 2026Ziqing Wang, Lili Zhao, Kaize DingLLM Agent Evaluation AI Agent Benchmarks
Northwestern University
Decide, Ask, or Defer: Clinical LLMs under Incomplete Evidence
Oct 3, 2026Mingzhan Yang, Weili WuClinical Decision-Making Clinical Language Model Evaluation
Department of Computer Science University of Texas at Dallas Richardson, TX 75080
Clinical Concept Centers in LLMs
Oct 2, 2026Aishik Nagar, Abhishek Vaidyanathan, Arun-Kumar Kaliya-Perumal +2Mechanistic Interpretability Causal Interventions in Language Models
Carnegie Mellon University (CMU) · National University of Singapore · SAP Asia Pte Ltd +2
Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage
Oct 1, 2026Manar Aljohani, Brandon Ho, Kenneth McKinley +2Clinical Decision-Making Language Model Bias Evaluation
Department of Computer Science Virginia Tech Blacksburg, VA, US · Division of Emergency Medicine University of Washington and Seattle Children’s Hospital Seattle, WA, USA · Children’s National Hospital Washington DC, USA
A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
Oct 1, 2026Zhangshu Joshua Jiang, Zina Ibrahim, James T. TeoFaithfulness of Language Model Explanations Temporal Reasoning in Language Models
DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London London, UK
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
Oct 1, 2026Negin Ashrafi, Jia Luo, Stacey M. Frumm +1LLM Safety Benchmarks LLM Auditing
Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Lowe Center for Thoracic Oncology, Dana-Farber Cancer Institute, and Harvard Medical School, Boston, MA, USA
UniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLI
Sep 30, 2026Marius Micluta-Campeanu, Claudiu Creanga, Ana-Maria Bucur +2Clinical Language Model Evaluation Clinical NLP
Interdisciplinary School of Doctoral Studies, ♡HLT Research Center University of Bucharest, Romania · Faculty of Mathematics and Computer Science
Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity
Sep 30, 2026Xinkai ChenDepression Detection CoT Reasoning
Independent Researcher
Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts
Sep 29, 2026Abinitha Gourabathina, Haoran Zhang, Yuexing Hao +2Clinical Triage Clinical Decision-Making
Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
Sep 29, 2026Lukas Thede, Yash Kumar Atri, David Chen +4Continual Learning for LLMs Factual Knowledge in Language Models
University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +4
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
Sep 29, 2026Xueting Fang, Zehui Li, Yang Yang +5Clinical Decision-Making Multi-Turn Dialogue Evaluation
Zhejiang University · Imperial College London · Nanchang University +2
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
Sep 29, 2026Zhangshu Joshua Jiang, Zina Ibrahim, James T. TeoClinical Reasoning Rubric-Based Evaluation
DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London, London, UK · Neurological Institute, Cleveland Clinic London, London, UK · King’s College London, London, UK +1
CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations
Sep 28, 2026Yusuf Kesmen, Aniruddha Mukherjee, Yena Chang +7Medical Diagnosis Multi-Turn Dialogue Evaluation
EPFL · Aarhus University
Understanding Clinical Cognitive Dialogues Using Large Language Models
Sep 28, 2026Vishalakshi Arumugam, Dan Schumacher, Veronica Rammouz +3Intent Classification Multi-Turn Dialogue Evaluation
College of AI, Cyber and Computing University of Texas at San Antonio, USA
Jev in Medicine: A Benchmark Evaluation
Sep 27, 2026Alfredo Madrid-García, Beatriz Merino-BarbanchoMedical QA Clinical Reasoning
Independent researcher Madrid, Spain
OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories
Sep 26, 2026Anqi Li, Zhixuan Ge, Yixuan Duan +9Clinical Decision-Making Oncology
Rice University · University of Illinois at Urbana-Champaign · University of Washington +3
A Living Benchmark for Information Retrieval from Electronic Health Records
Sep 24, 2026Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani +23Synthetic Benchmark Generation Clinical QA
Department of Biomedical Data Science, Stanford University, Stanford, CA · Department of Pathology, Stanford University, Stanford, CA · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA +6
Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
Sep 24, 2026Christine Park, Valerie Chen, Tim DettmersSynthetic Benchmark Generation Clinical Language Model Evaluation
Carnegie Mellon University
Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER
Sep 24, 2026Rakib Abdullah, Md. Maruful Islam MarufNamed Entity Recognition Multilingual Language Model Evaluation
Department of Artificial Intelligence and Data Science Green University of Bangladesh Dhaka, Bangladesh
Decomposing Error and Style in Automated Clinical Coding
Sep 21, 2026Han-Chin Shing, Jack Moriarty, Ryan Ware +6Annotator Disagreement Automated Medical Coding
Amazon
Misaligned Clinical Risk Classification and Cost Asymmetry in Open-Weight Large Language Models
Sep 21, 2026Star S. D. Liu, Xiyu Ding, Robert B. Barrett +3Cost-Sensitive Learning LLM Interpretability
Biomedical Informatics and Data Science, Johns Hopkins University School of Medicine, Baltimore, MD, USA · Microsoft, Redmond, WA, USA · National Institutes of Health, Bethesda, MD, USA
LLMs Anchor on Chief Complaint and Fail to Integrate Evidence in Sequential Clinical Triage
Sep 19, 2026Dipankar Srirag, Haokai Zhao, Ashutosh Kumar +6Clinical Triage Clinical Reasoning
University of New South Wales, Sydney · Independent Researcher · St Vincent’s Hospital, Sydney +2
HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication
Sep 17, 2026Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi +2Multilingual Language Model Evaluation Healthcare
SD-AI ERA, School of Computing, Queen’s University, Kingston, Ontario, Canada.
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
Sep 15, 2026Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto +2LLM Safety Benchmarks Healthcare
Department of Neurosurgery, National Hospital Organization Kagoshima Medical Center, Kagoshima, Japan · Department of Neurology, Imamura General Hospital, Kagoshima, Japan · Iwamoto Neurosurgery, Kanoya, Kagoshima, Japan +1
Challenges of Auditing: Variability in Outputs of Large Language Models for Health
Sep 15, 2026Yuan Pu, Yewon Chang, Furong Jia +4LLM Evaluation LLM Auditing
Department of Computer Science, Duke University, 308 Research Drive, Durham, 27708, NC. · Department of Medicine, Duke University, 40 Duke Medicine Circle, Durham, 27710, NC. · Geriatrics and Extended Care, Durham VA Health System, 508 Fulton Street, Durham, 27705, NC. +2
Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering
Sep 14, 2026Jiashuo Zhang, Yuling Chen, Yvonne Commodore-Mensah +1LLM Grounding LLM Answer Verification
Department of Computer Science Johns Hopkins University · School of Nursing Johns Hopkins University

Oct 6, 2026, cs.CLJ/K move · Enter open · S save

Large language models are vulnerable to incidental information in clinical documentation and reasoning

Krithik Vishwanath, Brandon Ye, Anton Alyakin, John E. Markert+3

Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Johns Hopkins University School of Medicine, Baltimore, MD, USA · Washington University School of Medicine in St. Louis, St. Louis, MO, USA+7

Medical Report Generation Clinical Language Model Evaluation Language Model Robustness Medical LLMs

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly relied upon to support ambient documentation and clinical reasoning. Here we examine the impact of a failure mode shared between these two applications by assessing their sensitivity to information incidental to the patient encounter. In 576 patient-clinician dialogues, we found that frontier models inserted small-talk exchanges into 35% of notes, while mean quality scores changed by at most 0.20 points on five-point scales. In 3.7% of frontier notes, models misattributed the asides or used them clinically. In 57 mock recorded consultations, background speech from a separate patient encounter at -10 dB leaked into 48.2% of transcripts, with contamination detected in 5.3% of downstream notes generated by four open-weight models. We propose a dual encoding hypothesis of clinical reasoning and distraction in LLMs, with preliminary evidence that LLM components associated with disruption by incidental information also support clinical reasoning. These findings support evaluating resistance to incidental information before clinical use, with safeguards that prevent contamination while preserving clinical reasoning.

Momentum

Related topics

Latest papers 259