Clinical Language Model Evaluation

Ask about this topic

Momentum

36 papers in the last four weeks, up 112% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 258

All topics
CardsList

  1. Large language models are vulnerable to incidental information in clinical documentation and reasoning

    Oct 6, 2026Krithik Vishwanath, Brandon Ye, Anton Alyakin +4Medical Report GenerationClinical Language Model Evaluation

    Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Johns Hopkins University School of Medicine, Baltimore, MD, USA · Washington University School of Medicine in St. Louis, St. Louis, MO, USA +7

  2. Language-model ratings of depression reflect the rater more than the patient

    Oct 6, 2026Baihan LinInter-Rater ReliabilityAnnotator Disagreement

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  3. Symphony for Text Generation: Benchmarking Clinical Note Generation

    Oct 6, 2026Daniel Varab, Victor Petrén Bach Hansen, Asbjørn W. Helge +8Medical Report GenerationClinical Language Model Evaluation

    Corti

  4. COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

    Oct 5, 2026Baihan LinRepresentational Similarity AnalysisPsychometric Validation

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  5. MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks

    Oct 5, 2026Ziqing Wang, Lili Zhao, Kaize DingLLM Agent EvaluationAI Agent Benchmarks

    Northwestern University

  6. Clinical Concept Centers in LLMs

    Oct 2, 2026Aishik Nagar, Abhishek Vaidyanathan, Arun-Kumar Kaliya-Perumal +2Mechanistic InterpretabilityCausal Interventions in Language Models

    Carnegie Mellon University (CMU) · National University of Singapore · SAP Asia Pte Ltd +2

  7. Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage

    Oct 1, 2026Manar Aljohani, Brandon Ho, Kenneth McKinley +2Clinical Decision-MakingLanguage Model Bias Evaluation

    Department of Computer Science Virginia Tech Blacksburg, VA, US · Division of Emergency Medicine University of Washington and Seattle Children’s Hospital Seattle, WA, USA · Children’s National Hospital Washington DC, USA

  8. A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined

    Oct 1, 2026Zhangshu Joshua Jiang, Zina Ibrahim, James T. TeoFaithfulness of Language Model ExplanationsTemporal Reasoning in Language Models

    DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London London, UK

  9. OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

    Oct 1, 2026Negin Ashrafi, Jia Luo, Stacey M. Frumm +1LLM Safety BenchmarksLLM Auditing

    Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Lowe Center for Thoracic Oncology, Dana-Farber Cancer Institute, and Harvard Medical School, Boston, MA, USA

  10. UniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLI

    Sep 30, 2026Marius Micluta-Campeanu, Claudiu Creanga, Ana-Maria Bucur +2Clinical Language Model EvaluationClinical NLP

    Interdisciplinary School of Doctoral Studies, ♡HLT Research Center University of Bucharest, Romania · Faculty of Mathematics and Computer Science

  11. Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

    Sep 30, 2026Xinkai ChenDepression DetectionCoT Reasoning

    Independent Researcher

  12. Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

    Sep 29, 2026Abinitha Gourabathina, Haoran Zhang, Yuexing Hao +2Clinical TriageClinical Decision-Making

    Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science

  13. MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models

    Sep 29, 2026Lukas Thede, Yash Kumar Atri, David Chen +4Continual Learning for LLMsFactual Knowledge in Language Models

    University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +4

  14. KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

    Sep 29, 2026Xueting Fang, Zehui Li, Yang Yang +5Clinical Decision-MakingMulti-Turn Dialogue Evaluation

    Zhejiang University · Imperial College London · Nanchang University +2

  15. A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses

    Sep 29, 2026Zhangshu Joshua Jiang, Zina Ibrahim, James T. TeoClinical ReasoningRubric-Based Evaluation

    DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London, London, UK · Neurological Institute, Cleveland Clinic London, London, UK · King’s College London, London, UK +1

  16. CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations

    Sep 28, 2026Yusuf Kesmen, Aniruddha Mukherjee, Yena Chang +7Medical DiagnosisMulti-Turn Dialogue Evaluation

    EPFL · Aarhus University

  17. Understanding Clinical Cognitive Dialogues Using Large Language Models

    Sep 28, 2026Vishalakshi Arumugam, Dan Schumacher, Veronica Rammouz +3Intent ClassificationMulti-Turn Dialogue Evaluation

    College of AI, Cyber and Computing University of Texas at San Antonio, USA

  18. Jev in Medicine: A Benchmark Evaluation

    Sep 27, 2026Alfredo Madrid-García, Beatriz Merino-BarbanchoMedical QAClinical Reasoning

    Independent researcher Madrid, Spain

  19. OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

    Sep 26, 2026Anqi Li, Zhixuan Ge, Yixuan Duan +9Clinical Decision-MakingOncology

    Rice University · University of Illinois at Urbana-Champaign · University of Washington +3

  20. A Living Benchmark for Information Retrieval from Electronic Health Records

    Sep 24, 2026Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani +23Synthetic Benchmark GenerationClinical QA

    Department of Biomedical Data Science, Stanford University, Stanford, CA · Department of Pathology, Stanford University, Stanford, CA · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA +6

  21. Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark

    Sep 24, 2026Christine Park, Valerie Chen, Tim DettmersSynthetic Benchmark GenerationClinical Language Model Evaluation

    Carnegie Mellon University

  22. Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

    Sep 24, 2026Rakib Abdullah, Md. Maruful Islam MarufNamed Entity RecognitionMultilingual Language Model Evaluation

    Department of Artificial Intelligence and Data Science Green University of Bangladesh Dhaka, Bangladesh

  23. Decomposing Error and Style in Automated Clinical Coding

    Sep 21, 2026Han-Chin Shing, Jack Moriarty, Ryan Ware +6Annotator DisagreementAutomated Medical Coding

    Amazon

  24. Misaligned Clinical Risk Classification and Cost Asymmetry in Open-Weight Large Language Models

    Sep 21, 2026Star S. D. Liu, Xiyu Ding, Robert B. Barrett +3Cost-Sensitive LearningLLM Interpretability

    Biomedical Informatics and Data Science, Johns Hopkins University School of Medicine, Baltimore, MD, USA · Microsoft, Redmond, WA, USA · National Institutes of Health, Bethesda, MD, USA

  25. LLMs Anchor on Chief Complaint and Fail to Integrate Evidence in Sequential Clinical Triage

    Sep 19, 2026Dipankar Srirag, Haokai Zhao, Ashutosh Kumar +6Clinical TriageClinical Reasoning

    University of New South Wales, Sydney · Independent Researcher · St Vincent’s Hospital, Sydney +2

  26. HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication

    Sep 17, 2026Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi +2Multilingual Language Model EvaluationHealthcare

    SD-AI ERA, School of Computing, Queen’s University, Kingston, Ontario, Canada.

  27. Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models

    Sep 15, 2026Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto +2LLM Safety BenchmarksHealthcare

    Department of Neurosurgery, National Hospital Organization Kagoshima Medical Center, Kagoshima, Japan · Department of Neurology, Imamura General Hospital, Kagoshima, Japan · Iwamoto Neurosurgery, Kanoya, Kagoshima, Japan +1

  28. Challenges of Auditing: Variability in Outputs of Large Language Models for Health

    Sep 15, 2026Yuan Pu, Yewon Chang, Furong Jia +4LLM EvaluationLLM Auditing

    Department of Computer Science, Duke University, 308 Research Drive, Durham, 27708, NC. · Department of Medicine, Duke University, 40 Duke Medicine Circle, Durham, 27710, NC. · Geriatrics and Extended Care, Durham VA Health System, 508 Fulton Street, Durham, 27705, NC. +2

  29. Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

    Sep 14, 2026Jiashuo Zhang, Yuling Chen, Yvonne Commodore-Mensah +1LLM GroundingLLM Answer Verification

    Department of Computer Science Johns Hopkins University · School of Nursing Johns Hopkins University

  30. K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

    Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9LLM Safety BenchmarksMental Health Counseling

    School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom +4

  31. KnowBench: Effort Reduction as a Unified, Deployment-Grounded Benchmark for Clinical AI

    Sep 14, 2026Jocelyn Kang, Caroline ZhangBenchmark DesignWorkflow Automation

    Knowtex Inc.

  32. Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML

    Sep 14, 2026Sidi Chang, Peiying ZhuLLM-as-a-JudgeLLM Auditing

    Blossom AI San Francisco, CA, USA

  33. When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation

    Sep 14, 2026Griffin Farrow, Lily Sijia Li, Jack Johnson +4LLM Hallucination DetectionRubric-Based Evaluation

    University of Oxford

  34. MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

    Sep 14, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Medical DiagnosisMulti-Turn Dialogue Evaluation

    MBZUAI · Cairo University · Ain Shams University +2

  35. A primer on evaluation methods for large language models in healthcare

    Sep 13, 2026Suzannah E McKinney, Phuc Vu, Samuel A Justice +7LLM EvaluationClinical Language Model Evaluation

    Mass General Brigham AI, Mass General Brigham, Boston, MA, USA · Generative AI Laboratory, School of Informatics, University of Edinburgh, Edinburgh, UK · University of Edinburgh Centre for Cardiovascular Science, Edinburgh, UK +5

  36. Towards a Deterministic Math Solver for Clinical Language Models

    Sep 12, 2026Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange +5Numerical Reasoning in Language ModelsLLM Reliability

    MIT Critical Data · UNC Chapel-Hil · University of Pavia +1

  37. Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment

    Sep 12, 2026Jiayu Huang, Zichen Tang, Qianhui Ling +2Reasoning Consistency in Language ModelsClinical Language Model Evaluation

    Beijing University of Posts and Telecommunications · Hypertension Center, Beijing Anzhen Hospital, Capital Medical University

  38. Scaling Clinical Judgment to Evaluate Medical AI

    Sep 11, 2026Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur +15LLM-as-a-JudgeLLM Fine-Tuning

    Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA · Stanford Division of Hospital Medicine, Stanford University, Stanford, CA +11

  39. The widening evaluation gap in medical large language model research 2023 to 2026

    Sep 11, 2026Raad Bin Tareaf, Murad Al-Rajab, Samia LoucifLLM EvaluationClinical Language Model Evaluation

    XU Exponential University of Applied Sciences, Data Science and Artificial Intelligence Cluster, Marlene-Dietrich-Allee 12B, 14482 Potsdam, Germany · College of Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates · College of Technological Innovation, Zayed University, Abu Dhabi, United Arab Emirates

  40. Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support

    Sep 9, 2026Jonathan A. Handler, Marlene I. Robles-Granda, Jacob E. Mefford +5Clinical Decision SupportClinical Decision-Making

    Clinical Intelligence and Advanced Data Lab, OSF HealthCare, Peoria, IL · Digital Innovation Development, OSF HealthCare, Peoria, IL · OnCall, OSF HealthCare, Peoria, IL +7

  41. Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics

    Sep 8, 2026Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei +6HealthcareMedical Diagnosis

    1A.I. Doctor Medical Assist LTD, Limassol, Cyprus. · 2Family Medicine Specialist, Kraków, Poland.

  42. LLM4CKD: Large Language Models for Early Stage Chronic Kidney Disease Screening

    Sep 3, 2026Muhammad Ashad Kabir, Sirajam MuniraMedical DiagnosisIn-Context Learning

    School of Computing, Mathematics and Engineering Charles Sturt University NSW, Australia · Department of Computer Science Rensselaer Polytechnic Institute NY, USA

  43. Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor

    Sep 2, 2026Rohith Reddy Bellibaltu, Manpreet Singh, Deepak Parashar +1Counterfactual EvaluationAlgorithmic Fairness

    Florida International University · Boston University, Boston, MA 02215 USA · Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, India +1

  44. Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA

    Sep 1, 2026Nishant Mishra, Ameen Abu-Hanna, Iacer CalixtoClinical Language Model EvaluationLanguage Model Robustness

    Department of Medical Informatics, Amsterdam UMC, University of Amsterdam · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands

  45. ClinTraceBench: Source-Verifiable Longitudinal Clinical Reasoning over EHR-Derived Dialogues

    Sep 1, 2026Huimin Wang, Zhengyi Zhao, Yutian ZhaoClinical QAClinical Reasoning

    Shenzhen University · The Chinese University of Hong Kong · Dealism

  46. Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

    Aug 31, 2026Junyi Yao, Baichuan Li, Zihao Zheng +1Benchmark DesignLLM Agent Evaluation

    Washington University in St. Louis, USA · Southern Methodist University, USA

  47. One note in three: a verified census of three deployed AI scribes, and the instrument that counted it

    Aug 31, 2026Sebastian Fox, Luke Markham, Ryan Lail +1Medical Report GenerationClinical Language Model Evaluation

    Composo

  48. LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

    Aug 31, 2026Sebastian Fox, Luke Markham, Ryan Lail +1LLM-as-a-JudgeClinical Language Model Evaluation

    Composo

  49. ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography

    Aug 31, 2026Mohammadsina Hassannia, Matthew A. Reyna, Reza SameniElectrocardiographyLLM Fine-Tuning

    Department of Biomedical Informatics Emory University Atlanta, GA 30322, USA · Department of Biomedical Engineering Georgia Institute of Technology and Department of Biomedical Informatics Emory University Atlanta, GA 30332, USA

  50. GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space

    Aug 31, 2026Boqi Chen, Xudong Liu, Yunke Ao +2HealthcareConstrained RL

    ETH Zurich · MBZUAI · University of Toronto

  51. Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models

    Aug 30, 2026Hermione Warr, Harry Anthony, Lilli J Freischem +3LLM EvaluationPET/CT Imaging

    Department of Engineering Science, University of Oxford, UK · Department of Physics, University of Oxford, UK · Department of Oncology, University of Oxford, UK +1

  52. Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions

    Aug 13, 2026Qingfang Liu, Qiao Jin, Joe D. Menke +2Clinical QAInformation Retrieval

    National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, MD, USA · National Library of Medicine, National Institutes of Health, Bethesda, MD, USA · School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA

  53. A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

    Aug 12, 2026Praveen Reddy, Charuta Mandke, Suvrankar Datta +4Retrieval-Augmented GenerationLLM Grounding

  54. Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMs

    Aug 12, 2026Alireza A. Safaei, Laura M. Vowels, Matthew J. Vowels +2Energy-Efficient MLLLM Safety Evaluation

    University of Isfahan Isfahan, Iran · University of Roehampton London, UK · Kivira Health New York City, New York, U.S.A.

  55. Locally Deployable Small Language Models for Emergency Department Decision Support: A Systematic Benchmark of Fine-Tuning Strategies

    Aug 10, 2026Qingfeng Zhang, Yuanxiong Guo, Yanmin GongLLM Fine-TuningClinical Decision Support

    The University of Texas at San Antonio, San Antonio, TX · Texas A&M University, Houston, TX

  56. TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent

    Aug 10, 2026Waleed Jamil, Raphael SchmittLLM Safety BenchmarksLLM Safety Evaluation

    1Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg, Germany

  57. An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer

    Aug 10, 2026Mengxian Lyu, Cheng Peng, Tim Jang +18Clinical Decision-MakingDomain Adaptation for LLMs

    Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA. · Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · University of Florida Health Cancer Institute, Gainesville, FL, USA +8

  58. When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information

    Aug 10, 2026Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam +1Hallucination in Language ModelsLLM Uncertainty Estimation

    School of Computing, Macquarie University, Sydney, NSW 2109, Australia

  59. RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care

    Aug 5, 2026Mouxiao Bian, Zhi Chen, Ruiyao Chen +8LLM Safety BenchmarksHealthcare

    Shanghai Artificial Intelligence Laboratory, Shanhai, China · Macau University of Science and Technology, Macau, China · First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China +1

  60. CARE-Bench: Benchmarking Patient-Facing LLM Triage

    Aug 4, 2026Yining Hua, Hongbin Na, Cyrus AyubchaLLM Safety BenchmarksClinical Triage

    1Harvard University · University of Technology Sydney