Clinical Language Model Evaluation

Overview

Clinical language model evaluation asks whether models can answer medical questions, interpret records, support conversations, and assist with decisions without causing harm. Researchers use exam-style benchmarks, clinician-authored rubrics, electronic health records, simulated consultations, and tests that vary language, missing information, patient behavior, or evidence; they also compare expert ratings with automated judges. Evaluation increasingly measures safety, omissions, evidence grounding, uncertainty, consistency, and workflow performance alongside accuracy. A central challenge is that strong scores on static tests or plausible reasoning may not transfer to dynamic, incomplete clinical encounters, while automated judges can disagree with clinicians and require validation.

Latest papers

CardsList

461 papers56 papers in the last four weeks, up 33% on the four weeks before. 0.4% of all new papers.

  1. Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction

    Nov 17, 2026Meilu Zhu, Xiaohan Li, Axiu MaoLLM Agent EvaluationBenchmark Validity

  2. Fine-Grained Japanese Clinical Named Entity Recognition: A Controlled Comparison of BERT and Open-Weight Large Language Models

    Nov 17, 2026Feng Han, Ayako Yagahara, Kenji Hirata +2Named Entity RecognitionLLM Information Extraction

  3. Beyond Acuity Accuracy: A Multidimensional Framework for Evaluating Patient-Facing Emergency Triage AI Tools

    Nov 17, 2026Ruio Heerschap, Henry Foote, Mahmoud Alwakeel +19Clinical TriageClinical Language Model Evaluation

  4. Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes

    Nov 15, 2026Indranil Mallick, Paromita RoyClinical Information ExtractionClinical Language Model Evaluation

  5. Offline-First Quantized AI: Design and Evaluation of an On-Device Maternal-Health Assistant for Low-Connectivity Environments

    Nov 15, 2026Okenla AbdulQahhar Abimbola, Maurice Abraham John, Adepoju Jeremiah Moyondafoluwa +10Local LLM DeploymentHealthcare

  6. ROUNDS: Source-Framed Diagnostic Revision after Multi-Turn Clinical Evidence

    Nov 14, 2026Jinrui Fang, Runhan Chen, Yining Wu +6interactive medical diagnosisClinical Language Model Evaluation

  7. Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes

    Nov 14, 2026Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly +4Clinical Language Model EvaluationRubric-Based Evaluation

  8. LABEL: Literature Asymmetry between Efficacy and Limits in Drug Knowledge and Language Models

    Nov 14, 2026S M Rayeed, Charles V. Stewart, Yao MaMedical LLMsClinical Language Model Evaluation

  9. When evaluation becomes intervention in clinical AI

    Nov 13, 2026Anthony Abou Mrad, Aviona Conti, Samuel Schmidgall +10LLM-as-a-JudgeClinical Language Model Evaluation

  10. Do Large Language Models Show Racial and Sex Bias in Pain Management? A Counterfactual Audit of GPT, Claude, Gemini, Llama, DeepSeek and Qwen

    Nov 11, 2026Richard J. Young, Alice M. MatthewsDemographic Bias in Language ModelsAlgorithmic Bias

  11. Hx2Dx: From Anamnesis to Diagnosis - A Multimodal Arena for Conversational Clinical Reasoning in LLMs

    Nov 10, 2026Federico Ravenda, Mirko Cesarini, Daniele Montagnani +3interactive medical diagnosisClinical Language Model Evaluation

  12. HandsOnAid: Building and Evaluating an Offline First-Aid Assistant for Emergency Response Scenarios

    Nov 10, 2026Rezwan Islam Salvi, Saiyma Sittul Muna, Mushfiqur Rahman MushfiqueMedical QAClinical Language Model Evaluation

  13. An evidence-traceable benchmark of clinical reasoning in large language models

    Nov 10, 2026Zhiqun Zhao, Xiaodong Yang, Bruce Gu +13clinical reasoning in language modelsClinical Language Model Evaluation

  14. Curbside Chatbots: Auditing the Mental Health Interventions Large Language Models Recommend to Clinicians, Caregivers, and Patients

    Nov 9, 2026Matthew Flathers, Jill Noorily, Phuong Anh Nguyen +14Medical LLMsLLM Auditing

  15. TrialDesignBench: Benchmarking Language Models for Phase-I Trial Design Against Clinical Knowledge

    Nov 9, 2026Yihan Hank Tang, Peng Yang, Jiayi Xin +5Benchmark DesignClinical Language Model Evaluation

  16. AI in the Clinical Documentation Loop: Effects of LLM-Induced Note Bloat and Diagnostic Overcall on Downstream LLMs

    Nov 7, 2026Jinghui Liu, Anthony Nguyen, Taylan Gurgencierror propagation in large language model inferenceMedical LLMs

  17. Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization

    Nov 6, 2026Magdalini Paschali, Asad Aali, Arogya Koirala +6Clinical Language Model EvaluationAdversarial Robustness of Language Models

  18. TrialJev: Open-Weight Single Pass Eligibility Scoring for Patient Trial Screening

    Nov 6, 2026Omar Ibrahim, Grant Hamblin, Benjamin Haibe-KainsMedical LLMsEfficient Language Model Inference

  19. Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?

    Nov 5, 2026K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson +3LLM-Assisted AnnotationMental Health

  20. Remembering Is Not Enough: Evaluating Timing and Omission in Longitudinal AI Mental Health Support

    Nov 5, 2026Eli Min, Mohit Chandra, Aamogh Sawant +8Mental Health Conversational AgentsMulti-Turn Language Model Safety Evaluation

  21. MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection

    Nov 3, 2026Omar NusratClinical Language Model EvaluationLanguage Model Error Detection

  22. Balancing Chain-of-Thought and Answer Supervision for Small Language Models on Medical Question Answering

    Nov 2, 2026Le Zhou, Xiao Luo, Chinmay Chandra +1Fine-TuningSupervised Fine-Tuning

  23. Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks

    Nov 2, 2026Viju Sudhi, Giulio Billi, Andreea Scrob +2Energy-Aware LLM InferenceLLM Evaluation

  24. From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

    Oct 30, 2026Navapat Nananukul, Mayank KejriwalMedical LLMsMedical QA