Clinician use of language models diverges from how the models are evaluated

Krithik Vishwanath; Haitong Lin; Anton Alyakin; Jin Vivian Lee; D. Brock Hewitt; Jie J. Yao; William Robert Small; Hammad A. Khan

cs.CLOct 8, 2026

Clinician use of language models diverges from how the models are evaluated

Authors: Krithik Vishwanath, Haitong Lin, Anton Alyakin, Jin Vivian Lee, D. Brock Hewitt, Jie J. Yao, William Robert Small, Hammad A. Khan, +14 more

Organizations: Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Department of Technology Management, NYU Tandon School of Engineering, New York University, New York, NY, USA · Washington University School of Medicine, St. Louis, MO, USA · Global AI Frontier Lab, New York University, New York, NY, USA · Department of Surgery, NYU Langone Health, New York, NY, USA · Department of Orthopedic Surgery, NYU Langone Health, New York, NY, USA · Department of MCIT Health Informatics, NYU Langone Health, New York, NY, USA · Department of Medicine, NYU Langone Health, New York, NY, USA · Division of Dermatology, Department of Medicine, NYU Langone Long Island, Mineola, NY, USA · Johns Hopkins University School of Medicine, Baltimore, MD, USA · Department of Cardiothoracic Surgery, Stanford University School of Medicine, Stanford, CA, USA · Department of Pathology, NYU Grossman School of Medicine, New York, NY, USA · Biomedical Data Science Hub, NYU Langone Health, New York, NY, USA · Department of Management and Organizations, NYU Stern School of Business, New York, NY, USA · Faculty of Medicine, Macau University of Science and Technology, Taipa, Macao, China · Department of Big Data and Biomedical AI, College of Future Technology, Peking University and Peking-Tsinghua Center for Life Sciences, Beijing, China · Department of Population Health, NYU Langone Health, New York, NY, USA · Department of Neuroscience, NYU Langone Health, New York, NY, USA · Department of Radiology, NYU Langone Health, New York, NY, USA · Neuroscience Institute, NYU Langone Health, New York, NY, USA · Center for Data Science, New York University, New York, NY, USA

Abstract

Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. A benchmark predicts performance in deployment only to the extent that its items resemble real use, yet whether benchmarks reflect the work these systems receive has rarely been measured. Here we analyze 127,833 queries sent by 6,342 physicians, advanced practice providers and nurses in 35 specialties to an institutional assistant during an eight-month roll-out. We characterize each query with RCQ-Map, a clinician-validated framework grounded in taxonomies of clinical questions and of LLM evaluation, which records its task, intent, answerability, missing information and potential harm. Documentation and administration (36.2%) and knowledge retrieval (28.9%) made up nearly two-thirds of use, and diagnosis 3.7%; more than a third of queries could not be answered well as posed. Applying RCQ-Map to 58 public benchmarks drawn from major evaluation suites and frontier model reports, which we assemble into the Clinical AI Benchmark Atlas, showed that the median benchmark contained no documentation requests and shared 31% of the task mix of real use, less than an even spread across task categories would. Benchmarks in suites designed to resemble clinical practice were individually no closer to real use than those used in frontier model reports. Benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given, and evaluation should be matched to real clinical use.

Figures & tables

Figure 1

	All	Attending physicians	Fellows	Residents	APPs	Registered nurses
Clinicians, n	6,342	1,664	157	771	946	2,804
Conversations, n (%)	127,833 (100.0)	37,383 (29.2)	2,121 (1.7)	23,630 (18.5)	23,906 (18.7)	40,793 (31.9)
Conversations per clinician, median (IQR)	4 (1–16)	4 (1–20)	4 (1–13)	8 (2–33)	6 (2–23)	3 (1–11)
Specialties represented, n	35	35	19	19	27	—
Kind of work (%)
Documentation & administration	36.2	32.1	37.3	38.0	35.5	39.1

Table 2

Figure 4

Figure 6

Figure 7

Figure 8

Figure 9

Figure 11

Block	Field (values)	What it captures	Grounding
Task & domain	Task category (10)	The deliverable that would satisfy the request: drug information; treatment and management; foundational knowledge; patient education; test and result interpretation; documentation and workflow; diagnosis and differential; coding and administrative; procedural guidance; other	Generic clinical-question taxonomies 44,49,51 ; clinical-information-system needs 50,53 ; MedHELM task families 16 ; LLM usage studies 22,26
	Question intent (12; 5 groups)	The motive behind the task (verification, fact look-up, clinical decision, dosing, documentation drafting, definition, mechanism, comparison, procedure, coding, result interpretation, other)	Web-search intent 56 ; generic question stems 44 ; well-built clinical question 57 ; question pursuit 43
	Clinical department (25); Medicine division (15)	The department (and Medicine division) that would own management, via a precedence ladder	Institutional ownership for governance and oversight (Methods)
	AMA use case (8)	Mapping to the AMA list of physician AI use cases, plus none of the above	AMA physician survey 1
Query properties	Patient-specific (binary)	About an identifiable patient	Patient-specific needs 49,53
Context	Patient or local information; current evidence (binary)	Information a safe answer needs that the query lacks	Unmet patient-specific needs 53 ; EHR grounding 58 ; resource gaps 59 ; retrieval augmentation 29,30

Table 15

Field	Krippendorff’s $\alpha$ (95% CI)	Gwet’s AC1	Pairwise agreement (%)	Unanimous (%)	Held-out clinician vs reference (%)	GPT-5.6 sol vs reference (%)	Difference, pts (95% CI)
Task & domain
Task category	0.64 (0.55–0.73)	0.67	70	59	84	79	-5 (-14 to 4)
Clinical domain	0.48 (0.38–0.57)	0.60	61	48	79	73	-6 (-17 to 4)
Medicine division	0.43 (0.33–0.52)	0.59	61	45	74	73	-2 (-12 to 8)
Question intent	0.45 (0.36–0.54)	0.51	55	38	69	67	-2 (-16 to 10)
Question intent (five groups)	0.61 (0.50–0.70)	0.68	74	62	84	82	-2 (-11 to 7)

Table 16

Variable	Model 1: OR (95% CI)	Model 2: OR (95% CI)
Concerns a specific patient	2.91 (2.64–3.22)	3.20 (2.88–3.55)
Names a medication	3.12 (2.89–3.37)	1.30 (1.18–1.44)
States a dose	1.13 (0.97–1.31)	1.93 (1.70–2.18)
Cites a laboratory or test result	0.93 (0.85–1.01)	0.99 (0.90–1.09)
Mentions imaging	0.62 (0.53–0.73)	0.76 (0.68–0.86)
Vulnerable population	1.24 (1.08–1.42)	1.37 (1.24–1.52)

Table 17

Explore similar work

Sep 29, 2026cs.CL

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment. All the code and data is available on https://zehui127.github.io/klinikebench/

Xueting Fang, Zehui Li, Yang Yang +5

Zhejiang University · Imperial College London · Nanchang University +2

Clinical Decision-Making Multi-Turn Dialogue Evaluation Clinical Reasoning Clinical Language Model Evaluation

Ask PDF

Jul 1, 2026cs.CL

Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention

Background: Expert-annotated benchmarks for non-English open-response clinical questions are scarce. LLM-as-a-judge systems may scale evaluation but require validation. Objective: To introduce MedQADE, a standardized German open-response clinical benchmark with physician reference annotations, and evaluate LLM-as-a-judge alignment, self- and intra-family bias, and abstention. Methods: The benchmark contains 3,800 question-answer sets with answers from five student LLMs and annotations from 10 physicians. All 10 rated the 200-question core; two primary raters assessed each of 3,600 extension questions, with the tenth resolving disagreements. Nine LLM evaluators assessed all sets. We assessed physician reliability, student-model accuracy, evaluator alignment, bias, and abstention. Results: Physicians showed moderate-to-substantial agreement on answer correctness (unweighted mean pairwise Cohen's kappa = 0.612) but limited agreement on question difficulty (Krippendorff's alpha = 0.208 using squared numeric-score distances). Student-model accuracy was 17.8%-66.0% and generally decreased with physician-rated difficulty. Gemini 3 Flash approached the leave-one-out physician reference (kappa = 0.694 vs 0.709). Four of five models rated their own responses more favorably than out-of-family evaluators; five of six intra-family comparisons were positive. Physician abstention increased with perceived difficulty. Seven of nine LLM evaluators abstained in no more than 0.51% of evaluations; the two strongest evaluators assigned definitive labels to every response. Conclusions: Strong LLM evaluators approached physician agreement, but evaluator bias and low observed abstention warrant physician validation and further assessment of selective deferral before fully automated evaluation. These results do not establish clinical safety.

William Philipp, Finn Fassbender, Daniel Fister +12

Section for Clinical Research-IT, Institute of Medical Biometry and Statistics, University of Luebeck, Luebeck, Germany · Medical Faculty, University of Tübingen, Tübingen, Germany · German Medical Students’ Association (bvmd), Berlin, Germany +9

Multilingual Language Model Evaluation LLM-as-a-Judge Selective Prediction Clinical QA Language Model Bias Evaluation Clinical Language Model Evaluation

Ask PDF

Apr 30, 2026cs.CL

HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

Millions of clinicians use ChatGPT to support clinical care, but evaluations of the most common use cases in model-clinician conversations are limited. We introduce HealthBench Professional, an open benchmark for evaluating large language models on real tasks that clinicians bring to ChatGPT in the course of their work. The benchmark is organized around three common use cases central to clinical practice: care consult, writing and documentation, and medical research. Each example includes a physician-authored conversation with ChatGPT for Clinicians and is scored via rubrics written and iteratively adjudicated by three or more physicians across three phases. HealthBench Professional examples were carefully selected for quality, representativeness, and difficulty for OpenAI's current frontier models, to enable continued measurement of progress. Difficult examples for recent OpenAI models were enriched by roughly 3.5 times relative to the candidate pool of 15,079 examples. Additionally, about one-third of examples involve physicians conducting deliberate adversarial testing of models. As a strong baseline, we also collected human physician responses for all tasks (unbounded time, specialist-matched, web access). The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians. We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care.

Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim +13

OpenAI

Healthcare Clinical Decision Support Clinical Language Model Evaluation Language Model Robustness Medical LLMs

Ask PDF