cs.CLSep 29, 2026

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

Authors: Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati, Vallijah Subasri, Caihua Shan

Organizations: Zhejiang University · Imperial College London · Nanchang University · University of Toronto · Microsoft

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment.

Figures & tables

Figure 1: How models are evaluated in KlinikeBench. A clinician authors the patient profile and hidden scoring key. The evaluated agent interacts with an LLM-simulated patient and executes clinical tools in a sandbox. The verifier scores the transcript for diagnosis, required tool use, action constraints, and must-ask coverage. This example fails because the diagnosis is incorrect and required tools and history topics are missed. Tool arguments, simulator messages, and tool checks are abbreviated.
BenchmarkMetricInteractionToolsSourceClinician verification
MedQA ( 2021 )MCQA acc.Single turnNoneBoard examsExam keys
MMLU-med. ( 2020 )MCQA acc.Single turnNoneExam materialSource answers
MedMCQA ( 2022 )MCQA acc.Single turnNoneEntry examsExpert questions
HealthBench Pro. ( 2026 )RubricNext replyVariesClinical chats≥\geq 3 raters/item
AgentClinic ( 2026 )Diag. acc.20 turnsTests + aidsClinical casesDialogue ratings
MedAgentBench ( 2025 )Task successMulti-stepFHIR APIsEHRsTask authoring
Table 1: Comparison with medical benchmarks and clinical-agent evaluations. KlinikeBench verification uses a survey of clinician judgments.
Figure 2: Clinicians author and revise tasks through live simulation. In a separate assessment, virtual-patient dialogues receive higher mean quality ratings than references (shown in Table 3 ).
Figure 4
ModelStrict pass@1DiagnosisToolsMust-ask
Claude Opus 5 ( Anthropic, 2026b )29.4%90.7%44.1%89.9%
Claude Opus 5.5 ( Anthropic, 2026c )28.8%90.7%49.8%84.4%
Claude Opus 4.8 ( Anthropic, 2026a )16.8%89.2%27.9%85.5%
Claude Sonnet 5 ( Anthropic, 2026e )13.8%87.4%24.0%85.1%
Claude Haiku 4.5 ( Anthropic, 2025a )2.7%72.1%8.1%70.6%
GPT-6-astra ( OpenAI, 2026f )24.3%87.4%39.3%88.0%
Table 2: Performance of 18 models on 333 tasks; all 31 appear in Table 7 . Scores are percentages: diagnosis accuracy, required-tool completion (Tools), and must-ask coverage. Strict pass@1 requires all criteria ( Section 2.2 ). Bold : overall best; underlining : best within families with multiple models. Citations link model reports or cards.
Figure 6
Figure 7
Figure 5: Each point ranks a clinician by strict pass@1 over 50 tasks under the indicated judge (a) or patient (b). Mini: GPT-5.4-mini; Gemini: Flash variants. Ties are slightly offset; full scores: Table 9 .
DimensionVirtual patientMTS-Dialog
Progressive history disclosure3.69±0.963.69\pm 0.963.43±1.053.43\pm 1.05
Emotional appropriateness3.50±0.963.50\pm 0.963.31±1.083.31\pm 1.08
Human-likeness3.76±1.073.76\pm 1.073.45±1.193.45\pm 1.19
Table 3: Clinician ratings by conversation source (mean ±\pm SD, scale 1–5). Thirty clinicians provide 1,050 ratings per dimension for 35 virtual-patient conversations and 450 for 15 MTS-Dialog references ( Abacha et al., 2023 ) . SD summarizes individual ratings.
Figure 6: Qwen3-4B diagnosis during GRPO. Left: five-step training means with ±\pm SD bands. Right: held-out accuracy with ±\pm SE bars (50 tasks).
Figure 7: Two passing trajectories on the same held-out case. Qwen3-4B (SFT + GRPO) and GPT-6-astra (zero-shot) satisfy the reported gates but differ in dialogue coherence and diagnosis recording. Excerpts are verbatim with omissions; the patient is Claude Opus 4.6.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Task-authoring interface. The illustrated PrimeKG view supports disease search and a guided sequence of disease selection, symptom selection, differentiating findings, LLM refinement, and clinician review. Workspace counters describe the authoring workspace.
Figure 9: Live-simulation and review interface. Clinicians inspect the case context, dialogue, execution counters, and tool calls, and annotate individual turns before returning to the editor. The displayed settings belong to this authoring session; benchmark evaluation follows Section 3.1 .
CorpusTotalPer taskMeanMedianMin–Max
Clinical tasks333Must-ask topics4.540–26
Distinct tool names140Required tools5.851–19
Clinical subcategories17Exposed tools10.7103–24
Appendix
Table 4: Corpus totals and per-task statistics.
FunctionNamesExample toolsExecution behavior
Patient records6get_patient_info , get_patient_history , get_patient_vitalsReturn demographics, history, medications, allergies, family history, or vital signs.
Tests and imaging3order_lab_test , get_lab_results , order_imagingStore orders and retrieve available laboratory results.
Treatment and referral3prescribe_medication , refer_to_specialist , check_drug_interactionsStore prescriptions or referrals; return an interaction-check response.
Diagnosis recording2record_diagnosis , record_differentialStore the final diagnosis or differential; only the final diagnosis ends the encounter.
Clinical knowledge4get_treatment_guidance , get_disease_symptomsExpose knowledge-query interfaces; return unavailable status in the current headless runtime.
Encounter memory3take_note , set_working_diagnosis , summarize_history_takenStore notes and interim hypotheses, or summarize questions already asked.
Appendix
Table 5: Clinical tool catalogue. Counts refer to distinct exposed names across the corpus. The first six rows partition the 21 built-in tools; the final three describe the 119 additional names. Examples are representative, not exhaustive.
SettingSFTRL (GRPO)
InitializationQwen3-4BSFT checkpoint
Learning rate10−510^{-5}10−610^{-6}
ScheduleCosine; 5% warm-upConstant
Batch32 sequences (global)8 tasks ×\times 8 rollouts
Training duration3 epochs315 steps planned; results through 30
Sequence limit20,480 tokens32k tokens
Appendix
Table 6: Training settings for Qwen3-4B. The two RL runs share these settings and differ only in the optimized reward. The SFT patient and judge generate and score the demonstration trajectories.
ModelStrict pass@1DiagnosisToolsMust-ask
Claude Opus 5 ( Anthropic, 2026b )29.4%90.7%44.1%89.9%
Claude Opus 5.5 ( Anthropic, 2026c )28.8%90.7%49.8%84.4%
Claude Opus 4.8 ( Anthropic, 2026a )16.8%89.2%27.9%85.5%
Claude Sonnet 5 ( Anthropic, 2026e )13.8%87.4%24.0%85.1%
Claude Sonnet 4.5 ( Anthropic, 2025b )9.9%82.3%25.8%76.7%
Claude Sonnet 4.6 ( Anthropic, 2026d )8.4%83.8%22.8%73.8%
Appendix
Table 7: Results (%) for 31 models on 333 tasks (metrics: Section 2.2 ). Strict pass@1 requires correct diagnosis ∧\land complete clinical actions ∧\land full must-ask coverage. Bold : overall best; underlining : best within families with multiple models.
Figure 10: Strict pass@1 versus estimated clinician cost per task for the 18 models with pricing data. Costs use the workbook’s average input, cached-input, and output tokens and listed token rates; patient and judge costs are excluded. The cost axis is logarithmic. Colors indicate model families.
ModelStep cappass@1DiagnosisToolsMust-ask coverageΔ\Delta [95% CI]
GPT-6-astraDefault28864688.9—
 4534886090.4+6  [−6,20]+6\;[-6,20]
 6032865691.2+4  [−8,16]+4\;[-8,16]
 None36885289.0+8  [−2,20]+8\;[-2,20]
Claude Opus 5Default34865489.8—
 4536846290.4+2  [−10,16]+2\;[-10,16]
Appendix
Table 8: Step-cap ablation for five clinicians on 50 matched tasks. Scores are percentages; Δ\Delta is the change in strict pass@1 from the default run, in percentage points, with the reported paired-bootstrap 95% interval. The ten-patient-turn limit remains fixed.
ConditionOP5G6GR4.7GE3.8OP5.5G5.6S5GE3.7
Judge changed; patient fixed to GPT-5.4-mini
GPT-5.4-mini (original)24302020261888
GPT-5.4-mini (repeat)26282420261866
GPT-5.4243220203224106
Gemini-3.5-flash22201812241466
Grok-4.522281814281686
Appendix
Table 9: Strict pass@1 (%) for all patient and judge substitutions. OP5/OP5.5: Claude Opus 5/5.5; S5: Claude Sonnet 5; G6/G5.6: GPT-6-astra/GPT-5.6-sol; GR4.7: Grok-4.7; GE3.8/GE3.7: Gemini-3.8/3.7-flash. Dashes mark unavailable scores.
StageStrict pass@1DiagnosisRequired toolsMust-ask
Base0.0%26.0%3.0%39.7%
SFT8.0%39.0%33.0%77.5%
SFT + RL (diagnosis)10.0%44.0%32.0%76.8%
SFT + RL (multi-gate)6.0%42.0%32.0%73.1%
Appendix
Table 10: Qwen3-4B on 50 held-out tasks. Base and SFT pool two rollouts per task; each RL checkpoint uses one. RL rows report step 30. All scores are percentages.
Figure 11: Qwen3-4B gate trajectories through GRPO step 50. The top row shows diagnosis and required-tool completion; the bottom row shows must-ask coverage and strict outcome. Blue and orange denote diagnosis-only and multi-gate rewards. Solid curves with shaded bands show five-step training means ±\pm SD; dashed curves with circles and error bars show held-out results ±\pm SE. Horizontal dotted and dash-dotted lines mark the zero-shot and SFT held-out baselines. Main-text comparisons use checkpoints through step 30.

Explore similar work

CardsList
Apr 30, 2026cs.CL

HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

Millions of clinicians use ChatGPT to support clinical care, but evaluations of the most common use cases in model-clinician conversations are limited. We introduce HealthBench Professional, an open benchmark for evaluating large language models on real tasks that clinicians bring to ChatGPT in the course of their work. The benchmark is organized around three common use cases central to clinical practice: care consult, writing and documentation, and medical research. Each example includes a physician-authored conversation with ChatGPT for Clinicians and is scored via rubrics written and iteratively adjudicated by three or more physicians across three phases. HealthBench Professional examples were carefully selected for quality, representativeness, and difficulty for OpenAI's current frontier models, to enable continued measurement of progress. Difficult examples for recent OpenAI models were enriched by roughly 3.5 times relative to the candidate pool of 15,079 examples. Additionally, about one-third of examples involve physicians conducting deliberate adversarial testing of models. As a strong baseline, we also collected human physician responses for all tasks (unbounded time, specialist-matched, web access). The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians. We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care.

Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim +13

OpenAI

Clinician TrustLarge Language Model BenchmarksPhysical Activities
AskPDF
Jul 28, 2026cs.CL

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating of diagnostic hypotheses, and continuous refinement of clinical reasoning. However, existing evaluations of multimodal large language models (MLLMs) typically rely on single-turn or isolated tasks, making it difficult to fully capture the complexity of real-world clinical diagnosis. To bridge this gap, we developed ClinMM-Bench, the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date. ClinMM-Bench contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties. We systematically evaluated 15 representative MLLMs using a two-level evaluation framework that assessed both diagnostic accuracy and diagnostic reasoning quality. Results showed that proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited across all models. In terms of diagnostic reasoning quality, current models can identify plausible diagnostic directions but still have considerable limitations in generating reliable diagnostic reasoning. Error analysis further identified five representative failure modes: information synthesis failure, knowledge mapping error, perception error, premature closure, and visual hallucination.

Rui Yang, Weihao Xuan, Yi Lin +23

Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA +24

Multimodal Clinical DataDiagnosis
AskPDF
Jun 2, 2026cs.MA

MeDxAgent: Multi-Agent Consultation for Interactive Medical Diagnosis

Large language models (LLMs) are increasingly used for health-related decision support. Yet most evaluations treat diagnosis as a single-shot task with complete information provided upfront, often as a multiple-choice selection. This diverges from clinical practice, where diagnosis is interactive and open-ended, involving sequential hypothesis refinement through targeted questioning. We address this gap. We build MeDxBench, a large-scale benchmark of 4,421 clinical cases across 20 specialties. We further propose MeDxAgent, a multi-agent consultation system for interactive diagnosis, and systematically study its prompt-, flow- and agent-level design choices. MeDxAgent achieves a 10.3% accuracy gain over the baseline on MeDxBench, closing 52.3% of the gap to a full-information oracle. We find that specific design choices: collecting demographics first, passing summarized dialogue for diagnosis, and feeding candidate diagnoses for targeted questioning, improve accuracy, mirroring how physicians reason, though their effect emerges fully only in combination. Code and dataset will be released upon publication.

Akshat Sanghvi, Naren Akash, Raza Imam +2

Microsoft Research India · Institute of AI and Innovation, AIG Hospitals · Mohamed bin Zayed University of Artificial Intelligence

DiagnosisMultiple-Choice Questions
AskPDF