cs.LGFeb 18, 2026
SaveLiveClin: A Live Clinical Benchmark without Leakage
Organizations: The Chinese University of Hong Kong, ShenZhen · Ant Healthcare, Ant Group · The Ohio State University · Zhejiang University
Abstract
The reliability of medical LLM evaluation is critically undermined by data contamination and knowledge obsolescence, leading to inflated scores on static benchmarks. To address these challenges, we introduce LiveClin, a live benchmark designed for approximating real-world clinical practice. Built from contemporary, peer-reviewed case reports and updated biannually, LiveClin ensures clinical currency and resists data contamination. Using a verified AI-human workflow involving 239 physicians, we transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway. The benchmark currently comprises 1,407 case reports and 6,605 questions. Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%. In benchmarking against human experts, Chief Physicians achieved the highest accuracy, followed closely by Attending Physicians, with both surpassing most models. LiveClin thus provides a continuously evolving, clinically grounded framework to guide the development of medical LLMs towards closing this gap and achieving greater reliability and real-world utility. Our data and code are publicly available at https://github.com/AQ-MedAI/LiveClin.
Figures & tables
Figure 1: An example from LiveClin simulating the entire clinical pathway. The case progresses from initial assessment to long-term management, with new clinical information and diverse imaging modalities (e.g., X-ray, MRI, pathology, CT) progressively introduced at each key decision point to challenge the model’s reasoning in an evolving scenario.
Figure 2: LLM Case Accuracy across datasets with different publication dates. Shaded regions indicate the knowledge cutoff for different models
Figure 3: Overview of the LiveClin Construction Pipeline. Our three-stage pipeline creates a dynamic and clinically authentic benchmark. (1) Case Construction: We source and sample recent, peer-reviewed case reports to build a contemporary data foundation. (2) Exam Generation: An iterative Generator-Critic architecture transforms static reports into sequential reasoning problems. (3) Quality Check: A synergistic AI-human workflow, featuring AI pre-screening and multi-physician verification, ensures factual and logical integrity.
Figure 4: Distribution of the physician reviewers involved in the quality assurance process. The charts detail the team’s composition by clinical specialty and professional rank (Chief Physician, Associate Chief Physician, and Attending Physician) for both the Annotation and Inspection stages.
Figure 5: Overview of LiveClin’s composition and statistics. The top-left panel details the case attrition funnel from construction pipeline and presents summary statistics of the final benchmark. Figure A shows the distribution of question focus across the entire clinical pathway. Figure B displays the distribution by ICD-10 chapters. Figures C and D detail the distribution of image and table modalities, respectively, highlighting the benchmark’s multimodal nature.
Figure 6: Overall case accuracy, showing models grouped by family and ordered reverse chronologically. Bar textures indicate model type and dashed lines represent physician reference levels.
Figure 7: Fine-grained Performance Analysis of Representative Models. Figure A shows Case accuracy (%) across 16 major ICD-10 chapters. Figure B details Distribution of error proportion (%) across case progression. Figure C displays Question accuracy (%) across five clinical stages.
Figure 8: Performance Analysis across Multimodal Sub-types. Question accuracy (%) on distinct image and table modalities. Modality sub-types are sorted by the average accuracy across all models.
| Method | Accuracy (%) | Trivial Ratio (%) | Time (hrs) † | Cost ($) |
| Physicians | 92.5 | 38.5 | 188.9 | 4534.30 |
| Generator | 84.5 | 16.5 | 0.13 | 35.34 |
| Generator-Critic | 93.0 | 5.5 | 0.45 | 221.69 |
| Generator-Critic-Judge | 89.5 * | 5.5 | 0.55 | 244.19 |
Table 1: Ablation study of exam generation methods. Accuracy is the physician-validated pass rate. Costs are normalized per 100 case reports. † Time denotes person-hours for human authors or API run-time for AI. * The accuracy on the Judge-approved subset is 98.4%.
| Stage I | Stage II | Stage III | Evaluation | Total (Cost/Time) |
| 0 / 0.8 | 3,500 / 0.5 | 45,000 / 9.0 | 5,000 / 1.0 | 53,500 / 11.3 |
Table 2: Estimated cost in USD and time in Days required for each stage of a biannual update cycle of LiveClin. Values are reported as Cost/Time .
| Time | GPT-5 | GPT-4.1 | InternVL 3.5-24B | Gemini 2.5 Pro | Claude 3.5 Sonnet | Qwen2.5-VL-72B | MedGemma-27B | LingShu-32B | HuatuoGPT-V-7B |
| H1 2025 | 35.5 | 31.8 | 34.4 | 33.7 | 27.3 | 17.2 | 15.6 | 20.1 | 9.2 |
| July 2025 | 36.8 | 33.2 | 35.8 | 35.1 | 28.7 | 18.6 | 16.9 | 21.4 | 10.5 |
| Aug 2025 | 34.1 | 30.4 | 33.0 | 32.3 | 26.0 | 15.8 | 14.2 | 18.7 | 8.0 |
Table 3: Monthly stability evaluation of the private leaderboard, showing Case Accuracy (%) for each model across the first half of 2025 (H1 2025), July 2025, and August 2025 sets.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| ICD-10 Chapters (Level 1) | Disease Clusters (Level 2) |
| §1: Infectious & Parasitic | Intestinal Inf. & Tuberculosis (A00-19); Zoonotic bacterial (A20-28); Bacterial, Spirochetal, Chlamydial Inf., etc. (A30-79); Viral Inf. (CNS, HIV, etc.) (A80-B34); Fungal & Protozoal Inf. (B35-64); Parasitic & Helminthic (B65-98) |
| §2: Neoplasms | Malignant Neoplasms of Lip, Oral, Pharynx (C00-14); Digestive Organs (C15-26); Respiratory & Intrathoracic (C30-39); Bone & Cartilage (C40-41); Skin (C43-44); Mesothelial & Soft Tissue (C45-49); Breast (C50); Female Genital Organs (C51-58); Male Genital Organs (C60-63); Urinary Tract (C64-68); Eye, Brain & CNS (C69-72); Thyroid & Endocrine (C73-75); Lymphoid Tissues (C76-96); Benign, In Situ. (C97, D00-48) |
| §3: Blood, Blood-forming Organs & Immune Mechanism | Nutr, Hemol& Aplast Anemias (D50-64); Coagulation Defects, Purpura & Hemorrhagic Cond. (D65-69); Blood-Forming Organ (D70-77); Immune Mechanism Disord. (D80-89) |
| §4: Endocrine, Nutritional & Metabolic | Thyroid Disorders. (E00-07); Diabetes mellitus (E10-14); Endocrine, Glucose Reg. & Nutritional Deficiencies (E15-64); Obesity & Metabolic Disorders (E65-90) |
| §5: Mental & Behavioural Disord. | Comprehensive Mental & Behavioural Disord. (F00-98) |
| §6: Nervous System | Inflammatory & systemic atrophic of CNS (G00-14); Extrapyramidal, Movement, & Degen. Disord. (G20-32); Demyelinating of CNS (G35-37); Episodic & paroxysmal disord. (G40-47); Nerve root & plexus disord. (G50-59); Polyneuropathies & other PNS disord. (G60-64); Muscle, Myoneural, & Paralytic Disord. (G70-99) |
Table 4: Hierarchical Taxonomy of the LiveClin Benchmark. The table displays the first two levels of the taxonomy. Level 3, the most granular tier, corresponds to the specific ICD-10 codes.
Figure 9: The prompt designed to classify a case report into one of the 16 high-level clinical chapters.
Figure 10: The prompt for the second classification stage. It uses the Level-1 result to provide a constrained set of options for Level-2 and Level-3 classification.
Figure 11: The prompt for the Generator agent, which initiates the creation of the clinical examination by converting a case report into a progressive set of MCQs.
Figure 12: The prompt for the Critic agent, designed to systematically evaluate the generated MCQs for clinical accuracy and cognitive complexity.
Figure 13: The prompt used during the iterative refinement loop. It provides the Generator with the Critic ’s feedback to guide the revision process.
Figure 14: The prompt for the Judge agent, which enforces a strict two-part validation focused on factual correctness and logical solvability.
Figure 15: The prompt provided to native Chinese-speaking physicians to validate the consistency between AI-generated and human-expert translations.
| Loop 1 | Loop 2 | Loop 3 | |
| Case Num | 1286 (91.4%) | 109 (7.8%) | 12 (0.9%) |
Table 5: Case counts and proportions for each revision loop in physician verification.
| Major Specialty (Total, %) | Subspecialty | Count |
| Internal Medicine (95, 39.8%) | Cardiology | 8 |
| Nephrology | 4 | |
| Gastroenterology | 7 | |
| Respiratory Medicine | 5 | |
| Hematology | 5 | |
| Endocrinology | 3 |
Table 6: Specialty distribution of the 239 physicians in the verification cohort.
Figure 16: The expert-defined logic used to cluster granular, AI-generated stage labels into the five primary clinical workflow categories.
Figure 17: The prompt used to instruct the o3 model to perform initial classification of images and tables.
Figure 18: The expert-defined logic used to group specific, validated modality labels into the broader categories shown in the final statistics.
| Clinical Pathway Stage | 0–20% | 20–40% | 40–60% | 60–80% | 80–100% |
| Proportion | 62.0% | 46.0% | 35.5% | 27.5% | 38.5% |
Table 7: Proportion of multimodal questions across clinical pathway stages, representing the percentage of questions requiring multimodal interpretation within each stage.
Figure 19: The conversational prompt structure used for evaluation. Context from previous turns is cumulatively added for subsequent questions within the same case.
| Major Specialty | Subspecialty | Count |
| Internal Medicine (31) | Cardiology | 3 |
| Nephrology | 1 | |
| Gastroenterology | 2 | |
| Respiratory Medicine | 2 | |
| Hematology | 2 | |
| Endocrinology | 1 |
Table 8: Specialty distribution of physicians in the baseline evaluation cohort, proportionally scaled from the full verification team to 78 participants.
| Model | Run 1 (%) | Run 2 (%) | Run 3 (%) | Std. Dev. |
| GPT-5 | 35.5 | 35.4 | 35.6 | 0.22 |
| o3 | 35.7 | 35.2 | 35.9 | 0.36 |
| GPT-4.1 | 31.8 | 31.7 | 31.9 | 0.10 |
| GPT-4.1-mini | 26.1 | 26.0 | 26.1 | 0.00 |
| GPT-4o | 14.7 | 14.7 | 14.8 | 0.12 |
| GPT-4o-mini | 14.8 | 14.8 | 14.9 | 0.08 |
Table 9: Benchmark stability across three independent runs. Scores are reported as Case Accuracy (%), with standard deviations across runs.
| Model | Zero-shot (%) | One-shot (%) | Three-shot (%) |
| GPT-5 | 35.5 | 35.8 (+0.3) | 35.1 (-0.4) |
| GPT-4.1 | 31.8 | 31.5 (-0.3) | 32.3 (+0.5) |
| InternVL 3.5-24B | 34.4 | 34.7 (+0.3) | 34.9 (+0.5) |
| Gemini-2.5 Pro | 33.7 | 33.1 (-0.6) | 32.9 (-0.8) |
| Claude-3.5 Sonnet | 27.3 | 27.0 (-0.3) | 27.9 (+0.6) |
| Qwen2.5-VL-72B | 17.2 | 17.4 (+0.2) | 17.0 (-0.2) |
Table 10: Case Accuracy (%) under different shot settings. Values in parentheses denote changes relative to zero-shot performance.
Figure 20: Instruction prompt for physicians during attribution analysis.
Figure 21: Instruction prompt for physicians during rarity annotation.
| Loop Num | 1 | 2 | 3 |
| Case Number (Proportion) | 1276 (90.7%) | 113 (8.0%) | 18 (1.1%) |
Table 11: Inter‑annotator agreement across annotation loops for rarity classification. Case counts are shown with their proportion in parentheses.
| Model | Rare (%) | Unrare (%) | Overall (%) | |
| GPT-5 | 35.65 | 35.05 | -0.60 | 35.5 |
| O3 | 35.99 | 34.11 | -1.88 | 35.7 |
| GPT-4.1 | 31.92 | 30.84 | -1.08 | 31.8 |
| GPT-4.1-mini | 26.32 | 25.00 | -1.32 | 26.1 |
| GPT-4o | 14.23 | 17.76 | 3.53 | 14.7 |
| GPT-4o-mini | 14.15 | 19.34 | 5.19 | 14.8 |
Table 12: Model Case Accuracy (%) on rare (1181 cases, 84%) and unrare (226 cases, 16%) subsets. denotes the difference between unrare and rare performance.
More in this topic
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 418 papers