Patient-specific rubric-based evaluation of large language models across longitudinal cancer care
Authors: Wei-Wei Zhang, Pengkai Wang, Yan Li, Zhijie Sang, Guan-Qun Zhou, Jun-Yan Li, Shu-Han Zhou, Pei-Wen Liu, Ting-Xi Tang, Min Tang, Zhitian Hou, Jianmin Wu, Jian Guan, i Lin, Hongxia Yang, Ying Sun
Organizations: Department of Radiation Oncology, Sun Yat-sen University Cancer Center, Guangzhou, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Pathology, Sun Yat-sen University Cancer Center, Guangzhou, China. · Department of Radiation Oncology, Nanfang Hospital, Southern Medical University, Guangzhou, China. · InfiX.ai, Hong Kong, China.
Large language models (LLMs) show promise in specialty medicine, yet existing evaluations largely focus on isolated clinical encounters rather than tracking how clinical responsibilities evolve across a longitudinal care pathway. Here we introduce NPC-RubricBench, a phase-aware benchmark for evaluating multistage clinical reasoning across the cancer care continuum, using nasopharyngeal carcinoma as a defined disease context. Curated from longitudinal records of 405 patients across two cancer centers, the benchmark comprises 941 patient-phase units across six sequential care phases and 17,688 patient-specific atomic criteria spanning five evaluation dimensions, with each assessment strictly restricted to decision-time clinical evidence. Evaluating 30 general-purpose and medical LLMs revealed that strong diagnostic performance does not carry over to downstream care. We evaluated 30 general-purpose and medical LLMs, showing that high diagnostic performance did not translate into improved downstream care. Instead, the dominant failure modes reversed across phases. Explicit decision errors dominated anatomical staging at 48.2%, whereas decision omissions dominated initial work-up and post-treatment follow-up at 33.4% and 30.4%, respectively. In the evaluated model set, medically adapted models showed higher omission rates than general-purpose models. Moreover, varying query granularity exposes a critical specification trade-off: broad open-ended requests degrade Decision Quality across all 30 models, while more detailed, checklist-like prompts raise Decision Quality but reduce Evidence Faithfulness and Safety Adherence. Furthermore, NPC-RubricBench demonstrates that static diagnostic benchmarks are insufficient to ensure consistent clinical reliability over time, highlighting the importance of phase-adaptive safeguards in multistage medical AI.
Figures & tables
Figure 1: NPC-RubricBench design and composition. a, Longitudinal evaluation setting. Longitudinal records are segmented into six ordered care phases. Patients are counted once across the internal and external cohorts, and each patient–phase unit is counted once. Not every patient contributes every phase. The staging example illustrates uncertainty from a limited biopsy. b, Clinical scenarios. Stacked bars show patient–phase counts across the defined clinical scenarios, using the same inventory as a and a shared absolute-count scale. Definitions and counts are provided in S1.4 and Table S1-B. c, Patient-specific atomic rubrics. Condensed examples illustrate patient-specific criteria for Decision Quality (DQ), Evidence Grounding (EG), Status Determination (SD), Evidence Faithfulness (EF) and Safety Adherence (SA). Schematic outcomes are grouped as Met/Partial/Error/Omission for DQ/EG/SD (upper) and Met/Not met for EF/SA (lower). Partial is permitted only for items with an explicit item-specific partial-credit allowance. Met/Not met denote positive requirements satisfied/unmet, with score factors 1/0. Rubric construction and patient-specific examples are described in S2.1 and S2.3. d, Rubric scale and composition. Annulus sector area encodes rubric-item counts by phase and dimension; n gives phase totals. Counts are of unique patient-specific rubric items. NPC, nasopharyngeal carcinoma; EBV, Epstein–Barr virus; EBER, EBV-encoded RNA; RT, radiotherapy; CCRT, concurrent chemoradiotherapy.
Figure 2: Performance across care phases and rubric dimensions. a, Phase-mean distributions. Phase-mean score distributions for 30 models under Focused formulation (text) in the main clinical evaluation set (internal and external cohorts). Scores use dimension-eligible patient–phase units on a bounded 0–100 scale. Violins span observed values; marks show the minimum, median and maximum. Wu, Work-up; Dx, Diagnosis; St, Staging; Tr, Treatment; Tx, Toxicity; Fu, Follow-up. b, Decision quality. DQ means for eight author-selected models, ordered as API-accessed general-purpose, locally deployed open-weight general-purpose and locally deployed open-weight medically adapted groups, with descending overall DQ within each group. c, Evidence grounding. EG means for the same models. d, Status determination. SD means for the same models. Panels b–d share a fixed model order and colours with Figs. 3 and 4c,d; labels are rounded scores. Complete phase-specific scores and denominators are reported in Table S5. DQ, Decision Quality; EG, Evidence Grounding; SD, Status Determination; EF, Evidence Faithfulness; SA, Safety Adherence. Model abbreviations: I, Instruct; T, Thinking/Think. Full names are listed in Table S3.
Figure 3: Model phase profiles and concept violations. a, Grouped model phase profiles. Grouped radars show five-dimensional scores across six phases for the same eight models as Fig. 2b–d and Fig. 4c,d, in API-accessed general-purpose, locally deployed open-weight general-purpose and locally deployed open-weight medically adapted groups. Colours are shared with Fig. 2; outer sectors identify dimensions. b, EF concept violations. EF concept-violation rates: the percentage of resolved, eligible main-evaluation patient–phase units with at least one applicable EF requirement judged Not met. Units with all applicable requirements judged Met remain in the denominator. All phases use one fixed model order. c, SA concept violations. SA concept-violation rates use the same unit and Not met event rule as b. d, Individual model phase profiles. Individual radars show the same scores as a, in the shared model order. Wu, work-up; Dx, diagnosis; St, staging; Tr, treatment; Tx, toxicity; Fu, follow-up. Panels b,c use full phase names. Model abbreviations: I, Instruct; T, Thinking/Think. Full names are listed in Table S3.
Figure 4: Six-class failure analysis across care phases and models. a, Six-class failure taxonomy. The six classes are Decision omission, Decision error, Evidence omission, Evidence error, Status-assessment omission and Status-assessment error. EF and SA violations are summarized separately. b, Error and omission occurrence by phase. Phase-specific item-level rates pool all 30 models in the main clinical evaluation set (internal and external cohorts). Error or Omission counts are divided by the preset applicable Rubric item total for each phase and DQ, EG or SD dimension. Met and Partial remain in the denominator; EF and SA are outside this inventory. Phase-level summaries are reported in Table S11; failure-rate definitions are provided in S7.3. c, Error and omission profiles by model. Bars show omission and error rates using each model’s preset Rubric opportunities pooled across phases and DQ, EG and SD, for the same eight models and order as Figs. 2 and 3. Pooled estimates and 95% confidence intervals for all 30 models are reported in Table S10. Confidence intervals use a patient-cluster bootstrap with 1,000 resamples. d, Selected errors and omissions across models. Bars show prevalence point estimates for the same eight models in four selected phase-by-failure examples. Full model-by-phase failure profiles are provided in Suppl. Fig. 3. Model abbreviations: I, Instruct; T, Thinking/Think. Full names are listed in Table S3.
Figure 5: Sensitivity to clinical task formulation. a, Clinical task formulation. Broad formulation, Focused formulation and Elaborated formulation prompts from the staging example; clinical records and base rubrics are held fixed. b, Paired score directions. Direction of paired score changes relative to Focused formulation in DQ, EG and SD across six care phases. Each segment is the arithmetic mean of within-model percentages across 30 models. For each phase, dimension and comparison, all models share the same set of cases with scores for both formulations. Unchanged denotes an absolute score difference of at most 10−8 points. Wk, Work-up; Dx, Diagnosis; St, Staging; Tr, Treatment; Tx, Toxicity; Fu, Follow-up. c, Secondary directions within DQ branches. Directions of EG, SD, EF and SA changes within selected DQ increase, unchanged or decrease branches. Percentages are calculated within each model on the common cases valid for DQ and the secondary dimension, then averaged equally over models with a nonzero DQ-branch denominator. These are conditional percentages, not percentages of all paired responses. In b,c, bottom, middle and top segments indicate increase (orange hatch), unchanged (grey) and decrease (blue hatch), respectively. Integer labels and outlined segments in b,c identify author-selected descriptive patterns.
Figure 6: Cross-cohort concordance and expert–judge agreement. a, Lin’s concordance correlation coefficients (CCC) between internal- and external-cohort mean scores across 30 models under the Focused formulation. b, Exact outcome agreement between three clinical experts and the LLM judge, overall and by dimension, calculated separately for each expert and averaged with equal expert weights. Panels b–c include 6,075 valid individual rating pairs from 108 responses across 12 purposively selected cases. c, Expert and LLM-Judge matched-item scores, calculated as equal-item factor means multiplied by 100 without zero clipping. Each point represents one expert–response–dimension combination; the diagonal indicates equality. These subset scores differ from the full-response weighted benchmark scores. Sampling, matching and scoring procedures are detailed in S4.5 and S7.6–S7.7. DQ, Decision Quality; EG, Evidence Grounding; SD, Status Determination; EF, Evidence Faithfulness; SA, Safety Adherence.
Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety. As patients and clinicians increasingly use LLMs for guidance on complex conditions such as pancreatic cancer, evaluation must extend beyond general medical knowledge. Existing frameworks, such as HealthBench, rely on simulated queries and lack disease-specific depth. Moreover, high rubric-based scores do not ensure factual correctness, underscoring the need to assess hallucinations. We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions from the Pancreatic Cancer Action Network (PanCAN). The resulting benchmark, PanCanBench, includes 3,130 question-specific criteria across 282 authentic patient questions. We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration. Models showed substantial variation in rubric-based completeness, with scores ranging from 46.5% to 82.3%. Factual errors were common, with hallucination rates (the percentages of responses containing at least one factual error) ranging from 6.0% for Gemini-2.5 Pro and GPT-4o to 53.8% for Llama-3.1-8B. Importantly, newer reasoning-optimized models did not consistently improve factuality: although o3 achieved the highest rubric score, it produced inaccuracies more frequently than other GPT-family models. Web-search integration did not inherently guarantee better responses. The average score changed from 66.8% to 63.9% for Gemini-2.5 Pro and from 73.8% to 72.8% for GPT-5 when web search was enabled. Synthetic AI-generated rubrics inflated absolute scores by 17.9 points on average while generally maintaining similar relative ranking.
Yimin Zhao, Sheela R. Damle, Simone E. Dekker +13
Department of Biostatistics, University of Washington · Clinical Research Division, Fred Hutch Cancer Center · Division of Hematology and Oncology, Department of Medicine, University of Washington +5
Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. A benchmark predicts performance in deployment only to the extent that its items resemble real use, yet whether benchmarks reflect the work these systems receive has rarely been measured. Here we analyze 127,833 queries sent by 6,342 physicians, advanced practice providers and nurses in 35 specialties to an institutional assistant during an eight-month roll-out. We characterize each query with RCQ-Map, a clinician-validated framework grounded in taxonomies of clinical questions and of LLM evaluation, which records its task, intent, answerability, missing information and potential harm. Documentation and administration (36.2%) and knowledge retrieval (28.9%) made up nearly two-thirds of use, and diagnosis 3.7%; more than a third of queries could not be answered well as posed. Applying RCQ-Map to 58 public benchmarks drawn from major evaluation suites and frontier model reports, which we assemble into the Clinical AI Benchmark Atlas, showed that the median benchmark contained no documentation requests and shared 31% of the task mix of real use, less than an even spread across task categories would. Benchmarks in suites designed to resemble clinical practice were individually no closer to real use than those used in frontier model reports. Benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given, and evaluation should be matched to real clinical use.
Krithik Vishwanath, Haitong Lin, Anton Alyakin +19
Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Department of Technology Management, NYU Tandon School of Engineering, New York University, New York, NY, USA · Washington University School of Medicine, St. Louis, MO, USA +18
Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce ClinConsensus, a Chinese medical benchmark of 2{,}500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose Clinician-Anchored Coverage Score (CACS), a physician-calibrated threshold metric instantiated at k=10, and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6% to 52.1%, whereas CACS@10 ranges from 17.8% to 32.9%, leaving a 19.2--21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness.