Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

Wei-Wei Zhang; Pengkai Wang; Yan Li; Zhijie Sang; Guan-Qun Zhou; Jun-Yan Li; Shu-Han Zhou; Pei-Wen Liu

cs.AIOct 26, 2026

Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

Authors: Wei-Wei Zhang, Pengkai Wang, Yan Li, Zhijie Sang, Guan-Qun Zhou, Jun-Yan Li, Shu-Han Zhou, Pei-Wen Liu,

Organizations: Department of Radiation Oncology, Sun Yat-sen University Cancer Center, Guangzhou, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Pathology, Sun Yat-sen University Cancer Center, Guangzhou, China. · Department of Radiation Oncology, Nanfang Hospital, Southern Medical University, Guangzhou, China. ·

Abstract

Large language models (LLMs) show promise in specialty medicine, yet existing evaluations largely focus on isolated clinical encounters rather than tracking how clinical responsibilities evolve across a longitudinal care pathway. Here we introduce NPC-RubricBench, a phase-aware benchmark for evaluating multistage clinical reasoning across the cancer care continuum, using nasopharyngeal carcinoma as a defined disease context. Curated from longitudinal records of 405 patients across two cancer centers, the benchmark comprises 941 patient-phase units across six sequential care phases and 17,688 patient-specific atomic criteria spanning five evaluation dimensions, with each assessment strictly restricted to decision-time clinical evidence. Evaluating 30 general-purpose and medical LLMs revealed that strong diagnostic performance does not carry over to downstream care. We evaluated 30 general-purpose and medical LLMs, showing that high diagnostic performance did not translate into improved downstream care. Instead, the dominant failure modes reversed across phases. Explicit decision errors dominated anatomical staging at 48.2%, whereas decision omissions dominated initial work-up and post-treatment follow-up at 33.4% and 30.4%, respectively. In the evaluated model set, medically adapted models showed higher omission rates than general-purpose models. Moreover, varying query granularity exposes a critical specification trade-off: broad open-ended requests degrade Decision Quality across all 30 models, while more detailed, checklist-like prompts raise Decision Quality but reduce Evidence Faithfulness and Safety Adherence. Furthermore, NPC-RubricBench demonstrates that static diagnostic benchmarks are insufficient to ensure consistent clinical reliability over time, highlighting the importance of phase-adaptive safeguards in multistage medical AI.

Figures & tables

Figure 1: NPC-RubricBench design and composition. a, Longitudinal evaluation setting. Longitudinal records are segmented into six ordered care phases. Patients are counted once across the internal and external cohorts, and each patient–phase unit is counted once. Not every patient contributes every phase. The staging example illustrates uncertainty from a limited biopsy. b, Clinical scenarios. Stacked bars show patient–phase counts across the defined clinical scenarios, using the same inventory as a and a shared absolute-count scale. Definitions and counts are provided in S1.4 and Table S1-B. c, Patient-specific atomic rubrics. Condensed examples illustrate patient-specific criteria for Decision Quality (DQ), Evidence Grounding (EG), Status Determination (SD), Evidence Faithfulness (EF) and Safety Adherence (SA). Schematic outcomes are grouped as Met/Partial/Error/Omission for DQ/EG/SD (upper) and Met/Not met for EF/SA (lower). Partial is permitted only for items with an explicit item-specific partial-credit allowance. Met/Not met denote positive requirements satisfied/unmet, with score factors 1/0. Rubric construction and patient-specific examples are described in S2.1 and S2.3. d, Rubric scale and composition. Annulus sector area encodes rubric-item counts by phase and dimension; n gives phase totals. Counts are of unique patient-specific rubric items. NPC, nasopharyngeal carcinoma; EBV, Epstein–Barr virus; EBER, EBV-encoded RNA; RT, radiotherapy; CCRT, concurrent chemoradiotherapy.

Figure 2: Performance across care phases and rubric dimensions. a, Phase-mean distributions. Phase-mean score distributions for 30 models under Focused formulation (text) in the main clinical evaluation set (internal and external cohorts). Scores use dimension-eligible patient–phase units on a bounded 0–100 scale. Violins span observed values; marks show the minimum, median and maximum. Wu, Work-up; Dx, Diagnosis; St, Staging; Tr, Treatment; Tx, Toxicity; Fu, Follow-up. b, Decision quality. DQ means for eight author-selected models, ordered as API-accessed general-purpose, locally deployed open-weight general-purpose and locally deployed open-weight medically adapted groups, with descending overall DQ within each group. c, Evidence grounding. EG means for the same models. d, Status determination. SD means for the same models. Panels b–d share a fixed model order and colours with Figs. 3 and 4c,d; labels are rounded scores. Complete phase-specific scores and denominators are reported in Table S5. DQ, Decision Quality; EG, Evidence Grounding; SD, Status Determination; EF, Evidence Faithfulness; SA, Safety Adherence. Model abbreviations: I, Instruct; T, Thinking/Think. Full names are listed in Table S3.

Figure 3: Model phase profiles and concept violations. a, Grouped model phase profiles. Grouped radars show five-dimensional scores across six phases for the same eight models as Fig. 2b–d and Fig. 4c,d, in API-accessed general-purpose, locally deployed open-weight general-purpose and locally deployed open-weight medically adapted groups. Colours are shared with Fig. 2; outer sectors identify dimensions. b, EF concept violations. EF concept-violation rates: the percentage of resolved, eligible main-evaluation patient–phase units with at least one applicable EF requirement judged Not met. Units with all applicable requirements judged Met remain in the denominator. All phases use one fixed model order. c, SA concept violations. SA concept-violation rates use the same unit and Not met event rule as b. d, Individual model phase profiles. Individual radars show the same scores as a, in the shared model order. Wu, work-up; Dx, diagnosis; St, staging; Tr, treatment; Tx, toxicity; Fu, follow-up. Panels b,c use full phase names. Model abbreviations: I, Instruct; T, Thinking/Think. Full names are listed in Table S3.

Figure 4: Six-class failure analysis across care phases and models. a, Six-class failure taxonomy. The six classes are Decision omission, Decision error, Evidence omission, Evidence error, Status-assessment omission and Status-assessment error. EF and SA violations are summarized separately. b, Error and omission occurrence by phase. Phase-specific item-level rates pool all 30 models in the main clinical evaluation set (internal and external cohorts). Error or Omission counts are divided by the preset applicable Rubric item total for each phase and DQ, EG or SD dimension. Met and Partial remain in the denominator; EF and SA are outside this inventory. Phase-level summaries are reported in Table S11; failure-rate definitions are provided in S7.3. c, Error and omission profiles by model. Bars show omission and error rates using each model’s preset Rubric opportunities pooled across phases and DQ, EG and SD, for the same eight models and order as Figs. 2 and 3. Pooled estimates and 95% confidence intervals for all 30 models are reported in Table S10. Confidence intervals use a patient-cluster bootstrap with 1,000 resamples. d, Selected errors and omissions across models. Bars show prevalence point estimates for the same eight models in four selected phase-by-failure examples. Full model-by-phase failure profiles are provided in Suppl. Fig. 3. Model abbreviations: I, Instruct; T, Thinking/Think. Full names are listed in Table S3.

Figure 5: Sensitivity to clinical task formulation. a, Clinical task formulation. Broad formulation, Focused formulation and Elaborated formulation prompts from the staging example; clinical records and base rubrics are held fixed. b, Paired score directions. Direction of paired score changes relative to Focused formulation in DQ, EG and SD across six care phases. Each segment is the arithmetic mean of within-model percentages across 30 models. For each phase, dimension and comparison, all models share the same set of cases with scores for both formulations. Unchanged denotes an absolute score difference of at most

10^{-8}

points. Wk, Work-up; Dx, Diagnosis; St, Staging; Tr, Treatment; Tx, Toxicity; Fu, Follow-up. c, Secondary directions within DQ branches. Directions of EG, SD, EF and SA changes within selected DQ increase, unchanged or decrease branches. Percentages are calculated within each model on the common cases valid for DQ and the secondary dimension, then averaged equally over models with a nonzero DQ-branch denominator. These are conditional percentages, not percentages of all paired responses. In b,c, bottom, middle and top segments indicate increase (orange hatch), unchanged (grey) and decrease (blue hatch), respectively. Integer labels and outlined segments in b,c identify author-selected descriptive patterns.

Figure 6: Cross-cohort concordance and expert–judge agreement. a, Lin’s concordance correlation coefficients (CCC) between internal- and external-cohort mean scores across 30 models under the Focused formulation. b, Exact outcome agreement between three clinical experts and the LLM judge, overall and by dimension, calculated separately for each expert and averaged with equal expert weights. Panels b–c include 6,075 valid individual rating pairs from 108 responses across 12 purposively selected cases. c, Expert and LLM-Judge matched-item scores, calculated as equal-item factor means multiplied by 100 without zero clipping. Each point represents one expert–response–dimension combination; the diagonal indicates equality. These subset scores differ from the full-response weighted benchmark scores. Sampling, matching and scoring procedures are detailed in S4.5 and S7.6–S7.7. DQ, Decision Quality; EG, Evidence Grounding; SD, Status Determination; EF, Evidence Faithfulness; SA, Safety Adherence.

Phase	Dim.	Secondary clinical targets	Criterion requirements in the evaluation rubrics
Work-up	DQ	Pathological confirmation; locoregional imaging; distant-metastasis assessment; laboratory, hepatitis and cardiopulmonary assessment; EBV-DNA testing	Recommend phase-appropriate investigations, preserving distinct prerequisites for disease confirmation and treatment tolerance.
Work-up	EG	Presenting symptoms and signs; outside investigations; completed in-hospital tests	Identify the findings available at the decision point; distinguish completed tests from proposed tests.
Work-up	EF	Fidelity to symptoms, care history and investigation results	Do not introduce patient facts or investigation results absent from the visible record.
Work-up	SD	Completion of pathology, imaging, metastasis assessment, EBV-DNA and pretreatment assessments	Determine which decision prerequisites are satisfied and which remain incomplete.
Work-up	SA	Premature conclusions; duplicate or unindicated investigations	Avoid definitive conclusions before prerequisites are met and avoid unjustified repeat or routine testing.
Diagnosis	DQ	Primary pathological diagnosis; unresolved lesion type; further confirmation; important comorbid diagnoses	State the diagnosis supported by the current pathology; request further confirmation when classification remains unresolved.

Table 7

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 419 papers

PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology
Mar 2, 2026Yimin Zhao, Sheela R. Damle, Simone E. Dekker +13Clinical Language Model Evaluation Medical QA
Department of Biostatistics, University of Washington · Clinical Research Division, Fred Hutch Cancer Center · Division of Hematology and Oncology, Department of Medicine, University of Washington +5
Clinician use of language models diverges from how the models are evaluated
Oct 8, 2026Krithik Vishwanath, Haitong Lin, Anton Alyakin +19Clinical Language Model Evaluation Medical LLMs
Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Department of Technology Management, NYU Tandon School of Engineering, New York University, New York, NY, USA · Washington University School of Medicine, St. Louis, MO, USA +18
ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs
Mar 2, 2026Xiang Zheng, Han Li, Wenjie Luo +15Clinical Language Model Evaluation Medical LLMs
Alibaba Group, China

Mar 2, 2026J/K move · Enter open · S save

PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng+12

Department of Biostatistics, University of Washington · Clinical Research Division, Fred Hutch Cancer Center · Division of Hematology and Oncology, Department of Medicine, University of Washington+5

Clinical Language Model Evaluation Medical QA Oncology

Open paper AskPDF arXiv

Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety. As patients and clinicians increasingly use LLMs for guidance on complex conditions such as pancreatic cancer, evaluation must extend beyond general medical knowledge. Existing frameworks, such as HealthBench, rely on simulated queries and lack disease-specific depth. Moreover, high rubric-based scores do not ensure factual correctness, underscoring the need to assess hallucinations. We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions from the Pancreatic Cancer Action Network (PanCAN). The resulting benchmark, PanCanBench, includes 3,130 question-specific criteria across 282 authentic patient questions. We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration. Models showed substantial variation in rubric-based completeness, with scores ranging from 46.5% to 82.3%. Factual errors were common, with hallucination rates (the percentages of responses containing at least one factual error) ranging from 6.0% for Gemini-2.5 Pro and GPT-4o to 53.8% for Llama-3.1-8B. Importantly, newer reasoning-optimized models did not consistently improve factuality: although o3 achieved the highest rubric score, it produced inaccuracies more frequently than other GPT-family models. Web-search integration did not inherently guarantee better responses. The average score changed from 66.8% to 63.9% for Gemini-2.5 Pro and from 73.8% to 72.8% for GPT-5 when web search was enabled. Synthetic AI-generated rubrics inflated absolute scores by 17.9 points on average while generally maintaining similar relative ranking.

Abstract

Clinical Language Model Evaluation

Explore similar work

PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

Clinician use of language models diverges from how the models are evaluated

ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs