PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology
Authors: Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh, Jeffrey T. Leek
Organizations: Department of Biostatistics, University of Washington · Clinical Research Division, Fred Hutch Cancer Center · Division of Hematology and Oncology, Department of Medicine, University of Washington · Allen Institute for AI · Department of Computer Science and Engineering, University of Washington · Pancreatic Cancer Action Network · Public Health Sciences, Biostatistics, Fred Hutchinson Cancer Center · Independent Researcher
Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety. As patients and clinicians increasingly use LLMs for guidance on complex conditions such as pancreatic cancer, evaluation must extend beyond general medical knowledge. Existing frameworks, such as HealthBench, rely on simulated queries and lack disease-specific depth. Moreover, high rubric-based scores do not ensure factual correctness, underscoring the need to assess hallucinations. We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions from the Pancreatic Cancer Action Network (PanCAN). The resulting benchmark, PanCanBench, includes 3,130 question-specific criteria across 282 authentic patient questions. We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration. Models showed substantial variation in rubric-based completeness, with scores ranging from 46.5% to 82.3%. Factual errors were common, with hallucination rates (the percentages of responses containing at least one factual error) ranging from 6.0% for Gemini-2.5 Pro and GPT-4o to 53.8% for Llama-3.1-8B. Importantly, newer reasoning-optimized models did not consistently improve factuality: although o3 achieved the highest rubric score, it produced inaccuracies more frequently than other GPT-family models. Web-search integration did not inherently guarantee better responses. The average score changed from 66.8% to 63.9% for Gemini-2.5 Pro and from 73.8% to 72.8% for GPT-5 when web search was enabled. Synthetic AI-generated rubrics inflated absolute scores by 17.9 points on average while generally maintaining similar relative ranking.
Figures & tables
Figure 1: Overview of how we use rubrics to evaluate large language model (LLM) responses to patient and caregiver questions. a) Collect de-identified patient and caregiver questions from the Pancreatic Cancer Action Network. b) Generate responses from LLMs. c) Grade the LLM responses using an AI judge operating under human-curated evaluation rubrics. d) Factual error analysis using LLM-as-a-judge.
Figure 2: Pipeline to create rubrics and evaluate responses to Pancreatic Cancer Action Network (PanCAN) hotline questions guided by human expertise. Phase 1) Two oncology fellows independently designed initial rubrics. Phase 2) Leverage AI to polish the human-written rubrics and perform ‘gut checks’ on the rubrics by comparing the scores of pairs of large and small models in the same family. Phase 3) AI merged the rubrics from the oncology fellows by semantically joining redundant rubric items and retaining unique items. Phase 4) A third oncology fellow reviewed and finalized the rubrics.
Figure 3: Pipeline to create rubrics and evaluate responses to Pancreatic Cancer Action Network (PanCAN) hotline questions guided by human expertise. a) Frequency of questions in each category b) Example questions for each question category.
Figure 4: Comprehensive benchmark study across 22 models. a) Barplot showing average score by model. b) UpSet plot showing the percentage of responses containing factual errors and the question categories where each model is most prone to mistakes. c) Average scores for questions with rubric items requiring references/citations, with and without web search. d) Table showing triggering rate, success rate when web search is used, key failure mode, and supportive link rate for GPT-5, Claude-Sonnet-4.5, and Gemini-2.5 pro with web search function.
Figure 5: Investigating the necessity of human involvement in LLM Evaluation. a) Paired barplot showing average score by model evaluated by human-curated versus AI-generated synthetic rubrics. b) Slope plot showing the ranking comparison of models evaluated by human rubrics and AI-generated rubrics.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Question
Are there studies for pancreatic cancer evaluating infusion of high vitamin C doses in conjunction with chemotherapy?
Original Rubric
High-dose vitamin C has not been proven to improve outcomes in pancreas or any other cancer. ( 10 points )
Grader A
The response does not mention that the evidence is not yet conclusive; high-dose vitamin C is not widely accepted as a standard treatment. (0/10)
Grader B
The response does not support the use of high-dose vitamin C. (10/10)
Polished Rubric
The response should explicitly state that high-dose vitamin C has not been proven to improve outcomes in pancreatic cancer or any other cancer. ( 10 points )
Appendix
Table S1 : Example of using AI to polish the human-written rubrics
Figure S1 : Comparison of large versus small model performance across human expert evaluations (Llama-3.1-8B vs Llama-3.1-70B; Qwen3-8B vs Qwen3-32B)
Figure S2 : Example of merging semantically same rubrics and retain unique items
Model
Temperature
GPT-5
Does not support temperature
GPT-4o
1
GPT-4.1
1
GPT-o3
1
GPT-o4-mini-high
1
Claude-opus-4
1
Appendix
Table S2 : Temperature settings for evaluated language models
Model Pair
Both Favor A
Both Favor B
Both Tie
Overall Agreement
GPT-5 vs o3
31.2%
7.1%
0.0%
38.3%
GPT-5 vs Grok-4
43.6%
0.0%
0.0%
43.6%
GPT-5 vs Gemini-2.5 Flash
51.4%
0.4%
0.0%
51.8%
o3 vs Gemini-2.5 Flash
57.8%
1.4%
0.0%
59.2%
o3 vs Grok-4
47.5%
2.1%
0.0%
49.6%
Grok-4 vs Gemini-2.5 Flash
28.7%
21.6%
0.0%
50.4%
Appendix
Table S3: Agreement between AI direct judgments and rubric-based evaluations across model pairs.
Metric
GPT-5
Gemini-2.5 Pro
Claude-Sonnet-4.5
Key Failure Mode
Missing specific information or required links/PMIDs
Provided generic redirect links instead of specific citations (PMIDs, DOIs, etc.)
Provided generic redirect links instead of specific citations (PMIDs, DOIs, etc.)
Appendix
Table S4 : Key failure modes in web-search evaluation.
Predicted
Actual
Negative (Correct)
Positive (Wrong)
Actual Negative
30
1
Actual Positive
4
65
Appendix
Table S5 : Confusion Matrix for Response Factuality Verification
Claims
Reasoning
PALB2 mutations may increase the risk of pancreatic cancer more than BRCA1 and BRCA2 mutations, according to some research.
PALB2 - 1–4% risk, BRCA1 up to 5% risk, BRCA2 up to 10% risk.
Human Kallikrein-related peptidase 6 (KLK6) is a potential biomarker for early stages of pancreatic cancer.
KLK6 is being studied in pancreatic cancer but has not been shown to be a biomarker of early stage disease.
Managing type 2 diabetes can reduce your risk of pancreatic cancer.
It is not yet known which disease causes the other or if they share a similar underlying cause.
A mutation in CDKN2A can affect treatment choices for pancreatic cancer.
No specific treatments based on CDKN2A mutation. There may be implications for cancer screening.
Appendix
Table S6 : Examples of incorrect claims
Figure S3 : Barplot showing number of wrong claims by model
Figure S4 : Percentage of wrong claims by model
Figure S5 : Distribution of number of tokens by model
Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.
Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4
Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.
While Large Language Models (LLMs) excel on standardized medical exams, high scores often fail to translate to high-quality responses for real-world medical queries. Current evaluations rely heavily on multiple-choice questions, failing to capture the unstructured, ambiguous, and long-tail complexities inherent in genuine user inquiries. To bridge this gap, we introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment. We compiled a massive dataset spanning Clinical Care, Wellness Health, and Professional Inquiry, comprising 20,821 single-turn queries and 3,853 multi-turn sessions. To objectively evaluate open-ended answers, we propose an automated scoring framework that integrates multi-model consensus with evidence-based retrieval to dynamically generate 220,617 fine-grained scoring rubrics (~9.8 per query). During evaluation, hierarchical weighting and safety constraints structurally quantify medical accuracy, key-point coverage, and risk interception, effectively mitigating the high costs and subjectivity of human grading. Experimental results demonstrate that the generated rubrics achieve a 91.8% concordance rate with clinical expert blind audits, establishing highly dependable medical reliability. Crucially, baseline evaluations on this benchmark reveal significant performance disparities among state-of-the-art models when navigating real-world clinical nuances, highlighting the limitations of conventional exam-based metrics. Ultimately, QuarkMedBench establishes a rigorous, reproducible yardstick for measuring LLM performance on complex health issues, while its framework inherently supports dynamic knowledge updates to prevent benchmark obsolescence.