CaseScope: A Multi-Field Benchmark for Case-Level Pathology Diagnosis with Multimodal LLMs
Authors: Wenchuan Zhang, Chengyang Zhang, Fengling Li, Ting Yin, Chen Shu, Jie Chen, Shuwan Zhang, Xiaoyan Wu, Zhang Zhang, Yuhao Yi, Jiancheng Lv, Hong Bu
Organizations: Department of Pathology, West China Hospital, Sichuan University · Institute of Clinical Pathology, West China Hospital, Sichuan University · College of Computer Science, Sichuan University · Department of Pathology, Shengjing Hospital of China Medical University · Department of Pathology, Renmin Hospital of Wuhan University
Pathologists rarely diagnose from a single image: they scan a section at low power for architecture, zoom into selected fields for cytology, and integrate the evidence across views and scales. Pathology benchmarks for multimodal large language models (MLLMs), however, mostly ask about one image or require whole-slide pipelines, and the few that show several fields draw nested regions from cancer slides of one public cohort. We introduce CaseScope, a benchmark of diagnosis from several curated fields of one lesion: CaseScope-Teaching, 436 cases from a residency-training image library, reconstructed, de-identified, harmonised against a WHO-based ontology and re-examined by a pathologist wherever an automated pre-review raised a doubt, and CaseScope-Atlas, 800 cases from 88 published atlases (387 of them figure series of one disease), labelled from the captions by two LLMs that must agree. They cover 843 entities across 18 organ systems and pose 4,679 items in six tasks, from open-ended and ten-way diagnosis with ontology-guided hard negatives under circular evaluation to the verification and location of morphological findings. Across twenty-eight open and four proprietary MLLMs, diagnoses are unstable -- the best open and proprietary models are correct under all ten option rotations on 17.9% and 61.7% of the teaching cases -- and wrong answers favour the classic look-alike; open models make little use of the additional fields on teaching cases, where the one-line specimen site matters more, although on atlas cases with three or more fields the fields count about as much as the site; and most open models verify findings barely above chance (median 62.0% vs. 50%, best 81.5%), gaining a median +3.4 points from images; proprietary models reach 83.2-93.2%. Rankings broadly replicate between the sources. The evaluation code is available at https://github.com/Wenchuan-Zhang/CaseScope.
Figures & tables
Figure 1
Benchmark
Size
Exp.
MF
Case
HN
Rob.
Single image or region
PathVQA [ 24 ]
32.8k
✗
✗
✗
✗
✗
Quilt-VQA [ 51 ]
1.3k
(✓)
✗
✗
✗
✗
PathQABench [ 41 ]
105
✓
✗
✓
(✓)
✗
PathMMU [ 52 ]
33.4k
✓
✗
✗
(✓)
✗
PathBench [ 55 ]
6.3k
✓
✗
–
(✓)
✗
Table 1: Pathology benchmarks for MLLMs (selection; full comparison and size scopes in Table S1 ). Size : number of evaluation items. Exp. : pathologist review; MF : separate fields of one lesion; Case : patient diagnosis; HN : close distractors; Rob. : circular evaluation or consistency under controlled input changes. (✓): partial; ✗: not reported; –: not applicable.
Figure 2: Composition of the two sources. (a) Share of each source’s cases per primary organ system, coloured by behaviour; numbers: cases. (b) Share of cases by number of H&E fields; a teaching case keeps all its fields, of which at most eight are shown to the models.
Figure 3: A1 diagnosis on two CaseScope cases. Top: the prompt, identical for both cases except for the number of fields and the site. Each card gives the diagnosis, an excerpt of the response (green/red: findings consistent/inconsistent with the reference; grey: the rest) or the differential the model returned, and an editorial note. Ticks/crosses follow the A1 judge.
Task
Teach.
Atlas
Chance
Metric
A1
Open-ended diagnosis
436
800
–
HDS, Top- k
A2
Diagnosis, 10-way MCQ
436
800
10%
CircularEval
B
Behaviour triage, 4-way
380
762
25%
CircularEval
C
Ordinal grading, 2–4-way
44
–
1/ n
Circ., QWK
R1
Finding verification, T/F †
390
195
50%
Stmt./item acc.
G1
Finding location, 2–8-way
–
436
1/ n
CircularEval
Table 2: Task suite. Items per source (Teach.: CaseScope-Teaching), counted before circular rotation. Chance is the vanilla (single-pass) expectation.
CaseScope-Teaching
CaseScope-Atlas
A1
A2
A2 v
B
Miss ↓
C
C κ
R1
Δimg
A1
A2
A2 v
B
Miss ↓
R1
Δimg
G1
Random choice
–
0.0
10.1
0.5
49.7
11.4
0.08
–
–
–
0.0
9.9
0.9
50.9
–
–
9.2
Proprietary MLLMs
Claude Sonnet 5.5
64.2
61.7
78.3
75.5
11.4
79.5
0.81
93.2
+10.2 ∗
41.6
49.1
68.5
61.5
21.6
80.8
+6.0 ∗
61.5
Claude Haiku 5.5
43.9
39.0
61.0
57.4
10.3
54.5
0.55
83.2
+19.5 ∗
28.8
29.9
52.9
47.0
25.4
73.6
+0.0
48.2
GPT-5.6-sol
43.5
49.3
67.4
55.5
19.4
61.4
0.64
92.9
+12.6 ∗
27.6
37.1
57.1
41.9
35.4
78.6
+5.6 ∗
45.6
Table 3: Results on CaseScope-Teaching and CaseScope-Atlas (%; tasks and metrics as in Sec. 4 ). A2, B, C, G1: CircularEval; A2 v : single-pass accuracy. Miss: critical miss rate. C κ : QWK. Δimg : R1 with minus without the images ( ∗ : 95% paired bootstrap interval excludes 0). † Single-image model given a mosaic of the fields. –: not applicable. Bold / underlined : best/second best per column except Δimg , separately for proprietary and open models. 95% half-widths reach 4.8 points on CaseScope-Teaching and 3.4 on CaseScope-Atlas; per-model intervals: Tables S6 and S7 .
Figure 4: Do models use the fields? Change of the open models’ A2 circular accuracy when only the highest-power field is given (left) or the specimen site is removed (right), against all fields with the site; paired case bootstrap, 95% CI; filled: p<0.05 ; numbers: change in points.
Benchmark
Venue
Visual input
Size
QA construction
Expert
Multi- field
Case dx
Hard neg.
Robust
Single-image and region-based benchmarks
PathVQA [ 24 ]
arXiv’20
1 image
32.8k QA
Rule-based
✗
✗
✗
✗
✗
Quilt-VQA [ 51 ]
CVPR’24
1 image
1.3k QA
LLM-generated
(✓)
✗
✗
✗
✗
PathQABench [ 41 ]
Nature’24
1 ROI
105 QA
Expert-written
✓
✗
✓
(✓)
✗
PathMMU [ 52 ]
ECCV’24
1 image
33.4k QA
LLM-generated
✓
✗
✗
(✓)
✗
PathBench [ 55 ]
TMI’25
1 image / 1 WSI
6.3k QA
NR
✓
✗
–
(✓)
✗
Table S1 : Pathology benchmarks for MLLMs: full version of Table 1. Visual input, reported dataset size, QA construction and evaluation properties are compared; subset scopes and special input settings are specified below.
Scale-VQA [ 47 ]
PathScale-VQA [ 46 ]
CaseScope (ours)
Status
MICCAI 2026
arXiv 2026, extends Scale-VQA
this work
Image source
177 TCGA WSIs, 15 organs
373 TCGA WSIs, 14 organ sites
residency teaching library (436 cases) and 88 published atlases (800 cases), 18 organ systems
One item
zoom path: a region selected at 10 × and corresponding regions at 40 × and 200 ×
teaching: rule-based questions and distractors from expert labels harmonised to a WHO-based ontology, LLM pre-review followed by pathologist adjudication, R1 statements LLM-generated; atlas: labels, statements and finding locations from the captions by LLMs; R1 statements and atlas labels verified by a pathologist
Table S2 : CaseScope vs. cross-scale pathology benchmarks. Facts as reported in [ 47 , 46 ] .
Figure S1 : From the library as filed to benchmark cases. The steps of this section, each with the count after it, shown on real items: a solitary fibrous tumour picture filed under four chapters and two names (C0025; three shown; SFT: solitary fibrous tumour), the re-encoded copy I0288 of picture I0020, the gross photograph I0019 leaving case C0065, and the five fields of the phyllodes case of Fig. 1 ordered from low to high power. The input is the representative set of Fig. 1b, as filed in the teaching library.
Diagnostic entity
T
A
Diagnostic entity
T
A
Oral & salivary (Teaching 30 cases; Atlas 31)
Acinic cell carcinoma
1
1
Granulomatous cheilitis
–
1
Adenoid cystic carcinoma
1
1
Kimura disease
1
–
Adenomatoid odontogenic tumour
–
1
Lichen planus
1
–
Ameloblastic carcinoma
1
1
Lymphangioma (lymphatic malformation)
1
–
Ameloblastic fibrosarcoma
–
1
Lymphoepithelial sialadenitis (Sjögren syndrome)
1
1
Table S4 : Diagnostic entities of CaseScope. T: CaseScope-Teaching cases of the entity filed under the organ system (an entity filed under several systems is listed in each); A: its CaseScope-Atlas case (totals in the last row). Grouped by organ system and alphabetised by the canonical English name, reading down the left list and then the right list within each group.
Figure S4 : One teaching case, five tasks: the prompts and the answers of twelve models. Case C0247 (malignant phyllodes tumour, breast). Top: the five fields and, per task, the prompt (its shared opening once), the options with the reference in green and, for R1, the eight statements with their reference truth values. Bottom: each model’s answers at rotation 0 (blue rows: proprietary models). A1: the diagnosis (green/red: reference entity or synonym/other), the differential the model returned (reference mentions in green) and, for the two reasoning models, an excerpt of the reasoning (green/red: findings consistent/inconsistent with the reference; ellipses mark omissions). A2, B, C: the option chosen, the number of rotations answered correctly and the option’s text. R1: the four judgements per level (green: matches the reference) and whether all four were right; Patho-R1-7B returned no parseable high-power answer. Qwen3.5-9B reached the output limit while still reasoning on A1, C and low-power R1 and answered after the stop-thinking instruction (Sec. S10 ).
Figure S5 : G1 on an atlas item. Five panels of a Bartholin gland cyst (vulva) from one atlas entry, each with its own atlas caption (the captions are not shown to the models); the described finding is the caption of image 4, the reference (green frame). Middle: the prompt and its options. Bottom: each model’s answer at option rotation 0, the image it names, the number of option rotations answered correctly (the options rotate, the images keep their order) and the response: the letter alone or, for the two reasoning models, excerpts of the reasoning (green/red: statements consistent/inconsistent with the panels and their captions). Qwen3.5-9B reached the output limit while still reasoning and answered after the stop-thinking instruction (Sec. S10 ).
Figure S6 : Teaching vs. Atlas. Each point is a model: A2 circular accuracy (left) and A1 hierarchical score (right) on CaseScope-Teaching and on CaseScope-Atlas; dashed: equal scores; ρ : Spearman correlation over the models shown.
Figure S7 : R1 with and without the images on CaseScope-Teaching (left) and CaseScope-Atlas (right) (the proprietary, then the open models, each block in the order of Teaching accuracy with images). Share of finding statements judged correctly from the text alone (grey) and with the fields (violet; open: gain not significant, paired case bootstrap); Δ : with minus without the images, in points.
A1
A2
A2 v
B
Miss ↓
C
C κ
R1
Δimg
Random choice
–
0.0
10.1
0.5
49.7
11.4
0.08
–
–
Proprietary MLLMs
Claude Sonnet 5.5
64.2 ± 4.5
61.7 ± 4.4
78.3
75.5
11.4
79.5
0.81
93.2
+10.2 ∗
Claude Haiku 5.5
43.9 ± 4.6
39.0 ± 4.6
61.0
57.4
10.3
54.5
0.55
83.2
+19.5 ∗
GPT-5.6-sol
43.5 ± 4.5
49.3 ± 4.8
67.4
55.5
19.4
61.4
0.64
92.9
+12.6 ∗
GPT-5.6-luna
33.5 ± 4.1
31.9 ± 4.2
57.3
51.1
33.6
56.8
0.61
87.1
+15.4 ∗
Table S6 : Complete CaseScope-Teaching results with intervals (%). Columns and marks as in Table 3; ± : half-width of the 95% case-bootstrap interval.
A1
A2
A2 v
B
Miss ↓
R1
Δimg
G1
Random choice
–
0.0
9.9
0.9
50.9
–
–
9.2
Proprietary MLLMs
Claude Sonnet 5.5
41.6 ± 3.3
49.1 ± 3.4
68.5
61.5
21.6
80.8
+6.0 ∗
61.5 ± 5.0
Claude Haiku 5.5
28.8 ± 3.1
29.9 ± 3.2
52.9
47.0
25.4
73.6
+0.0
48.2 ± 4.7
GPT-5.6-sol
27.6 ± 3.1
37.1 ± 3.4
57.1
41.9
35.4
78.6
+5.6 ∗
45.6 ± 4.9
GPT-5.6-luna
19.8 ± 2.8
24.1 ± 3.0
46.6
32.7
46.4
73.1
+8.5 ∗
39.9 ± 4.6
Table S7 : Complete CaseScope-Atlas results with intervals (%). Columns and marks as in Table 3; ± as in Table S6 , also for G1. G1 rotates the options while the images keep their order: naming one image, whichever it is, in every rotation is right on 31.5% of the items on average (always image 2: 31.7%), independent guesses on 8.5%.
CaseScope-Teaching
CaseScope-Atlas
Model
All
High-power only
No site
All
High-power only
No site
Qwen2.5-VL-3B
7.3
7.3 (+0.0)
2.3 (−5.0)
10.0
6.4 (−3.6)
7.8 (−2.3)
Qwen2.5-VL-7B
6.0
8.0 (+2.1)
1.4 (−4.6)
9.8
8.0 (−1.8)
6.1 (−3.6)
Qwen2.5-VL-32B
7.6
5.5 (−2.1)
1.8 (−5.7)
11.0
7.0 (−4.0)
6.4 (−4.6)
Qwen2-VL-7B
7.8
7.3 (−0.5)
2.3 (−5.5)
8.2
7.1 (−1.1)
6.6 (−1.6)
InternVL2.5-8B
8.5
8.7 (+0.2)
2.1 (−6.4)
8.8
7.5 (−1.2)
5.4 (−3.4)
Table S8 : Field-set ablation on CaseScope-Teaching and CaseScope-Atlas (A2, CircularEval, %). Same cases and options, different evidence: all fields with the specimen site (All), only the highest-power field, all fields without the specimen site; in brackets the change from All in points.
CaseScope-Teaching
CaseScope-Atlas
Model
Vanilla
Circular
Content- consistent
Top letter (share)
Vanilla
Circular
Content- consistent
Top letter (share)
Proprietary MLLMs
Claude Sonnet 5.5
78.3
61.7
67.2
D (10)
68.5
49.1
57.1
C (11)
Claude Haiku 5.5
61.0
39.0
46.8
B (12)
52.9
29.9
36.9
B (11)
GPT-5.6-sol
67.4
49.3
58.7
C (11)
57.1
37.1
49.2
D (11)
GPT-5.6-luna
57.3
31.9
40.4
D (11)
46.6
24.1
32.4
D (11)
Table S9 : Consistency under option rotation (A2, %) on CaseScope-Teaching and CaseScope-Atlas. Vanilla: mean accuracy over the ten rotations; circular: correct in all ten. Content-consistent: the same option text is chosen in all ten rotations, right or wrong. Top letter: the most frequently chosen position and its share of answers (uniform = 10%). † Single-image model given a numbered mosaic of the fields. Bold / underlined : best/second-best model in the vanilla and circular columns of each source, ranked separately among the proprietary and among the open models.
CaseScope-Teaching
CaseScope-Atlas
Model
HDS
Exact
Top-3
F
L
W
B
N
HDS
Exact
Top-3
F
L
W
B
N
Proprietary MLLMs
Claude Sonnet 5.5
64.2
60.6
80.7
7.3
15.8
2.1
14.2
0.0
41.6
37.8
55.1
7.8
35.2
4.6
14.6
0.0
Claude Haiku 5.5
43.9
39.4
65.1
8.9
22.9
2.5
26.1
0.0
28.8
25.4
38.9
6.9
38.0
7.9
21.9
0.0
GPT-5.6-sol
43.5
39.9
57.1
7.1
31.7
6.2
15.1
0.0
27.6
25.2
33.9
4.8
38.4
15.6
16.0
0.0
GPT-5.6-luna
33.5
30.3
51.6
6.4
35.1
6.0
22.2
0.0
19.8
17.8
25.8
4.1
43.8
13.4
21.0
0.0
Table S11 : Open-ended diagnosis (A1) on CaseScope-Teaching and CaseScope-Atlas (%), graded by the higher of the LLM judge and the ontology match (Sec. 4). HDS: hierarchical diagnostic score (exact entity 1, same family 0.5). Errors, as a share of all answers: F within the family, L another family of the same lineage, W wrong lineage, B wrong behaviour, N no answer. Bold / underlined : best/second-best model in the HDS, exact and top-3 columns of each source, ranked separately among the proprietary and among the open models.
Figure 21
Figure S9 : A2 single-pass accuracy (%) per organ system on (a) CaseScope-Teaching and (b) CaseScope-Atlas, for the same models (proprietary models in the top four rows) and the same organ systems (number of cases in brackets; chance: 10%).
Multimodal large language models (MLLMs) are increasingly used to analyze pathology images. However, dominant multimodal benchmarks in pathology mainly score final diagnostic answers, captions, or reports. These evaluations provide limited insight into whether a model understands the multiscale visual content needed for pathology reasoning and decision-making. We introduce PathVU, a vision-anchored benchmark for fine-grained and multiscale visual understanding in computational pathology. Built from 23 public pathology imaging datasets with human-supervised labels and spatial annotations, PathVU evaluates MLLM understanding in two fields of view: Region FOV for high-resolution local regions and Slide FOV for macro whole-slide views. By converting raw annotations into deterministic task targets, PathVU enables programmatic scoring of region localization, visual recognition, quantity estimation, spatial reasoning, and insufficient-context judgment. The benchmark contains 14 VQA-style tasks, 61,673 images, and 308,070 samples across 28 organs and 7,253,526 annotations. Evaluating 18 representative general-purpose, medical-domain, and pathology-oriented MLLMs, we observe substantial limitations even in advanced models on fine-grained visual tasks across multiscale pathology images. PathVU provides a reproducible basis for developing and evaluating pathology MLLMs with explicit multiscale visual understanding.
Zongyi Chen, Yu Liang, Jie Lin +1
National Institute for Data Science in Health and Medicine, Xiamen University · Institute of Artificial Intelligence Xiamen University, Xiamen, China · Department of Computer Science School of Informatics Xiamen University, Xiamen, China
State-of-the-art pathology foundation models, trained on millions of histology tiles, can fail to preserve tissue similarity when comparisons cross slide or institution boundaries. We show that general-purpose multimodal LLMs, without being trained as pathology foundation models, consistently outperform these specialized models in cross-domain histological similarity judgments. Using a relative similarity framework that we release as the MOSAIC (Model Similarity Assessment across Institutions and Cohorts) benchmark, we evaluate 17 models across 6 datasets and find that pathology encoders often rank same-institution, different-disease tiles as more similar than same-disease, different-institution tiles, a clinically dangerous failure mode invisible to standard within-domain evaluations. LLMs appear less susceptible to this failure, likely because they perform semantic visual comparison of morphology and tissue architecture rather than relying on shortcut features tied to acquisition context. Scaling training data does not resolve the problem for pathology encoders, implicating the learning objective rather than data coverage. Our results expose a fundamental robustness gap in current pathology foundation models and establish multimodal LLMs as a viable alternative for cross-institutional retrieval, dataset harmonization, and multi-site quality control. Code and data will be released upon acceptance.
Whole-slide image (WSI) diagnosis requires identifying diagnostically relevant regions, examining them across magnifications, and integrating multi-scale evidence. However, most existing pathology benchmarks evaluate models on pre-cropped patches or pre-extracted slide features, leaving their ability to acquire evidence directly from gigapixel WSIs largely untested. We introduce PathAgentBench, a benchmark for evaluating evidence-seeking vision-language models (VLMs) across four complementary capabilities: image-to-text matching for evidence interpretation, text-to-image retrieval for evidence verification, diagnostic-region localization for evidence acquisition, and multi-scale reasoning for evidence integration. The benchmark is organized as a diagnostic tree that links nested regions across magnifications with scale-specific findings and path-level diagnoses. It contains 1,822 TCGA WSIs and 17,135 diagnostic paths annotated by ten board-certified pathologists. An additional private cohort of 190 breast cancer WSIs with detailed annotations is used to evaluate autonomous whole-slide exploration. We evaluate 20 general-purpose, medical, and pathology-specialized models. Leading open-weight models achieve over 93% accuracy in multi-scale reasoning and over 50% accuracy in both cross-modal matching tasks. In contrast, diagnostic-region localization remains challenging: the best text-guided mean intersection-over-union is below 0.09, underperforming a simple center-based heuristic. During autonomous exploration, the unconditional hit rate decreases from 0.522 at low magnification to 0.185 at intermediate magnification and 0.020 at high magnification. These results reveal a pronounced gap between reasoning over curated evidence and acquiring that evidence directly from WSIs. PathAgentBench provides a unified framework for measuring and improving evidence-seeking pathology models.
Dankai Liao, Tianyi Zhang, Yufeng Wu +6
Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore