CaseScope: A Multi-Field Benchmark for Case-Level Pathology Diagnosis with Multimodal LLMs
Authors: Wenchuan Zhang, Chengyang Zhang, Fengling Li, Ting Yin, Chen Shu, Jie Chen, Shuwan Zhang, Xiaoyan Wu, Zhang Zhang, Yuhao Yi, Jiancheng Lv, Hong Bu
Organizations: Department of Pathology, West China Hospital, Sichuan University · Institute of Clinical Pathology, West China Hospital, Sichuan University · College of Computer Science, Sichuan University · Department of Pathology, Shengjing Hospital of China Medical University · Department of Pathology, Renmin Hospital of Wuhan University
Pathologists rarely diagnose from a single image: they scan a section at low power for architecture, zoom into selected fields for cytology, and integrate the evidence across views and scales. Pathology benchmarks for multimodal large language models (MLLMs), however, mostly ask about one image or require whole-slide pipelines, and the few that show several fields draw nested regions from cancer slides of one public cohort. We introduce CaseScope, a benchmark of diagnosis from several curated fields of one lesion: CaseScope-Teaching, 436 cases from a residency-training image library, reconstructed, de-identified, harmonised against a WHO-based ontology and re-examined by a pathologist wherever an automated pre-review raised a doubt, and CaseScope-Atlas, 800 cases from 88 published atlases (387 of them figure series of one disease), labelled from the captions by two LLMs that must agree. They cover 843 entities across 18 organ systems and pose 4,679 items in six tasks, from open-ended and ten-way diagnosis with ontology-guided hard negatives under circular evaluation to the verification and location of morphological findings. Across twenty-eight open and four proprietary MLLMs, diagnoses are unstable -- the best open and proprietary models are correct under all ten option rotations on 17.9% and 61.7% of the teaching cases -- and wrong answers favour the classic look-alike; open models make little use of the additional fields on teaching cases, where the one-line specimen site matters more, although on atlas cases with three or more fields the fields count about as much as the site; and most open models verify findings barely above chance (median 62.0% vs. 50%, best 81.5%), gaining a median +3.4 points from images; proprietary models reach 83.2-93.2%. Rankings broadly replicate between the sources. The evaluation code is available at https://github.com/Wenchuan-Zhang/CaseScope.
Figures & tables
Figure 1
Benchmark
Size
Exp.
MF
Case
HN
Rob.
Single image or region
PathVQA [ 24 ]
32.8k
✗
✗
✗
✗
✗
Quilt-VQA [ 51 ]
1.3k
(✓)
✗
✗
✗
✗
PathQABench [ 41 ]
105
✓
✗
✓
(✓)
✗
PathMMU [ 52 ]
33.4k
✓
✗
✗
(✓)
✗
PathBench [ 55 ]
6.3k
✓
✗
–
(✓)
✗
Table 1: Pathology benchmarks for MLLMs (selection; full comparison and size scopes in Table S1 ). Size : number of evaluation items. Exp. : pathologist review; MF : separate fields of one lesion; Case : patient diagnosis; HN : close distractors; Rob. : circular evaluation or consistency under controlled input changes. (✓): partial; ✗: not reported; –: not applicable.
Figure 2: Composition of the two sources. (a) Share of each source’s cases per primary organ system, coloured by behaviour; numbers: cases. (b) Share of cases by number of H&E fields; a teaching case keeps all its fields, of which at most eight are shown to the models.
Figure 3: A1 diagnosis on two CaseScope cases. Top: the prompt, identical for both cases except for the number of fields and the site. Each card gives the diagnosis, an excerpt of the response (green/red: findings consistent/inconsistent with the reference; grey: the rest) or the differential the model returned, and an editorial note. Ticks/crosses follow the A1 judge.
Task
Teach.
Atlas
Chance
Metric
A1
Open-ended diagnosis
436
800
–
HDS, Top- k
A2
Diagnosis, 10-way MCQ
436
800
10%
CircularEval
B
Behaviour triage, 4-way
380
762
25%
CircularEval
C
Ordinal grading, 2–4-way
44
–
1/ n
Circ., QWK
R1
Finding verification, T/F †
390
195
50%
Stmt./item acc.
G1
Finding location, 2–8-way
–
436
1/ n
CircularEval
Table 2: Task suite. Items per source (Teach.: CaseScope-Teaching), counted before circular rotation. Chance is the vanilla (single-pass) expectation.
CaseScope-Teaching
CaseScope-Atlas
A1
A2
A2 v
B
Miss ↓
C
C κ
R1
Δimg
A1
A2
A2 v
B
Miss ↓
R1
Δimg
G1
Random choice
–
0.0
10.1
0.5
49.7
11.4
0.08
–
–
–
0.0
9.9
0.9
50.9
–
–
9.2
Proprietary MLLMs
Claude Sonnet 5.5
64.2
61.7
78.3
75.5
11.4
79.5
0.81
93.2
+10.2 ∗
41.6
49.1
68.5
61.5
21.6
80.8
+6.0 ∗
61.5
Claude Haiku 5.5
43.9
39.0
61.0
57.4
10.3
54.5
0.55
83.2
+19.5 ∗
28.8
29.9
52.9
47.0
25.4
73.6
+0.0
48.2
GPT-5.6-sol
43.5
49.3
67.4
55.5
19.4
61.4
0.64
92.9
+12.6 ∗
27.6
37.1
57.1
41.9
35.4
78.6
+5.6 ∗
45.6
Table 3: Results on CaseScope-Teaching and CaseScope-Atlas (%; tasks and metrics as in Sec. 4 ). A2, B, C, G1: CircularEval; A2 v : single-pass accuracy. Miss: critical miss rate. C κ : QWK. Δimg : R1 with minus without the images ( ∗ : 95% paired bootstrap interval excludes 0). † Single-image model given a mosaic of the fields. –: not applicable. Bold / underlined : best/second best per column except Δimg , separately for proprietary and open models. 95% half-widths reach 4.8 points on CaseScope-Teaching and 3.4 on CaseScope-Atlas; per-model intervals: Tables S6 and S7 .
Figure 4: Do models use the fields? Change of the open models’ A2 circular accuracy when only the highest-power field is given (left) or the specimen site is removed (right), against all fields with the site; paired case bootstrap, 95% CI; filled: p<0.05 ; numbers: change in points.
Benchmark
Venue
Visual input
Size
QA construction
Expert
Multi- field
Case dx
Hard neg.
Robust
Single-image and region-based benchmarks
PathVQA [ 24 ]
arXiv’20
1 image
32.8k QA
Rule-based
✗
✗
✗
✗
✗
Quilt-VQA [ 51 ]
CVPR’24
1 image
1.3k QA
LLM-generated
(✓)
✗
✗
✗
✗
PathQABench [ 41 ]
Nature’24
1 ROI
105 QA
Expert-written
✓
✗
✓
(✓)
✗
PathMMU [ 52 ]
ECCV’24
1 image
33.4k QA
LLM-generated
✓
✗
✗
(✓)
✗
PathBench [ 55 ]
TMI’25
1 image / 1 WSI
6.3k QA
NR
✓
✗
–
(✓)
✗
Table S1 : Pathology benchmarks for MLLMs: full version of Table 1. Visual input, reported dataset size, QA construction and evaluation properties are compared; subset scopes and special input settings are specified below.
Scale-VQA [ 47 ]
PathScale-VQA [ 46 ]
CaseScope (ours)
Status
MICCAI 2026
arXiv 2026, extends Scale-VQA
this work
Image source
177 TCGA WSIs, 15 organs
373 TCGA WSIs, 14 organ sites
residency teaching library (436 cases) and 88 published atlases (800 cases), 18 organ systems
One item
zoom path: a region selected at 10 × and corresponding regions at 40 × and 200 ×
teaching: rule-based questions and distractors from expert labels harmonised to a WHO-based ontology, LLM pre-review followed by pathologist adjudication, R1 statements LLM-generated; atlas: labels, statements and finding locations from the captions by LLMs; R1 statements and atlas labels verified by a pathologist
Table S2 : CaseScope vs. cross-scale pathology benchmarks. Facts as reported in [ 47 , 46 ] .
Figure S1 : From the library as filed to benchmark cases. The steps of this section, each with the count after it, shown on real items: a solitary fibrous tumour picture filed under four chapters and two names (C0025; three shown; SFT: solitary fibrous tumour), the re-encoded copy I0288 of picture I0020, the gross photograph I0019 leaving case C0065, and the five fields of the phyllodes case of Fig. 1 ordered from low to high power. The input is the representative set of Fig. 1b, as filed in the teaching library.
Diagnostic entity
T
A
Diagnostic entity
T
A
Oral & salivary (Teaching 30 cases; Atlas 31)
Acinic cell carcinoma
1
1
Granulomatous cheilitis
–
1
Adenoid cystic carcinoma
1
1
Kimura disease
1
–
Adenomatoid odontogenic tumour
–
1
Lichen planus
1
–
Ameloblastic carcinoma
1
1
Lymphangioma (lymphatic malformation)
1
–
Ameloblastic fibrosarcoma
–
1
Lymphoepithelial sialadenitis (Sjögren syndrome)
1
1
Table S4 : Diagnostic entities of CaseScope. T: CaseScope-Teaching cases of the entity filed under the organ system (an entity filed under several systems is listed in each); A: its CaseScope-Atlas case (totals in the last row). Grouped by organ system and alphabetised by the canonical English name, reading down the left list and then the right list within each group.
Figure S4 : One teaching case, five tasks: the prompts and the answers of twelve models. Case C0247 (malignant phyllodes tumour, breast). Top: the five fields and, per task, the prompt (its shared opening once), the options with the reference in green and, for R1, the eight statements with their reference truth values. Bottom: each model’s answers at rotation 0 (blue rows: proprietary models). A1: the diagnosis (green/red: reference entity or synonym/other), the differential the model returned (reference mentions in green) and, for the two reasoning models, an excerpt of the reasoning (green/red: findings consistent/inconsistent with the reference; ellipses mark omissions). A2, B, C: the option chosen, the number of rotations answered correctly and the option’s text. R1: the four judgements per level (green: matches the reference) and whether all four were right; Patho-R1-7B returned no parseable high-power answer. Qwen3.5-9B reached the output limit while still reasoning on A1, C and low-power R1 and answered after the stop-thinking instruction (Sec. S10 ).
Figure S5 : G1 on an atlas item. Five panels of a Bartholin gland cyst (vulva) from one atlas entry, each with its own atlas caption (the captions are not shown to the models); the described finding is the caption of image 4, the reference (green frame). Middle: the prompt and its options. Bottom: each model’s answer at option rotation 0, the image it names, the number of option rotations answered correctly (the options rotate, the images keep their order) and the response: the letter alone or, for the two reasoning models, excerpts of the reasoning (green/red: statements consistent/inconsistent with the panels and their captions). Qwen3.5-9B reached the output limit while still reasoning and answered after the stop-thinking instruction (Sec. S10 ).
Figure S6 : Teaching vs. Atlas. Each point is a model: A2 circular accuracy (left) and A1 hierarchical score (right) on CaseScope-Teaching and on CaseScope-Atlas; dashed: equal scores; ρ : Spearman correlation over the models shown.
Figure S7 : R1 with and without the images on CaseScope-Teaching (left) and CaseScope-Atlas (right) (the proprietary, then the open models, each block in the order of Teaching accuracy with images). Share of finding statements judged correctly from the text alone (grey) and with the fields (violet; open: gain not significant, paired case bootstrap); Δ : with minus without the images, in points.
A1
A2
A2 v
B
Miss ↓
C
C κ
R1
Δimg
Random choice
–
0.0
10.1
0.5
49.7
11.4
0.08
–
–
Proprietary MLLMs
Claude Sonnet 5.5
64.2 ± 4.5
61.7 ± 4.4
78.3
75.5
11.4
79.5
0.81
93.2
+10.2 ∗
Claude Haiku 5.5
43.9 ± 4.6
39.0 ± 4.6
61.0
57.4
10.3
54.5
0.55
83.2
+19.5 ∗
GPT-5.6-sol
43.5 ± 4.5
49.3 ± 4.8
67.4
55.5
19.4
61.4
0.64
92.9
+12.6 ∗
GPT-5.6-luna
33.5 ± 4.1
31.9 ± 4.2
57.3
51.1
33.6
56.8
0.61
87.1
+15.4 ∗
Table S6 : Complete CaseScope-Teaching results with intervals (%). Columns and marks as in Table 3; ± : half-width of the 95% case-bootstrap interval.
A1
A2
A2 v
B
Miss ↓
R1
Δimg
G1
Random choice
–
0.0
9.9
0.9
50.9
–
–
9.2
Proprietary MLLMs
Claude Sonnet 5.5
41.6 ± 3.3
49.1 ± 3.4
68.5
61.5
21.6
80.8
+6.0 ∗
61.5 ± 5.0
Claude Haiku 5.5
28.8 ± 3.1
29.9 ± 3.2
52.9
47.0
25.4
73.6
+0.0
48.2 ± 4.7
GPT-5.6-sol
27.6 ± 3.1
37.1 ± 3.4
57.1
41.9
35.4
78.6
+5.6 ∗
45.6 ± 4.9
GPT-5.6-luna
19.8 ± 2.8
24.1 ± 3.0
46.6
32.7
46.4
73.1
+8.5 ∗
39.9 ± 4.6
Table S7 : Complete CaseScope-Atlas results with intervals (%). Columns and marks as in Table 3; ± as in Table S6 , also for G1. G1 rotates the options while the images keep their order: naming one image, whichever it is, in every rotation is right on 31.5% of the items on average (always image 2: 31.7%), independent guesses on 8.5%.
CaseScope-Teaching
CaseScope-Atlas
Model
All
High-power only
No site
All
High-power only
No site
Qwen2.5-VL-3B
7.3
7.3 (+0.0)
2.3 (−5.0)
10.0
6.4 (−3.6)
7.8 (−2.3)
Qwen2.5-VL-7B
6.0
8.0 (+2.1)
1.4 (−4.6)
9.8
8.0 (−1.8)
6.1 (−3.6)
Qwen2.5-VL-32B
7.6
5.5 (−2.1)
1.8 (−5.7)
11.0
7.0 (−4.0)
6.4 (−4.6)
Qwen2-VL-7B
7.8
7.3 (−0.5)
2.3 (−5.5)
8.2
7.1 (−1.1)
6.6 (−1.6)
InternVL2.5-8B
8.5
8.7 (+0.2)
2.1 (−6.4)
8.8
7.5 (−1.2)
5.4 (−3.4)
Table S8 : Field-set ablation on CaseScope-Teaching and CaseScope-Atlas (A2, CircularEval, %). Same cases and options, different evidence: all fields with the specimen site (All), only the highest-power field, all fields without the specimen site; in brackets the change from All in points.
CaseScope-Teaching
CaseScope-Atlas
Model
Vanilla
Circular
Content- consistent
Top letter (share)
Vanilla
Circular
Content- consistent
Top letter (share)
Proprietary MLLMs
Claude Sonnet 5.5
78.3
61.7
67.2
D (10)
68.5
49.1
57.1
C (11)
Claude Haiku 5.5
61.0
39.0
46.8
B (12)
52.9
29.9
36.9
B (11)
GPT-5.6-sol
67.4
49.3
58.7
C (11)
57.1
37.1
49.2
D (11)
GPT-5.6-luna
57.3
31.9
40.4
D (11)
46.6
24.1
32.4
D (11)
Table S9 : Consistency under option rotation (A2, %) on CaseScope-Teaching and CaseScope-Atlas. Vanilla: mean accuracy over the ten rotations; circular: correct in all ten. Content-consistent: the same option text is chosen in all ten rotations, right or wrong. Top letter: the most frequently chosen position and its share of answers (uniform = 10%). † Single-image model given a numbered mosaic of the fields. Bold / underlined : best/second-best model in the vanilla and circular columns of each source, ranked separately among the proprietary and among the open models.
CaseScope-Teaching
CaseScope-Atlas
Model
HDS
Exact
Top-3
F
L
W
B
N
HDS
Exact
Top-3
F
L
W
B
N
Proprietary MLLMs
Claude Sonnet 5.5
64.2
60.6
80.7
7.3
15.8
2.1
14.2
0.0
41.6
37.8
55.1
7.8
35.2
4.6
14.6
0.0
Claude Haiku 5.5
43.9
39.4
65.1
8.9
22.9
2.5
26.1
0.0
28.8
25.4
38.9
6.9
38.0
7.9
21.9
0.0
GPT-5.6-sol
43.5
39.9
57.1
7.1
31.7
6.2
15.1
0.0
27.6
25.2
33.9
4.8
38.4
15.6
16.0
0.0
GPT-5.6-luna
33.5
30.3
51.6
6.4
35.1
6.0
22.2
0.0
19.8
17.8
25.8
4.1
43.8
13.4
21.0
0.0
Table S11 : Open-ended diagnosis (A1) on CaseScope-Teaching and CaseScope-Atlas (%), graded by the higher of the LLM judge and the ontology match (Sec. 4). HDS: hierarchical diagnostic score (exact entity 1, same family 0.5). Errors, as a share of all answers: F within the family, L another family of the same lineage, W wrong lineage, B wrong behaviour, N no answer. Bold / underlined : best/second-best model in the HDS, exact and top-3 columns of each source, ranked separately among the proprietary and among the open models.
Figure 21
Figure S9 : A2 single-pass accuracy (%) per organ system on (a) CaseScope-Teaching and (b) CaseScope-Atlas, for the same models (proprietary models in the top four rows) and the same organ systems (number of cases in brackets; chance: 10%).
National Institute for Data Science in Health and Medicine, Xiamen University · Institute of Artificial Intelligence Xiamen University, Xiamen, China · Department of Computer Science School of Informatics Xiamen University, Xiamen, China