VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

Luca L. Weishaupt; Simone de Brot; Javier Asin; Llorenç Grau-Roma; Nic G. Reitsam; Andrew H. Song; Dongmin Bang; Stefan T. Kaluziak

cs.CVAug 26, 2026

VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

Authors: Luca L. Weishaupt, Simone de Brot, Javier Asin, Llorenç Grau-Roma, Nic G. Reitsam, Andrew H. Song, Dongmin Bang, Stefan T. Kaluziak, +4 more

Organizations: Harvard-MIT HST · Mass General Brigham · Harvard Medical School · COMPATH, University of Bern · UC Davis · University of Augsburg · UT MD Anderson Cancer Center · TU Dresden · University of Lausanne

Abstract

Pathology vision-language models are advancing rapidly, yet existing benchmarks remain focused on human tissue, particularly oncology, leaving non-human pathology largely unaddressed. This gap is especially important in toxicologic pathology, where microscopic tissue examination of laboratory animals is a core component of preclinical drug safety assessment. To address it, we introduce VIPER, the first expert-curated benchmark for vision-language model evaluation in toxicologic pathology. VIPER contains 1,251 questions associated with 419 H&E-stained rat histology images across seven organ systems, covering multiple-choice, KPrim, and free-text formats. All questions were curated and validated by board-certified veterinary pathologists. In total, we benchmarked 16 models, including two newly introduced veterinary-pathology models, seven human pathology-specialized models, and seven general-purpose frontier models. The results identify a substantial domain gap between veterinary and human pathology, expose the risk of over-diagnosis of normal tissue in frontier models, and show that domain-specific training remains critical for visually grounded predictions. VIPER data and evaluation code are available at https://github.com/mahmoodlab/viper.

Figures & tables

Figure 1: VIPER benchmark overview. (a) VIPER contains 419 H&E-stained rat histology images from seven organ systems and 1,251 questions spanning multiple-choice (MCQ), KPrim, and free-text. (b) Overview of the proposed expert-guided data curation pipeline. (c) Reader study set up with three veterinary pathologists certified by the European College of Veterinary Pathologists (ECVP), denoted VP1–VP3, and a board-certified physician pathologist (PP1) trained in human pathology. (d) Example of questions from VIPER. VP: Veterinary Pathologist, PP: Physician Pathologist.

Model	Domain	MCQ	KPrim	Free-Text	Overall
$\textsc{ToxScribe}_{\text{Qwen}}$	Veterinary pathology	67.1 [63.2,71.0]	61.8 [58.1,65.6]	58.3 [54.4,62.4]	62.4 [60.2,64.6]
$\textsc{ToxScribe}_{\text{Gemma}}$	Veterinary pathology	65.2 [61.4,69.0]	64.1 [60.4,67.9]	54.3 [50.2,58.3]	61.2 [59.0,63.5]
GPT-5.4	General-purpose	58.5 [54.2,62.5]	54.3 [50.6,58.1]	55.1 [50.8,59.3]	56.0 [53.7,58.3]
Gemma 4	General-purpose	60.7 [56.7,64.7]	54.1 [50.2,57.9]	48.3 [44.0,52.6]	54.4 [52.0,56.7]
Qwen 3.5-27B	General-purpose	60.0 [56.0,63.8]	46.6 [42.9,50.4]	50.6 [46.4,54.9]	52.4 [50.2,54.7]
PathChat+	Human pathology	58.7 [54.7,62.9]	41.5 [37.8,45.4]	52.7 [48.7,56.7]	51.0 [48.7,53.3]

Table 1: VIPER benchmark results. Overall score and breakdown by question type. Best result per column in bold , second-best underlined .

n=1,251{}

questions. 95% bootstrap confidence intervals (10,000 resamples). MCQ scores are mean across 5 cyclic-shift rotations of the answer ordering.

Figure 2: Qualitative comparison of three model responses across representative questions in VIPER. Each panel shows an H&E image from rat tissue together with the benchmark question, the ground-truth answer authored by $\mathrm{VP}_{1}$ (used as the evaluation reference, see subsection 3.3 ), and responses from ToxScribe , PathChat+, and GPT-5.4. The examples illustrate four of the seven VIPER question categories ( Appendix Table 2 ): a over-reading probe, distinguishing normal histomorphology from pathologic change in salivary gland, b spatial localization, identifying the in-image location of a focus of altered tubular epithelium in kidney, c anatomy identification, recognizing a non-salivary tissue type within a salivary gland section, and d pathology identification, naming the predominant glandular epithelial change in prostate / seminal vesicle. Underlined text marks the core answer content.

		MCQ		KPrim		Free-text
Rater	Set	$n$	Acc. [95% CI]	$n$	ETH [95% CI]	$n$	Score [95% CI]
$\mathrm{VP}_{2}$	A	30	93.3 [83.3, 100.0]	37	75.7 [63.5, 86.5]	33	76.8 [66.0, 86.5]
$\mathrm{VP}_{2}$	B	29	93.1 [82.8, 100.0]	33	72.7 [60.6, 84.8]	38	49.3 [37.0, 61.5]
$\mathrm{VP}_{3}$	A	30	83.3 [70.0, 96.7]	37	81.1 [70.3, 90.5]	33	75.5 [64.5, 85.3]
$\mathrm{PP}_{1}$	B	29	72.4 [55.2, 86.2]	33	65.2 [51.5, 78.8]	38	50.1 [37.2, 62.9]

Table 2: Pathologist reader performance on VIPER by question format. Scores are computed against the

\mathrm{VP}_{1}

gold standard and reported as percent (

0

–

100

). MCQ is scored as percent correct. KPrim uses the ETH half-point rule (4/4 correct statements

\rightarrow 100

, 3/4

\rightarrow 50

\leq 2/4\rightarrow 0

) and the reported value is the mean across questions. Free-text is scored on

[0,100]

by an LLM judge ( gpt-5.4 as

0.7\cdot

diagnostic accuracy +

0.3\cdot

completeness, both rated 0–10, mapped to percent), with the per-question scoring rubric included in the prompt. Bracketed values are 95% bootstrap CIs (

10^{4}

item-level resamples).

Reader	Question type	$\Delta$ [%]	95% CI [%]	$p$
$\mathrm{VP}_{3}$	MCQ	31.6	$[7.9,52.3]$	0.003
$\mathrm{VP}_{3}$	KPrim	35.6	$[18.4,52.1]$	<0.001
$\mathrm{PP}_{1}$	MCQ	35.7	$[11.8,59.4]$	0.003
$\mathrm{PP}_{1}$	KPrim	12.4	$[-6.1,30.3]$	0.096

Table 3: Effect of image access on reader performance.

\Delta=\mathrm{acc}_{+\mathrm{img}}-\mathrm{acc}_{-\mathrm{img}}

. Positive values mean the image helps. 95% CIs and one-sided

p

-values from

10^{4}

-resample unpaired bootstrap of

\Delta

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Organ system	Named structures	Images	MCQ	KPrim	Free-Text	Total
Urinary	kidney, urinary bladder, ureter, urethra	137	137	135	136	408
Hepatobiliary	liver	86	86	86	86	258
Endocrine	thyroid	56	56	56	56	168
Male reproductive and sex glands	prostate, ampullary gland, seminal vesicle, epididymis, ejaculatory duct, coagulating gland, periurethral gland, testis	50	50	49	50	149
Digestive	salivary gland, stomach, colon, duodenum, small intestine, large intestine, pancreas	40	40	38	40	118
Respiratory	lung	28	28	28	28	84

Appendix

Appendix Table 1 : VIPER composition by organ system. VIPER covers 23 distinct named anatomic structures drawn from seven organ systems.

Category	$n$	Definition	Prototype example
Anatomy Identification	362	Name, verify, or identify a normal anatomic structure, tissue type, region, or cell type. Includes true/false verification of anatomy claims (either polarity) and “unremarkable $X$ is represented” seeds.	Q : “Which renal region is represented?” A : Cortex
Over-Reading Probe	240	Hallucination-resistance probe. Two routes: true/false “ $X$ is within normal limits / changes are absent” (answer true), or a specific false disease claim (answer false). Excludes false-anatomy claims.	Q : “Mild myositis is present.” A : false
Spatial Localization	227	“Where in the image is $X$ ?” where the answer is a spatial region (quadrant, corner, edge, zone). Target may be anatomy or a lesion; output shape defines the category.	Q : “Where is tubular necrosis most evident?” A : Top third and subcapsular along the right edge
Pathology Identification	221	Recognise that a specific pathology is present: open-ended naming of a real diagnosis, true/false verification of a lesion (answer true), or interpret-region-of-interest yielding a pathologic entity.	Q : “Define the main histopathologic change.” A : Suppurative-necrotising hepatitis, focal
Feature Characterization	78	Describe attributes, distribution, grade, severity, or functional state of a named feature, lesion, or organ. Includes organ functional states (thyroid activity, bladder distension).	Q : “Add two descriptors for the hepatocellular necrosis.” A : Coagulative, acute, focal
Artifact Recognition	63	Technical, processing, scanning, or slide-quality issues: artefacts, tissue absence, coverslip edges, autolysis, slide completeness.	Q : “Name the artefactual change.” A : Stain precipitate

Appendix

Appendix Table 2: VIPER question categories.

n

: number of questions per category. Definitions and prototype examples are drawn from the seed annotations.

		MCQ (choice-level)			KPrim (per-statement)
Pair	Set	$n$	Agr.	$\kappa$	$n$	Agr.	$\kappa$
VP1 $\times$ VP2	A	30	0.933	0.917	148	0.858	0.716
VP1 $\times$ VP2	B	29	0.931	0.914	132	0.848	0.697
VP1 $\times$ VP3	A	30	0.833	0.792	148	0.905	0.811
VP1 $\times$ HP1	B	29	0.724	0.655	132	0.818	0.636
VP2 $\times$ VP3	A	30	0.867	0.833	148	0.899	0.797

Appendix

Appendix Table 3 : Pairwise inter-annotator agreement. MCQ uses choice-level agreement (5 categories,

p_{e}=1/5

). KPrim uses per-statement T/F agreement (

p_{e}=1/2

). VP1 is the benchmark author treated as a rater who always picks the correct choice. VP3

\times

HP1 has no overlapping questions.

Set	Question type	$n$	Krippendorff’s $\alpha$	Fleiss’s $\kappa$
A (3 vet. pathologists)	MCQ	30	0.875	0.874
A (3 vet. pathologists)	KPrim	148	0.775	0.774
B (2 vet. + 1 physician pathologist)	MCQ	29	0.789	0.786
B (2 vet. + 1 physician pathologist)	KPrim	132	0.637	0.636

Appendix

Appendix Table 4: Summary of pairwise agreements among pathologists. Three-rater reliability per set, summarizing all pairwise agreements in a single number. Krippendorff’s

\alpha

and Fleiss’s

\kappa

use different chance-correction conventions but produce near-identical values on this data. MCQ is analyzed at the choice level (5 categories); KPrim is analyzed per statement (2 categories). Set A raters are

\mathrm{VP}_{1},\mathrm{VP}_{2},\mathrm{VP}_{3}

; Set B raters are

\mathrm{VP}_{1},\mathrm{VP}_{2},\mathrm{PP}_{1}

Pair	Set	$n$	Mean score (a)	Mean score (b)	Pearson $r$	ICC(2,1)
$\mathrm{VP}_{2}\times\mathrm{VP}_{3}$	A	33	76.8	75.5	0.651	0.658
$\mathrm{VP}_{2}\times\mathrm{PP}_{1}$	B	38	49.3	50.1	0.422	0.428

Appendix

Appendix Table 5: Free-text inter-rater reliability on the two sets where two readers overlap. Each response is scored on

[0,100]

by a frozen two-axis LLM judge ( gpt-5.4 ) with the per-question scoring rubric included in the prompt. Mean scores per rater are shown in percent; reliability is reported as ICC(2,1), the absolute-agreement single-measurement intraclass correlation, which is the continuous-score analogue of Cohen’s

\kappa

and the standard choice when the scoring unit is a continuous number rather than a discrete category. Pearson

r

is shown for reference; it measures linear association but ignores absolute level. ICC and Pearson

r

are unitless and reported as decimals.

Model	GPT-5.4	GPT-5.4 (replicate)	Claude Opus 4.7
$\textsc{ToxScribe}_{\text{Qwen}}$	$58.3~[54.3,62.2]$	$59.1~[55.2,63.2]$	$56.0~[52.1,59.8]$
GPT-5.4	$55.0~[50.8,59.3]$	$56.0~[51.8,60.2]$	$54.4~[50.2,58.5]$
PathChat+	$52.7~[48.7,56.7]$	$53.8~[49.8,57.6]$	$50.3~[46.5,54.1]$

Appendix

Appendix Table 6: Free-text composite scores under independent LLM judges. Each cell is the mean composite score over the same

418{}

free-text predictions, scored under the same prompt and composite formula as in Table 1 . Values are in percent with 95% percentile bootstrap CIs (

10^{4}

item-level resamples). The first column reproduces the headline GPT-5.4 judge values from Table 1 ; the second column is a replicate scoring with the same GPT-5.4 judge to bound within-judge stochasticity; the third column is an independent judge from a different vendor (Anthropic).

Comparison type	Model A	Model B	$\Delta$ (%) [95% CI]	$p$ -value
Best vet pathology vs. best human pathology	ToxScribe (Qwen3.5)	PathChat+	+11.5 [+9.0, +13.9]	$<$ 0.001
Best vet pathology vs. best frontier model	ToxScribe (Qwen3.5)	GPT-5.4	+6.5 [+4.0, +9.0]	$<$ 0.001
Best frontier model vs. best human pathology	GPT-5.4	PathChat+	+5.0 [+2.4, +7.6]	$<$ 0.001

Appendix

Appendix Table 7 : Pair-wise comparison between lead models. Paired item-level bootstrap significance tests across model-category leaders on the VIPER benchmark (

n=1,251{}

questions, 10,000 resamples, one-sided

p

-value for

\Delta\leq 0

). Category leaders are the highest-scoring model in each group.

\Delta

is defined as the difference in accuracies: acc(Model A) - acc(Model B).

	Anatomy identification	Over-reading probe	Spatial localization	Pathology identification	Feature characterization	Artifact recognition	Feature quantification
$n$	362	240	227	221	78	63	60
ToxScribe (Qwen)	60.9 [55.8, 65.9]	77.2 [71.6, 82.5]	54.0 [47.0, 60.5]	50.3 [44.0, 56.5]	62.9 [53.1, 72.4]	57.0 [44.9, 68.8]	45.1 [32.1, 57.5]
ToxScribe (Gemma)	57.0 [51.4, 62.4]	80.5 [75.7, 85.5]	50.3 [44.1, 56.6]	48.3 [42.1, 54.2]	61.0 [50.4, 71.6]	62.7 [50.2, 75.0]	54.6 [41.4, 67.9]
GPT-5.4	60.4 [55.2, 65.5]	55.1 [48.1, 61.5]	51.4 [44.4, 58.2]	49.2 [42.8, 55.8]	61.0 [49.9, 71.8]	55.4 [44.3, 66.1]	42.9 [29.9, 55.9]
Gemma 4	50.0 [44.8, 55.3]	60.1 [53.5, 66.5]	52.3 [45.5, 59.4]	44.5 [37.8, 51.0]	44.9 [32.9, 56.5]	57.2 [44.8, 68.8]	44.0 [31.4, 56.8]
Qwen 3.5-27B	50.5 [45.1, 55.8]	58.8 [52.3, 65.4]	42.5 [35.8, 48.7]	43.6 [37.2, 50.5]	50.1 [40.2, 60.3]	46.0 [33.4, 59.8]	39.5 [27.0, 52.3]

Appendix

Appendix Table 8: VIPER mean performance (%) by question category. 95% bootstrap confidence intervals (2,000 item-level resamples). Best mean per column in bold.

n

is the number of questions assigned to that category on the frozen

1,251{}

-question set. Cell means are item-level averages within each (model, category); they are not directly comparable to the leaderboard overall scores (mean-of-means across question types).

Model	Urinary ( $n$ =408)	Hepatobiliary ( $n$ =258)	Endocrine ( $n$ =168)	Male Repro. ( $n$ =149)	Digestive ( $n$ =118)	Respiratory ( $n$ =84)	Cardiovascular ( $n$ =66)
ToxScribe (Qwen3.5)	65.0	64.3	67.0	43.7	62.1	65.9	66.0
ToxScribe (Gemma 4)	65.3	62.0	69.5	38.9	54.1	67.1	66.9
GPT-5.4	53.9	59.2	60.4	40.4	60.1	58.9	68.6
Gemma 4	54.5	55.6	63.9	35.0	50.0	61.1	67.3
Qwen 3.5-27B	54.4	55.1	53.4	33.5	51.9	59.6	61.4
PathChat+	55.8	50.2	54.6	35.1	52.6	45.0	55.9

Appendix

Appendix Table 9 : Performance by organ system. Overall score (%) per (model, organ system) on the canonical

n=1,251{}

bench, grouped into the seven organ systems of Appendix Table 1 . MCQ scores are the mean across 5 cyclic-shift rotations of the answer ordering. Best per column in bold , second-best underlined . Column headers list the organ system and its question count

n

Question type	$n$	$\mathrm{VP}_{2}$ score	$\mathrm{PP}_{1}$ score	$\Delta$ [95% CI]	$p$
MCQ	29	93.1	72.4	20.7 $[6.9,37.9]$	0.001
KPrim	33	72.7	65.2	7.6 $[-9.1,24.2]$	0.208
Free text	38	49.3	50.1	-0.8 $[-14.0,13.1]$	0.553

Appendix

Appendix Table 10: Comparison between

\mathrm{VP}_{2}

and

\mathrm{PP}_{1}

on Set B. Paired vet-versus-physician test on Set B (both

\mathrm{VP}_{2}

and

\mathrm{PP}_{1}

answered the same items with image visible). Per-rater scores and

\Delta=\mathrm{acc}(\mathrm{VP}_{2})-\mathrm{acc}(\mathrm{PP}_{1})

are reported in percent; positive

\Delta

means the veterinary pathologist outperforms the physician pathologist. 95% CIs and

p

-values are from

10^{4}

-resample paired bootstrap;

p

is the one-sided probability that

\Delta\leq 0

. Only one physician reader (

\mathrm{PP}_{1}

) participated, which limits the generalizability of the KPrim and free-text null results.

Model	Question type	Normal	Black	Random	No image	$\Delta$ (Normal $-$ No)
ToxScribe (Qwen3.5)	MCQ	67.1	45.1	45.8	43.2	$-$ 24.0
	KPrim	61.8	19.2	–	13.2	$-$ 48.7
	Free-text	58.3	11.4	–	5.7	$-$ 52.6
PathChat+	MCQ	58.7	36.3	34.1	27.5	$-$ 31.2
	KPrim	41.5	18.4	–	15.0	$-$ 26.6
	Free-text	52.7	10.2	–	26.5	$-$ 26.1

Appendix

Appendix Table 11: Image ablation across lead models on VIPER. “Normal” is the question’s correct image; “Black” a solid black image; “Random” a randomly-chosen other benchmark image; “No image” drops the image entirely. Right-most column is the Normal

-

No-image gap. The Random condition is reported on MCQ only; the KPrim and free-text random-image conditions used an earlier protocol and are omitted for comparability.

Model	Atlas	EduContent	PubMed	PathCLS	SocialPath	Overall
	$n{=}799$	$n{=}1{,}683$	$n{=}2{,}787$	$n{=}1{,}632$	$n{=}1{,}567$	$n{=}8{,}468$
ToxScribe (Qwen)	78.7 [75.8,81.6]	69.1 [66.8,71.2]	69.0 [67.2,70.7]	42.1 [39.7,44.5]	70.3 [68.0,72.5]	65.0 [64.0,66.0]
ToxScribe (Gemma)	77.6 [74.6,80.5]	69.8 [67.6,71.9]	72.3 [70.7,74.0]	52.1 [49.7,54.5]	70.5 [68.2,72.7]	68.1 [67.1,69.1]
PathChat+	84.0 [81.5,86.5]	72.9 [70.8,75.0]	75.2 [73.6,76.9]	51.3 [48.9,53.7]	71.0 [68.7,73.3]	70.2 [69.2,71.2]
GPT-5.4	74.2 [71.2,77.2]	70.2 [67.9,72.3]	73.2 [71.6,74.9]	56.2 [53.8,58.6]	69.8 [67.6,72.1]	68.8 [67.8,69.8]

Appendix

Appendix Table 12 : Cross-benchmark generalization on PathMMU. (

n=8{,}468

; 95% bootstrap CIs, 10k resamples). Both ToxScribe variants show some transfer to human pathology: ToxScribe (Gemma) is statistically similar to GPT-5.4 (

\Delta=-0.7

pp,

P=0.20

, paired bootstrap). PathChat+ is the best performer, leading over ToxScribe (Qwen) by 2.1% (

P<0.001

Explore similar work

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology
Oct 8, 2026Fanqi Cheng, Kuo Gong, Shangke Liu +15VLM Evaluation Medical VLMs
New York University, New York, NY, USA · Columbia University, New York, NY, USA · Weill Cornell Medicine, New York, NY, USA +9
Do Pathology Vision-Language Models Truly See Pathology?
Jul 23, 2026Chengyang Zhang, Wenchuan Zhang, Bo Li +10VLM Evaluation Computational Pathology
College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore +2
PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?
Jul 30, 2026Zongyi Chen, Yu Liang, Jie Lin +1Computational Pathology Medical VQA
National Institute for Data Science in Health and Medicine, Xiamen University · Institute of Artificial Intelligence Xiamen University, Xiamen, China · Department of Computer Science School of Informatics Xiamen University, Xiamen, China

Oct 8, 2026J/K move · Enter open · S save

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology

Fanqi Cheng, Kuo Gong, Shangke Liu, Beidi Zhao+14

New York University, New York, NY, USA · Columbia University, New York, NY, USA · Weill Cornell Medicine, New York, NY, USA+9

VLM Evaluation Medical VLMs Medical Image Benchmarks Computational Pathology Cross-Modal Retrieval

Open paper AskPDF arXiv

Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized. Existing pathology VLM benchmarks largely rely on canonical closed-set prompts or perturb only generic templates, treating language as a fixed evaluation component rather than a variable axis of model behavior. In clinical practice, however, diagnostic language varies across reports, institutions, and candidate diagnoses. We introduce PathLang, a language-centered and clinically grounded zero-shot benchmark. PathLang holds the underlying slides, ground-truth labels, and image-text evaluation direction fixed while systematically varying only the diagnostic language, so that performance differences reflect how a diagnosis is phrased rather than what is imaged. The language variation follows how pathologists actually rephrase diagnoses (terminology, specificity, and reporting style), and all prompts and candidate pools are validated by six board-certified pathologists. PathLang covers four task families: (1) zero-shot classification with image-text alignment analysis, (2) cross-modal retrieval, (3) paraphrase robustness, including semantic-equivalence paraphrases, length and reporting-style variation, and prompt ensembling, and (4) open-vocabulary diagnosis retrieval over four candidate pools with distinct forms of semantic competition. Across nine VLMs and five public datasets spanning four organs, we find that performance is highly sensitive to clinically equivalent paraphrases, varies substantially across forms of semantic competition, and that image-text alignment quality does not necessarily translate into inter-class separability. We release the prompt corpus, candidate pools, pre-computed text embeddings, and evaluation code at https://anonymous.4open.science/r/PathLang.

Abstract

Explore similar work

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology

Do Pathology Vision-Language Models Truly See Pathology?

PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?