Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models

Wenhao Zhang; Zhongliang Zhou; Shiyuan Zhang; Yiqing Yang; Pinqiao Wang; Lehan Yang; Hanyin Wang; John Kang

cs.CVSep 26, 2026

Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models

Authors: Wenhao Zhang, Zhongliang Zhou, Shiyuan Zhang, Yiqing Yang, Pinqiao Wang, Lehan Yang, Hanyin Wang, John Kang,

Organizations: University of Virginia, Charlottesville · Merck & Co., Inc.

Abstract

Pathology vision-language models (VLMs) are conventionally evaluated by accuracy, but accuracy alone does not measure evidence use: it may conflate dataset contamination, prior knowledge, and image evidence. In a motivating study of lymph-node metastasis prediction, we found that most public pathology VLMs showed minimal differences when changing from feeding the models with whole-slide images, an annotated lesion, or no image at all. To better understand the specific features leveraged by these models, this paper presents two contributions aimed at disentangling these factors. First, we present CleanSlide, a TCGA-based VQA benchmark designed to eliminate image- and question-side contamination. It contains 149K audited multiple-choice questions over 9,985 slides, with patient- and tissue-source-disjoint splits. Every question is audited for option shortcuts, stem leakage, cross-split duplication, and blind solvability. Second, we propose Pair-DPO, a preference loss over counterfactual slide pairs from the same question and source. By controlling for shared confounding factors, Pair-DPO cancels out the question-attributable signal and leaves image evidence as the source of preference. Specifically, each pair consists of two real slides with opposite, verified findings, introducing neither editing artifacts nor unverified labels for diffuse or graded features such as invasion, necrosis, and tumor grade. Experiments show that our method gains 15.29% from image evidence on the CleanSlide, compared with 2.81% for the best published model. On the external CPTAC and BCNB cohorts, our method achieves accuracies of 57.6% and 59.0%, outperforming all other evaluated models by 9.7% and 3.4%, respectively. We will release the benchmark and code.

Figures & tables

Figure 1: CAMELYON16 motivating study (n=129 slides; accuracy, %, with 95% bootstrap CIs). We evaluate 17 models across three evidence conditions—blank, whole slide, and lesion and test whether performance improves as increasingly relevant evidence is provided. $\Delta$ denotes the accuracy difference between whole-slide and lesion conditions.

Figure 2: Overview of CleanSlide and Pair-DPO. (a) CleanSlide splits TCGA slides so that train, validation and test share no patient, site or slide. Its questions are audited for option shortcuts, stem leakage, cross-split duplication and blind solvability, and are checked by pathologists. (b) Pair-DPO asks one question on two real slides of the same cancer type whose reports give opposite findings. Each answer must be preferred on its own slide over the other slide, so the prior cancels within the pair and no image is edited.

Figure 3: Ablations. (a) Sensitivity against the number of steps for the cross-entropy control and for Pair-DPO. (b) Sensitivity against the number of real counterfactual pairs available for training. (c) Sensitivity against the pair-loss weight $\lambda$ at two step budgets.

Figure 4: Natural-image transfer on the decontaminated NaturalBench test (1,182 pairs, 585 complete groups). G-acc is the official group accuracy and Sens the paired sensitivity.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Cross-benchmark comparison. Left: scale of pathology VQA benchmarks; the bar is the number of whole-slide images and the text gives slides and questions. Right: question cleanliness under the four audits of Section 3.1 , in %, lower is cleaner; $*$ marks the cleanest value in each column. Blind solvability is the text-only accuracy of Qwen3-32B. Grey cells are benchmarks that cannot be audited because their data are not released.

	Criterion	All imported	Morphologic families
Text-level integrity
C5	Stem does not give the answer away	100.00 ∗	100.00 ∗
C5	No partial hint either	98.10 [96.09–99.83]	99.70 [99.10–100.00]
C6	Wording is not broken	100.00 ∗	100.00 ∗
C6	Wording is natural	99.71 [99.20–100.00]	99.70 [99.10–100.00]
C4	All distractors plausible but wrong	96.18 [92.70–98.84]	100.00 ∗

Appendix

Table 5: Pathologist quality scores (%). Each score estimates the share of imported questions that meet the criterion. Task families are weighted by their share of the imported tier, and brackets give stratified bootstrap 95% CIs. ∗ No reviewed question failed this criterion. Morphologic families are morphology, microscopy and diagnosis, which hold 83.90% of the imported questions.

Arm	G-acc	Acc	Sens
Qwen3-VL-8B
zero-shot	30.26 [26.50–33.85]	77.71 [76.28–79.13]	1.129 [1.079–1.179]
$+$ CE control	38.86 [35.50–42.22]	80.61 [79.29–81.86]	1.184 [1.145–1.222]
$+$ retrieved image [ Xing et al., 2025 ]	37.44 [33.50–41.37]	80.37 [79.00–81.77]	1.161 [1.119–1.204]
$+$ sym. response [ Liu et al., 2025 ]	37.61 [33.50–41.71]	80.54 [79.06–81.94]	1.260 [1.217–1.302]
$+$ blind-anchored [ Wu et al., 2025b ]	41.03 [36.92–44.96]	81.56 [80.03–82.97]	1.239 [1.198–1.281]

Appendix

Table 6: Natural-image transfer on the decontaminated NaturalBench test (1,182 pairs, 585 complete groups), with group-clustered bootstrap 95% CIs on every number. G-acc is the official group accuracy, which counts a group only when all four of its answers are correct, Acc is the per-question accuracy over both images and Sens is the paired sensitivity of Section 3.2 . The CE control and pair-DPO pool three seeds and the other arms are single seeds. All trained arms see the same data, the same schedule and the same number of updates. For each backbone, bold marks the best and underline the second-best trained arm.

Pathology VQA

See all 16 papers

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models
Oct 19, 2026Wenhao Zhang, Zhongliang Zhou, Yi Ping +2Medical VLMs VLM Evaluation
School of Data Science, University of Virginia, Charlottesville, VA 22903, USA · Merck & Co., Inc., Rahway, NJ 07065, USA
PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image
Jul 21, 2026Dankai Liao, Tianyi Zhang, Yufeng Wu +6VLM Evaluation Medical VLMs
Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore
Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks
Jul 14, 2026Wenhao Zhang, Zhongliang Zhou, John Kang +1VLM Evaluation Data Leakage
University of Virginia · Merck & Co., Inc.

Oct 19, 2026J/K move · Enter open · S save

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

Wenhao Zhang, Zhongliang Zhou, Yi Ping, Zirui Wang+1

School of Data Science, University of Virginia, Charlottesville, VA 22903, USA · Merck & Co., Inc., Rahway, NJ 07065, USA

Medical VLMs VLM Evaluation Visual Evidence Sensitivity in VLMs Faithfulness in VLMs

Open paper AskPDF arXiv

Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced and test four claims with controlled probes that reuse released checkpoints and public data, varying one input at a time. Report text adds prognostic signal: adding a frozen embedding of the stage-redacted report to a slide embedding raises the pooled C-index by 0.019 for progression-free interval and by 0.016 for overall survival across 1{,}633 TCGA patients, and restoring the stage statement adds nothing. Prompts add no measurable accuracy at inference: with features and pooling fixed, zero-shot classification is statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping. Slide-level question answering is barely grounded: SlideChat answers 74.82% of SlideBench-VQA(TCGA) items correctly with the slide and 74.41% without it, whereas its language backbone alone reaches 43.52%. Paraphrasing leaves accuracy unchanged but flips 8--23% of answers. Self-reported gains from agentic navigation and few-shot language priors cannot yet be attributed to language. Language thus adds real but narrow value, and we propose minimum controls that would expose these failure modes in future evaluations.

Abstract

Pathology VQA

Explore similar work

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks