PathoJev: Efficient Pathology Decisions from Shared Visual Context

Chengyang Zhang; Wenchuan Zhang; Bo Li; Xinyu Liu; Jiaming Yang; Mengran Li; Jie Chen; Wei Ju

cs.CVOct 29, 2026

PathoJev: Efficient Pathology Decisions from Shared Visual Context

Authors: Chengyang Zhang, Wenchuan Zhang, Bo Li, Xinyu Liu, Jiaming Yang, Mengran Li, Jie Chen, Wei Ju,

Organizations: College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore · School of Intelligent Systems Engineering, Sun Yat-sen University

Abstract

Pathology workflows require multiple visual judgments about the same slide, from identifying tumor type to assessing grade and invasion. Generating and parsing a separate textual response for each structured question adds latency between visual evidence and usable decisions. Building on the System One paradigm, we introduce PathoJev, a pathology model that directly predicts answer probabilities for whole-slide and region-level assessment. PathoJev encodes each slide once and evaluates independent question suffixes in batches over the shared visual context. Reusing the hybrid backbone's recurrent states further limits the computation that grows with slide-context length for each additional question. Across three whole-slide benchmarks, the 9B and 27B models achieve mean accuracies of 77.2% and 78.1%, exceeding SlideChat by 5.4 and 6.3 percentage points. Both scales also maintain competitive accuracy on five ROI benchmarks. On WSI-Bench, the 9B model runs 9.1x faster than SlideChat. Shared context accelerates eight-question workloads by 6.9x and 5.9x over separate prefix encoding, rising to 11.2x and 8.9x for 64 questions. These results provide a reusable probabilistic interface for combining multiple visual judgments in whole-slide and local tissue assessment. Code: https://github.com/ChyaZhang/PathoJev.

Figures & tables

Figure 1: PathoJev: shared visual context for multiple pathology decisions. (a,b) Autoregressive answers versus direct probabilities from isolated questions sharing slide context. Re-reading context includes accessing cached prefix keys and values. (c,d) Whole-slide and ROI accuracy–latency comparisons. Accuracy averages each model’s available results across three WSI or five ROI benchmarks. Latency is GPU forward time per question on WSI-Bench and PathMMU-tiny, respectively. Zero-shot WSI baselines use thumbnails.

Figure 2: Overview of PathoJev. Left: WSI patch features and ROI images enter through separate visual pathways. Center: WSI question rows reuse a shared visual prefix independently, with fixed-size recurrent states and shared attention keys and values; a typed head returns answer probabilities. Right: supported output types, test-set decision examples, and marginal computation per additional question as slide-context length increases. The marginal-cost comparison uses Qwen3-8B as a softmax-only baseline.

Model	Region of Interest							Whole Slide
	PathMMU tiny	PathVQA	Quilt VQA	OmniMed VQA	MedXpert QA	Avg.	Latency (s/q)	WSI VQA	WSI Bench	SlideBench TCGA	Avg.	Latency (s/q)
7–13B models
Patho-R1-7B (reasoning) [ 29 ] †	67.6	57.1	70.8	69.7	16.7	56.4	2.291	44.1	67.8	53.9	55.3	2.367
HuatuoGPT-Vision-7B [ 4 ] †	59.5	66.6	64.4	51.5	12.2	50.8	0.068	47.7	66.8	51.1	55.2	0.081
Quilt-LLaVA-v1.5-7B [ 23 ] †	46.4	56.7	70.3	46.9	17.8	47.6	0.066	39.7	49.4	37.4	42.2	0.066
PathGen-LLaVA-13B [ 26 ] †	71.4	62.7	68.2	68.9	31.1	60.5	0.114	43.8	69.1	52.4	55.1	0.116

Table 1: Benchmark accuracy (%,

\uparrow

) and inference latency (s/q,

\downarrow

). Avg. is the unweighted mean over five ROI or three whole-slide benchmarks. Best and second-best results within each model-size group are bold and underlined , with ties marked equally. Latency is measured on PathMMU-tiny for ROI inputs and WSI-Bench for whole-slide inputs, at batch size one.

Variant	Region of Interest							Whole Slide
	PathMMU tiny	PathVQA	Quilt VQA	OmniMed VQA	MedXpert QA	Avg.	Latency (s/q)	WSI VQA	WSI Bench	SlideBench TCGA	Avg.	Latency (s/q)
Autoregressive SFT	74.3	93.8	66.8	68.8	30.0	66.7	0.215	63.8	89.0	74.2	75.7	0.119
PathoJev-9B	71.9	94.6	70.8	75.3	25.6	67.6	0.090	65.4	90.4	75.9	77.2	0.039
$\Delta$ / speedup	$-2.4$	$+0.8$	$+4.1$	$+6.5$	$-4.4$	$+0.9$	$2.4\times$	$+1.5$	$+1.4$	$+1.7$	$+1.5$	$3.0\times$

Table 2: Decision readout versus autoregressive fine-tuning at 9B. Both models use the Qwen3.5-9B backbone, the same training data, two-stage schedule, and step counts; PathoJev starts from JEV-9B and the autoregressive model from Qwen3.5-9B. Accuracy is reported in percent and latency in seconds per question. Avg. follows Table 1 . Best and second-best results are bold and underlined .

Setting	$N=1$	$N=8$	$N=64$	Speedup
PathoJev-9B
Separate encoding	0.12	0.95	7.63	—
Shared prefix	0.12	0.14	0.68	$11.2\times$
PathoJev-27B
Separate encoding	0.28	2.23	17.86	—
Shared prefix	0.28	0.38	2.01	$8.9\times$

Table 3: Prefix-sharing ablation. Median time (seconds per slide,

\downarrow

) over 40 WSI-Bench slides with

N

questions. Speedups compare shared with separate encoding at

N=64

. Questions are repeated when

N

exceeds the available set.

Figure 3: Multi-question inference on WSI-Bench. Median forward time over 40 slides; bands and error bars show 95% bootstrap confidence intervals. Shared-prefix methods cache slide context; separate encoding repeats prefill. Questions are cycled beyond the available set. Offline feature extraction is excluded.

Figure 4: Marginal cost of longer slide contexts. Per-question time and memory increments are calculated from measurements at

N=1

and

N=16

after prefix prefill. Memory uses a log scale. All systems store one copy of the prefix keys and values.

(a) Calibration: ECE (percentage points, $\downarrow$ )
Benchmark	PathoJev-9B	PathoJev-27B
WSI-VQA	6.7	10.5
WSI-Bench	2.0	2.1
SlideBench-TCGA	2.2	4.5

Table 4: Calibration and accuracy at different coverage levels. ECE uses 15 confidence bins at

\tau_{t}=1

without post-hoc calibration. Coverage is the fraction of questions retained by confidence;

\Delta

is the accuracy gain in percentage points.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Stage A	Stage B
Trainable parameters	$\phi$	$\phi,\Delta,\psi$
Training slides used	30%	100%
Epochs	1	2
Optimization steps	244	1,624
Slides per step	8	8
Questions per slide	$\leq 32$	$\leq 64$

Appendix

Table A1 : Two-stage training schedule, shared by the 9B and 27B models. Question counts are per optimization step.

Source	Input	Questions
SlideInstruct	WSI	48,699
WSI-Bench	WSI	34,799
WSI-VQA	WSI	2,311
CF-Path	WSI	126,467
PathVQA	ROI	9,751
PMC-VQA (pathology)	ROI	8,204

Appendix

Table A2 : Training mixture used for pathology adaptation.

	Training	Validation	Test
Questions	126,467	36,372	38,088
Slides	2,926	627	628
Cases	2,721	595	623
TSS groups	111	66	76

Appendix

Table A3 : CF-Path split sizes. TSS denotes tissue source site.

Benchmark / subset	Questions
Whole-slide assessment
WSI-VQA	390
WSI-Bench (closed-ended)	1,141
SlideBench-TCGA	3,127
Local tissue assessment
PathMMU-tiny	1,139

Appendix

Table A4 : Evaluation subsets and question counts.

Benchmark	$\Delta$	Lower	Upper
WSI-VQA	$+1.5$	$-1.8$	$+4.9$
WSI-Bench	$+1.4$	$0.0$	$+2.8$
SlideBench-TCGA	$+1.7$	$+0.5$	$+2.9$
PathMMU-tiny	$-2.4$	$-4.5$	$-0.3$
PathVQA	$+0.8$	$+0.1$	$+1.5$
Quilt-VQA	$+4.1$	$0.0$	$+8.2$

Appendix

Table A5 : Accuracy differences between PathoJev-9B and autoregressive SFT, in percentage points, with paired-bootstrap 95% confidence intervals.

Measure	$32\times 32$	$64\times 64$
Median visual tokens	499	1,755
WSI-VQA	67.7	65.9
WSI-Bench	88.0	84.7
SlideBench-TCGA	73.6	68.8
PathMMU-tiny	75.9	74.4
PathVQA	93.4	93.0

Appendix

Table A6 : Grid-resolution comparison under the initial training-data configuration. Accuracy (%); visual token counts are medians over 300 sampled slides.

Benchmark	With image	No image	Decrease
PathoJev-9B
WSI-VQA	65.4	58.7	6.7
WSI-Bench	90.4	80.3	10.1
SlideBench-TCGA	75.9	63.9	12.0
PathMMU-tiny	71.9	54.5	17.4
PathVQA	94.6	62.1	32.5

Appendix

Table A7 : Visual input ablation. Accuracy (%) with and without the image, and the decrease after image removal in percentage points.

Model	$N=1$	$N=8$	$N=64$
Autoregressive SFT	0.16	0.84	6.34
WSI-LLaVA	0.07	0.31	2.18
SlideChat	0.68	1.47	8.93
PathoJev-9B	0.12	0.14	0.68

Appendix

Table A8 : Median total forward time (seconds per slide,

\downarrow

) over 40 WSI-Bench slides. All models reuse the slide-prefix cache.

Figure E1 : Question scaling on SlideBench-TCGA. Median total forward time over 40 slides; shading and capped error bars show pointwise 95% bootstrap confidence intervals. Timing and repeated-question construction follow the WSI-Bench analysis.

Measure	256	1,024	4,096
PathoJev-9B
Marginal time (ms/question)	1.46	1.61	2.48
Marginal memory (MB/question)	12.1	12.1	37.4
Recurrent layers (ms)	58.1	58.2	58.3
Softmax attention (ms)	19.0	20.9	33.8
PathoJev-27B

Appendix

Table A9 : Costs at three slide-context lengths

L_{s}

with one shared prefix KV copy in every system. Marginal costs are per question; layer timings aggregate 16 rows.

Figure F1 : Eight benchmark questions over one slide. PathoJev-9B answers all eight SlideBench-TCGA questions for TCGA-XF-AAMJ through a batched question evaluation after shared-prefix prefill. Bars show the probability assigned to each predicted answer; the right column shows SlideChat outputs. These are the benchmark’s question set, including its staging item. The displayed 0.13 s and 2.05 s are median times over 40 slides at $N=8$ .

Figure F2 : The same question across different slides. Top: one correctly answered example per ground-truth grade, selected with seed 0. Bottom: predicted grade counts and question-level accuracy over all 40 WSI-Bench slides asked this question. PathoJev-9B achieves 80.0% accuracy with images; without images it predicts well differentiated for all slides and achieves 5.0%. Insets show $400\,\mu\mathrm{m}$ fields for display; the model receives the whole-slide grid.

Figure F3 : Answer distributions for whole-slide decisions. Examples answered correctly by PathoJev-9B and incorrectly by SlideChat. (a) The model assigns 0.96 probability to papillary renal cell carcinoma. (b) It assigns 0.57 to focal angioinvasion, with the remaining probability distributed across the other options. Selection and display crops follow Appendix F.1 .

Figure F4 : Probability readout across five ROI benchmarks. One microscopy question per benchmark answered correctly by both PathoJev-9B and PathoJev-27B. Paired bars show each model’s probabilities over the supplied alternatives. Screen-recorded frames are cropped to the slide viewer for display.

Figure F5 : Confidence differences between model scales. Both models answer each question correctly, with examples selected for a top-1 probability increase of at least 0.2 from 9B to 27B. The displayed probabilities increase from 0.63 to 0.99 on the WSI example and from 0.49 to 0.95 on the ROI example.

Pathology Visual Question Answering

See all 18 papers

Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning
Feb 28, 2026Wentao Huang, Weimin Lyu, Peiliang Lou +8Pathology Visual Question Answering Medical Visual Question Answering
Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA +1
PathoArgus: Advancing Evidence-Grounded Long-Context Visual Reasoning across Gigapixel Whole-Slide and Multi-Slide Case Contexts
Aug 18, 2026Bowen Liu, Qixiang Zhang, Xiaomeng LiPathology Visual Question Answering Medical Visual Question Answering
The Hong Kong University of Science and Technology
Look Beyond the Question: Preserve-Focus-Fuse for Whole-Slide Image Question Answering
Oct 21, 2026Xiangyu Li, Jincheng Li, Tianle Du +6Visual Question Answering Pathology Visual Question Answering
Tianjin University · Shanghai AI Laboratory · University of Science and Technology of China +6

Feb 28, 2026J/K move · Enter open · S save

Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning

Wentao Huang, Weimin Lyu, Peiliang Lou, Qingqiao Hu+7

Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA+1

Pathology Visual Question Answering Medical Visual Question Answering Medical VLMs Whole-Slide Image Analysis

Open paper AskPDF arXiv

Computational pathology has advanced rapidly in recent years, driven by domain-specific image encoders and growing interest in using vision-language models to answer natural-language questions about diseases. Yet, the core problem behind pathology question-answering remains unsolved, considering that a gigapixel slide contains far more information than necessary for a given question. Pathologists naturally navigate tissue and morphology complexity by scanning broadly, and zooming in selectively according to the clinical questions. Current models, in contrast, rely on uniform patch sampling or broad attention maps, often attending equally to irrelevant regions while overlooking key visual evidence. In this work, we try to bring models closer to how humans actually examine slides. We propose a question-guided, tissue-aware, and coarse-to-fine retrieval framework, HistoSelect, that consists of two key components: a group sampler that identifies question-relevant tissue regions, followed by a patch selector that retrieves the most informative patches within those regions. By selecting only the most informative patches, our method becomes significantly more efficient: reducing visual token usage by 70% on average, while improving accuracy across three pathology QA tasks. Evaluated on 356,000 question-answer pairs, our approach outperforms existing methods and produces answers grounded in interpretable, pathologist-consistent regions. Our results suggest that bringing human-like search and attention patterns into WSI reasoning is a promising direction for building practical and reliable pathology VLMs. Code is available at https://github.com/winston52/HistoSelect.

Abstract

Pathology Visual Question Answering

Explore similar work

Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning

PathoArgus: Advancing Evidence-Grounded Long-Context Visual Reasoning across Gigapixel Whole-Slide and Multi-Slide Case Contexts

Look Beyond the Question: Preserve-Focus-Fuse for Whole-Slide Image Question Answering