PathoJev: Efficient Pathology Decisions from Shared Visual Context
Authors: Chengyang Zhang, Wenchuan Zhang, Bo Li, Xinyu Liu, Jiaming Yang, Mengran Li, Jie Chen, Wei Ju, Yuhao Yi, Hong Bu, Jiancheng Lv
Organizations: College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore · School of Intelligent Systems Engineering, Sun Yat-sen University
Pathology workflows require multiple visual judgments about the same slide, from identifying tumor type to assessing grade and invasion. Generating and parsing a separate textual response for each structured question adds latency between visual evidence and usable decisions. Building on the System One paradigm, we introduce PathoJev, a pathology model that directly predicts answer probabilities for whole-slide and region-level assessment. PathoJev encodes each slide once and evaluates independent question suffixes in batches over the shared visual context. Reusing the hybrid backbone's recurrent states further limits the computation that grows with slide-context length for each additional question. Across three whole-slide benchmarks, the 9B and 27B models achieve mean accuracies of 77.2% and 78.1%, exceeding SlideChat by 5.4 and 6.3 percentage points. Both scales also maintain competitive accuracy on five ROI benchmarks. On WSI-Bench, the 9B model runs 9.1x faster than SlideChat. Shared context accelerates eight-question workloads by 6.9x and 5.9x over separate prefix encoding, rising to 11.2x and 8.9x for 64 questions. These results provide a reusable probabilistic interface for combining multiple visual judgments in whole-slide and local tissue assessment. Code: https://github.com/ChyaZhang/PathoJev.
Figures & tables
Figure 1: PathoJev: shared visual context for multiple pathology decisions. (a,b) Autoregressive answers versus direct probabilities from isolated questions sharing slide context. Re-reading context includes accessing cached prefix keys and values. (c,d) Whole-slide and ROI accuracy–latency comparisons. Accuracy averages each model’s available results across three WSI or five ROI benchmarks. Latency is GPU forward time per question on WSI-Bench and PathMMU-tiny, respectively. Zero-shot WSI baselines use thumbnails.
Figure 2: Overview of PathoJev. Left: WSI patch features and ROI images enter through separate visual pathways. Center: WSI question rows reuse a shared visual prefix independently, with fixed-size recurrent states and shared attention keys and values; a typed head returns answer probabilities. Right: supported output types, test-set decision examples, and marginal computation per additional question as slide-context length increases. The marginal-cost comparison uses Qwen3-8B as a softmax-only baseline.
Model
Region of Interest
Whole Slide
PathMMU tiny
PathVQA
Quilt VQA
OmniMed VQA
MedXpert QA
Avg.
Latency (s/q)
WSI VQA
WSI Bench
SlideBench TCGA
Avg.
Latency (s/q)
7–13B models
Patho-R1-7B (reasoning) [ 29 ] †
67.6
57.1
70.8
69.7
16.7
56.4
2.291
44.1
67.8
53.9
55.3
2.367
HuatuoGPT-Vision-7B [ 4 ] †
59.5
66.6
64.4
51.5
12.2
50.8
0.068
47.7
66.8
51.1
55.2
0.081
Quilt-LLaVA-v1.5-7B [ 23 ] †
46.4
56.7
70.3
46.9
17.8
47.6
0.066
39.7
49.4
37.4
42.2
0.066
PathGen-LLaVA-13B [ 26 ] †
71.4
62.7
68.2
68.9
31.1
60.5
0.114
43.8
69.1
52.4
55.1
0.116
Table 1: Benchmark accuracy (%, ↑ ) and inference latency (s/q, ↓ ). Avg. is the unweighted mean over five ROI or three whole-slide benchmarks. Best and second-best results within each model-size group are bold and underlined , with ties marked equally. Latency is measured on PathMMU-tiny for ROI inputs and WSI-Bench for whole-slide inputs, at batch size one.
Variant
Region of Interest
Whole Slide
PathMMU tiny
PathVQA
Quilt VQA
OmniMed VQA
MedXpert QA
Avg.
Latency (s/q)
WSI VQA
WSI Bench
SlideBench TCGA
Avg.
Latency (s/q)
Autoregressive SFT
74.3
93.8
66.8
68.8
30.0
66.7
0.215
63.8
89.0
74.2
75.7
0.119
PathoJev-9B
71.9
94.6
70.8
75.3
25.6
67.6
0.090
65.4
90.4
75.9
77.2
0.039
Δ / speedup
−2.4
+0.8
+4.1
+6.5
−4.4
+0.9
2.4×
+1.5
+1.4
+1.7
+1.5
3.0×
Table 2: Decision readout versus autoregressive fine-tuning at 9B. Both models use the Qwen3.5-9B backbone, the same training data, two-stage schedule, and step counts; PathoJev starts from JEV-9B and the autoregressive model from Qwen3.5-9B. Accuracy is reported in percent and latency in seconds per question. Avg. follows Table 1 . Best and second-best results are bold and underlined .
Setting
N=1
N=8
N=64
Speedup
PathoJev-9B
Separate encoding
0.12
0.95
7.63
—
Shared prefix
0.12
0.14
0.68
11.2×
PathoJev-27B
Separate encoding
0.28
2.23
17.86
—
Shared prefix
0.28
0.38
2.01
8.9×
Table 3: Prefix-sharing ablation. Median time (seconds per slide, ↓ ) over 40 WSI-Bench slides with N questions. Speedups compare shared with separate encoding at N=64 . Questions are repeated when N exceeds the available set.
Figure 3: Multi-question inference on WSI-Bench. Median forward time over 40 slides; bands and error bars show 95% bootstrap confidence intervals. Shared-prefix methods cache slide context; separate encoding repeats prefill. Questions are cycled beyond the available set. Offline feature extraction is excluded.
Figure 4: Marginal cost of longer slide contexts. Per-question time and memory increments are calculated from measurements at N=1 and N=16 after prefix prefill. Memory uses a log scale. All systems store one copy of the prefix keys and values.
(a) Calibration: ECE (percentage points, ↓ )
Benchmark
PathoJev-9B
PathoJev-27B
WSI-VQA
6.7
10.5
WSI-Bench
2.0
2.1
SlideBench-TCGA
2.2
4.5
Table 4: Calibration and accuracy at different coverage levels. ECE uses 15 confidence bins at τt=1 without post-hoc calibration. Coverage is the fraction of questions retained by confidence; Δ is the accuracy gain in percentage points.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Stage A
Stage B
Trainable parameters
ϕ
ϕ,Δ,ψ
Training slides used
30%
100%
Epochs
1
2
Optimization steps
244
1,624
Slides per step
8
8
Questions per slide
≤32
≤64
Appendix
Table A1 : Two-stage training schedule, shared by the 9B and 27B models. Question counts are per optimization step.
Source
Input
Questions
SlideInstruct
WSI
48,699
WSI-Bench
WSI
34,799
WSI-VQA
WSI
2,311
CF-Path
WSI
126,467
PathVQA
ROI
9,751
PMC-VQA (pathology)
ROI
8,204
Appendix
Table A2 : Training mixture used for pathology adaptation.
Table A4 : Evaluation subsets and question counts.
Benchmark
Δ
Lower
Upper
WSI-VQA
+1.5
−1.8
+4.9
WSI-Bench
+1.4
0.0
+2.8
SlideBench-TCGA
+1.7
+0.5
+2.9
PathMMU-tiny
−2.4
−4.5
−0.3
PathVQA
+0.8
+0.1
+1.5
Quilt-VQA
+4.1
0.0
+8.2
Appendix
Table A5 : Accuracy differences between PathoJev-9B and autoregressive SFT, in percentage points, with paired-bootstrap 95% confidence intervals.
Measure
32×32
64×64
Median visual tokens
499
1,755
WSI-VQA
67.7
65.9
WSI-Bench
88.0
84.7
SlideBench-TCGA
73.6
68.8
PathMMU-tiny
75.9
74.4
PathVQA
93.4
93.0
Appendix
Table A6 : Grid-resolution comparison under the initial training-data configuration. Accuracy (%); visual token counts are medians over 300 sampled slides.
Benchmark
With image
No image
Decrease
PathoJev-9B
WSI-VQA
65.4
58.7
6.7
WSI-Bench
90.4
80.3
10.1
SlideBench-TCGA
75.9
63.9
12.0
PathMMU-tiny
71.9
54.5
17.4
PathVQA
94.6
62.1
32.5
Appendix
Table A7 : Visual input ablation. Accuracy (%) with and without the image, and the decrease after image removal in percentage points.
Model
N=1
N=8
N=64
Autoregressive SFT
0.16
0.84
6.34
WSI-LLaVA
0.07
0.31
2.18
SlideChat
0.68
1.47
8.93
PathoJev-9B
0.12
0.14
0.68
Appendix
Table A8 : Median total forward time (seconds per slide, ↓ ) over 40 WSI-Bench slides. All models reuse the slide-prefix cache.
Figure E1 : Question scaling on SlideBench-TCGA. Median total forward time over 40 slides; shading and capped error bars show pointwise 95% bootstrap confidence intervals. Timing and repeated-question construction follow the WSI-Bench analysis.
Measure
256
1,024
4,096
PathoJev-9B
Marginal time (ms/question)
1.46
1.61
2.48
Marginal memory (MB/question)
12.1
12.1
37.4
Recurrent layers (ms)
58.1
58.2
58.3
Softmax attention (ms)
19.0
20.9
33.8
PathoJev-27B
Appendix
Table A9 : Costs at three slide-context lengths Ls with one shared prefix KV copy in every system. Marginal costs are per question; layer timings aggregate 16 rows.
Figure F1 : Eight benchmark questions over one slide. PathoJev-9B answers all eight SlideBench-TCGA questions for TCGA-XF-AAMJ through a batched question evaluation after shared-prefix prefill. Bars show the probability assigned to each predicted answer; the right column shows SlideChat outputs. These are the benchmark’s question set, including its staging item. The displayed 0.13 s and 2.05 s are median times over 40 slides at N=8 .
Figure F2 : The same question across different slides. Top: one correctly answered example per ground-truth grade, selected with seed 0. Bottom: predicted grade counts and question-level accuracy over all 40 WSI-Bench slides asked this question. PathoJev-9B achieves 80.0% accuracy with images; without images it predicts well differentiated for all slides and achieves 5.0%. Insets show 400μm fields for display; the model receives the whole-slide grid.
Figure F3 : Answer distributions for whole-slide decisions. Examples answered correctly by PathoJev-9B and incorrectly by SlideChat. (a) The model assigns 0.96 probability to papillary renal cell carcinoma. (b) It assigns 0.57 to focal angioinvasion, with the remaining probability distributed across the other options. Selection and display crops follow Appendix F.1 .
Figure F4 : Probability readout across five ROI benchmarks. One microscopy question per benchmark answered correctly by both PathoJev-9B and PathoJev-27B. Paired bars show each model’s probabilities over the supplied alternatives. Screen-recorded frames are cropped to the slide viewer for display.
Figure F5 : Confidence differences between model scales. Both models answer each question correctly, with examples selected for a top-1 probability increase of at least 0.2 from 9B to 27B. The displayed probabilities increase from 0.63 to 0.99 on the WSI example and from 0.49 to 0.95 on the ROI example.
Computational pathology has advanced rapidly in recent years, driven by domain-specific image encoders and growing interest in using vision-language models to answer natural-language questions about diseases. Yet, the core problem behind pathology question-answering remains unsolved, considering that a gigapixel slide contains far more information than necessary for a given question. Pathologists naturally navigate tissue and morphology complexity by scanning broadly, and zooming in selectively according to the clinical questions. Current models, in contrast, rely on uniform patch sampling or broad attention maps, often attending equally to irrelevant regions while overlooking key visual evidence. In this work, we try to bring models closer to how humans actually examine slides. We propose a question-guided, tissue-aware, and coarse-to-fine retrieval framework, HistoSelect, that consists of two key components: a group sampler that identifies question-relevant tissue regions, followed by a patch selector that retrieves the most informative patches within those regions. By selecting only the most informative patches, our method becomes significantly more efficient: reducing visual token usage by 70% on average, while improving accuracy across three pathology QA tasks. Evaluated on 356,000 question-answer pairs, our approach outperforms existing methods and produces answers grounded in interpretable, pathologist-consistent regions. Our results suggest that bringing human-like search and attention patterns into WSI reasoning is a promising direction for building practical and reliable pathology VLMs. Code is available at https://github.com/winston52/HistoSelect.
Wentao Huang, Weimin Lyu, Peiliang Lou +8
Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA +1
Whole-slide pathology reasoning requires models to integrate gigapixel-scale visual evidence across complete case-linked slides, yet current question-answering benchmarks primarily measure final answer accuracy--a metric vulnerable to linguistic priors and benchmark regularities, and insufficient to establish that predictions are grounded in the supplied tissue. We introduce PathoArgus-Bench, a benchmark and evaluation protocol that explicitly tests the full evidence chain: availability, accessibility, use, and responsiveness. PathoArgus-Bench comprises 22,078 four-choice questions from 4,913 patients across 15 TCGA projects, covering six pathology capabilities across three levels of evidence demand, and operates under a fixed reader budget that retains only a small fraction of the gigapixel context. To further isolate evidence-grounded reasoning, we contribute ESG (Evidence State Quartets), a controlled set of 483 quartets where the question text is fixed while the target WSI set is moved, replaced, or removed, requiring consistent predictions across all states. Evaluating 20 general-purpose, medical, and pathology-specific systems reveals a stark gap: while GPT-5.6 achieves 57.09% overall accuracy and 57.04% on ESG, it correctly completes only 19 of 483 quartets (3.93% QExact), exposing that row-level accuracy does not translate into reliable evidence grounding. We also introduce PathoArgus, a fixed-budget reader that allocates context via question relevance and spatial coverage, attaining 50.39% overall accuracy yet only 1.86% QExact--demonstrating that improved context access alone does not ensure consistent evidence-based prediction. Our benchmark and diagnostics establish that acquiring useful whole-slide context is necessary but far from sufficient, and call for a shift from answer-centric to evidence-grounded evaluation in computational pathology.
Bowen Liu, Qixiang Zhang, Xiaomeng Li
The Hong Kong University of Science and Technology
A question about a whole-slide image rarely names every morphological finding needed to answer it. Question-guided patch selection can therefore overlook supporting morphology, while slide-level compression can dilute focal evidence. Inspired by how pathologists combine broad slide review, question-directed examination, and integration of their findings, we introduce BEYONDPATH, a Preserve-Focus-Fuse framework for whole-slide image question answering. Preserve learns a question-agnostic Morphology Bank through answer supervision across multiple questions about each training slide. Focus retrieves questionspecific evidence from all foreground patches, including regions outside the Bank. Fuse integrates the preserved morphology at the retrieved evidence positions through Morphology-on-Visual-Evidence Fusion, producing a fixed 96-token visual prefix. On WSI-Bench, BEYONDPATH achieves an Avg-9 of 0.769 and leads all eight reported report-generation metrics using the same checkpoint. It also achieves 0.639 zero-shot overall accuracy on SlideBench-BCNB. Controlled ablations show complementary benefits from morphology preservation and question-directed retrieval, and support cross-question supervision and fusion at the evidence positions. These results support looking beyond the question's explicit cues: preserving broader slide morphology while retaining the ability to seek evidence for each question.
Xiangyu Li, Jincheng Li, Tianle Du +6
Tianjin University · Shanghai AI Laboratory · University of Science and Technology of China +6