PathJev: Decision-Native Vision-Language Modelling for Histopathology

Cristian Tommasino; Antonio M. Rinaldi

cs.CVOct 23, 2026

PathJev: Decision-Native Vision-Language Modelling for Histopathology

Authors: Cristian Tommasino, Antonio M. Rinaldi

Organizations: DIETI, University of Naples Federico II, Naples, Italy

Abstract

Vision-language models used for decisions in histopathology rarely return calibrated probabilities over the candidates a user cares about. Decision-native models write a question and a candidate set chosen at run time into the input and return a normalised distribution from a small decision head in one forward pass. We build PathJev-P, a decision-native model for histopathology: MedGemma-1.5-4B with LoRA adapters and a decision head, trained on 40,000 patch states from three public collections whose expert labels are turned into typed questions with organ-specific candidate sets and a "none of these" option. In a comparison over 3 seeds with an analysis plan fixed before testing, on test sets without documented overlap with the training data of the backbone, PathJev-P is more accurate than a generative model trained with the same backbone, data and selection rule, the zero-shot pathology model CONCH and a general medical decision model (balanced accuracy 0.541 against 0.492, 0.345 and 0.213); its advantage over the generative model comes from two organs absent from training. The calibration advantage depends on how the generative model's probabilities are read out. With the summed candidate log-likelihood used in previous work, PathJev-P has the lower calibration error (ECE 0.204 against 0.279); with a length-normalised read-out of the same generative model, examined post hoc, the two have similar mean calibration error (0.201) and the generative model is better calibrated on the unseen organs, where PathJev-P remains over-confident. A supervised linear probe on a pathology foundation model is better calibrated in every setting. In a qualitative whole-slide demonstration, PathJev-P labels unsupervised tissue clusters with abstention and separates carcinoma from other tissue while running the model on a fraction of the tiles.

Figures & tables

Figure 1: Overview of PathJev. (1) Expert labels of six training sets from three public collections are turned deterministically into typed decisions: a patch carries a Choice (one of $K$ candidates, including “none of these”), Noul (yes/no) and Score (ordered levels) question; the orange candidate is the label. (2) The image is encoded by MedSigLIP and enters the Gemma-3 decoder as tokens, followed by the question and all candidates, each closed by a marker token. One forward pass yields hidden states at the markers; a decision head shared by all questions sums a per-candidate score, a question–candidate interaction and a set-attention term, and a softmax returns the distribution. A generative read-out, as implemented here, needs one forward pass per candidate. Training: Stage A fits the head on the frozen 4-bit backbone; Stage B adds LoRA adapters. Whole-slide use is shown in Figure 3 . Patches: SICAPv2 (CC BY 4.0). Bar heights in (2) are illustrative.

Set	Organ	Role	Classes	States	Groups	Group unit
HepatoBench	liver	clean, unseen domain	7	5,000 of 87,301	5,000	image §
HiESD	stomach	clean, unseen domain	10	5,000 of 23,679	44	patient
BRACS	breast	clean, in domain	7	570	30	patient
SICAPv2	prostate	clean, in domain	4	2,122	21	patient
SPIDER colorectal	colon	seen source	13	5,000 of 7,485	158	slide
SPIDER thorax	lung	seen source	13	5,000 of 5,569	38	slide

Table 1: Test sets of the main Choice question. States: fixed class-balanced draw shared by all methods (of the available states). Groups: units resampled by the cluster bootstrap. SPIDER has no patient identifiers, so its groups are finer than patients. § HepatoBench releases no patient or slide identifiers and consecutive image indices appear to come from the same tissue, so its images are resampled as independent units, which understates the uncertainty of comparisons that involve it.

Trained on our data
	Clean average		Unseen organs		In domain		Seen source
Method	bAcc $\uparrow$	ECE $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$
PathJev-P (ours)	0.541 ±.015	0.204 ±.002	0.390 ±.015	0.301 ±.012	0.692 ±.019	0.108 ±.013	0.781 ±.017	0.049 ±.025
(b) Gen-SFT	0.492 ±.006	0.279 ±.006	0.277 ±.026	0.446 ±.017	0.707 ±.015	0.113 ±.018	0.810 ±.006	0.044 ±.011
Zero-shot
(a) OmniMed-Jev	0.213	0.517	0.253	0.540	0.173	0.493	0.212	0.577

Table 2: Balanced accuracy (bAcc) and expected calibration error (ECE) on the main Choice question, averaged over the sets of each group; the clean average covers the unseen organs (HepatoBench, HiESD) and the in-domain sets (BRACS, SICAPv2). Trained methods: mean

\pm

SD over 3 seeds. Bold: best language-model-based value per column. † On the unseen organs the probe is fitted with labels of the test domain.

Figure 2: Reliability diagrams of the main Choice question before temperature scaling, one panel per clean test set: accuracy against top-candidate confidence in 15 equal-width bins (every non-empty bin; marker area

\propto

share of the set’s states). Trained methods: one line per seed. Sparse bins (small markers) are noisy. The dashed diagonal is perfect calibration.

	Temperature fitted on
Method	none	val. mix	in-domain val.	other unseen organ	same set (oracle)
PathJev-P (ours)	2.39	2.08	1.99	1.78	1.71
(b) Gen-SFT	3.70	2.87	2.66	2.03	1.92
(a) OmniMed-Jev	3.76	1.98	2.09	1.97	1.97
(c) CONCH	4.41	1.77	2.22	1.77	1.75
(e) Inner-Jev	5.06	2.15	2.18	2.17	2.11

Table 3: Post hoc: negative log-likelihood on the unseen organs (lower is better) after one temperature per method, fitted on different data; seed mean for trained methods. Bold: best language-model-based value per column. The oracle is fitted on the evaluated set itself and is not a deployable procedure. Brier scores in Table 14 .

Figure 3: Whole-slide use of PathJev. Tiles of the slide are embedded with H-optimus-0 and clustered (PCA, Gaussian mixture with the number of components chosen by BIC, soft memberships $r_{ik}$ ). PathJev-P is run on the 16 most typical tiles of each cluster (4 shown for two clusters; one forward pass per tile answers all questions), and the cluster’s distribution $P(c\mid k)$ is the mean of their raw probabilities. A cluster abstains when the top probability of its tissue-type distribution (11 candidates) is below 0.5. Cluster probabilities are propagated to every tile as $P_{i}(c)=\sum_{k}r_{ik}P(c\mid k)$ ; their calibration was not evaluated. HiESD slide of Figure 4 (CC BY 4.0); bar heights are illustrative.

Panel of Figure 4 — Figure 4: Whole-slide demonstration on two of the four HiESD slides (top: slide with adenocarcinoma; bottom: slide without; the other two are in Appendix G ). Panels: thumbnail; region annotation (tiles whose mask is $\geq$ 90% one class); GMM clusters of H-optimus-0 tile features; cluster labels from PathJev-P (grey: cluster abstains, top probability $<0.5$ ); PathJev-P applied to every tile (reference); probability of adenocarcinoma propagated from the clusters. The number of clusters chosen by BIC reached the upper bound of the search range (30) on three of the four slides.

Slide	Patient	Tiles	Labelled	Clusters	tiles	clusters	all tiles	clusters	all tiles	clusters	all tiles
					PathJev-P	AUROC		Sensitivity		Specificity
1a7624bc	P22	2,731	713	30	471	0.983	0.946	0.912	0.912	0.993	0.845
46e7dccd	P27	7,165	2,073	30	480	0.942	0.928	0.920	0.824	0.896	0.879
c6a80707	P38	3,279	1,388	30	480	–	–	–	–	1.000	0.970
e492f29a	P22	1,043	230	18	288	–	–	–	–	1.000	0.987

Table 4: Whole-slide demonstration, adenocarcinoma presence question against the region annotation (descriptive; metrics added after inspecting slide 1a7624bc). Labelled: tiles whose annotation is at least 90% one class, the only tiles evaluated. PathJev-P tiles: cluster representatives run through the model. Sensitivity and specificity at

P(\text{yes})>0.5

; –: no carcinoma in the slide.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Clean average		Unseen organs		In domain
Method	bAcc $\uparrow$	ECE $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$
PathJev-P (ours)	0.542	0.199	0.392	0.290	0.692	0.108
(a) OmniMed-Jev	0.213	0.491	0.254	0.489	0.173	0.493
(c) CONCH	0.349	0.415	0.335	0.510	0.363	0.319
(e) Inner-Jev	0.288	0.609	0.230	0.696	0.346	0.521
(d) Linear probe †	0.718	0.049	0.760	0.024	0.677	0.074

Appendix

Table 5: Secondary analysis (plan amendment A1): the main metrics on the full test files instead of the capped draws, for every method except the generative baseline (not scored on the full files for compute reasons); seed mean for PathJev-P. BRACS and SICAPv2 are not capped, so the in-domain columns equal Table 2 .

Unseen organs
Set	Metric	PathJev-P	Gen-SFT	OmniMed-Jev	CONCH	Inner-Jev	Probe †
HepatoBench	bAcc $\uparrow$	0.512	0.414	0.267	0.354	0.303	0.971
	ECE $\downarrow$	0.305	0.426	0.575	0.496	0.525	0.007
	NLL $\downarrow$	1.862	3.240	3.798	3.911	3.671	0.098
HiESD	bAcc $\uparrow$	0.267	0.140	0.239	0.301	0.158	0.572
	ECE $\downarrow$	0.297	0.465	0.506	0.478	0.746	0.034

Appendix

Table 6: Main Choice question per test set (no averaging over sets): balanced accuracy, ECE and NLL before temperature scaling; seed mean for trained methods. The probe is cross-fitted within HepatoBench and HiESD.

Quantity	PathJev-P (3 seeds)	Gen-SFT (3 seeds)
Selected configuration	lr $2\times 10^{-4}$ , 3 epochs	lr $2\times 10^{-4}$ , budget of 5 PathJev-P epochs
Training items per pass	40,000 states	170,390 single-decision examples
Optimiser steps at checkpoint	200 (A) + 2,500 (B)	5,000
Planned steps (whole run)	3,950	6,250
Items per optimiser step	32	32
Passes at checkpoint (Stage B)	2.00	0.94

Appendix

Table 7: Training budget of the evaluated checkpoints (from the run folders).

Dataset	MedSigLIP / MedGemma 1.5	OmniMed-Jev fine-tuning	CONCH v1	H-optimus-0
SPIDER (HISTAI)	U	N	U	U
BRACS	U	N	U	U
SICAPv2	U	N	U	U
HepatoBench	U	N	R	U
HiESD	U	N	R	U

Appendix

Table 8: Overlap between datasets and model training data. D: documented; N: none documented and the model’s fine-tuning data are listed; U: undisclosed training data, overlap cannot be excluded; R: dataset released after the model; the release date alone does not exclude overlap, because the images may predate it.

	H1a: $\Delta$ ECE		H1b: $\Delta$ bAcc		Supported
HepatoBench groups	vs Gen-SFT	vs CONCH	vs Gen-SFT	vs CONCH	H1a	H1b
Image (primary)	$-$ 0.075	$-$ 0.199	+0.049	+0.196	yes	yes
	[ $-$ 0.088, $-$ 0.064]	[ $-$ 0.237, $-$ 0.158]	[+0.038, +0.063]	[+0.161, +0.218]
Blocks of 50	$-$ 0.075	$-$ 0.199	+0.049	+0.196	yes	yes
	[ $-$ 0.089, $-$ 0.063]	[ $-$ 0.237, $-$ 0.156]	[+0.038, +0.063]	[+0.161, +0.218]
Blocks of 200	$-$ 0.075	$-$ 0.199	+0.049	+0.196	yes	yes

Appendix

Table 9: A3.1: H1a/H1b (clean macro, PathJev-P minus comparator, seed mean, 95% marginal intervals) with different HepatoBench groups, and without HepatoBench.

	Clean average			Unseen organs			In domain
Read-out	bAcc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
PathJev-P (reference)	0.541	0.204	1.586	0.390	0.301	2.394	0.692	0.108	0.777
Gen-SFT, sum (primary)	0.492	0.279	2.257	0.276	0.446	3.700	0.709	0.112	0.814
Gen-SFT, sum + end-of-answer	0.492	0.279	2.257	0.276	0.446	3.700	0.709	0.112	0.814
Gen-SFT, length-normalised	0.492	0.201	1.508	0.289	0.110	1.965	0.696	0.293	1.050

Appendix

Table 10: A3.4: generative baseline with the pre-registered (summed) read-out of the candidate log-likelihoods, the same plus the end-of-answer token, and a length-normalised read-out, seed mean; main Choice question. First row: PathJev-P, from the runs of Table 2 , for reference. Bold: best value per column.

		Clean bAcc $\uparrow$			Clean ECE $\downarrow$					“other” (%)
Method	Perm.	orig.	mean	range	orig.	mean	range	TV $\downarrow$	flips (%) $\downarrow$	orig.	perm.
PathJev-P	10	0.541	0.545	0.542–0.549	0.204	0.199	0.197–0.202	0.050	7.3	0.0	0.0
Gen-SFT	5	0.498	0.501	0.493–0.509	0.281	0.275	0.266–0.288	0.060	7.6	0.0	0.0
OmniMed-Jev	10	0.213	0.239	0.215–0.251	0.517	0.481	0.448–0.525	0.147	18.9	49.8	46.1

Appendix

Table 11: A3.5: random candidate orders on the clean sets. All methods use the same ten orders; Gen-SFT (seed 1 only) uses the first five. “orig.” is the stored run in the original order (Gen-SFT: seed 1, so it differs from the three-seed mean of Table 2 ); permuted runs used L40 GPUs and the stored runs H100, so part of the flips are numerical. TV: mean total-variation distance between a state’s distribution under a permutation and under the original order; flips: share of states whose predicted candidate changes; “other”: share of predictions of the “other” candidate, in the original order and over the permutations. Bold: best value per column.

	Pre-resize 448 (primary)				Pre-resize 896
Group	bAcc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$	bAcc $\uparrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$
Clean average	0.213	0.517	3.269	1.185	0.207	0.519	3.291	1.191
Unseen organs	0.253	0.540	3.765	1.200	0.251	0.542	3.776	1.202
In domain	0.173	0.493	2.774	1.169	0.164	0.495	2.806	1.181
Seen source	0.212	0.577	5.261	1.269	0.210	0.581	5.348	1.277

Appendix

Table 12: A3.3: released OmniMed-Jev with images pre-resized to 448 (primary) or 896 pixels.

	HepatoBench		HiESD		BRACS		SICAPv2
Method	max TV	bAcc	max TV	bAcc	max TV	bAcc	max TV	bAcc
PathJev-P	$3.6\times 10^{-3}$	0.143	$6.3\times 10^{-3}$	0.100	$3.6\times 10^{-3}$	0.143	$9.5\times 10^{-4}$	0.250
Gen-SFT	0	0.143	0	0.100	0	0.143	0	0.250
OmniMed-Jev	$1.1\times 10^{-3}$	0.000	$7.3\times 10^{-4}$	0.100	$1.3\times 10^{-3}$	0.000	$3.7\times 10^{-4}$	0.000
Chance	–	0.143	–	0.100	–	0.143	–	0.250

Appendix

Table 13: A3.2: 100 states per clean set with every image replaced by a uniform grey image. Max TV: largest total-variation distance between two outputs within the set. Constant predictions with balanced accuracy at chance show that, in this check, the constant text of the input does not separate the classes; they say nothing about pretraining contamination or patient independence.

	Temperature fitted on
Method	none	val. mix	in-domain val.	other unseen organ	same set (oracle)
PathJev-P (ours)	0.843	0.790	0.772	0.723	0.715
(b) Gen-SFT	1.074	0.979	0.948	0.834	0.796
(a) OmniMed-Jev	1.200	0.821	0.863	0.818	0.817
(c) CONCH	1.094	0.778	0.880	0.773	0.777
(e) Inner-Jev	1.349	0.876	0.886	0.892	0.860

Appendix

Table 14: Post hoc: Brier score on the unseen organs after one temperature per method, as in Table 3 .

		Raw			Validation temperature
Set	Method	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HepatoBench	PathJev-P	0.305	0.750	1.862	0.247	0.698	1.592
	Gen-SFT	0.426	0.960	3.240	0.360	0.888	2.436
	OmniMed-Jev	0.575	1.225	3.798	0.178	0.798	1.878
	CONCH	0.496	1.091	3.911	0.077	0.760	1.577
	Inner-Jev	0.525	1.160	3.671	0.144	0.859	2.023

Appendix

Table 15: Calibration per clean test set before (raw) and after a temperature fitted on the validation mix (scaled); seed mean for trained methods.

Set	Method	Acc $\uparrow$	Macro-F1 $\uparrow$	aECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AURC $\downarrow$
HepatoBench	PathJev-P	0.512 ±.022	0.459 ±.043	0.305 ±.017	0.750 ±.030	1.862 ±.186	0.254 ±.011
	Gen-SFT	0.414 ±.041	0.338 ±.058	0.426 ±.026	0.960 ±.063	3.240 ±.282	0.375 ±.054
	OmniMed-Jev	0.267	0.255	0.575	1.225	3.798	0.518
	CONCH	0.354	0.387	0.496	1.091	3.911	0.654
	Inner-Jev	0.303	0.184	0.525	1.160	3.671	0.580
	Probe †	0.971	0.971	0.005	0.045	0.098	0.003

Appendix

Table 16: Secondary metrics of the main Choice question, before temperature scaling; mean

\pm

SD over seeds for trained methods. † Cross-fitted within HepatoBench and HiESD (Section 5.1 ).

Question	Method	AUROC $\uparrow$	ECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$
HepatoBench: tumour	PathJev-P	0.893 ±.005	0.103 ±.031	0.498 ±.115	0.132 ±.024
	Gen-SFT	0.883 ±.023	0.163 ±.038	0.757 ±.205	0.180 ±.033
	OmniMed-Jev	0.734	0.107	0.593	0.117
	Inner-Jev	0.707	0.787	4.438	0.781
HiESD: adenocarcinoma	PathJev-P	0.897 ±.005	0.086 ±.007	0.463 ±.028	0.127 ±.004
	Gen-SFT	0.882 ±.014	0.142 ±.010	0.726 ±.047	0.166 ±.010

Appendix

Table 17: Exploratory (not pre-registered): Noul and Score decisions on the clean test sets, before temperature scaling; seed mean for trained methods. Only HiESD has a Score question among the unseen sets.

Panel of Figure 5 — Figure 5: The two HiESD slides of the demonstration not shown in Figure 4 (top: slide with adenocarcinoma; bottom: slide without). Panels as in Figure 4 .

Computational Pathology

Computational pathology uses algorithms to analyze digitized tissue images for diagnosis, prognosis, biomarker discovery, and clinical decision support.

See all 277 papers

Vision-Language Models as Zero-Annotation Oracles in Histopathology
Jun 15, 2026Vishal Jain, Giorgio Buzzanca, Sarah Cechnicka +6Computational Pathology Histopathology Image Segmentation
Imperial College London, London, UK · Leiden University Medical Center, Leiden, The Netherlands · KU Leuven, Leuven, Belgium +3
See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology
Sep 28, 2026Chengyang Zhang, Wenchuan Zhang, Bo Li +8Computational Pathology Medical VQA
College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore +1
HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction
Mar 20, 2026Ruicheng Yuan, Zhenxuan Zhang, Anbang Wang +5Computational Pathology Medical Report Generation
College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · Department of Bioengineering and Imperial-X, Imperial College London, London, UK · Department of Pathology, Xiangtan Maternal and Child Health Hospital, Xiangtan, Hunan, China +1

Jun 15, 2026J/K move · Enter open · S save

Vision-Language Models as Zero-Annotation Oracles in Histopathology

Vishal Jain, Giorgio Buzzanca, Sarah Cechnicka, Maarten Naesens+5

Imperial College London, London, UK · Leiden University Medical Center, Leiden, The Netherlands · KU Leuven, Leuven, Belgium+3

Computational Pathology Histopathology Image Segmentation Medical VLMs Few-Shot Medical Image Segmentation Zero-Shot Medical Image Segmentation Whole-Slide Image Analysis

Open paper AskPDF arXiv

Foreground segmentation is the critical first step of every computational pathology pipeline, yet existing methods rely on hand-tuned heuristics or supervised models that overfit to narrow stain and scanner distributions, failing silently on specialised stains such as Jones silver or Elastica van Gieson. We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task and leverages general-purpose vision-language models (VLMs) as zero-annotation oracles. Our key insight is that tissue-versus-background discrimination is a natural-image recognition problem, not a histopathological one, so VLMs trained on internet-scale corpora generalise where domain-specific models cannot. We introduce Leica-75, a benchmark of 75 renal transplant whole-slide images spanning three stain families. On Leica-75, our method achieves the highest segmentation quality on out-of-distribution stains (Dice 0.858 +/- 0.027 on Jones, 0.853 +/- 0.041 on EVG) with 7x lower cross-stain variance than the best supervised baseline, while remaining competitive on in-distribution H&E. Few-shot prompting with automatically curated exemplars (Auto-context) rescues hard cases on Stress-32 (n=32), a curated stress-test subset (Dice 0.470 to 0.819 for the 2B model). VLM-based annotation review matches human expert consensus (kappa=0.989 for blur detection; mean precision/recall grading accuracy 0.708 vs. human 0.646 for segmentation mask review). The resulting pseudo-labels are used to distil lightweight student models that are as performant as the teacher model while running for a fraction of the cost. Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.

Abstract

Computational Pathology

Explore similar work

Vision-Language Models as Zero-Annotation Oracles in Histopathology

See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology

HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction