Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity

Yishu Zhang; Yun Li; Daiwei Zhang

cs.CVSep 26, 2026

Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity

Authors: Yishu Zhang, Yun Li, Daiwei Zhang

Organizations: University of North Carolina at Chapel Hill

Abstract

State-of-the-art pathology foundation models, trained on millions of histology tiles, can fail to preserve tissue similarity when comparisons cross slide or institution boundaries. We show that general-purpose multimodal LLMs, without being trained as pathology foundation models, consistently outperform these specialized models in cross-domain histological similarity judgments. Using a relative similarity framework that we release as the MOSAIC (Model Similarity Assessment across Institutions and Cohorts) benchmark, we evaluate 17 models across 6 datasets and find that pathology encoders often rank same-institution, different-disease tiles as more similar than same-disease, different-institution tiles, a clinically dangerous failure mode invisible to standard within-domain evaluations. LLMs appear less susceptible to this failure, likely because they perform semantic visual comparison of morphology and tissue architecture rather than relying on shortcut features tied to acquisition context. Scaling training data does not resolve the problem for pathology encoders, implicating the learning objective rather than data coverage. Our results expose a fundamental robustness gap in current pathology foundation models and establish multimodal LLMs as a viable alternative for cross-institutional retrieval, dataset harmonization, and multi-site quality control. Code and data will be released upon acceptance.

Figures & tables

Figure 1: Schematic of the relative similarity evaluation framework.

Figure 2: Accuracy for relative similarity analysis across all the experiments.

Pathology foundation models
	HER2ST						CCTGS							TIGER
Method	Adi	Bre	Can	Con	Imm	All	Adi	Lam	Lym	Mus	Nor	Tum	All	Hea	Nec	Str	Tum	All
UNI2H	0.61	0.57	0.38	0.38	0.36	0.46	0.60	0.58	0.56	0.62	0.55	0.62	0.59	0.39	0.21	0.09	0.13	0.19
H-optimus-0	0.59	0.53	0.36	0.40	0.32	0.44	0.59	0.56	0.57	0.59	0.51	0.51	0.55	0.40	0.20	0.17	0.17	0.22
Midnight	0.60	0.56	0.41	0.37	0.40	0.46	0.76	0.70	0.69	0.76	0.75	0.68	0.72	0.50	0.12	0.11	0.16	0.20
CONCH v1.5	0.43	0.52	0.51	0.41	0.38	0.45	0.47	0.59	0.58	0.66	0.53	0.55	0.56	0.32	0.19	0.18	0.20	0.21

Table 1: Cross-slide tile relative similarity accuracy on HER2ST, CCTGS, and TIGER, broken down by positive class

c_{+}

. For each column, the best method is in bold and the second best is underlined . “All” refers to the overall accuracy across all classes. Class abbreviations are listed in Table 15 .

Figure 3: Within-slide (left) and cross-slide (right) relative similarity accuracy on HER2ST. Pathology foundation models lead within-slide accuracy, while LLMs achieve competitive or higher cross-slide accuracy.

Pathology foundation models
Method	CAMELYON16	TCGA	BEETLE
UNI2H	0.61	0.31	0.56
H-optimus-0	0.40	0.18	0.45
Midnight	0.64	0.36	0.48
CONCH v1.5	0.57	0.37	0.47
Virchow2	0.63	0.44	0.57

Table 2: Cross-institution relative similarity accuracy on CAMELYON16 (2 institutions, 2 classes), TCGA (3 institutions, 2 classes), and BEETLE (4 institutions, 3 classes). For each column, the best method is in bold and the second best is underlined . Per-institution breakdowns are in Appendix C.1 .

Figure 4: Example cross-institution comparisons on CAMELYON16. Each row shows one comparison: reference $t_{r}$ (left), positive $t_{p}$ (center, green border, same class from a different institution), negative $t_{n}$ (right, red border, different class from the same institution). Correctness of each model’s choice is shown on the right (✓ correct, $\times$ incorrect).

Figure 5: Within-institution vs cross-institution relative similarity accuracy on CAMELYON16 (left) and TCGA (right). Each point is a model.

Figure 6: Relative similarity accuracy vs model cost/size across all six datasets.

	HER2ST (cross-slide)		CAMELYON16 (cross-inst.)
Prompt	Gemini 3 Flash	GPT-5 Nano	Gemini 3 Flash	GPT-5 Nano
Biology-focused (default)	0.549	0.570	0.705	0.590
Minimal	0.510	0.560	0.668	0.555

Table 3: Effect of LLM prompt on relative similarity accuracy.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Pathology foundation models
Method	Adipose	Breast gl.	Cancer	Connective	Immune inf.	All
UNI2H	0.775	0.759	0.664	0.757	0.712	0.728
H-optimus-0	0.709	0.691	0.682	0.748	0.756	0.719
Midnight	0.706	0.732	0.670	0.695	0.700	0.697
CONCH v1.5	0.706	0.736	0.691	0.689	0.724	0.705
Virchow2	0.741	0.718	0.755	0.739	0.709	0.735

Appendix

Table 4: Within-slide tile relative similarity accuracy on HER2ST, broken down by tissue class.

Figure 7: Within-slide vs cross-slide relative similarity accuracy on HER2ST (overall accuracy). Each point is a method, colored/marked by category (pathology foundation models = blue circles, general foundation models = green squares, LLMs = red triangles) and labeled with an abbreviated name. Dashed diagonal:

y=x

; dotted gray lines: chance (0.5). All methods fall below the diagonal, confirming that cross-slide discrimination is harder. Pathology foundation models cluster in the lower-right (high within-slide, low cross-slide), while LLMs maintain higher cross-slide accuracy relative to their within-slide performance.

Figure 8: Per-class pair confusion for cross-slide comparisons on HER2ST. Rows indicate the positive class $c_{+}$ (shared by reference $t_{r}$ and positive $t_{p}$ ) and columns the negative class $c_{-}$ . Higher values (warmer colors) indicate greater confusion.

Figure 9: Per-class pair confusion for within-slide comparisons on HER2ST. Rows indicate the positive class $c_{+}$ and columns the negative class $c_{-}$ . Higher values (warmer colors) indicate greater confusion.

Figure 10: Per-class pair confusion for cross-slide comparisons on CCTGS. Rows indicate the positive class $c_{+}$ and columns the negative class $c_{-}$ . Higher values (warmer colors) indicate greater confusion.

Figure 11: Per-class pair confusion for cross-slide comparisons on TIGER. Rows indicate the positive class $c_{+}$ and columns the negative class $c_{-}$ . Higher values (warmer colors) indicate greater confusion.

Pathology foundation models
Method	Radboud normal	Radboud tumor	Utrecht normal	Utrecht tumor	All
UNI2H	0.340	0.770	0.440	0.900	0.613
H-optimus-0	0.390	0.310	0.490	0.400	0.398
Midnight	0.610	0.670	0.550	0.730	0.640
CONCH v1.5	0.460	0.630	0.410	0.760	0.565
Virchow2	0.500	0.660	0.630	0.710	0.625

Appendix

Table 5: Cross-institution tile relative similarity accuracy on CAMELYON16.

Pathology foundation models
Method	TSS-22 normal	TSS-22 tumor	TSS-56 normal	TSS-56 tumor	TSS-66 normal	TSS-66 tumor	All
UNI2H	0.190	0.540	0.220	0.450	0.090	0.380	0.312
H-optimus-0	0.020	0.040	0.200	0.300	0.200	0.300	0.177
Midnight	0.110	0.080	0.470	0.590	0.300	0.590	0.357
CONCH v1.5	0.300	0.410	0.300	0.480	0.300	0.400	0.365
Virchow2	0.330	0.470	0.300	0.530	0.430	0.580	0.440

Appendix

Table 6: Cross-institution tile relative similarity accuracy on TCGA.

Pathology foundation models
Method	NKI inv.	NKI nec.	NKI non-inv.	RUMC inv.	RUMC nec.	RUMC non-inv.	SCH inv.	SCH nec.	SCH non-inv.	TCGA inv.	TCGA nec.	TCGA non-inv.	All
UNI2H	0.650	0.750	0.800	0.620	0.670	0.670	0.300	0.310	0.320	0.400	0.680	0.520	0.557
H-optimus-0	0.600	0.570	0.430	0.800	0.410	0.490	0.410	0.310	0.390	0.500	0.170	0.280	0.447
Midnight	0.350	0.740	0.860	0.120	0.630	0.700	0.140	0.420	0.590	0.190	0.290	0.700	0.477
CONCH v1.5	0.450	0.470	0.450	0.700	0.710	0.590	0.390	0.370	0.360	0.360	0.310	0.430	0.466
Virchow2	0.390	0.810	0.740	0.660	0.520	0.850	0.600	0.170	0.480	0.420	0.420	0.720	0.565

Appendix

Table 7: Cross-institution tile relative similarity accuracy on BEETLE.

Pathology foundation models
Method	Radboud normal	Radboud tumor	Utrecht normal	Utrecht tumor	All
UNI2H	0.610	0.970	0.530	0.940	0.763
H-optimus-0	0.570	0.890	0.680	0.890	0.758
Midnight	0.730	0.970	0.680	0.930	0.828
CONCH v1.5	0.610	0.950	0.560	0.910	0.758
Virchow2	0.710	0.960	0.790	0.950	0.853

Appendix

Table 8: Within-institution tile relative similarity accuracy on CAMELYON16.

Pathology foundation models
Method	TSS-22 normal	TSS-22 tumor	TSS-56 normal	TSS-56 tumor	TSS-66 normal	TSS-66 tumor	All
UNI2H	0.730	0.760	0.430	0.690	0.490	0.830	0.655
H-optimus-0	0.720	0.820	0.450	0.730	0.460	0.920	0.683
Midnight	0.760	0.750	0.520	0.650	0.570	0.890	0.690
CONCH v1.5	0.710	0.600	0.450	0.690	0.470	0.870	0.632
Virchow2	0.780	0.720	0.500	0.720	0.540	0.840	0.683

Appendix

Table 9: Within-institution tile relative similarity accuracy on TCGA.

Pathology foundation models
Method	NKI inv.	NKI nec.	NKI non-inv.	RUMC inv.	RUMC nec.	RUMC non-inv.	SCH inv.	SCH nec.	SCH non-inv.	TCGA inv.	TCGA nec.	TCGA non-inv.	All
UNI2H	0.930	0.940	0.980	0.770	1.000	0.945	0.945	0.905	0.740	0.785	1.000	0.780	0.893
H-optimus-0	0.885	0.950	0.965	0.845	1.000	0.910	0.985	0.795	0.715	0.815	1.000	0.795	0.888
Midnight	0.940	0.965	0.950	0.870	1.000	0.945	0.930	0.905	0.770	0.905	0.995	0.755	0.910
CONCH v1.5	0.840	0.960	0.805	0.870	0.995	0.905	0.970	0.825	0.640	0.765	0.975	0.755	0.859
Virchow2	0.945	0.955	0.970	0.835	1.000	0.975	0.980	0.870	0.800	0.790	0.975	0.765	0.905

Appendix

Table 10: Within-institution tile relative similarity accuracy on BEETLE.

Figure 12: Per-class pair confusion for cross-institution comparisons on BEETLE. Rows indicate the positive class $c_{+}$ and columns the negative class $c_{-}$ . Higher values (warmer colors) indicate greater confusion.

Figure 13: LLM relative similarity accuracy vs input-token price. Top row: cross-slide accuracy on HER2ST, CCTGS, and TIGER (Table 1 ). Bottom row: cross-institution accuracy on CAMELYON16, TCGA, and BEETLE (Tables 5 , 6 , and 7 ). Each point is one LLM. Dashed line indicates chance (0.5).

Figure 14: CAMELYON16 cross-institution relative similarity accuracy vs model cost. Left: pathology foundation model accuracy vs training data size. Center: general foundation model accuracy vs training data size (Gemini Emb. 2 training size estimated at

\sim

1B). Right: LLM accuracy vs price per million input tokens. Dashed line indicates chance (50%).

Figure 15: TCGA cross-institution relative similarity accuracy vs model cost. Left: pathology foundation model accuracy vs training data size. Center: general foundation model accuracy vs training data size. Right: LLM accuracy vs price per million input tokens. Dashed line indicates chance (50%).

Figure 16: BEETLE cross-institution relative similarity accuracy vs model cost. Left: pathology foundation model accuracy vs training data size. Center: general foundation model accuracy vs training data size. Right: LLM accuracy vs price per million input tokens. Dashed line indicates chance (50%).

Figure 17: General foundation model relative similarity accuracy vs training data size across all six datasets. Color indicates dataset; marker shape indicates method. Lines connect methods within the same dataset. Dashed line indicates chance (0.5). Training data sizes for LLM-derived encoders are estimates based on the upstream vision encoder’s pretraining data (Table 11 ).

Model	Training size	Source	URL	Quote
Pathology foundation models
UNI2H	$>$ 350K WSIs	HuggingFace model card	https://huggingface.co/MahmoodLab/UNI2-h	“Over 200 million image tiles sampled from over 350k diverse H&E and IHC slides sourced from Mass General Brigham.”
H-optimus-0	$>$ 500K WSIs	HuggingFace model card	https://huggingface.co/bioptimus/H-optimus-0	“The model is a 1.1B parameter vision transformer trained on a proprietary collection of more than 500,000 H&E stained whole slide histology images.”
Midnight	12K WSIs	Tolkach et al. (2025)	https://arxiv.org/abs/2504.05186	“We trained our first FM on the 12k TCGA WSIs alone.”
CONCH v1.5	1.26M pairs	Ding et al. (2024)	https://arxiv.org/abs/2411.19666	“CONCHv1.5, an extended version of CONCH, which was trained with 1.26 million image-caption pairs using the CoCa training objective.”
Virchow2	3.1M WSIs	Zimmermann et al. (2024)	https://arxiv.org/abs/2408.00738	“each trained with 3.1 million histopathology whole slide images”

Appendix

Table 11: Training data sizes for foundation models used in cost/size vs accuracy analysis (Figure 6 ). Sizes marked with

\dagger

are estimates based on the upstream vision encoder’s pretraining data, as the multimodal training data size is not separately disclosed.

Model	Price per 1M input tokens (USD)
Gemini 3 Flash	$0.50
Gemini 3.1 Flash Lite	$0.25
Qwen3.5	$0.39
GLM-4.6V	$0.30
Gemma 3	$0.08
GPT-5 Nano	$0.05

Appendix

Table 12: LLM input-token prices used in cost vs accuracy analysis (Figure 6 ).

	HER2ST (cross-slide)			CAMELYON16 (cross-inst.)
Method	Acc.	95% CI	$n$	Acc.	95% CI	$n$
Pathology foundation models
UNI2H	0.459	[0.437, 0.480]	2000	0.613	[0.565, 0.660]	400
H-optimus-0	0.439	[0.416, 0.461]	2000	0.398	[0.350, 0.445]	400
Midnight	0.464	[0.442, 0.485]	2000	0.640	[0.593, 0.685]	400
CONCH v1.5	0.450	[0.428, 0.472]	2000	0.565	[0.517, 0.613]	400

Appendix

Table 13: Bootstrap 95% confidence intervals for overall accuracy on HER2ST (cross-slide,

n=2000

) and CAMELYON16 (cross-institution,

n=400

). Models are grouped by category.

Experiment	Comparison	Acc. A	Acc. B	$\chi^{2}$	$p$ -value
HER2ST (cross-slide)	GPT-5 Nano vs Midnight	0.570	0.464	58.1	$<$ 0.0001
HER2ST (cross-slide)	GPT-5 Nano vs DINOv2	0.570	0.549	2.0	0.153
CAMELYON16 (cross-inst.)	Gem. 3.1 F. Lite vs Midnight	0.730	0.640	11.8	$<$ 0.001
CAMELYON16 (cross-inst.)	Gem. 3.1 F. Lite vs Gemini Emb. 2	0.730	0.610	26.9	$<$ 0.0001

Appendix

Table 14: McNemar’s test for key pairwise comparisons. Each test uses paired comparisons where both models returned valid responses. The best LLM is compared against the best pathology foundation model and the best general foundation model for each experiment.

Dataset	Abbreviation	Full name
HER2ST	Adi	Adipose
	Bre	Breast glandular
	Can	Cancer
	Con	Connective
	Imm	Immune infiltrate
CCTGS	Adi	Adipose

Appendix

Table 15: Tissue class abbreviations used in Table 1 .

Model	License
Pathology foundation models
UNI2H	CC-BY-NC-ND 4.0
H-optimus-0	Apache 2.0
Midnight	MIT
CONCH v1.5	CC-BY-NC-ND 4.0
Virchow2	CC-BY-NC-ND 4.0

Appendix

Table 16: Licenses for models used in this paper.

Explore similar work

Jul 10, 2026cs.CV

ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts

Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones. Here we present ALICE, a unified foundation model trained through multi-stage agglomerative distillation that sequentially distills eight vision-only, vision-language, and slide-level teacher models into dedicated modules of a single backbone. ALICE is pretrained on 24,985,184 tile-level pathology images and 155,604 high-resolution images, and evaluated across 21 task scenarios, 96 downstream tasks, and 48 data sources, spanning region-of-interest tissue analysis, vision-language multimodal evaluation, and whole-slide clinical assessment. In all three evaluation settings, ALICE achieved the best average rank among task-matched pathology foundation models. These results demonstrate that agglomerative distillation can consolidate complementary capabilities from specialized models into a unified backbone for broad computational pathology applications. The model is available at https://github.com/WonderLandxD/ALICE.

Jiawen Li, Tian Guan, Huijuan Shi +5

Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Department of Pathology, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Engineering Science, University of Oxford, Oxford, UK +1

Pathology Foundation Models Probe-Logit Distillation

Ask PDF

Jun 10, 2026cs.CV

How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology

General-purpose large language models (LLMs) are routinely used as baselines when evaluating specialized pathology models on whole-slide images (WSIs). Because WSIs exceed contemporary model context limits, LLM baselines routinely use small, high-magnification patches processed independently via majority voting, without systematic evaluation of seemingly inconsequential design choices such as patch size, patch count, and magnification. Generalist LLMs have consistently underperformed specialized systems, reinforcing the perception that domain-specific training or architectural adaptation is necessary for pathology tasks involving WSIs. Here, we conduct a systematic factorial analysis of four input design factors: inference mode, patch size, magnification, and patch count. We demonstrate that prior studies have overstated the gap between specialized models and general-purpose LLMs by choosing non-optimized input configurations. On the MultiPathQA benchmark, switching to a single balanced configuration (large patches at lower magnification, processed jointly) raises GPT-5 from 15.1% to 39.5% on cancer-type classification (TCGA) and from 38.1% to 62.9% on organ classification (GTEx). Per-task optimization yields further gains up to 43.9% (TCGA) and 71.6% (GTEx). The same configuration generalizes to two other models and to a fully held-out CPTAC cohort, where it improves Gemini 3 Flash by 23.4 percentage points without any task-specific tuning.

Kian R. Weihrauch, Thomas A. Buckley, William Lotter +1

Massachusetts Institute of Technology, Cambridge, MA · Harvard Medical School, Boston, MA · Dana-Farber Cancer Institute, Boston, MA

Pathology Foundation Models Model Size Large Language Model Baselines Tuning

Ask PDF

Jul 5, 2026cs.CV

The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models

How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) across eleven clinically realistic perturbations and a dissimilarity-driven Non-Redundant K-fold validation (NR-Kfold) protocol. We introduce a Perturbation Performance Index (PPI) to summarise accuracy trends under controlled perturbation sweeps and analyse robustness scaling with parameter count. We show that PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns: mid-sized models such (UNI2/Virchow-2 etc.) achieve comparable or greater resilience than larger systems. NR-Kfold analysis further reveals systematic accuracy loss and increased variability when training-test similarity is broken, underscoring the need for explicit distribution-shift evaluation. These findings suggest that the next generation of pathology foundation models must prioritise data quality, multimodality information and domain alignment over parameter count to achieve genuine clinical reliability.

Dhyey Yajnik, Amina Asif, Fayyaz Minhas

Predictive Systems in Biomedicine Lab, Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom

Pathology Foundation Models Computational Pathology Cross Validation Medical World Model Fragility Generalisation+1

Ask PDF