Scaling Vision Foundation Models for Toxicologic Pathology

Jessica Loo; Theodoros Pissas; Byunghak Kang; Vincent A. Ella; Xiaoyan Zhang; Kevin Sweeney; Walter Georgescu; Prateek Katiyar

cs.CVOct 19, 2026

Scaling Vision Foundation Models for Toxicologic Pathology

Authors: Jessica Loo, Theodoros Pissas, Byunghak Kang, Vincent A. Ella, Xiaoyan Zhang, Kevin Sweeney, Walter Georgescu, Prateek Katiyar,

Organizations: Novartis Institutes for Biomedical Research Inc., Cambridge, MA, USA · Novartis Pharma AG, Basel, Switzerland · Novartis Ireland Limited, Dublin, Ireland

Abstract

Here, we present TOX-FM, a family of non-clinical PFMs trained on large-scale diverse histopathology datasets of over one million whole slide images consisting of over 60 organs across six animal species from over 14,000 toxicology studies across six decades. We introduce a comprehensive benchmark of 18 non-clinical downstream tasks, incorporate an established benchmark of 9 clinical downstream tasks, and evaluate TOX-FM against state-of-the-art clinical PFMs across both non-clinical and clinical domains. Notably, we explore open questions surrounding domain-specific training and cross-domain generalization, offering insights into the opportunities and challenges of applying PFMs in the non-clinical domain.

Figures & tables

Figure 1: Examples of PCA feature maps of Midnight-12k (best performing clinical PFM) and TOX-FM-1M-ViT-H-CL (best performing non-clinical PFM) across different domains and scales. Non-clinical images are taken from the tissue task (rat, 2.0 mpp) and clinical images are taken from the MoNuSAC task (0.25 mpp).

Figure 2: (A) UMAP visualization of the hierarchical k-means clustering results at 4.0 mpp. Each point corresponds to one of the 100,000 bottom-level cluster centroids with the color indicating the corresponding top-level cluster centroid. Examples of the patches from the 16 largest top-level clusters are shown. Overall, the hierarchical k-means approach produces a morphologically coherent organization of the embedding space, with top-level clusters forming contiguous regions that correspond to visually distinct histological patterns. (B) Examples of spatially linked patches selected by the progressive multi-scale strategy from the lowest magnification (4.0 mpp) to highest magnification (0.5 mpp).

Domain	Model	Architecture	Training Magnification (mpp)
Non-Clinical	TOX-FM-100K-ViT-L	ViT-L	0.5, 1.0, 2.0, 4.0
	TOX-FM-100K-ViT-H	ViT-H	0.5, 1.0, 2.0, 4.0
	TOX-FM-1M-ViT-H	ViT-H	0.5, 1.0, 2.0, 4.0
	TOX-FM-1M-ViT-H-CL	ViT-H	0.5, 1.0, 2.0, 4.0
Clinical	Prov-GigaPath	ViT-g	0.5
	H-optimus-0	ViT-g	0.5

Table 1: Overview of non-clinical and clinical PFMs for evaluation. The number of parameters for ViT-L, ViT-H, and ViT-g architectures are approximately 305M, 636M, and 1.1B parameters, respectively.

Domain	Type	Task	Magnification (mpp)	Number of Classes
Non-Clinical	Patch Classification (Linear Probe)	Species	4.0	6
		Tissue (Dog)	2.0	23
		Tissue (Monkey)	2.0	10
		Tissue (Mouse)	2.0	33
		Tissue (Rabbit)	2.0	9
		Tissue (Rat)	2.0	30

Table 2: Overview of non-clinical and clinical downstream tasks for evaluation of PFMs.

Figure 3: (A) Distribution of task weights and overall performance (mean and standard deviation) of all models across a comprehensive set of 18 non-clinical and 9 clinical downstream tasks. (B) Examples of attention weights from attention probe evaluation for the artifact patch classification non-clinical task using embeddings from TOX-FM-1M-ViT-H-CL. The attention weights were used to aggregate the patch token embeddings for patch classification. (C) Examples of segmentation masks for the lesion segmentation non-clinical task using embeddings from TOX-FM-1M-ViT-H-CL.

Figure 4: Average pairwise distribution distances (mean and standard deviation) between the training and task datasets across three metrics. Metrics are averaged across 5 random subsets of 100,000 patches from each dataset.

Figure 5: Average representation similarity metrics (mean and standard deviation) of TOX-FM models with respect to Midnight-12k and H-optimus-0. Metrics are averaged across 5 random subsets of 100,000 patches from the clinical task datasets. We observed very small variability for both metrics across the random subsets. We report the mutual kNN metric with k=100 but note that the same trend was observed for k=1 and k=10.

Figure 6: Overall performance (mean and standard deviation) trajectories of TOX-FM-1M-ViT-H and TOX-FM-1M-ViT-H-CL on non-clinical validation downstream tasks during training on TOX-FM-1M .

Figure 7: Overall performance (mean and standard deviation) of TOX-FM models and the corresponding -Gram models on spatially resolved non-clinical (artifact and lesion) and clinical (segmentation) downstream tasks.

Figure 8: Examples of PCA feature maps across magnifications and concentric fields of view on an H&E WSI of femur tissue from a rat. The black boxes on each image indicate the field of view at the next higher magnification.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Distribution of candidate and selected patches across tissue types using hierarchical k-means on a subset of 36,000 WSIs with corresponding tissue information. For the validation analysis, the method was independently applied at each magnification with a fixed sampling ratio of 25%.

Component	Patch Classification (Linear Probe)	Patch Classification (Attention Probe)	Slide Classification	Segmentation
Method	Logistic Regression	ABMIL	ABMIL	U-Net
Loss	Cross-entropy	Cross-entropy	Cross-entropy	Lovász
Learning rate	-	0.01, 0.001, 0.0001	0.01, 0.003, 0.001	0.01, 0.001, 0.0001
Scheduler	-	Cosine	Cosine	Cosine
Weight decay	-	0.001, 0.0001	0.001, 0.0001	0.001, 0.0001
Optimizer	L-BFGS	AdamW	AdamW	AdamW

Appendix

Table A1: Hyperparameter settings for non-clinical downstream tasks.

Component	Patch Classification	Slide Classification	Segmentation
Learning rate	0.00003, 0.0003 , 0.003	0.0001, 0.001 , 0.01	0.0002, 0.002 , 0.02
Weight decay	0.001, 0.01 , 0.1	0.001, 0.01 , 0.1	0.001, 0.01 , 0.1

Appendix

Table A2: Hyperparameter settings for the eva evaluation framework for clinical downstream tasks. The bold values indicate the defaults. All other hyperparameter settings followed the official eva repository defaults.

Type	Task	ImageNet-ViT-B	DINOv3-LVD-1689M-ViT-L	Prov-GigaPath	H-optimus-0	Midnight-12k	TOX-FM-100K-ViT-L	TOX-FM-100K-ViT-H	TOX-FM-1M-ViT-H	TOX-FM-1M-ViT-H-CL
Patch Classification (Linear Probe)	Species	0.5627 (0.0019)	0.6322 (0.0062)	0.7461 (0.0046)	0.7579 (0.0045)	0.7746 (0.0062)	0.8246 (0.0047)	0.8345 (0.0066)	0.8904 (0.0019)	0.8985 (0.0037)
	Tissue (Dog)	0.8319 (0.0018)	0.8596 (0.0042)	0.9182 (0.0035)	0.9136 (0.0011)	0.9271 (0.0024)	0.9363 (0.0031)	0.9399 (0.0042)	0.9610 (0.0022)	0.9633 (0.0027)
	Tissue (Monkey)	0.9231 (0.0041)	0.9373 (0.0045)	0.9646 (0.0038)	0.9604 (0.0037)	0.9736 (0.0023)	0.9783 (0.0023)	0.9797 (0.0021)	0.9840 (0.0022)	0.9865 (0.0020)
	Tissue (Mouse)	0.7595 (0.0053)	0.8215 (0.0052)	0.8819 (0.0044)	0.8770 (0.0041)	0.8817 (0.0047)	0.9099 (0.0070)	0.9158 (0.0053)	0.9092 (0.0057)	0.9152 (0.0052)
	Tissue (Rabbit)	0.9292 (0.0029)	0.9418 (0.0020)	0.9636 (0.0029)	0.9633 (0.0019)	0.9648 (0.0024)	0.9709 (0.0017)	0.9710 (0.0023)	0.9741 (0.0006)	0.9751 (0.0016)
	Tissue (Rat)	0.8101 (0.0043)	0.8435 (0.0020)	0.9163 (0.0053)	0.9171 (0.0034)	0.9173 (0.0043)	0.9596 (0.0027)	0.9629 (0.0027)	0.9545 (0.0032)	0.9578 (0.0028)

Appendix

Table A3: Average performance (mean and standard deviation) of all models on non-clinical downstream tasks. Performance is reported based on macro F1 for classification tasks and mIOU for segmentation tasks. Performance is averaged across multiple runs. All tasks were assigned a weight of 1.0, except tissue tasks, which were assigned a weight of 0.2, to enable a more balanced contribution across task types and magnifications.

Type	Task	ImageNet-ViT-B	DINOv3-LVD-1689M-ViT-L	Prov-GigaPath	H-optimus-0	Midnight-12k	TOX-FM-100K-ViT-L	TOX-FM-100K-ViT-H	TOX-FM-1M-ViT-H	TOX-FM-1M-ViT-H-CL
Patch Classification	BACH	0.6332 (0.0204)	0.7164 (0.0048)	0.7678 (0.0144)	0.7704 (0.0102)	0.9051 (0.0043)	0.7158 (0.0053)	0.7275 (0.0094)	0.7973 (0.0119)	0.8366 (0.0060)
	BreakHis	0.7271 (0.0073)	0.7798 (0.0030)	0.8348 (0.0123)	0.8091 (0.0138)	0.8557 (0.0015)	0.7425 (0.0034)	0.6385 (0.0043)	0.7616 (0.0131)	0.8059 (0.0033)
	CRC	0.9371 (0.0047)	0.9278 (0.0023)	0.9531 (0.0012)	0.9562 (0.0037)	0.9665 (0.0003)	0.9577 (0.0009)	0.9610 (0.0013)	0.9710 (0.0007)	0.9739 (0.0009)
	Gleason Arvaniti	0.6894 (0.0096)	0.7451 (0.0051)	0.7274 (0.0161)	0.7709 (0.0052)	0.7991 (0.0021)	0.7469 (0.0027)	0.7456 (0.0050)	0.6995 (0.0103)	0.6992 (0.0114)
	Patch Camelyon	0.8534 (0.0019)	0.8856 (0.0023)	0.9450 (0.0008)	0.9421 (0.0010)	0.9305 (0.0016)	0.8874 (0.0029)	0.8887 (0.0014)	0.9131 (0.0006)	0.9121 (0.0027)
Slide Classification	Camelyon16 (Small)	0.6937 (0.0200)	0.6577 (0.0613)	0.8041 (0.0207)	0.8269 (0.0253)	0.8459 (0.0213)	0.7803 (0.0191)	0.7255 (0.0377)	0.7501 (0.0670)	0.7522 (0.0455)

Appendix

Table A4: Average performance (mean and standard deviation) of all models on clinical downstream tasks. Performance is reported based on metrics as defined in the official eva repository. Performance is averaged across multiple runs.

Figure A2: Examples of PCA feature maps of Midnight-12k (best performing clinical PFM) and TOX-FM-1M-ViT-H-CL (best performing non-clinical PFM) across different domains at the training resolution. Non-clinical images are taken from the tissue task (rat, 2.0 mpp) and clinical images are taken from the MoNuSAC task (0.25 mpp).

Figure A3: UMAP visualizations of the training and task datasets across the embedding spaces of six models. Each panel is fit independently on the respective model’s embeddings.

Type	Task	TOX-FM-100K-ViT-L	TOX-FM-100K-ViT-L-Gram	TOX-FM-100K-ViT-H	TOX-FM-100K-ViT-H-Gram	TOX-FM-1M-ViT-H	TOX-FM-1M-ViT-H-Gram	TOX-FM-1M-ViT-H-CL	TOX-FM-1M-ViT-H-CL-Gram
Patch Classification (Linear Probe)	Artifact	0.6009 (0.0085)	0.6031 (0.0081)	0.6112 (0.0049)	0.6163 (0.0084)	0.5993 (0.0074)	0.6089 (0.0094)	0.5916 (0.0046)	0.6144 (0.0059)
	Lesion	0.5657 (0.0093)	0.5681 (0.0080)	0.5889 (0.0072)	0.5960 (0.0078)	0.5732 (0.0055)	0.5828 (0.0086)	0.5839 (0.0057)	0.5807 (0.0080)
Patch Classification (Attention Probe)	Artifact	0.7380 (0.0066)	0.7391 (0.0066)	0.7455 (0.0039)	0.7483 (0.0051)	0.7435 (0.0052)	0.7505 (0.0056)	0.7529 (0.0047)	0.7481 (0.0070)
	Lesion	0.6392 (0.0082)	0.6410 (0.0048)	0.6467 (0.0052)	0.6477 (0.0078)	0.6374 (0.0063)	0.6551 (0.0046)	0.6582 (0.0029)	0.6463 (0.0056)
Segmentation	Artifact	0.2796 (0.0065)	0.2763 (0.0101)	0.3015 (0.0021)	0.2862 (0.0027)	0.2950 (0.0061)	0.2886 (0.0101)	0.2888 (0.0069)	0.2924 (0.0016)
	Lesion	0.4852 (0.0020)	0.4884 (0.0027)	0.4868 (0.0078)	0.4873 (0.0037)	0.4838 (0.0082)	0.4809 (0.0044)	0.4955 (0.0078)	0.4886 (0.0081)

Appendix

Table A5: Average performance (mean and standard deviation) of TOX-FM models and the corresponding -Gram models on artifact and lesion non-clinical downstream tasks. Performance is reported based on macro F1 for classification tasks and mIOU for segmentation tasks. Performance is averaged across multiple runs.

Type	Task	TOX-FM-100K-ViT-L	TOX-FM-100K-ViT-L-Gram	TOX-FM-100K-ViT-H	TOX-FM-100K-ViT-H-Gram	TOX-FM-1M-ViT-H	TOX-FM-1M-ViT-H-Gram	TOX-FM-1M-ViT-H-CL	TOX-FM-1M-ViT-H-CL-Gram
Segmentation	BCSS	0.6022 (0.0017)	0.6064 (0.0020)	0.6034 (0.0033)	0.6020 (0.0044)	0.6078 (0.0020)	0.6118 (0.0032)	0.6070 (0.0029)	0.6105 (0.0023)
	MoNuSAC	0.6452 (0.0046)	0.6508 (0.0025)	0.6461 (0.0028)	0.6569 (0.0043)	0.6363 (0.0039)	0.6399 (0.0047)	0.6504 (0.0063)	0.6576 (0.0044)
Overall		0.6237 (0.0035)	0.6286 (0.0023)	0.6248 (0.0031)	0.6295 (0.0043)	0.6220 (0.0031)	0.6259 (0.0041)	0.6287 (0.0049)	0.6340 (0.0035)

Appendix

Table A6: Average performance (mean and standard deviation) of TOX-FM models and the corresponding -Gram models on segmentation clinical downstream tasks. Performance is reported based on metrics as defined in the official eva repository. Performance is averaged across multiple runs.

Figure A4: Examples of PCA feature maps across magnifications and concentric fields of view on an H&E WSI of kidney tissue from a rat. The black boxes on each image indicate the field of view at the next higher magnification.

Figure A5: Examples of PCA feature maps across magnifications and concentric fields of view on an H&E WSI of lung tissue from a monkey. The black boxes on each image indicate the field of view at the next higher magnification.

Pathology Foundation Models

Pathology foundation models are pretrained AI systems that learn reusable tissue-image representations for diagnosis, biomarker discovery, and clinical workflows.

See all 89 papers

Benchmarking Pathology Foundation Models for Spatial Domain Understanding
May 25, 2026Bokai Zhao, Yiyang Zhang, Yuanchi Zhu +6Pathology Foundation Models Medical Image Benchmarks
School of Artificial Intelligence, University of Chinese Academy of Sciences. · Brainnetome Center, Institute of Automation, Chinese Academy of Sciences. · Beijing Key Laboratory of Brainnetome and Brain-Computer Interface, Institute of Automation, Chinese Academy of Sciences. +2
VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology
Aug 26, 2026Luca L. Weishaupt, Simone de Brot, Javier Asin +9VLM Evaluation Pathology VQA
Harvard-MIT HST · Mass General Brigham · Harvard Medical School +6
Discrepancy Minimization Improves Cross-Hospital Robustness in Digital Pathology
May 24, 2026Ben Vardi, Dana Schonberger, Yuval Friedmann +4Computational Pathology Unsupervised Domain Adaptation
Reichman University, Herzliya, Israel · Institute of Pathology, Sheba Medical Center, Ramat-Gan, Israel · Technion - Israel Institute of Technology, Haifa, Israel

May 25, 2026J/K move · Enter open · S save

Benchmarking Pathology Foundation Models for Spatial Domain Understanding

Bokai Zhao, Yiyang Zhang, Yuanchi Zhu, Hanqing Chao+5

School of Artificial Intelligence, University of Chinese Academy of Sciences. · Brainnetome Center, Institute of Automation, Chinese Academy of Sciences. · Beijing Key Laboratory of Brainnetome and Brain-Computer Interface, Institute of Automation, Chinese Academy of Sciences.+2

Pathology Foundation Models Medical Image Benchmarks Whole-Slide Image Analysis

Open paper AskPDF arXiv

Pathology foundation models (PFMs) have emerged as a core approach for learning transferable representations from whole slide images (WSIs), and they are typically benchmarked through downstream clinical endpoints. While such task level evaluations are indispensable, they offer limited insight into what the representations themselves encode, particularly whether PFM embeddings can distinguish meaningful tissue regions and capture their spatial relationships. We present SpaPath-Bench, a representation level benchmark designed to diagnose spatial representation capability in PFMs. SpaPath-Bench formulates spatial domain identification (SDI) on paired whole slide image and spatial transcriptomics (ST) data as a diagnostic task. It curates 42 public paired WSI and ST slides, enables large scale evaluation across 19 encoders and seven SDI methods, and measures partition quality using three complementary criteria: unsupervised spatial coherence, transcriptomics referenced agreement, and expert referenced agreement. Across 83K runs, SpaPath-Bench reveals that different pretraining paradigms capture distinct aspects of tissue spatial architecture, and it provides practical guidance for building the next generation of spatially aware computational pathology models. Code and data pipelines are publicly available at https://bokai-zhao.github.io/SpaPath-benchboard/.

Abstract

Pathology Foundation Models

Explore similar work

Benchmarking Pathology Foundation Models for Spatial Domain Understanding

VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

Discrepancy Minimization Improves Cross-Hospital Robustness in Digital Pathology