Here, we present TOX-FM, a family of non-clinical PFMs trained on large-scale diverse histopathology datasets of over one million whole slide images consisting of over 60 organs across six animal species from over 14,000 toxicology studies across six decades. We introduce a comprehensive benchmark of 18 non-clinical downstream tasks, incorporate an established benchmark of 9 clinical downstream tasks, and evaluate TOX-FM against state-of-the-art clinical PFMs across both non-clinical and clinical domains. Notably, we explore open questions surrounding domain-specific training and cross-domain generalization, offering insights into the opportunities and challenges of applying PFMs in the non-clinical domain.
Figures & tables
Figure 1: Examples of PCA feature maps of Midnight-12k (best performing clinical PFM) and TOX-FM-1M-ViT-H-CL (best performing non-clinical PFM) across different domains and scales. Non-clinical images are taken from the tissue task (rat, 2.0 mpp) and clinical images are taken from the MoNuSAC task (0.25 mpp).
Figure 2: (A) UMAP visualization of the hierarchical k-means clustering results at 4.0 mpp. Each point corresponds to one of the 100,000 bottom-level cluster centroids with the color indicating the corresponding top-level cluster centroid. Examples of the patches from the 16 largest top-level clusters are shown. Overall, the hierarchical k-means approach produces a morphologically coherent organization of the embedding space, with top-level clusters forming contiguous regions that correspond to visually distinct histological patterns. (B) Examples of spatially linked patches selected by the progressive multi-scale strategy from the lowest magnification (4.0 mpp) to highest magnification (0.5 mpp).
Domain
Model
Architecture
Training Magnification (mpp)
Non-Clinical
TOX-FM-100K-ViT-L
ViT-L
0.5, 1.0, 2.0, 4.0
TOX-FM-100K-ViT-H
ViT-H
0.5, 1.0, 2.0, 4.0
TOX-FM-1M-ViT-H
ViT-H
0.5, 1.0, 2.0, 4.0
TOX-FM-1M-ViT-H-CL
ViT-H
0.5, 1.0, 2.0, 4.0
Clinical
Prov-GigaPath
ViT-g
0.5
H-optimus-0
ViT-g
0.5
Table 1: Overview of non-clinical and clinical PFMs for evaluation. The number of parameters for ViT-L, ViT-H, and ViT-g architectures are approximately 305M, 636M, and 1.1B parameters, respectively.
Domain
Type
Task
Magnification (mpp)
Number of Classes
Non-Clinical
Patch Classification (Linear Probe)
Species
4.0
6
Tissue (Dog)
2.0
23
Tissue (Monkey)
2.0
10
Tissue (Mouse)
2.0
33
Tissue (Rabbit)
2.0
9
Tissue (Rat)
2.0
30
Table 2: Overview of non-clinical and clinical downstream tasks for evaluation of PFMs.
Figure 3: (A) Distribution of task weights and overall performance (mean and standard deviation) of all models across a comprehensive set of 18 non-clinical and 9 clinical downstream tasks. (B) Examples of attention weights from attention probe evaluation for the artifact patch classification non-clinical task using embeddings from TOX-FM-1M-ViT-H-CL. The attention weights were used to aggregate the patch token embeddings for patch classification. (C) Examples of segmentation masks for the lesion segmentation non-clinical task using embeddings from TOX-FM-1M-ViT-H-CL.
Figure 4: Average pairwise distribution distances (mean and standard deviation) between the training and task datasets across three metrics. Metrics are averaged across 5 random subsets of 100,000 patches from each dataset.
Figure 5: Average representation similarity metrics (mean and standard deviation) of TOX-FM models with respect to Midnight-12k and H-optimus-0. Metrics are averaged across 5 random subsets of 100,000 patches from the clinical task datasets. We observed very small variability for both metrics across the random subsets. We report the mutual kNN metric with k=100 but note that the same trend was observed for k=1 and k=10.
Figure 6: Overall performance (mean and standard deviation) trajectories of TOX-FM-1M-ViT-H and TOX-FM-1M-ViT-H-CL on non-clinical validation downstream tasks during training on TOX-FM-1M .
Figure 7: Overall performance (mean and standard deviation) of TOX-FM models and the corresponding -Gram models on spatially resolved non-clinical (artifact and lesion) and clinical (segmentation) downstream tasks.
Figure 8: Examples of PCA feature maps across magnifications and concentric fields of view on an H&E WSI of femur tissue from a rat. The black boxes on each image indicate the field of view at the next higher magnification.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Distribution of candidate and selected patches across tissue types using hierarchical k-means on a subset of 36,000 WSIs with corresponding tissue information. For the validation analysis, the method was independently applied at each magnification with a fixed sampling ratio of 25%.
Component
Patch Classification (Linear Probe)
Patch Classification (Attention Probe)
Slide Classification
Segmentation
Method
Logistic Regression
ABMIL
ABMIL
U-Net
Loss
Cross-entropy
Cross-entropy
Cross-entropy
Lovász
Learning rate
-
0.01, 0.001, 0.0001
0.01, 0.003, 0.001
0.01, 0.001, 0.0001
Scheduler
-
Cosine
Cosine
Cosine
Weight decay
-
0.001, 0.0001
0.001, 0.0001
0.001, 0.0001
Optimizer
L-BFGS
AdamW
AdamW
AdamW
Appendix
Table A1: Hyperparameter settings for non-clinical downstream tasks.
Component
Patch Classification
Slide Classification
Segmentation
Learning rate
0.00003, 0.0003 , 0.003
0.0001, 0.001 , 0.01
0.0002, 0.002 , 0.02
Weight decay
0.001, 0.01 , 0.1
0.001, 0.01 , 0.1
0.001, 0.01 , 0.1
Appendix
Table A2: Hyperparameter settings for the eva evaluation framework for clinical downstream tasks. The bold values indicate the defaults. All other hyperparameter settings followed the official eva repository defaults.
Type
Task
ImageNet-ViT-B
DINOv3-LVD-1689M-ViT-L
Prov-GigaPath
H-optimus-0
Midnight-12k
TOX-FM-100K-ViT-L
TOX-FM-100K-ViT-H
TOX-FM-1M-ViT-H
TOX-FM-1M-ViT-H-CL
Patch Classification (Linear Probe)
Species
0.5627 (0.0019)
0.6322 (0.0062)
0.7461 (0.0046)
0.7579 (0.0045)
0.7746 (0.0062)
0.8246 (0.0047)
0.8345 (0.0066)
0.8904 (0.0019)
0.8985 (0.0037)
Tissue (Dog)
0.8319 (0.0018)
0.8596 (0.0042)
0.9182 (0.0035)
0.9136 (0.0011)
0.9271 (0.0024)
0.9363 (0.0031)
0.9399 (0.0042)
0.9610 (0.0022)
0.9633 (0.0027)
Tissue (Monkey)
0.9231 (0.0041)
0.9373 (0.0045)
0.9646 (0.0038)
0.9604 (0.0037)
0.9736 (0.0023)
0.9783 (0.0023)
0.9797 (0.0021)
0.9840 (0.0022)
0.9865 (0.0020)
Tissue (Mouse)
0.7595 (0.0053)
0.8215 (0.0052)
0.8819 (0.0044)
0.8770 (0.0041)
0.8817 (0.0047)
0.9099 (0.0070)
0.9158 (0.0053)
0.9092 (0.0057)
0.9152 (0.0052)
Tissue (Rabbit)
0.9292 (0.0029)
0.9418 (0.0020)
0.9636 (0.0029)
0.9633 (0.0019)
0.9648 (0.0024)
0.9709 (0.0017)
0.9710 (0.0023)
0.9741 (0.0006)
0.9751 (0.0016)
Tissue (Rat)
0.8101 (0.0043)
0.8435 (0.0020)
0.9163 (0.0053)
0.9171 (0.0034)
0.9173 (0.0043)
0.9596 (0.0027)
0.9629 (0.0027)
0.9545 (0.0032)
0.9578 (0.0028)
Appendix
Table A3: Average performance (mean and standard deviation) of all models on non-clinical downstream tasks. Performance is reported based on macro F1 for classification tasks and mIOU for segmentation tasks. Performance is averaged across multiple runs. All tasks were assigned a weight of 1.0, except tissue tasks, which were assigned a weight of 0.2, to enable a more balanced contribution across task types and magnifications.
Type
Task
ImageNet-ViT-B
DINOv3-LVD-1689M-ViT-L
Prov-GigaPath
H-optimus-0
Midnight-12k
TOX-FM-100K-ViT-L
TOX-FM-100K-ViT-H
TOX-FM-1M-ViT-H
TOX-FM-1M-ViT-H-CL
Patch Classification
BACH
0.6332 (0.0204)
0.7164 (0.0048)
0.7678 (0.0144)
0.7704 (0.0102)
0.9051 (0.0043)
0.7158 (0.0053)
0.7275 (0.0094)
0.7973 (0.0119)
0.8366 (0.0060)
BreakHis
0.7271 (0.0073)
0.7798 (0.0030)
0.8348 (0.0123)
0.8091 (0.0138)
0.8557 (0.0015)
0.7425 (0.0034)
0.6385 (0.0043)
0.7616 (0.0131)
0.8059 (0.0033)
CRC
0.9371 (0.0047)
0.9278 (0.0023)
0.9531 (0.0012)
0.9562 (0.0037)
0.9665 (0.0003)
0.9577 (0.0009)
0.9610 (0.0013)
0.9710 (0.0007)
0.9739 (0.0009)
Gleason Arvaniti
0.6894 (0.0096)
0.7451 (0.0051)
0.7274 (0.0161)
0.7709 (0.0052)
0.7991 (0.0021)
0.7469 (0.0027)
0.7456 (0.0050)
0.6995 (0.0103)
0.6992 (0.0114)
Patch Camelyon
0.8534 (0.0019)
0.8856 (0.0023)
0.9450 (0.0008)
0.9421 (0.0010)
0.9305 (0.0016)
0.8874 (0.0029)
0.8887 (0.0014)
0.9131 (0.0006)
0.9121 (0.0027)
Slide Classification
Camelyon16 (Small)
0.6937 (0.0200)
0.6577 (0.0613)
0.8041 (0.0207)
0.8269 (0.0253)
0.8459 (0.0213)
0.7803 (0.0191)
0.7255 (0.0377)
0.7501 (0.0670)
0.7522 (0.0455)
Appendix
Table A4: Average performance (mean and standard deviation) of all models on clinical downstream tasks. Performance is reported based on metrics as defined in the official eva repository. Performance is averaged across multiple runs.
Figure A2: Examples of PCA feature maps of Midnight-12k (best performing clinical PFM) and TOX-FM-1M-ViT-H-CL (best performing non-clinical PFM) across different domains at the training resolution. Non-clinical images are taken from the tissue task (rat, 2.0 mpp) and clinical images are taken from the MoNuSAC task (0.25 mpp).
Figure A3: UMAP visualizations of the training and task datasets across the embedding spaces of six models. Each panel is fit independently on the respective model’s embeddings.
Type
Task
TOX-FM-100K-ViT-L
TOX-FM-100K-ViT-L-Gram
TOX-FM-100K-ViT-H
TOX-FM-100K-ViT-H-Gram
TOX-FM-1M-ViT-H
TOX-FM-1M-ViT-H-Gram
TOX-FM-1M-ViT-H-CL
TOX-FM-1M-ViT-H-CL-Gram
Patch Classification (Linear Probe)
Artifact
0.6009 (0.0085)
0.6031 (0.0081)
0.6112 (0.0049)
0.6163 (0.0084)
0.5993 (0.0074)
0.6089 (0.0094)
0.5916 (0.0046)
0.6144 (0.0059)
Lesion
0.5657 (0.0093)
0.5681 (0.0080)
0.5889 (0.0072)
0.5960 (0.0078)
0.5732 (0.0055)
0.5828 (0.0086)
0.5839 (0.0057)
0.5807 (0.0080)
Patch Classification (Attention Probe)
Artifact
0.7380 (0.0066)
0.7391 (0.0066)
0.7455 (0.0039)
0.7483 (0.0051)
0.7435 (0.0052)
0.7505 (0.0056)
0.7529 (0.0047)
0.7481 (0.0070)
Lesion
0.6392 (0.0082)
0.6410 (0.0048)
0.6467 (0.0052)
0.6477 (0.0078)
0.6374 (0.0063)
0.6551 (0.0046)
0.6582 (0.0029)
0.6463 (0.0056)
Segmentation
Artifact
0.2796 (0.0065)
0.2763 (0.0101)
0.3015 (0.0021)
0.2862 (0.0027)
0.2950 (0.0061)
0.2886 (0.0101)
0.2888 (0.0069)
0.2924 (0.0016)
Lesion
0.4852 (0.0020)
0.4884 (0.0027)
0.4868 (0.0078)
0.4873 (0.0037)
0.4838 (0.0082)
0.4809 (0.0044)
0.4955 (0.0078)
0.4886 (0.0081)
Appendix
Table A5: Average performance (mean and standard deviation) of TOX-FM models and the corresponding -Gram models on artifact and lesion non-clinical downstream tasks. Performance is reported based on macro F1 for classification tasks and mIOU for segmentation tasks. Performance is averaged across multiple runs.
Type
Task
TOX-FM-100K-ViT-L
TOX-FM-100K-ViT-L-Gram
TOX-FM-100K-ViT-H
TOX-FM-100K-ViT-H-Gram
TOX-FM-1M-ViT-H
TOX-FM-1M-ViT-H-Gram
TOX-FM-1M-ViT-H-CL
TOX-FM-1M-ViT-H-CL-Gram
Segmentation
BCSS
0.6022 (0.0017)
0.6064 (0.0020)
0.6034 (0.0033)
0.6020 (0.0044)
0.6078 (0.0020)
0.6118 (0.0032)
0.6070 (0.0029)
0.6105 (0.0023)
MoNuSAC
0.6452 (0.0046)
0.6508 (0.0025)
0.6461 (0.0028)
0.6569 (0.0043)
0.6363 (0.0039)
0.6399 (0.0047)
0.6504 (0.0063)
0.6576 (0.0044)
Overall
0.6237 (0.0035)
0.6286 (0.0023)
0.6248 (0.0031)
0.6295 (0.0043)
0.6220 (0.0031)
0.6259 (0.0041)
0.6287 (0.0049)
0.6340 (0.0035)
Appendix
Table A6: Average performance (mean and standard deviation) of TOX-FM models and the corresponding -Gram models on segmentation clinical downstream tasks. Performance is reported based on metrics as defined in the official eva repository. Performance is averaged across multiple runs.
Figure A4: Examples of PCA feature maps across magnifications and concentric fields of view on an H&E WSI of kidney tissue from a rat. The black boxes on each image indicate the field of view at the next higher magnification.
Figure A5: Examples of PCA feature maps across magnifications and concentric fields of view on an H&E WSI of lung tissue from a monkey. The black boxes on each image indicate the field of view at the next higher magnification.
Pathology foundation models are pretrained AI systems that learn reusable tissue-image representations for diagnosis, biomarker discovery, and clinical workflows.
Pathology foundation models (PFMs) have emerged as a core approach for learning transferable representations from whole slide images (WSIs), and they are typically benchmarked through downstream clinical endpoints. While such task level evaluations are indispensable, they offer limited insight into what the representations themselves encode, particularly whether PFM embeddings can distinguish meaningful tissue regions and capture their spatial relationships. We present SpaPath-Bench, a representation level benchmark designed to diagnose spatial representation capability in PFMs. SpaPath-Bench formulates spatial domain identification (SDI) on paired whole slide image and spatial transcriptomics (ST) data as a diagnostic task. It curates 42 public paired WSI and ST slides, enables large scale evaluation across 19 encoders and seven SDI methods, and measures partition quality using three complementary criteria: unsupervised spatial coherence, transcriptomics referenced agreement, and expert referenced agreement. Across 83K runs, SpaPath-Bench reveals that different pretraining paradigms capture distinct aspects of tissue spatial architecture, and it provides practical guidance for building the next generation of spatially aware computational pathology models. Code and data pipelines are publicly available at https://bokai-zhao.github.io/SpaPath-benchboard/.
Bokai Zhao, Yiyang Zhang, Yuanchi Zhu +6
School of Artificial Intelligence, University of Chinese Academy of Sciences. · Brainnetome Center, Institute of Automation, Chinese Academy of Sciences. · Beijing Key Laboratory of Brainnetome and Brain-Computer Interface, Institute of Automation, Chinese Academy of Sciences. +2
Pathology vision-language models are advancing rapidly, yet existing benchmarks remain focused on human tissue, particularly oncology, leaving non-human pathology largely unaddressed. This gap is especially important in toxicologic pathology, where microscopic tissue examination of laboratory animals is a core component of preclinical drug safety assessment. To address it, we introduce VIPER, the first expert-curated benchmark for vision-language model evaluation in toxicologic pathology. VIPER contains 1,251 questions associated with 419 H&E-stained rat histology images across seven organ systems, covering multiple-choice, KPrim, and free-text formats. All questions were curated and validated by board-certified veterinary pathologists. In total, we benchmarked 16 models, including two newly introduced veterinary-pathology models, seven human pathology-specialized models, and seven general-purpose frontier models. The results identify a substantial domain gap between veterinary and human pathology, expose the risk of over-diagnosis of normal tissue in frontier models, and show that domain-specific training remains critical for visually grounded predictions. VIPER data and evaluation code are available at https://github.com/mahmoodlab/viper.
Luca L. Weishaupt, Simone de Brot, Javier Asin +9
Harvard-MIT HST · Mass General Brigham · Harvard Medical School +6
Pathology foundation models (PFMs) have advanced rapidly in recent years and support training classifiers for a range of histopathology tasks. However, their robustness across hospitals remains limited: performance often degrades when training a classifier on data from one hospital and evaluating it on another target hospital. We address this challenge by fine-tuning PFMs with a local maximum mean discrepancy (LMMD) objective that applies to two settings: domain adaptation, where unlabeled target-hospital data is available, and domain generalization, where target-hospital data is unavailable at all. Experiments at both the patch- and slide-level show consistent improvements across multiple PFMs and tasks.
Ben Vardi, Dana Schonberger, Yuval Friedmann +4
Reichman University, Herzliya, Israel · Institute of Pathology, Sheba Medical Center, Ramat-Gan, Israel · Technion - Israel Institute of Technology, Haifa, Israel