Revisiting the Role of Foundation Models in Cell-Level Histopathological Image Analysis under Small-Patch Constraints -- Effects of Training Data Scale and Blur Perturbations on CNNs and Vision Transformers
Organizations: Division of Gastrointestinal Surgery, Department of Surgery, Kobe University Graduate School of Medicine, 7-5-2 Kusunoki-cho, Chuo-ku,, Kobe 650-0017, Hyogo, Japan · Association of Medical Artificial Intelligence Curation, 505, Sakae Members Office Building, 4-16-8 Sakae, Naka-ku, Nagoya 460-0008, Aichi, Japan · Center for Medical Education and Clinical Training, Kindai University Faculty of Medicine, 1-14-1 Miharadai, Minami-ku, Sakai 590-0111, Osaka, Japan · Department of Biophysics, Kobe University Graduate School of Health Sciences, 7-10-2 Tomogaoka, Suma-ku, Kobe 654-0142, Hyogo, Japan
Background and objective: Cell-level pathological image analysis requires working with extremely small image patches (40x40 pixels), far below standard ImageNet resolutions. It remains unclear whether modern deep learning architectures and foundation models can learn robust and scalable representations under this constraint. We systematically evaluated architectural suitability and data-scale effects for small-patch cell classification. Methods: We analyzed 303 colorectal cancer specimens with CD103/CD8 immunostaining, generating 185,432 annotated cell images. Eight task-specific architectures were trained from scratch at multiple data scales (FlagLimit: 256--16,384 samples per class), and three foundation models were evaluated via linear probing and fine-tuning after resizing inputs to 224x224 pixels. Robustness to blur was assessed using pre- and post-resize Gaussian perturbations. Results: Task-specific models improved consistently with increasing data scale, whereas foundation models saturated at moderate sample sizes. A Vision Transformer optimized for small patches (CustomViT) achieved the highest accuracy, outperforming all foundation models with substantially lower inference cost. Blur robustness was comparable across architectures, with no qualitative advantage observed for foundation models. Conclusion: For cell-level classification under extreme spatial constraints, task-specific architectures are more effective and efficient than foundation models once sufficient training data are available. Higher clean accuracy does not imply superior robustness, and large pre-trained models offer limited benefit in the small-patch regime.
Figures & tables
Figure 1: A schematic phylogenetic overview of major deep learning architectures. Early multilayer perceptrons gave rise to convolutional neural networks, which evolved through deeper residual designs and efficiency-oriented variants. In parallel, recurrent models led to attention-based transformers, from which Vision Transformer emerged as a direct adaptation to visual tasks. The diagram also illustrates how transformer design principles subsequently influenced modern convolutional architectures such as ConvNeXt, reflecting cross-paradigm convergence in recent model development.
Figure 2: Trade-off between inference speed and classification performance at flag_limit = 4096. Each point represents a model’s macro-F1 score (y-axis) versus its inference time per patch (x-axis). CustomViT achieves the highest classification performance while maintaining inference costs more than an order of magnitude lower than foundation models such as UNI. Task-specific convolutional models (CNN, ResNet, NIN, SE-ResNet) offer sub-millisecond to few-millisecond inference but plateau at substantially lower performance. Foundation models (ResNet-RS50, CTransPath, UNI) incur markedly higher inference latency, with UNI requiring approximately 25 ms per patch despite strong fine-tuning performance. This demonstrates that CustomViT provides a favorable balance between accuracy and computational efficiency for large-scale patch-based analysis.
Figure 3: Scaling behavior of classification performance with increasing training set size ( flag_limit ). Macro F1 scores under clean conditions are shown for task-specific models and foundation models with fine-tuning of the last layer (FT_last). CustomViT exhibits monotonic performance gains and surpasses foundation models at moderate data scales, whereas conventional convolutional networks saturate. EfficientNet shows early gains but was not evaluated at the largest scale due to excessive training time.
Figure 4: Robustness to strong blur ( σ=1.6 ) under a fixed annotation budget (flag_limit = 4096). Macro-F1 scores are shown for clean images (blue), post-resize blur (orange), and pre-resize blur (green). Blur is applied either after resizing to 40×40 pixels (post) or before resizing with resolution-corrected strength (pre). All models show pronounced performance degradation at this severity, with comparable sensitivity to pre- and post-resize blur.
Model
Blur
σ
Macro-F1
Δ Macro-F1
ResNet
pre
0.1
0.553384
-0.000187
0.2
0.553114
-0.000457
0.4
0.548343
-0.005228
0.8
0.526534
-0.027037
1.6
0.435413
-0.118158
post
0.1
0.553571
0
Table 5: Robustness to blur at different severities ( σ ) under a fixed annotation budget (flag_limit = 4096). Macro-F1 scores and relative changes from the clean baseline ( Δ Macro-F1) are reported. For each blur type (pre-resize and post-resize), the label is shown only in the first row of the corresponding block.
Figure 5: Accuracy degradation rate with increasing blur severity. Bars indicate the slope of accuracy change per unit increase in log2(1+σ) , with negative values representing faster performance degradation. Pre-resize (optical) and post-resize (digital) blur are shown separately.
Figure 6: Training dynamics of ResNet over 50 epochs. The left panel shows training and validation loss curves, exhibiting smooth exponential decay and early stabilization. The right panel displays corresponding accuracy trajectories, with validation accuracy plateauing after approximately 30 epochs. These trends indicate stable optimization and the absence of severe overfitting.
Figure 7: Normalized confusion matrix for ResNet classification across 16 cell types. Diagonal entries represent correct classification rates, while off-diagonal entries indicate misclassification patterns. Epithelial and structurally distinctive classes (e.g., adenocarcinoma NOS, adenoma, cytotoxic T cell, dendritic cell, smooth muscle cell) show strong diagonal dominance. In contrast, morphologically heterogeneous or weakly discriminative classes (e.g., endothelium, neutrophil, plasma cell, stromal cell) exhibit increased confusion with related cell populations.
Figure 8: Training dynamics of CustomViT over 60 epochs. Both training and validation loss decrease smoothly, accompanied by monotonic improvements in accuracy. The close alignment between training and validation curves indicates stable convergence despite the higher model capacity. Compared with convolutional baselines, CustomViT achieves substantially higher validation accuracy without evidence of optimization instability.
Figure 9: Normalized confusion matrix for CustomViT across 16 cell types. Strong diagonal dominance is observed for nearly all classes, reflecting improved feature separability relative to convolutional architectures. Residual confusion is primarily confined to biologically heterogeneous categories such as stromal cells and other lymphocytes, consistent with intrinsic label ambiguity rather than optimization failure.
Figure 10: Training dynamics of UNI with fine-tuning of the last layers (FT_last). Rapid initial convergence is followed by stable refinement of both loss and accuracy. The early saturation of training accuracy reflects the strong representational prior provided by large-scale self-supervised pretraining.
Figure 11: Normalized confusion matrix for UNI with FT_last. Overall classification accuracy is high, with well-separated epithelial and immune cell populations. Remaining errors are concentrated in morphologically overlapping or weakly stained classes, suggesting that residual limitations arise from data characteristics rather than insufficient model capacity.
In recent years, foundation models such as CLIP, DINO,and CONCH have demonstrated remarkable domain generalization and unsupervised feature extraction capabilities across diverse imaging tasks. However, systematic and independent evaluations of these models for pixel-level semantic segmentation in histopathology remain scarce. In this study, we propose a robust benchmarking approach to asses 10 foundational models on four histopathological datasets covering both morphological tissue-region and cellular/nuclear segmentation tasks. Our method leverages attention maps of foundation models as pixel-wise features, which are then classified using a machine learning algorithm, XGBoost, enabling fast, interpretable, and model-agnostic evaluation without finetuning. We show that the vision language foundation model, CONCH performed the best across datasets when compared to vision-only foundation models, with PathDino as close second. Further analysis shows that models trained on distinct histopathology cohorts capture complementary morphological representations, and concatenating their features yields superior segmentation performance. Concatenating features from CONCH, PathDino and CellViT outperformed individual models across all the datasets by 7.95% (averaged across the datasets), suggesting that ensembles of foundation models can better generalize to diverse histopathological segmentation tasks.
Lavish Ramchandani, Aashay Tinaikar, Dev Kumar Das +2
How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) across eleven clinically realistic perturbations and a dissimilarity-driven Non-Redundant K-fold validation (NR-Kfold) protocol. We introduce a Perturbation Performance Index (PPI) to summarise accuracy trends under controlled perturbation sweeps and analyse robustness scaling with parameter count. We show that PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns: mid-sized models such (UNI2/Virchow-2 etc.) achieve comparable or greater resilience than larger systems. NR-Kfold analysis further reveals systematic accuracy loss and increased variability when training-test similarity is broken, underscoring the need for explicit distribution-shift evaluation. These findings suggest that the next generation of pathology foundation models must prioritise data quality, multimodality information and domain alignment over parameter count to achieve genuine clinical reliability.
Dhyey Yajnik, Amina Asif, Fayyaz Minhas
Predictive Systems in Biomedicine Lab, Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
Pathology foundation models (FMs) produce powerful tile-level representations which remain sensitive to scanner and staining variability, undermining deployment across laboratories. We develop a novel fine-tuning recipe that improves the robustness of pathology FMs to acquisition factors. Applied to ten different FMs, our fine-tuning strategy consistently improves robustness for every model as well as downstream performance, with no observed trade-off. On average, it raises the PathoROB robustness index by 23% (from 0.72 to 0.87) and increases the overall cross-benchmark performance by 43% on Patho-Bench, HEST and THUNDER combined, with individual gains reaching up to 72% in robustness (Phikon-v2) and 76% in performance (Midnight-12k). We publicly release the fine-tuned versions of Phikon-v2 (Phaet) and Midnight-12k (Mascaret) at https://huggingface.co/wearewaiv/models.
Alexandre Filiot, Oskar Thaeter, Benoit Schmauch +1
1Waiv · Institute of Pathology, Technical University of Munich · School of Computation, Information and Technology, Technical University of Munich