Plug-and-Play Logit Fusion for Heterogeneous Pathology Foundation Models
Authors: Gexin Huang, Anqi Li, Yusheng Tan, Beidi Zhao, Gang Wang, Zu-Hua Gao, Xiaoxiao Li
Organizations: University of British Columbia, Vancouver, BC, Canada · Vector Institute, Toronto, ON, Canada · Washington University in St. Louis, St. Louis, MO, USA
Pathology foundation models (FMs) have become central to computational histopathology, offering strong transfer performance across a wide range of diagnostic and prognostic tasks. The rapid proliferation of pathology foundation models creates a model-selection bottleneck: no single model is uniformly best, yet exhaustively adapting and validating many candidates for each downstream endpoint is prohibitively expensive. We address this challenge with a lightweight and novel model fusion strategy, LogitProd, which treats independently trained FM-based predictors as fixed experts and learns sample-adaptive fusion weights over their slide-level outputs. The fusion operates purely on logits, requiring no encoder retraining and no feature-space alignment across heterogeneous backbones. We further provide a theoretical analysis showing that the optimal weighted product fusion is guaranteed to perform at least as well as the best individual expert under the training objective. We systematically evaluate LogitProd on 22 benchmarks spanning WSI-level classification, tile-level classification, gene mutation prediction, and discrete-time survival modeling. LogitProd ranks first on 20/22 tasks and improves the average performance across all tasks by ~3% over the strongest single expert. LogitProd enables practitioners to upgrade heterogeneous FM-based pipelines in a plug-and-play manner, achieving multi-expert gains with ∼12× lower training cost than feature-fusion alternatives.
Figures & tables
Figure 1: Overview of LogitProd. Frozen FM experts output logits; LogitProd derives confidence/entropy/disagreement cues to predict sample-adaptive weights and fuses experts via weighted product fusion, enabling efficient multi-task prediction without re-encoding or feature alignment.
Figure 2: Evaluation across 22 pathology tasks. a–f, Gene mutation prediction (mAUC): a , mean across five genes; b–f , per-gene performance with prevalence. g–i, TIL classification (AUC) across six datasets. j, WSI-level tumour diagnosis (AUC). k, Breast carcinoma subtyping (AUC). l–q, C-index distributions across six TCGA cohorts for all FM-based experts and LogitProd. Box plots summarize cross-validation folds. r, Task-wide rank heatmap. The rightmost column reports mean rank, and dashed lines separate task groups.
Pathology foundation models (PFMs) provide strong tile-level representations via self-supervised pre-training on large-scale pathology images. Yet, PFMs are developed under diverse and often opaque data, architecture, and objective choices, inducing latent representational biases that limit robustness and obscure what each model specialises in. We present AdaFusion, a lightweight adaptive fusion framework that integrates complementary signals from multiple frozen PFMs through (1) low-dimensional feature compression and (2) a sample-conditioned gating module that reweights model-wise (and optionally channel-wise) contributions. Beyond improving predictive accuracy, AdaFusion provides contribution-driven interpretation that offers evidence consistent with model-specific preferences and synergistic interactions across tissue phenotypes. We evaluate AdaFusion on three public benchmarks spanning treatment response prediction, prostate cancer grading, and spatial gene expression inference. AdaFusion consistently outperforms individual PFMs and other fusion baselines, while providing interpretable tissue visualisation which aligns model preferences with morphological patterns. Code is available at: https://github.com/xyx-98/PathoOracle.
Yuxiang Xiao, Yang Hu, Bin Li +5
School of Computer Science and Engineering, South China University of Technology, China · School of Computing and Mathematical Sciences, University of Leicester, UK · Leicester Cancer Research Centre, University of Leicester, UK +4
Pathology vision-language foundation models vary in performance across diseases and tasks, with no single model consistently performing best. The high cost of expert pathology annotation can also limit the labeled data available for task-specific adaptation. Combining complementary pretrained representations is a potential approach to these limitations, yet learning an effective fusion from few labeled examples remains challenging. We introduce Few-shot Fusion Foundation Models of Computational Pathology (FFM-CP), which is a framework that combines multiple pathology vision-language models in the few-shot learning setting. The framework first aligns heterogeneous representations using a closed-form Orthogonal Procrustes transformation estimated from corresponding support images. This alignment preserves within-model feature geometry without training an additional alignment network. Within the aligned space, a unified graph enables information exchange across backbones by jointly refining support-image features and visual and textual class prototypes. These refined representations support complementary text-prototype and case-retrieval branches that capture semantic class knowledge and within-class visual variation, respectively. Each branch learns to combine predictions from all ordered backbone pairs, allowing queries encoded by one model to draw on evidence represented by another. We evaluate three backbone combinations on six histopathology datasets at 4, 8, and 16 shots per class. FFM-CP achieves higher mean macro-F1 than the strongest individually adapted member of each fused set in 50 of 54 comparisons. These findings suggest that combining complementary pretrained representations can improve histopathological classification when annotations are limited.
Anh-Tien Nguyen, Trung DQ. Dang, Nghiem Tuong Diep +9
Institute for Predictive Deep Learning for Medicine and Healthcare, Giessen University, Germany · Department of Medical Informatics, University Medicine Gottingen, Germany · Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark +8
Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored. This work systematically evaluates FM-based representations on a suite of computational pathology tasks across two real-world commercial cohorts, IH-BC and IH-NSCLC, drawn from the licensed in-house (IH) oncology dataset. The analysis focuses on two modalities, whole-slide images and transcriptomic profiles, drawn from the IH multimodal data. We first benchmark unimodal probing performance across five FMs on eight downstream classification tasks, and find that image and omics representations carry complementary predictive signals. Then we investigate whether multimodal fusion can yield additional gains over unimodal baselines by comparing three image-omics fusion strategies built on paired representations. The trustworthiness of selected unimodal and multimodal pipelines is further assessed through conformal prediction. Our results show that FM representations achieve competitive performance on out-of-distribution data and that multimodal fusion helps mainly when no single modality dominates the signal. Conformal prediction reveals that in the majority of cases where a point prediction fails, the true diagnosis remains recoverable within the prediction set, reinforcing the value of uncertainty-aware inference for clinical support.
Jingyu Hu, Giuseppe Tripodi, Reed Naidoo +2
The Alan Turing Institute, London, United Kingdom · University of Manchester, Manchester, United Kingdom · The Institute of Cancer Research, London, United Kingdom +1