Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology
Organizations: Program of Computational Biology and Bioinforamtics, Yale University, USA · Broad Institute of MIT and Harvard, USA · These authors contributed equally to this work. · Department of Statistics and Data Science, Northwestern University, USA · Department of Computer Science, Northeastern University, USA · Department of Pathology, Mass General Brigham, Harvard Medical School, USA · Cancer Program, Broad Institute of Harvard and MIT, USA · Data Science Program, Dana-Farber Cancer Institute, USA · Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, USA · Department of Pathology, Yale University, USA · These authors contribute equally to this project as human experts · Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania, USA · Department of Pathology and Laboratory Medicine, University of California, San Francisco, USA · Department of Pathology and Laboratory Medicine, KK Women’s and Children’s Hospital, SGD · Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, USA
Abstract
Predicting immune biomarkers associated with the tumor immune microenvironment (TIME) is critical for advancing precision oncology, yet existing approaches are largely limited to single image modalities and suffer from insufficient resolution and incomplete utilization of complementary clinical and biological information. Here we introduce MixTIME, a multimodal foundation model that leverages a mixture-of-experts (MoE) architecture to integrate pathology foundation models trained across distinct modalities: image only (UNIv2), image text (CONCHv1.5), and image transcriptomic (STPath) representations for pixel-level and slide-level prediction of multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin (HE) whole-slide images. MixTIME employs a learnable router to dynamically weight expert contributions and is trained with a distribution- and tendency-aware loss function. Benchmarked on two datasets of different scales, MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics. The predicted mIF profiles substantially enhance downstream tasks, including spatial domain identification, survival prediction, and AI-assisted pathology report generation validated by expert pathologists from multiple institutes across the world. Furthermore, MixTIME enables longitudinal tracking of protein expression dynamics across clinical time points and reveals protein gene interaction patterns linked to drug resistance and immune suppression in tumor microenvironments. Collectively, MixTIME provides a scalable framework for multimodal biomarker discovery and clinical translation in computational pathology.