CoM3eT: A foundation model for medical image analysis through federated, multidimensional context integration
Authors: J. Raphael Schäfer, Kai Geissler, Till Nicke, Chiara Tappermann, Karoline Heber, Eike Petersen, Habib Mergan, Lars Ole Schwen, +22 more
Organizations: Fraunhofer Institute for Digital Medicine MEVIS · Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany · Institute of Diagnostic and Interventional Radiology, Medizinische Hochschule Hannover, Hannover, Germany · Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA · Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin und Humboldt-Universität zu Berlin, Berlin, Germany · Institute of Pathology, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin und Humboldt-Universität zu Berlin, Berlin, Germany · Dr. Senckenberg Institutes of Pathology, Neuropathology and Human Genetics, Goethe University Frankfurt and University Medical Center Frankfurt, Frankfurt am Main, Germany · Department of Diagnostic and Interventional Radiology, Tübingen University Hospital, Tübingen, Germany · Digital & Automation, Siemens Healthineers AG, Forchheim, Germany · Research & Clinical Translation for Magnetic Resonance, Siemens Healthineers AG, Erlangen, Germany · Department of Radiology, University Hospital Erlangen and Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Medical Centre for Information and Communication Technology, University Hospital Erlangen, Erlangen, Germany · Imaging Science Institute, Uniklinik Erlangen, Erlangen, Germany · Fraunhofer Institute for Toxicology and Experimental Medicine ITEM, Hannover, Germany
Medical foundation models improve generalization when training AI models with limited labeled data, but remain confined to a single specialty, such as pathology or radiology, and to either sparse or dense outputs, such as classification or segmentation. Here, we present CoM3eT (Co-representation Multidimensional Multitask Medical Transformer), a medical vision foundation model that unifies pathology and radiology, sparse and dense predictions, and two- and higher-dimensional inputs by modeling multidimensional context with attention. CoM3eT outperformed other medical foundation models in an open competition spanning five tomographic, four whole-specimen, and three two-dimensional datasets, covering sparse and dense prediction tasks as well as report generation. When adapted across diverse clinical applications, training fewer than 2.5% of parameters achieved performance comparable to full fine-tuning, enabling research without access to high-performance GPU clusters. Applied to federated learning across hospitals, this approach achieved performance comparable to pooled-data training over internet connections and with consumer-grade hardware.
Figures & tables
Figure 1: Study overview. A, CoM³eT’s pretraining includes multidimensional images from multiple medical imaging modalities together with natural image data into a single model via multitask learning. B, the image transformer models global context from outputs of the vision backbone. C, the pyramid transformer uses this context to extend FM segmentation to multidimensional images such as 3D volumes. D-F, pretraining uses supervised labels across vision-language, classification and segmentation tasks. G, CoM³eT participated in UNICORN, the first benchmark competition for medical FMs, which compared frozen FMs on a diverse set of medical imaging tasks. H, performance comparison between CoM³eT, a specialized reference model, and CoM³eT restricted to individual patches. I, CoM³eT’s architecture supports partial fine-tuning. We compare full and partial fine-tuning and apply partial fine-tuning in a real-world federated learning setting.
Figure 2: Model overview. A, First, CoM³eT’s vision backbone encodes image inputs, representing a multidimensional medical image as a group of patch tokens and feature pyramids. In the case of multidimensional images, the image transformer then computes global context. For multidimensional segmentation tasks, the pyramid transformer uses the image transformer’s output tokens to enrich the feature pyramids with global context. B, CoM³eT creates two types of features. Segmentation tasks use features with their spatial shape intact (hyperpixel tokens), while classification tasks use feature vectors (patch tokens). C, The image transformer is an encoder-only transformer that models interactions between patch tokens to add global multidimensional context while preserving one representation per input. D, The pyramid transformer uses the outputs of the image transformer to inject global multidimensional context into feature maps.
Figure 3: Results overview. A, Mean task-specific metrics from the UNICORN competition (each task’s metric scaled to the range from random-guessing performance to maximum performance). Left, CoM³eT compared with a theoretical best ensemble composed of the strongest competing FM for each task. Middle and right, pathology and radiology results, respectively, compared with the second- and third-place FMs or combinations of FMs. B, Fine-tuning results with partial and full fine-tuning. C, GPU time, measured on an A100, for different fine-tuning settings. For multidimensional images with between 1 and 32 patch tokens, training based on cached patch tokens (backbone caching) has almost zero cost for partial fine-tuning (black). The blue and orange lines overlap because partial fine-tuning and no fine-tuning incur nearly identical cost. D, GPU time for multidimensional images with 64 or more patch tokens. Full fine-tuning cannot be used for these images. E, Time required per update for each fine-tuning setting for CoM³eT and its larger variant, CoM³eT-Large. In federated learning, a full update step requires exchanging the full model state twice, first from the sites to the server and then from the server back to the sites. Unless stated otherwise, “CoM³eT” refers to the CoM³eT-Base variant.
Figure 4: Test data. A, The study included three 3D and two 2D segmentation tasks. Grouped inputs are shown as rows within boxes, and tasks that depend on positional information are connected by a line. B, Four classification tasks were included, one of them ungrouped. C, Synthetic tasks supported model development and ablation studies. D, Segmentation results. Without the image and pyramid transformer (CoM³eT-2D), common problems include discontinuities in tubular structures (red) and missing segments in uncertain regions (purple), other errors remain for CoM³eT (blue).
CoM³eT Variant
Vision backbone
#Parameters transformer
Dimensionality
Variant
#Parameters
Image T.
Pyramid T.
Patch token
Hyperpixel token
Tiny
Swin-T
27,582,570
1,054,208
529,120
256
32
UNICORN
Swin-B
86,905,848
4,205,568
2,105,472
512
32
FL
Swin-B
86,905,848
4,205,568
2,105,472
512
64
Base
Swin-B
86,905,848
4,205,568
2,105,472
512
64
Large
Swin-L
195,220,980
16,799,744
8,403,904
1024
128
Table 1: Architecture statistics. Each 2D input (such as an image or image patch) is described by the “Patch token size”, which refers to the dimension of the feature vector (patch token) produced by the encoder for that input. “Hyperpixel token size” refers to the dimension of the hyperpixel token (the embedding corresponding to each spatial location within the input) as produced by the decoder.
Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM2, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM2 equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable O(1/T) convergence and generalization guarantees. FM2 further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.
Shengchao Chen, Ting Shu
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance.
Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5
Radiological artificial intelligence has advanced rapidly, yet most systems remain narrowly task-specific, data-intensive, and fragile under domain shift. Foundation models promise more transferable and data-efficient solutions, but existing approaches are limited in scale, evaluated narrowly, and often assume that a single pretrained model can support diverse downstream tasks. Here we present nnFoundation, complementary convolutional and transformer-based 3D radiological foundation models. Developed within the Human Radiome Project (THRP), nnFoundation is trained on 2.1 million CT, MRI, and PET image volumes from 125 institutional and public datasets. We evaluate them across 108 tasks spanning segmentation, detection, classification, report generation, and image retrieval, including evaluations under domain shift, by external partners and in low-data and low-compute regimes. Across all task types, our convolution- and transformer-based nnFoundation models consistently outperform both prior 3D foundation models and training from scratch, establishing state-of-the-art performance for radiological imaging. However, performance follows a consistent task-dependent structure: the convolutional nnFoundation model dominates spatially localized tasks, whereas the transformer-based nnFoundation model excels in tasks requiring global semantic reasoning and in frozen-feature settings. Dynamically aligning the foundation model topology with the dataset characteristics post-hoc further improves transfer across heterogeneous 3D settings. These results show that transferable 3D radiological performance is governed not by a single universal model, but by the interplay of scalable pretraining, complementary architectures, and dataset-aware adaptation. We release nnFoundation models integrated into nnU-Net and nnDetection, enabling immediate application across established radiology workflows.
Constantin Ulrich Harsy, Tassilo Wald, Karol Gotkowski +80