CoM$^3$eT: A foundation model for medical image analysis through federated, multidimensional context integration

J. Raphael Schäfer; Kai Geissler; Till Nicke; Chiara Tappermann; Karoline Heber; Eike Petersen; Habib Mergan; Lars Ole Schwen

cs.CVAug 17, 2026

CoM $^3$ eT: A foundation model for medical image analysis through federated, multidimensional context integration

Authors: J. Raphael Schäfer, Kai Geissler, Till Nicke, Chiara Tappermann, Karoline Heber, Eike Petersen, Habib Mergan, Lars Ole Schwen, +22 more

Organizations: Fraunhofer Institute for Digital Medicine MEVIS · Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany · Institute of Diagnostic and Interventional Radiology, Medizinische Hochschule Hannover, Hannover, Germany · Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA · Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin und Humboldt-Universität zu Berlin, Berlin, Germany · Institute of Pathology, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin und Humboldt-Universität zu Berlin, Berlin, Germany · Dr. Senckenberg Institutes of Pathology, Neuropathology and Human Genetics, Goethe University Frankfurt and University Medical Center Frankfurt, Frankfurt am Main, Germany · Department of Diagnostic and Interventional Radiology, Tübingen University Hospital, Tübingen, Germany · Digital & Automation, Siemens Healthineers AG, Forchheim, Germany · Research & Clinical Translation for Magnetic Resonance, Siemens Healthineers AG, Erlangen, Germany · Department of Radiology, University Hospital Erlangen and Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Medical Centre for Information and Communication Technology, University Hospital Erlangen, Erlangen, Germany · Imaging Science Institute, Uniklinik Erlangen, Erlangen, Germany · Fraunhofer Institute for Toxicology and Experimental Medicine ITEM, Hannover, Germany

Abstract

Medical foundation models improve generalization when training AI models with limited labeled data, but remain confined to a single specialty, such as pathology or radiology, and to either sparse or dense outputs, such as classification or segmentation. Here, we present CoM $^3$ eT (Co-representation Multidimensional Multitask Medical Transformer), a medical vision foundation model that unifies pathology and radiology, sparse and dense predictions, and two- and higher-dimensional inputs by modeling multidimensional context with attention. CoM $^3$ eT outperformed other medical foundation models in an open competition spanning five tomographic, four whole-specimen, and three two-dimensional datasets, covering sparse and dense prediction tasks as well as report generation. When adapted across diverse clinical applications, training fewer than 2.5% of parameters achieved performance comparable to full fine-tuning, enabling research without access to high-performance GPU clusters. Applied to federated learning across hospitals, this approach achieved performance comparable to pooled-data training over internet connections and with consumer-grade hardware.

Figures & tables

Figure 1: Study overview. A, CoM³eT’s pretraining includes multidimensional images from multiple medical imaging modalities together with natural image data into a single model via multitask learning. B, the image transformer models global context from outputs of the vision backbone. C, the pyramid transformer uses this context to extend FM segmentation to multidimensional images such as 3D volumes. D-F, pretraining uses supervised labels across vision-language, classification and segmentation tasks. G, CoM³eT participated in UNICORN, the first benchmark competition for medical FMs, which compared frozen FMs on a diverse set of medical imaging tasks. H, performance comparison between CoM³eT, a specialized reference model, and CoM³eT restricted to individual patches. I, CoM³eT’s architecture supports partial fine-tuning. We compare full and partial fine-tuning and apply partial fine-tuning in a real-world federated learning setting.

Figure 2: Model overview. A, First, CoM³eT’s vision backbone encodes image inputs, representing a multidimensional medical image as a group of patch tokens and feature pyramids. In the case of multidimensional images, the image transformer then computes global context. For multidimensional segmentation tasks, the pyramid transformer uses the image transformer’s output tokens to enrich the feature pyramids with global context. B, CoM³eT creates two types of features. Segmentation tasks use features with their spatial shape intact (hyperpixel tokens), while classification tasks use feature vectors (patch tokens). C, The image transformer is an encoder-only transformer that models interactions between patch tokens to add global multidimensional context while preserving one representation per input. D, The pyramid transformer uses the outputs of the image transformer to inject global multidimensional context into feature maps.

Figure 3: Results overview. A, Mean task-specific metrics from the UNICORN competition (each task’s metric scaled to the range from random-guessing performance to maximum performance). Left, CoM³eT compared with a theoretical best ensemble composed of the strongest competing FM for each task. Middle and right, pathology and radiology results, respectively, compared with the second- and third-place FMs or combinations of FMs. B, Fine-tuning results with partial and full fine-tuning. C, GPU time, measured on an A100, for different fine-tuning settings. For multidimensional images with between 1 and 32 patch tokens, training based on cached patch tokens (backbone caching) has almost zero cost for partial fine-tuning (black). The blue and orange lines overlap because partial fine-tuning and no fine-tuning incur nearly identical cost. D, GPU time for multidimensional images with 64 or more patch tokens. Full fine-tuning cannot be used for these images. E, Time required per update for each fine-tuning setting for CoM³eT and its larger variant, CoM³eT-Large. In federated learning, a full update step requires exchanging the full model state twice, first from the sites to the server and then from the server back to the sites. Unless stated otherwise, “CoM³eT” refers to the CoM³eT-Base variant.

Figure 4: Test data. A, The study included three 3D and two 2D segmentation tasks. Grouped inputs are shown as rows within boxes, and tasks that depend on positional information are connected by a line. B, Four classification tasks were included, one of them ungrouped. C, Synthetic tasks supported model development and ablation studies. D, Segmentation results. Without the image and pyramid transformer (CoM³eT-2D), common problems include discontinuities in tubular structures (red) and missing segments in uncertain regions (purple), other errors remain for CoM³eT (blue).

CoM³eT Variant	Vision backbone		#Parameters transformer		Dimensionality
	Variant	#Parameters	Image T.	Pyramid T.	Patch token	Hyperpixel token
Tiny	Swin-T	27,582,570	1,054,208	529,120	256	32
UNICORN	Swin-B	86,905,848	4,205,568	2,105,472	512	32
FL	Swin-B	86,905,848	4,205,568	2,105,472	512	64
Base	Swin-B	86,905,848	4,205,568	2,105,472	512	64
Large	Swin-L	195,220,980	16,799,744	8,403,904	1024	128

Table 1: Architecture statistics. Each 2D input (such as an image or image patch) is described by the “Patch token size”, which refers to the dimension of the feature vector (patch token) produced by the encoder for that input. “Hyperpixel token size” refers to the dimension of the hyperpixel token (the embedding corresponding to each spatial location within the input) as produced by the decoder.

Explore similar work

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Jul 15, 2026Shengchao Chen, Ting ShuMultimodal Federated Learning Cross-Modal Alignment
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
MultiMedVision: Multi-Modal Medical Vision Framework
May 9, 2026Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5Medical Imaging Foundation Models Vision Transformer
Emory University · Yale University
nnFoundation: 3D Foundation Models for Radiology
Sep 22, 2026Constantin Ulrich Harsy, Tassilo Wald, Karol Gotkowski +803D Foundation Models Medical Imaging Foundation Models

Jul 15, 2026J/K move · Enter open · S save

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Shengchao Chen, Ting Shu

School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia

Multimodal Federated Learning Cross-Modal Alignment Medical VQA Medical Image Classification Multimodal Foundation Models Federated Learning+1

Open paper AskPDF arXiv

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM

^2

, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM

^2

equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable

O(1/\sqrt{T})

convergence and generalization guarantees. FM

^2

further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.

Abstract

Explore similar work

FM2^22: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

MultiMedVision: Multi-Modal Medical Vision Framework

nnFoundation: 3D Foundation Models for Radiology

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging