FM^2: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Simultaneous assessment of medical imaging and patient records is often required in clinical diagnosis. However, standard machine learning algorithms cannot analyze these data types together. Meanwhile, compliance with HIPAA and GDPR can constrain centralized aggregation of sensitive patient data. This leaves a crucial void of secure fusion of visual and textual context across distant networks. Thus, we present OmniMed-FL, a controlled systems study of multimodal federated learning for five-class clinical condition classification (Normal, Pneumonia, COVID-19, Pleural Effusion, Cardiomegaly). Our proxy corpus pairs 3,000 public chest radiographs with 3,000 class-conditioned synthetic notes, matched by class, not by patient. The framework benchmarks eight fusion strategies, three initializations, four missing-text imputation rules, and matched federated baselines under non-IID Dirichlet partitioning across 3 to 20 hospital clients. As all notes are synthetic and pairing is not patient-level, these are descriptive proxy comparisons, not estimates of diagnostic performance or deployment readiness. Within those limits with clients ($K=5$) and severe skew ($α=0.1$), local-only training achieves a macro-F1 score of 0.297, FedAvg achieves $0.662\pm0.074$, FedProx $0.737\pm0.085$, a matched FedMME-style one-shot ensemble $0.647\pm0.080$, and our SCAFFOLD-AdamW adaptation $0.070\pm0.015$, the 0.075 FedProx-FedAvg gap falling inside the wider of the two two-seed standard deviations. Over a $4\times3$ grid, label skew costs up to 0.27 F1 whereas a near-sevenfold client increase costs at most 0.10, while bidirectional volume grows linearly to 183.5 GiB at $K=20$. Multimodal fusion leads on both corpora, scoring 0.956 against 0.934 for text and 0.664 for images on the synthetic corpus and 0.906 against 0.880 and 0.737 on the radiograph corpus, for $2.3\times$ the model state of text alone.

Federated learning enables collaborative model training across medical institutions without sharing raw data, but its performance is often limited by domain heterogeneity across clients. Existing approaches to address this challenge fall into two main paradigms: model-side personalization, which adapts model parameters to each client, and data-side harmonization, which reduces inter-client variation at the input level. Despite their widespread use, these strategies have not been systematically compared. In this work, we conduct a comprehensive study across six medical imaging settings-colon polyp, skin lesion, and breast tumor segmentation, and tuberculosis CXR, brain tumor, and breast tumor classification-covering diverse types of domain shift. We evaluate a broad set of state-of-the-art harmonization and personalization methods under a unified framework. Our results reveal a conditional trade-off driven by the nature of heterogeneity: harmonization is more effective when variation is primarily appearance-based (e.g., CXR classification), while personalization performs better when differences are structural (e.g., colon polyp segmentation). When inter-client variation is limited, both strategies perform similarly. These findings demonstrate that the effectiveness of adaptation in federated medical imaging depends on the type and magnitude of domain shift rather than the strategy alone. We provide practical guidelines for selecting between harmonization and personalization and highlight directions for future hybrid approaches that combine both paradigms. Code is available at https://github.com/ChamaniS/WhenToAdapt.

Purpose: Developing generalizable medical image segmentation models is challenging because imaging data are distributed across institutions and differ in modality and acquisition protocol. Federated learning (FL) enables collaborative training without centralizing raw medical images, but cross-modality domain shifts between computed tomography (CT) and magnetic resonance imaging (MRI) can substantially reduce model performance. This study investigates augmentation-driven cross-modality FL for abdominal organ and whole-heart segmentation. Methods: We evaluate convolution-based spatial augmentation, frequency-domain argumentation, domain-specific normalization, and global intensity nonlinear (GIN) augmentation for multimodal segmentation. Abdominal organ segmentation and whole-heart segmentation are first evaluated using a 2D U-Net framework. For whole-heart segmentation, we additionally perform native 3D experiments using a self-configuring nnU-Net architecture on the CARE-WHS 2026 dataset, enabling evaluation of whether the observed cross-modality FL behavior persists when moving from slice-based 2D segmentation to volumetric 3D segmentation. Results: GIN provides the most consistent cross-modality performance among the evaluated approaches in the original 2D experiments. For pancreas segmentation, the Dice similarity coefficient (DSC) improved from 0.073 to 0.437 when CT data were incorporated through federated cross-modality training. In 3D whole-heart segmentation, FedGIN improved mean DSC over FedAvg from 0.8696 to 0.8901 on the unseen CT center and from 0.7160 to 0.7956 on the unseen MRI center. Relative to centralized GIN training, FedGIN retained 92.4% of performance on unseen CT data and achieved comparable performance on unseen MRI data (0.7956 versus 0.7937).