Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation
Authors: Hong Liu, Dong Wei, Qian Dai, Xian Wu, Yefeng Zheng, Liansheng Wang
Organizations: National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, 361005, Fujian, China · Jarvis Research Center, Tencent YouTu Lab, Shenzhen, 518075, Guangdong, China · Department of Computer Science at School of Informatics, Xiamen University, Xiamen, 361005, Fujian, China · Medical Artificial Intelligence Lab, Westlake University, Hangzhou, 310030, Zhejiang, China
Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, some FL participants may possess only a subset of the complete imaging modalities, posing intermodal heterogeneity as a challenge to effectively training a global model on all participants' data. Meanwhile, each participant expects a personalized model tailored to its local data characteristics in FL. This work proposes a new FL framework with federated modality-specific encoders and partially personalized multimodal fusion decoders (FedMEPD) to address the two concurrent issues. Specifically, FedMEPD employs an exclusive encoder for each modality to account for the intermodal heterogeneity. While these encoders are fully federated, the decoders are partially personalized to meet individual needs -- using the discrepancy between global and local parameter updates to dynamically determine which decoder filters are personalized. Implementation-wise, a server with full-modal data employs a fusion decoder to fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation. Moreover, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the model parameters. Conversely, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up for the information loss due to absent modalities. FedMEPD is validated on the BraTS 2018 and 2020 multimodal brain tumor segmentation benchmarks. Results show that it outperforms various up-to-date methods for multimodal and personalized FL, and its novel designs are effective.
Figures & tables
Figure 1: (a) Example images of the four modalities in BraTS 2020 ( Menze and others, 2014 ) demonstrating severe intermodal heterogeneities, and corresponding tumor regions: blue: edema; yellow: enhancing tumor; and green: necrotic and non-enhancing tumor core. (b) To address the heterogeneities caused by different modal combinations across various sites, this work proposes adopting different aggregation strategies for modality-specific encoders (fully federated) and multi-modal fusion decoders (partially federated, partially personalized), to facilitate both common knowledge sharing and effective personalization.
Figure 2: Overview of the proposed FedMEPD framework. The server has four modality-specific encoders (one for each modality) and a multimodal fusion decoder, whose fused features are clustered to produce multimodal anchors. Each client has a fully federated modality-specific encoder for each modality and a partially federated (partially personalized) fusion decoder. A localized adaptive calibration via cross-attention (LACCA) module calibrates the clients’ missing-modal representations toward the server’s multimodal anchors.
Figure 3: Illustration of the proposed localized adaptive calibration via cross-attention (LACCA) module.
Figure 4: Example segmentation results of BraTS 2018 for a random subject in the test set of each site, by our method and the other three best-performing methods in terms of the average client mDSC in Table 1 . The red arrows highlight the advantages of our method. The Dice similarity coefficients (%) are overlaid for reference. Blue: necrotic and non-enhancing tumor core, yellow: enhancing tumor, and green: edema.
Figure 5: (a) Total ratio of federated decoder parameters of all clients concerning different personalization patience P . (b) Ratio of federated decoder parameters for each client with P=10 .
Trustworthy multimodal fusion in clinical settings requires handling incomplete and heterogeneous modality subsets across institutions, where privacy constraints prohibit centralized data sharing. Federated learning (FL) mitigates data-sharing constraints but suffers from client-specific missing modalities, where institutions possess incomplete multimodal subsets, degrading fusion quality and segmentation performance. While FL and weak supervision have been studied separately, their joint use with image-level labels under heterogeneous missing modalities remains unaddressed. We propose MOSAIC, the first modality-agnostic federated framework for weakly supervised binary tumor segmentation under client-specific missing modalities. We introduce a client-specific modality-alignment module that fuses available channels into a shared latent space without prior knowledge of modality identity, a spectral prototype alignment loss that reconciles cross-client distribution shift using compact non-invertible frequency-domain statistics, and a dedicated federated refinement network that denoises the resulting CAM pseudo-labels into accurate masks, breaking the accuracy ceiling of weak supervision. Experiments on three multi-institutional brain tumor benchmarks (FeTS2022, BraTS-MEN, and BraTS-SSA) demonstrate significant improvements over all image, box, and point-supervised baselines, approaching fully supervised accuracy using only image-level labels and reaching 0.84 Dice on FeTS2022. Dynamic new client addition enables previously unseen institutions to join an already-trained federation within 0.01-0.04 Dice without retraining. Code is available at https://github.com/Tarun2201/MOSAIC.
Tarun Kumar Garg, Vaanathi Sundaresan
Department of IISc Mathematics Initiative, Indian Institute of Science, Bengaluru 560012, Karnataka, India · Department of Computational and Data Sciences, Indian Institute of Science, Bengaluru 560012, Karnataka, India
Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME
Peibo Song, Xiaotian Xue, Jinshuo Zhang +5
School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab +2
Federated learning (FL) has become a promising paradigm for collaborative medical image analysis, yet existing frameworks remain tightly coupled to task-specific backbones and are fragile under heterogeneous imaging modalities. Such constraints hinder real-world deployment, where institutions vary widely in modality distributions and must support diverse downstream tasks. To address this limitation, we propose OmniFM, a modality- and task-agnostic FL framework that unifies training across classification, segmentation, super-resolution, visual question answering, and multimodal fusion without re-engineering the optimization pipeline. OmniFM builds on a key frequency-domain insight: low-frequency spectral components exhibit strong cross-modality consistency and encode modality-invariant anatomical structures. Accordingly, OmniFM integrates (i) Global Spectral Knowledge Retrieval to inject global frequency priors, (ii) Embedding-wise Cross-Attention Fusion to align representations, and (iii) Prefix-Suffix Spectral Prompting to jointly condition global and personalized cues, together regularized by a Spectral-Proximal Alignment objective that stabilizes aggregation. Experiments on real-world datasets show that OmniFM consistently surpasses state-of-the-art FL baselines across intra- and cross-modality heterogeneity, achieving superior results under both fine-tuning and training-from-scratch setups.
Meilin Liu, Jiaying Wang, Jing Shan
School of Software, Shenyang University of Technology