Organizations: Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University · Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University · Alibaba International Digital Commerce Group · College of Computer Science and Technology, Zhejiang University
Multimodal Federated Learning (MFL) enables clients with heterogeneous data modalities to collaboratively train models without sharing raw data, offering a privacy-preserving framework that leverages complementary cross-modal information. However, existing methods often overlook personalized client performance and struggle with modality/task discrepancies, as well as model heterogeneity. To address these challenges, we propose FedAFD, a unified MFL framework that enhances client and server learning. On the client side, we introduce a bi-level adversarial alignment strategy to align local and global representations within and across modalities, mitigating modality and task gaps. We further design a granularity-aware fusion module to integrate global knowledge into the personalized features adaptively. On the server side, to handle model heterogeneity, we propose a similarity-guided ensemble distillation mechanism that aggregates client representations on shared public data based on feature similarity and distills the fused knowledge into the global model. Extensive experiments conducted under both IID and non-IID settings demonstrate that FedAFD achieves superior performance and efficiency for both the client and the server.
Figures & tables
Figure 1: The challenges of multimodal federated learning
Figure 2: Overview of our proposed FedAFD comprising three steps: ① Server is trained on a public dataset and extracts global public features. ② With delivered global representations and encoders, clients train local models on private data by granularity-aware local and global fusion, enhanced with bi-level adversarial feature alignment. ③ Clients extract local public features of the public dataset and then sends it to the server. The server performs adaptive aggregation and updates the global model via similarity-guided ensemble distillation.
Size
Clients
Server
CIFAR-100
AGNEWS
Flickr30k
MS-COCO
acc@1
acc@1
i2t R@1
t2i R@1
i2t R@1
t2i R@1
rsum R@1
10k
33.18
51.98
32.48
25.68
33.98
26.18
60.16
20k
32.03
50.81
34.50
27.39
39.90
31.91
71.81
30k
30.08
50.35
36.88
28.64
43.66
34.43
78.09
Table 1: FedAFD with varying public data sizes. The evaluation metric is top-1 accuracy(acc@1) for CIFAR-100/AGNEWS classification, top-1 Recall(R@1) for Flickr30k and MS-COCO cross-modal retrieval (i2t/t2i).
Table 2: Performance comparison with baselines on diverse clients and the server under both IID and Non-IID settings. #rounds denotes the communication round number when they are close to convergence on the server, i.e. , rsum R@1 equals 57.50.
Method
Clients
Server
CIFAR-100
AGNEWS
Flickr30k
MS-COCO
acc@1
acc@1
i2t R@1
t2i R@1
i2t R@1
t2i R@1
rsum R@1
FedAFD
33.18
51.98
32.48
25.68
33.98
26.18
60.16
w/o BAA
33.56
49.03
32.13
25.56
33.70
25.59
59.29
w/o GFF
24.94
44.46
22.23
18.31
33.88
25.84
59.72
w/o SED
32.21
50.20
31.38
25.59
33.82
25.74
59.56
Table 3: Ablation Study of FedAFD under Non-IID Setting
Figure 3: T-SNE analysis of feature discrepancy on public data. Locally-trained and FedAFD encoders are compared.
Figure 4: T-SNE visualization of different features on CIFAR-100, in terms of both image category and feature types.
Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability. However, most existing MMFL methods cast multimodal training as a joint optimization problem, overlooking a key bottleneck: modality competition, where dominant modalities suppress weaker ones and lead to suboptimal global models. To address this, we propose FedMChain, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases. This phase-wise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an error-compensated regularizer. On the server side, we employ a sparse sign-guided aggregation strategy that leverages directional sign agreement for robust intra-modality aggregation, avoids destructive averaging, and supports less frequent synchronization to reduce communication overhead. Extensive experiments on multimodal benchmarks demonstrate that FedMChain consistently improves predictive performance while requiring less frequent communication than baselines.
Zixin Zhang, Fan Qi, Shuai Li +2
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a Multimodal Federated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (ii) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance.
Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China · School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China
Federated learning (FL) enables collaborative model training across distributed devices without sharing raw data, but applying FL to multi-modal settings introduces significant challenges. Clients typically possess heterogeneous modalities and model architectures, making it difficult to align feature spaces efficiently while preserving privacy and minimizing communication costs. To address this, we introduce CoMFed, a Communication-Efficient Multi-Modal Federated Learning framework that uses learnable projection matrices to generate compressed latent representations. A latent-space regularizer aligns these representations across clients, improving cross-modal consistency and robustness to outliers. Experiments on human activity recognition benchmarks show that CoMFed achieves competitive accuracy with minimal overhead.
Mohamed Badi, Chaouki Ben Issaid, Mehdi Bennis
Center for Wireless Communications, University of Oulu, Oulu 90014, Finland