Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus

Mohamed Badi; Chaouki Ben Issaid; Mehdi Bennis

cs.LGMar 19, 2026

Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus

Authors: Mohamed Badi, Chaouki Ben Issaid, Mehdi Bennis

Organizations: Center for Wireless Communications, University of Oulu, Oulu 90014, Finland

Abstract

Federated learning (FL) enables collaborative model training across distributed devices without sharing raw data, but applying FL to multi-modal settings introduces significant challenges. Clients typically possess heterogeneous modalities and model architectures, making it difficult to align feature spaces efficiently while preserving privacy and minimizing communication costs. To address this, we introduce CoMFed, a Communication-Efficient Multi-Modal Federated Learning framework that uses learnable projection matrices to generate compressed latent representations. A latent-space regularizer aligns these representations across clients, improving cross-modal consistency and robustness to outliers. Experiments on human activity recognition benchmarks show that CoMFed achieves competitive accuracy with minimal overhead.

Figures & tables

Algorithm	Uplink Comp.	Comm. Cost (40 / 50 / 58%)
CoMFed	$\mathcal{O}(\|\mathcal{M}\|d)$	269 KB / 410 KB / 768 KB
Harmony (UniFL)	$\mathcal{O}(\|\mathbf{w}\|)$	4.39 GB / 9.57 GB / NA
FedMD	$\mathcal{O}(\|\mathcal{D}_{\text{public}}\|\!\cdot\!\|\mathcal{M}\|)$	723 KB / 1.73 MB / NA
FedIoT	$\mathcal{O}(\|\mathbf{w^{g}}\|)$	NA / NA / NA

TABLE I: Cumulative communication overhead required to first achieve each target accuracy (USC dataset). “NA” denotes that the accuracy threshold was not attained.

Fig. 1: Average test accuracy over training iterations (USC dataset)

Fig. 2: Average test accuracy over training iterations (DeepSense blockage-severity dataset)

Fig. 3: Accuracy comparison among algorithms and across modality subsets (USC-dataset).

Fig. 4: Accuracy comparison among algorithms and across modality subsets (DeepSense-dataset).

Fig. 5: Robustness to attacks median vs mean methods for USC-dataset (

\gamma

denotes the number of Byzantine clients).

Fig. 6: Average accuracy vs communication cost for different projection dimensions (USC-dataset).

Multimodal Federated Learning

Multimodal federated learning trains models across data silos with different data types while keeping raw data local.

See all 51 papers

Boosting Multimodal Federated Learning via Chained Modality Optimization
Jun 1, 2026Zixin Zhang, Fan Qi, Shuai Li +2Multimodal Federated Learning Federated Learning Aggregation
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Prototype-guided Bilateral Alignment Multimodal Federated Learning
Sep 30, 2026Tianchi Liao Tianchi_Liao, Lele Fu, Sheng Huang +3Multimodal Federated Learning Federated Learning Aggregation
Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China · School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China
FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation
Mar 5, 2026Min Tan, Junchao Ma, Yinfu Feng +6Multimodal Federated Learning Federated Learning
Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University · Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University · Alibaba International Digital Commerce Group +1

Jun 1, 2026J/K move · Enter open · S save

Boosting Multimodal Federated Learning via Chained Modality Optimization

Zixin Zhang, Fan Qi, Shuai Li, Xiaoshan Yang+1

College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China

Multimodal Federated Learning Federated Learning Aggregation Federated Learning

Open paper AskPDF arXiv

Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability. However, most existing MMFL methods cast multimodal training as a joint optimization problem, overlooking a key bottleneck: modality competition, where dominant modalities suppress weaker ones and lead to suboptimal global models. To address this, we propose FedMChain, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases. This phase-wise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an error-compensated regularizer. On the server side, we employ a sparse sign-guided aggregation strategy that leverages directional sign agreement for robust intra-modality aggregation, avoids destructive averaging, and supports less frequent synchronization to reduce communication overhead. Extensive experiments on multimodal benchmarks demonstrate that FedMChain consistently improves predictive performance while requiring less frequent communication than baselines.

Abstract

Multimodal Federated Learning

Explore similar work

Boosting Multimodal Federated Learning via Chained Modality Optimization

Prototype-guided Bilateral Alignment Multimodal Federated Learning

FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation