FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis

Xianxun Zhu; Zezhong Sun; Imad Rida; Erik Cambria; Junqi Su; Rui Wang; Hui Chen

cs.LGFeb 28, 2026

FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis

Authors: Xianxun Zhu, Zezhong Sun, Imad Rida, Erik Cambria, Junqi Su, Rui Wang, Hui Chen

Organizations: Shanghai University, China · Nanyang Technological University, Singapore · Macquarie University, Australia · North China Electric Power University, China ·

Abstract

Multimodal sentiment analysis in federated learning environments faces significant challenges due to missing modalities, heterogeneous data distributions, and unreliable client updates. Existing federated approaches often struggle to maintain robust performance under these practical conditions. In this paper, we propose FedUAF, a unified multimodal federated learning framework that addresses these challenges through uncertainty-aware fusion and reliability-guided aggregation. FedUAF explicitly models modality-level uncertainty during local training and leverages client reliability to guide global aggregation, enabling effective learning under incomplete and noisy multimodal data. Extensive experiments on CMU-MOSI and CMU-MOSEI demonstrate that FedUAF consistently outperforms state-of-the-art federated baselines across various missing-modality patterns and Non-IID settings. Moreover, FedUAF exhibits superior robustness against noisy clients, highlighting its potential for real-world multimodal federated applications.

Figures & tables

Fig. 1: Overview of the proposed FedUAF framework.

Dataset	$\rho_{m}$	Method	IID					Non-IID
Dataset	$\rho_{m}$	Method	0.2	0.4	0.6	0.8	1.0	0.2	0.4	0.6	0.8	1.0
MOSI	0.2	FedAvg	0.88 $\pm$ 0.06	0.90 $\pm$ 0.07	0.91 $\pm$ 0.07	0.96 $\pm$ 0.04	0.96 $\pm$ 0.06	0.90 $\pm$ 0.08	0.95 $\pm$ 0.07	1.01 $\pm$ 0.02	1.07 $\pm$ 0.05	1.13 $\pm$ 0.09
		FedProx	0.82 $\pm$ 0.04	0.89 $\pm$ 0.04	0.90 $\pm$ 0.13	0.95 $\pm$ 0.05	0.98 $\pm$ 0.04	0.88 $\pm$ 0.06	0.93 $\pm$ 0.09	0.98 $\pm$ 0.04	1.04 $\pm$ 0.11	1.10 $\pm$ 0.07
		FedMSplit	0.85 $\pm$ 0.04	0.88 $\pm$ 0.03	0.91 $\pm$ 0.09	0.89 $\pm$ 0.04	0.97 $\pm$ 0.04	0.89 $\pm$ 0.02	0.91 $\pm$ 0.03	0.96 $\pm$ 0.03	1.02 $\pm$ 0.04	1.07 $\pm$ 0.08
		MIFL	0.84 $\pm$ 0.02	0.87 $\pm$ 0.03	0.90 $\pm$ 0.03	0.93 $\pm$ 0.04	0.96 $\pm$ 0.09	0.85 $\pm$ 0.11	0.90 $\pm$ 0.03	0.99 $\pm$ 0.06	1.06 $\pm$ 0.04	1.05 $\pm$ 0.05
		FedMAC	0.83 $\pm$ 0.02	0.86 $\pm$ 0.03	0.89 $\pm$ 0.06	0.92 $\pm$ 0.03	0.95 $\pm$ 0.04	0.84 $\pm$ 0.02	0.88 $\pm$ 0.11	0.93 $\pm$ 0.03	0.98 $\pm$ 0.04	1.02 $\pm$ 0.04

TABLE I: Performance (MAE

\downarrow

) under different missing-modality ratios in both IID and Non-IID settings. Results are reported as mean

\pm

standard deviation over 3 runs. Best and second-best results are highlighted in bold and underline , respectively.

Fig. 2: Ablation study of FedUAF under severe missing-modality (

\rho_{m}=0.8

) and strong data heterogeneity (Non-IID intensity

=1.0

). Results on CMU-MOSI (top) and CMU-MOSEI (bottom) are reported in terms of MAE (mean

\pm

std over 3 runs).

Fig. 3: Robustness to noisy clients under severe missing-modality (

\rho_{m}=0.8

) and strong data heterogeneity (Non-IID intensity

=1.0

). Results on CMU-MOSI (top) and CMU-MOSEI (bottom) are reported in terms of MAE.

Multimodal Federated Learning

Multimodal federated learning trains models across data silos with different data types while keeping raw data local.

See all 51 papers

FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation
Mar 5, 2026Min Tan, Junchao Ma, Yinfu Feng +6Multimodal Federated Learning Federated Learning
Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University · Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University · Alibaba International Digital Commerce Group +1
Multimodal Federated Learning under Dual-Axis Modality Missingness
Aug 10, 2026Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin +5Multimodal Federated Learning Missing-Modality Learning
1KAIST · 2CUHK · 3HKUST
Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus
Mar 19, 2026Mohamed Badi, Chaouki Ben Issaid, Mehdi BennisMultimodal Federated Learning Federated Learning
Center for Wireless Communications, University of Oulu, Oulu 90014, Finland

Mar 5, 2026J/K move · Enter open · S save

FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation

Min Tan, Junchao Ma, Yinfu Feng, Jiajun Ding+5

Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University · Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University · Alibaba International Digital Commerce Group+1

Multimodal Federated Learning Federated Learning Heterogeneous Federated Learning Multimodal Fusion Federated Knowledge Distillation

Open paper AskPDF arXiv

Multimodal Federated Learning (MFL) enables clients with heterogeneous data modalities to collaboratively train models without sharing raw data, offering a privacy-preserving framework that leverages complementary cross-modal information. However, existing methods often overlook personalized client performance and struggle with modality/task discrepancies, as well as model heterogeneity. To address these challenges, we propose FedAFD, a unified MFL framework that enhances client and server learning. On the client side, we introduce a bi-level adversarial alignment strategy to align local and global representations within and across modalities, mitigating modality and task gaps. We further design a granularity-aware fusion module to integrate global knowledge into the personalized features adaptively. On the server side, to handle model heterogeneity, we propose a similarity-guided ensemble distillation mechanism that aggregates client representations on shared public data based on feature similarity and distills the fused knowledge into the global model. Extensive experiments conducted under both IID and non-IID settings demonstrate that FedAFD achieves superior performance and efficiency for both the client and the server.

Abstract

Multimodal Federated Learning

Explore similar work

FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation

Multimodal Federated Learning under Dual-Axis Modality Missingness

Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus