cs.LGNov 2, 2026

Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity

Authors: Xiaofei Wu, Rongmei Liang, Jian Qing Shi

Organizations: Yunnan Key Laboratory of Statistical Modeling and Data Analysis, Yunnan University; School of Mathematics and Statistics, Yunnan University, Dongwaihuan South Road, Chenggong District, Kunming, Yunnan, 650500, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, 518055, China · Guangdong Provincial/Zhuhai Key Laboratory of Interdisciplinary Research and Application for Data Science, Beijing Normal-Hong Kong Baptist University, 2000 Jintong Road, Tangjiawan, Zhuhai, Guangdong, 519087, China

Abstract

Multimodal fusion is studied under hybrid federation, where data are partitioned across samples and feature blocks. The proposed two-dimensional federated smooth multiple-kernel large-margin learning method (2D-FedSMKL) integrates availability-corrected modality scores, group-kernel regularization, and smooth worst-site risk aggregation. For fixed maps and propensities, the objective is convex and smooth. Curvature bounds and exact synchronous block-gradient equivalence are established for binary and multiclass scores, giving a conditional accelerated rate. An oracle-propensity binary risk bound and a propensity-perturbation result are also derived. Simulations and two public activity-recognition benchmarks assess site heterogeneity and modality absence. Worst-site aggregation improves worst-site performance under heterogeneity, including reversed and size-balanced designs. On the benchmarks, validation selects the correction; the selected model is not significantly worse than the strongest reference at the worst site, and group-regularization benefits vary by dataset.

Figures & tables

Explore similar work

CardsList
  1. Multimodal Federated Learning under Dual-Axis Modality Missingness

    Aug 10, 2026Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin +5Multimodal Federated LearningMissing-Modality Learning

  2. Prototype-guided Bilateral Alignment Multimodal Federated Learning

    Sep 30, 2026Tianchi Liao Tianchi_Liao, Lele Fu, Sheng Huang +3Multimodal Federated LearningFederated Learning Aggregation