cs.LGOct 30, 2026

Role-Aware Server Optimization for Multimodal Federated Learning with Missing Modalities

Authors: Liuyin Shi, Wenbo He

Organizations: McMaster University

Abstract

Multimodal federated learning (MFL) trains a shared model by aggregating client updates without exchanging raw data. Under missing modalities, clients adapt active modality encoders and a shared prediction block (fusion and classifier) to different input subsets. This raises two server-side questions: which clients should contribute to each component, and how should their aggregates be incorporated across rounds? We observe that an absent-modality encoder can still adapt through missing-input handling. Availability- based aggregation can therefore omit its update while retaining jointly learned prediction-block updates. Contributor selection also shapes server momentum, a weighted history of aggregated updates. We observe that prediction-block momentum can improve or reduce performance depending on the encoder update policy. We therefore propose Role-Aware Server Optimization (RASO), which coordinates contributor selection with role-specific server updates. RASO selects contributors through prediction dependence and inclusion in local optimization, retains encoder momentum, and synchronizes the shared prediction block to its current aggregate. Experiments across datasets and fusion architectures demonstrate the effectiveness of RASO under missing modalities.

Figures & tables

Explore similar work

CardsList
  1. FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation

    Mar 5, 2026Min Tan, Junchao Ma, Yinfu Feng +6Multimodal Federated LearningFederated Learning

  2. Boosting Multimodal Federated Learning via Chained Modality Optimization

    Jun 1, 2026Zixin Zhang, Fan Qi, Shuai Li +2Multimodal Federated LearningFederated Learning Aggregation

  3. Multimodal Federated Learning under Dual-Axis Modality Missingness

    Aug 10, 2026Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin +5Multimodal Federated LearningMissing-Modality Learning