Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity
Authors: Xiaofei Wu, Rongmei Liang, Jian Qing Shi
Organizations: Yunnan Key Laboratory of Statistical Modeling and Data Analysis, Yunnan University; School of Mathematics and Statistics, Yunnan University, Dongwaihuan South Road, Chenggong District, Kunming, Yunnan, 650500, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, 518055, China · Guangdong Provincial/Zhuhai Key Laboratory of Interdisciplinary Research and Application for Data Science, Beijing Normal-Hong Kong Baptist University, 2000 Jintong Road, Tangjiawan, Zhuhai, Guangdong, 519087, China
Multimodal fusion is studied under hybrid federation, where data are partitioned across samples and feature blocks. The proposed two-dimensional federated smooth multiple-kernel large-margin learning method (2D-FedSMKL) integrates availability-corrected modality scores, group-kernel regularization, and smooth worst-site risk aggregation. For fixed maps and propensities, the objective is convex and smooth. Curvature bounds and exact synchronous block-gradient equivalence are established for binary and multiclass scores, giving a conditional accelerated rate. An oracle-propensity binary risk bound and a propensity-perturbation result are also derived. Simulations and two public activity-recognition benchmarks assess site heterogeneity and modality absence. Worst-site aggregation improves worst-site performance under heterogeneity, including reversed and size-balanced designs. On the benchmarks, validation selects the correction; the selected model is not significantly worse than the strongest reference at the worst site, and group-regularization benefits vary by dataset.
Figures & tables
Figure 1: Illustrative sample and modality partitions in hybrid federated fusion. Sites group distinct patient records vertically; independently held modality blocks form the horizontal partition. Original device icons depict baseline records, ECG monitoring and wrist/ankle inertial sensors (IMUs). Check marks indicate observed blocks and crosses indicate record-level absence; the baseline remains observed. IDs, devices and availability patterns are schematic, not clinical study data. Computational row groups need not coincide with statistical sites. Raw feature blocks remain with their custodians. 2D-Fed denotes 2D-FedSMKL.
Figure 2: Availability-corrected additive fusion for one sample. Observed modalities are inverse-probability weighted, whereas an absent block contributes zero rather than an imputed signal. Under the stated missing-at-random assumption and with the true observation propensities, the correction recovers each full-data linear score contribution in expectation. Estimated propensities are clipped to limit instability, which can introduce bias.
Figure 3: Smooth worst-site aggregation. Log-sum-exp provides a differentiable approximation to the largest site risk, and its gradient gives the adaptive weight qs . Decreasing ρ approaches worst-site training; increasing ρ approaches π -weighted mean-risk training.
Method
Mean AUC
Worst-site AUC
Mean BA
Worst-site BA
Complete-case SMKL
0.675 (0.024)
0.583 (0.030)
0.621 (0.025)
0.546 (0.027)
Zero-fill SMKL
0.740 (0.018)
0.613 (0.034)
0.673 (0.016)
0.574 (0.032)
IPW mean SMKL
0.736 (0.018)
0.612 (0.035)
0.669 (0.014)
0.577 (0.029)
2D-FedSMKL
0.723 (0.022)
0.622 (0.037)
0.659 (0.018)
0.584 (0.028)
Oracle full SMKL
0.751 (0.016)
0.623 (0.037)
0.681 (0.015)
0.581 (0.036)
Table 1: Predictive performance in the primary simulation setting (35% target optional-modality absence and heterogeneity h=0.8 ). Entries are mean (standard deviation) over 20 replicates. The oracle uses fully observed modalities and is not feasible in practice.
Figure 4: Compact simulation trends for the three incomplete-sample fusion methods. The legend label 2D-Fed denotes the proposed 2D-FedSMKL method. Left: the robust objective separates from mean-risk alternatives as site heterogeneity increases. Right: the methods remain close at moderate modality absence, with only modest separation at severe absence. Error bars show standard deviations across matched replicates.
Method
Mean macro-F1
Worst-site macro-F1
Mean BA
Worst-site BA
MHEALTH
Pooled linear SVM
0.949 (0.007)
0.849 (0.025)
0.949 (0.007)
0.859 (0.027)
Complete-case CS
0.763 (0.021)
0.587 (0.060)
0.767 (0.021)
0.592 (0.062)
FedAvg-CS
0.925 (0.012)
0.856 (0.029)
0.925 (0.012)
0.858 (0.028)
FedProx-CS
0.924 (0.012)
0.856 (0.029)
0.925 (0.012)
0.858 (0.028)
Zero-fill SMKL
0.927 (0.011)
0.856 (0.032)
0.927 (0.011)
0.855 (0.033)
Table 2: Public-data results with fold-fitted preprocessing for source validation and final source-only representations, under 35% controlled MAR block absence. Entries are mean (sample SD) over ten matched masks. Zero-fill SMKL, and the zero-filled inference of complete-case CS, place an absent block at its observed training mean, which is mean imputation on standardized blocks. The oracle uses unmasked data and is fitted once, so its SD is zero across masks.
Method
Worst-site macro-F1
Δ vs IPW mean
MHEALTH
IPW mean
0.850
–
2D-FedSMKL (IPW, robust)
0.857
+0.007 [-0.010, +0.024]
Conditional imputation
0.877
+0.027 [+0.008, +0.047]
Conditional imputation + robust
0.882
+0.032 [+0.008, +0.056]
UCI-HAR
Table 3: Availability correction under the primary protocol (35% target absence, ten masks, 1400 rounds). Worst-site macro-F1 and paired difference against IPW mean, with the 95% interval over ten masks. The last row is the configuration selected on validation data and is the model summarised in the abstract.
Design
h
2D-FedSMKL
IPW mean
Zero-fill
Δ vs IPW mean
Original
0.8
0.622
0.612
0.613
+0.010 [+0.003, +0.017]
Original
1.2
0.535
0.486
0.483
+0.049 [+0.038, +0.060]
Reversed
0.8
0.669
0.670
0.674
-0.000 [-0.004, +0.004]
Reversed
1.2
0.604
0.586
0.588
+0.018 [+0.009, +0.028]
Balanced
0.8
0.659
0.646
0.651
+0.014 [+0.005, +0.022]
Balanced
1.2
0.606
0.581
0.585
+0.025 [+0.014, +0.035]
Table S1: Design sensitivity of the simulation. Worst-site AUC at 35% absence (20 replicates); the last column is the paired difference of 2D-FedSMKL against IPW mean. Original assigns the largest decision-boundary shift to the smallest site; reversed assigns it to the largest site; balanced uses site sizes (187,187,186) .
Figure S1: Simulation stress tests. The legend label 2D-Fed denotes the proposed 2D-FedSMKL method. Left: site heterogeneity at 35% optional-modality absence. Right: controlled optional-modality absence at h=0.8 . Points are means of 20 replicates.
Missing
h
Comparator
AUC difference (95% CI)
Win rate
0.35
0.8
Complete-case SMKL
0.039 [0.023, 0.055]
85%
0.35
0.8
Zero-fill SMKL
0.009 [0.002, 0.016]
75%
0.35
0.8
IPW mean SMKL
0.010 [0.004, 0.016]
80%
0.35
1.2
Complete-case SMKL
0.066 [0.045, 0.086]
85%
0.35
1.2
Zero-fill SMKL
0.051 [0.041, 0.061]
100%
0.35
1.2
IPW mean SMKL
0.049 [0.039, 0.059]
100%
Table S2: Paired improvement in worst-site AUC of 2D-FedSMKL over non-oracle baselines. Intervals are normal-approximation 95% confidence intervals over 20 matched replicates.
Figure S2: Objective value minus the minimum observed over the complete reference trace, floored at 10−12 for the logarithmic axis. The reference is the best observed value over the trace, not a known J∗ , and the plot is a numerical diagnostic.
Rows
Columns
Max. gradient error
KiB/round
1
1
4.34e-17
0.75
3
2
4.16e-17
11.00
3
4
4.51e-17
28.50
6
4
5.72e-17
30.75
9
8
1.04e-16
68.00
Table S3: Numerical equivalence and estimated synchronous communication for two-dimensional partitions.
Dataset
ρ
SVM C
Fed learning rate
Fed rounds
MHEALTH
0.12
10
0.05
350
UCI-HAR
0.20
1
0.05
200
Table S4: Frozen primary settings after source-only validation. Smooth hyperparameters not listed as selected are fixed as specified in S2.
Dataset
Comparator
Difference
95% interval
MHEALTH
Zero-fill SMKL
+0.001
[-0.024, +0.025]
MHEALTH
IPW mean SMKL
+0.007
[-0.010, +0.024]
MHEALTH
2D-FedSMKL no MKL
-0.001
[-0.014, +0.011]
MHEALTH
Complete-case CS
+0.270
[+0.225, +0.315]
MHEALTH
FedAvg-CS
+0.000
[-0.017, +0.018]
MHEALTH
FedProx-CS
+0.000
[-0.017, +0.018]
Table S5: Paired differences in worst-site macro-F1 after fold-fitted source validation: 2D-FedSMKL minus comparator. Intervals are 95% paired t intervals over ten matched masks, conditional on the fixed benchmark samples.
Dataset
Method
Worst F1 (SD)
Gradient-norm range
Max gap bound
MHEALTH
Zero-fill
0.856 (0.032)
[1.44e-05, 7.02e-05]
4.10e-06
MHEALTH
IPW mean
0.850 (0.032)
[1.43e-04, 1.61e-04]
2.16e-05
MHEALTH
2D-Fed
0.857 (0.027)
[1.33e-04, 1.62e-04]
2.18e-05
MHEALTH
No-MKL
0.858 (0.031)
[1.93e-04, 2.17e-04]
3.94e-05
UCI-HAR
Zero-fill
0.777 (0.006)
[3.54e-04, 3.94e-04]
1.30e-04
UCI-HAR
IPW mean
0.787 (0.010)
[3.40e-04, 4.12e-04]
1.41e-04
Table S6: Final primary smooth-model diagnostics over ten masks. The no-MKL fit is an additional primary ablation. Norms and certificates concern the empirical training objective; reaching a budget or a relative-step rule is not a claim of exact minimization. 2D-Fed denotes 2D-FedSMKL.
MHEALTH
UCI HAR
Method
35%
65%
35%
65%
Mean zero
0.857 (0.027)
0.769 (0.031)
0.775 (0.008)
0.742 (0.005)
Robust zero
0.865 (0.021)
0.787 (0.025)
0.776 (0.008)
0.747 (0.006)
Mean IPW
0.847 (0.027)
0.767 (0.039)
0.781 (0.009)
0.751 (0.007)
Robust IPW
0.863 (0.030)
0.787 (0.031)
0.781 (0.008)
0.755 (0.006)
Robust IPW no-MKL
0.863 (0.029)
0.806 (0.024)
0.777 (0.010)
0.753 (0.007)
Table S7: Fixed-representation component comparison. Entries are mean (SD) worst-site macro-F1 over 20 matched masks; settings are frozen before evaluation.
Dataset
Absence
Contrast
Difference
95% interval
MHEALTH
35%
Robust given IPW
+0.0157
[+0.0063, +0.0251]
MHEALTH
35%
Robust given zero
+0.0078
[-0.0075, +0.0232]
MHEALTH
35%
IPW given mean
-0.0102
[-0.0234, +0.0031]
MHEALTH
35%
IPW given robust
-0.0022
[-0.0110, +0.0065]
MHEALTH
35%
Interaction
+0.0079
[-0.0060, +0.0219]
MHEALTH
35%
MKL given robust IPW
-0.0006
[-0.0105, +0.0092]
Table S8: Prespecified secondary component study with common ρ=0.05 and a 350-round cap. Entries are paired differences in worst-site macro-F1 over 20 matched masks with unadjusted 95% t intervals. Robust given IPW is robust minus mean risk under IPW, and analogously for the other simple effects. Interaction is (robust IPW minus mean IPW) minus (robust zero minus mean zero); MKL is full minus no-MKL under robust IPW. These settings are not selected by the nested primary validation.
Dataset
Penalty
Block 1
Block 2
Block 3
Block 4
MHEALTH
Full MKL
0.625 (0.009)
0.058 (0.008)
0.171 (0.009)
0.145 (0.007)
MHEALTH
No-MKL
0.554 (0.007)
0.103 (0.006)
0.186 (0.007)
0.156 (0.007)
UCI-HAR
Full MKL
0.789 (0.010)
0.211 (0.010)
–
–
UCI-HAR
No-MKL
0.761 (0.010)
0.239 (0.010)
–
–
Table S9: Descriptive coefficient-norm shares at 65% absence in the secondary study: mean (SD) over 20 masks. Shares use smoothed modality Frobenius norms normalized to sum to one in both fits; no-MKL shares are post-fit summaries rather than optimized MKL weights. These are not causal importance measures.
Dataset
Absence
Fits
At cap
Gradient norm
Gap certificate
MHEALTH
10%
80
80/80
0.00154 / 0.00177
0.00198 / 0.00261
MHEALTH
35%
100
100/100
0.00201 / 0.00253
0.00335 / 0.00532
MHEALTH
65%
130
130/130
0.00278 / 0.00573
0.00642 / 0.0274
UCI-HAR
10%
80
80/80
0.00397 / 0.00422
0.0131 / 0.0148
UCI-HAR
35%
100
100/100
0.00395 / 0.00437
0.013 / 0.0159
UCI-HAR
65%
130
130/130
0.00689 / 0.00792
0.0396 / 0.0523
Table S10: Terminal diagnostics for all 620 prespecified secondary fits, including sensitivity variants. Gradient norm and empirical objective-gap certificate are shown as median / maximum. For this strongly convex objective, J(θ)−J∗≤∥∇J(θ)∥2/(2μ) with μ=λ0ηb=0.0006 . The certificate concerns numerical optimization, not generalization; cap attainment is not itself convergence.
Dataset
Setting
Worst F1
Difference
Lower clipped
ESS ratio
MHEALTH
Full, 0.10
0.798
+0.000
0.0%
0.885
MHEALTH
Full, 0.05
0.798
+0.000
0.0%
0.885
MHEALTH
Full, 0.20
0.799
+0.001
4.7%
0.889
MHEALTH
Site-only, 0.10
0.796
-0.001
0.0%
0.963
UCI-HAR
Full, 0.10
0.754
+0.000
0.0%
0.887
UCI-HAR
Full, 0.05
0.754
+0.000
0.0%
0.887
Table S11: Propensity sensitivity at 65% absence using 10 matched masks. ESS ratio is the average modality-specific effective sample size divided by observed training records. Differences compare each setting with baseline-plus-site propensity and clipping floor 0.10.
Dataset
ep min
ep max
Δp min
Δp max
Bw
MHEALTH
0.0799
0.1617
11451.2
23164.9
25.88
UCI-HAR
0.0208
0.1460
8580.9
60116.9
18.30
Table S12: Observed propensity deviation and the perturbation constant of Corollary 2. ep=max∣p^−p∣ over training rows and optional modalities; Δp=Bwκep/pmin2 with Bw from Corollary 1 and pmin=0.08 .
Dataset
Method
ΔJ
∥g350∥
∥g1400∥
Δ F1
Max ∣Δ F1 ∣
MHEALTH
2D-Fed
-1.12e-04
2.04e-03
1.53e-04
+0.0000
0.0000
MHEALTH
IPW mean
-1.08e-04
1.99e-03
1.55e-04
+0.0000
0.0000
MHEALTH
Zero-fill
-2.13e-05
9.09e-04
5.67e-05
+0.0031
0.0093
UCI-HAR
2D-Fed
-4.69e-04
4.04e-03
3.48e-04
+0.0014
0.0056
UCI-HAR
IPW mean
-4.12e-04
3.84e-03
3.67e-04
+0.0025
0.0036
UCI-HAR
Zero-fill
-4.31e-04
3.98e-03
3.82e-04
+0.0010
0.0021
Table S13: Numerical budget sensitivity on three prespecified matched masks. Changes are 1400-budget minus 350-budget values; gradients are terminal Euclidean norms. Diagnostic test metrics were not used to select the primary budget. This pilot diagnostic uses ρ=0.12 for MHEALTH and 0.05 for HAR; the final primary HAR ρ was subsequently reselected as 0.20 . 2D-Fed denotes 2D-FedSMKL.
Method
35% absence
65% absence
MHEALTH
Zero-fill (mean imputation)
0.858
0.763
Conditional imputation
0.875
0.776
IPW mean
0.848
0.778
2D-FedSMKL (IPW, robust)
0.856
0.785
Conditional imputation + robust
0.884
0.816
Table S14: Correction and aggregation in the secondary budget (350 rounds, 20 masks). Worst-site macro-F1, mean over masks. At 35% absence the conditional-imputation variant is +0.035[+0.021,+0.049] above IPW mean on MHEALTH and −0.000[−0.005,+0.004] on UCI-HAR.
Dataset
IPW
Conditional imputation
Selected
MHEALTH
0.8168
0.8282
Conditional imputation
UCI-HAR
0.5549
0.5769
Conditional imputation
Table S15: Correction selection on ten inner-validation masks (1400 rounds, 35% absence). Entries are mean worst-site macro-F1; the selection rule prefers the higher value, with ties to the IPW form.
Dataset
ρ=0.03
0.05
0.08
0.12
0.20
MHEALTH
0.8016
0.8093
0.8169
0.8168
0.8089
UCI-HAR
0.5463
0.5481
0.5512
0.5533
0.5549
Table S16: Validation of ρ on ten validation masks at the full 1400-round budget, 35% absence. Entries are mean worst-site macro-F1 over the ten masks.
Data
Layout/method
Gap
∥∇J∥
Target round
MiB
MHEALTH
1x1 FISTA
1.12e-04
2.04e-03
188
232.8
MHEALTH
1x1 GD
4.40e-02
6.84e-02
–
232.8
MHEALTH
2x2 FISTA
1.12e-04
2.04e-03
188
463.9
MHEALTH
2x2 GD
4.40e-02
6.84e-02
–
463.9
MHEALTH
4x4 FISTA
1.12e-04
2.04e-03
188
926.0
MHEALTH
4x4 GD
4.40e-02
6.84e-02
–
926.0
Table S17: Multiclass hybrid message-interface audit on fixed public-data caches with 35% imposed absence (seed 20261101). All methods use 350 accepted rounds and the same objective. Gap is relative to an independently optimized best observed central objective, not a known optimum. MiB counts float64 application payloads in the explicit separate-coordinator topology; it is not measured network traffic. A dash indicates that the 10−3 reference-gap target was not reached.
Formulation
Data and target
Coordination and guarantee object
Relation to the present study
SimpleMKL [ 20 ]
Kernel combination for supervised prediction; a foundation for block-norm regularization.
Classifier and kernel-weight optimization; the weighted-norm/mixed-norm connection.
Supplies the block-norm foundation. The present target couples this principle to corrected modality scores and site-risk aggregation.
FedHD [ 9 ]
Learning from hybrid data, including non-uniform holding patterns.
Local updates and gradient tracking for communication-efficient hybrid optimization.
Provides a hybrid optimization comparison. The present target explicitly couples propensity-corrected scores, group regularization and a specified site-risk aggregate.
Zhang et al. [ 10 ]
Sample/feature heterogeneity with global/local model matching.
Feature-extractor/classifier coordination through model matching.
Addresses representation/model compatibility; the present interface assumes feature correspondence and does not perform that matching.
Partial scores, site-risk scalars, norm summaries and block gradients. Exact synchronous reductions recover the common objective and gradient.
Target-specific convexity/curvature and equivalence properties; conditional accelerated convergence and an oracle-propensity binary risk bound. Guarantees concern the stated objective and exact synchronous reductions.
Table S18: Technical positioning relative to selected foundations. The comparisons concern the stated formulations, rather than asserting that an existing method cannot be extended. Hybrid data ownership and sample-level modality absence are distinct: the present method assumes aligned records and known feature-block correspondence before optimization.
Family
Training information and inference handling
Comparison role
Pooled linear SVM
Central access to all masked feature vectors and labels; zero filling during training and inference.
Statistical reference under centralized access.
Complete-case CS/SMKL
Fit on rows with every optional block observed; infer on all masked test rows after zero filling. The public CS implementation sets the MKL penalty to zero.
Joint training-selection/inference strategy; its gap does not isolate loss of training rows alone.
FedAvg-CS/FedProx-CS
Horizontal site learners have complete zero-filled feature vectors for their own rows; exchange model parameters.
Horizontal statistical references; they do not enforce separate column custodians.
Mean/robust × zero-fill/IPW
The same fixed representation and masked rows; cross site aggregation (mean/robust) with test and training score handling (zero-fill/IPW).
Factorial controls for aggregation, availability correction and their interaction.
No-MKL
Same corrected inputs and robust site objective as the full method, with λ1=0 .
Control for group-kernel regularization; group weights describe the fitted coefficients rather than causal modality importance.
Oracle full SMKL
Central access to complete training and test modalities, without artificial removal.
Infeasible full-information reference, not a deployment competitor.
Table S19: Information access and roles of the predictive references and computational controls. All feasible predictors are evaluated on the same masked test population; complete-case methods are not scored only on a complete-test subset. Centralized/horizontal references do not independently establish feasibility under column-wise feature custody.
Figure S3: Spatial schematic of the two-dimensional ownership layout. Rows partition samples and columns partition feature/modality blocks; the dashed tile illustrates an unavailable block. Tile thickness is illustrative, not a third partition dimension, sample size, risk or measured quantity. Computational row groups need not coincide with statistical sites.
Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.
Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a Multimodal Federated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (ii) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance.
Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China · School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China
Multimodal federated learning (MFL) trains a shared model by aggregating client updates without exchanging raw data. Under missing modalities, clients adapt active modality encoders and a shared prediction block (fusion and classifier) to different input subsets. This raises two server-side questions: which clients should contribute to each component, and how should their aggregates be incorporated across rounds? We observe that an absent-modality encoder can still adapt through missing-input handling. Availability- based aggregation can therefore omit its update while retaining jointly learned prediction-block updates. Contributor selection also shapes server momentum, a weighted history of aggregated updates. We observe that prediction-block momentum can improve or reduce performance depending on the encoder update policy. We therefore propose Role-Aware Server Optimization (RASO), which coordinates contributor selection with role-specific server updates. RASO selects contributors through prediction dependence and inclusion in local optimization, retains encoder momentum, and synchronizes the shared prediction block to its current aggregate. Experiments across datasets and fusion architectures demonstrate the effectiveness of RASO under missing modalities.