Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity
Organizations: Yunnan Key Laboratory of Statistical Modeling and Data Analysis, Yunnan University; School of Mathematics and Statistics, Yunnan University, Dongwaihuan South Road, Chenggong District, Kunming, Yunnan, 650500, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, 518055, China · Guangdong Provincial/Zhuhai Key Laboratory of Interdisciplinary Research and Application for Data Science, Beijing Normal-Hong Kong Baptist University, 2000 Jintong Road, Tangjiawan, Zhuhai, Guangdong, 519087, China
Abstract
Multimodal fusion is studied under hybrid federation, where data are partitioned across samples and feature blocks. The proposed two-dimensional federated smooth multiple-kernel large-margin learning method (2D-FedSMKL) integrates availability-corrected modality scores, group-kernel regularization, and smooth worst-site risk aggregation. For fixed maps and propensities, the objective is convex and smooth. Curvature bounds and exact synchronous block-gradient equivalence are established for binary and multiclass scores, giving a conditional accelerated rate. An oracle-propensity binary risk bound and a propensity-perturbation result are also derived. Simulations and two public activity-recognition benchmarks assess site heterogeneity and modality absence. Worst-site aggregation improves worst-site performance under heterogeneity, including reversed and size-balanced designs. On the benchmarks, validation selects the correction; the selected model is not significantly worse than the strongest reference at the worst site, and group-regularization benefits vary by dataset.
Figures & tables
| Method | Mean AUC | Worst-site AUC | Mean BA | Worst-site BA |
|---|---|---|---|---|
| Complete-case SMKL | 0.675 (0.024) | 0.583 (0.030) | 0.621 (0.025) | 0.546 (0.027) |
| Zero-fill SMKL | 0.740 (0.018) | 0.613 (0.034) | 0.673 (0.016) | 0.574 (0.032) |
| IPW mean SMKL | 0.736 (0.018) | 0.612 (0.035) | 0.669 (0.014) | 0.577 (0.029) |
| 2D-FedSMKL | 0.723 (0.022) | 0.622 (0.037) | 0.659 (0.018) | 0.584 (0.028) |
| Oracle full SMKL | 0.751 (0.016) | 0.623 (0.037) | 0.681 (0.015) | 0.581 (0.036) |
| Method | Mean macro-F1 | Worst-site macro-F1 | Mean BA | Worst-site BA |
|---|---|---|---|---|
| MHEALTH | ||||
| Pooled linear SVM | 0.949 (0.007) | 0.849 (0.025) | 0.949 (0.007) | 0.859 (0.027) |
| Complete-case CS | 0.763 (0.021) | 0.587 (0.060) | 0.767 (0.021) | 0.592 (0.062) |
| FedAvg-CS | 0.925 (0.012) | 0.856 (0.029) | 0.925 (0.012) | 0.858 (0.028) |
| FedProx-CS | 0.924 (0.012) | 0.856 (0.029) | 0.925 (0.012) | 0.858 (0.028) |
| Zero-fill SMKL | 0.927 (0.011) | 0.856 (0.032) | 0.927 (0.011) | 0.855 (0.033) |
| Method | Worst-site macro-F1 | vs IPW mean |
|---|---|---|
| MHEALTH | ||
| IPW mean | 0.850 | – |
| 2D-FedSMKL (IPW, robust) | 0.857 | +0.007 [-0.010, +0.024] |
| Conditional imputation | 0.877 | +0.027 [+0.008, +0.047] |
| Conditional imputation + robust | 0.882 | +0.032 [+0.008, +0.056] |
| UCI-HAR | ||
| Design | 2D-FedSMKL | IPW mean | Zero-fill | vs IPW mean | |
|---|---|---|---|---|---|
| Original | 0.8 | 0.622 | 0.612 | 0.613 | +0.010 [+0.003, +0.017] |
| Original | 1.2 | 0.535 | 0.486 | 0.483 | +0.049 [+0.038, +0.060] |
| Reversed | 0.8 | 0.669 | 0.670 | 0.674 | -0.000 [-0.004, +0.004] |
| Reversed | 1.2 | 0.604 | 0.586 | 0.588 | +0.018 [+0.009, +0.028] |
| Balanced | 0.8 | 0.659 | 0.646 | 0.651 | +0.014 [+0.005, +0.022] |
| Balanced | 1.2 | 0.606 | 0.581 | 0.585 | +0.025 [+0.014, +0.035] |
| Missing | Comparator | AUC difference (95% CI) | Win rate | |
|---|---|---|---|---|
| 0.35 | 0.8 | Complete-case SMKL | 0.039 [0.023, 0.055] | 85% |
| 0.35 | 0.8 | Zero-fill SMKL | 0.009 [0.002, 0.016] | 75% |
| 0.35 | 0.8 | IPW mean SMKL | 0.010 [0.004, 0.016] | 80% |
| 0.35 | 1.2 | Complete-case SMKL | 0.066 [0.045, 0.086] | 85% |
| 0.35 | 1.2 | Zero-fill SMKL | 0.051 [0.041, 0.061] | 100% |
| 0.35 | 1.2 | IPW mean SMKL | 0.049 [0.039, 0.059] | 100% |
| Rows | Columns | Max. gradient error | KiB/round |
|---|---|---|---|
| 1 | 1 | 4.34e-17 | 0.75 |
| 3 | 2 | 4.16e-17 | 11.00 |
| 3 | 4 | 4.51e-17 | 28.50 |
| 6 | 4 | 5.72e-17 | 30.75 |
| 9 | 8 | 1.04e-16 | 68.00 |
| Dataset | SVM | Fed learning rate | Fed rounds | |
|---|---|---|---|---|
| MHEALTH | 0.12 | 10 | 0.05 | 350 |
| UCI-HAR | 0.20 | 1 | 0.05 | 200 |
| Dataset | Comparator | Difference | 95% interval |
|---|---|---|---|
| MHEALTH | Zero-fill SMKL | +0.001 | [-0.024, +0.025] |
| MHEALTH | IPW mean SMKL | +0.007 | [-0.010, +0.024] |
| MHEALTH | 2D-FedSMKL no MKL | -0.001 | [-0.014, +0.011] |
| MHEALTH | Complete-case CS | +0.270 | [+0.225, +0.315] |
| MHEALTH | FedAvg-CS | +0.000 | [-0.017, +0.018] |
| MHEALTH | FedProx-CS | +0.000 | [-0.017, +0.018] |
| Dataset | Method | Worst F1 (SD) | Gradient-norm range | Max gap bound |
|---|---|---|---|---|
| MHEALTH | Zero-fill | 0.856 (0.032) | [1.44e-05, 7.02e-05] | 4.10e-06 |
| MHEALTH | IPW mean | 0.850 (0.032) | [1.43e-04, 1.61e-04] | 2.16e-05 |
| MHEALTH | 2D-Fed | 0.857 (0.027) | [1.33e-04, 1.62e-04] | 2.18e-05 |
| MHEALTH | No-MKL | 0.858 (0.031) | [1.93e-04, 2.17e-04] | 3.94e-05 |
| UCI-HAR | Zero-fill | 0.777 (0.006) | [3.54e-04, 3.94e-04] | 1.30e-04 |
| UCI-HAR | IPW mean | 0.787 (0.010) | [3.40e-04, 4.12e-04] | 1.41e-04 |
| MHEALTH | UCI HAR | |||
|---|---|---|---|---|
| Method | 35% | 65% | 35% | 65% |
| Mean zero | 0.857 (0.027) | 0.769 (0.031) | 0.775 (0.008) | 0.742 (0.005) |
| Robust zero | 0.865 (0.021) | 0.787 (0.025) | 0.776 (0.008) | 0.747 (0.006) |
| Mean IPW | 0.847 (0.027) | 0.767 (0.039) | 0.781 (0.009) | 0.751 (0.007) |
| Robust IPW | 0.863 (0.030) | 0.787 (0.031) | 0.781 (0.008) | 0.755 (0.006) |
| Robust IPW no-MKL | 0.863 (0.029) | 0.806 (0.024) | 0.777 (0.010) | 0.753 (0.007) |
| Dataset | Absence | Contrast | Difference | 95% interval |
|---|---|---|---|---|
| MHEALTH | 35% | Robust given IPW | +0.0157 | [+0.0063, +0.0251] |
| MHEALTH | 35% | Robust given zero | +0.0078 | [-0.0075, +0.0232] |
| MHEALTH | 35% | IPW given mean | -0.0102 | [-0.0234, +0.0031] |
| MHEALTH | 35% | IPW given robust | -0.0022 | [-0.0110, +0.0065] |
| MHEALTH | 35% | Interaction | +0.0079 | [-0.0060, +0.0219] |
| MHEALTH | 35% | MKL given robust IPW | -0.0006 | [-0.0105, +0.0092] |
| Dataset | Penalty | Block 1 | Block 2 | Block 3 | Block 4 |
|---|---|---|---|---|---|
| MHEALTH | Full MKL | 0.625 (0.009) | 0.058 (0.008) | 0.171 (0.009) | 0.145 (0.007) |
| MHEALTH | No-MKL | 0.554 (0.007) | 0.103 (0.006) | 0.186 (0.007) | 0.156 (0.007) |
| UCI-HAR | Full MKL | 0.789 (0.010) | 0.211 (0.010) | – | – |
| UCI-HAR | No-MKL | 0.761 (0.010) | 0.239 (0.010) | – | – |
| Dataset | Absence | Fits | At cap | Gradient norm | Gap certificate |
|---|---|---|---|---|---|
| MHEALTH | 10% | 80 | 80/80 | 0.00154 / 0.00177 | 0.00198 / 0.00261 |
| MHEALTH | 35% | 100 | 100/100 | 0.00201 / 0.00253 | 0.00335 / 0.00532 |
| MHEALTH | 65% | 130 | 130/130 | 0.00278 / 0.00573 | 0.00642 / 0.0274 |
| UCI-HAR | 10% | 80 | 80/80 | 0.00397 / 0.00422 | 0.0131 / 0.0148 |
| UCI-HAR | 35% | 100 | 100/100 | 0.00395 / 0.00437 | 0.013 / 0.0159 |
| UCI-HAR | 65% | 130 | 130/130 | 0.00689 / 0.00792 | 0.0396 / 0.0523 |
| Dataset | Setting | Worst F1 | Difference | Lower clipped | ESS ratio |
|---|---|---|---|---|---|
| MHEALTH | Full, 0.10 | 0.798 | +0.000 | 0.0% | 0.885 |
| MHEALTH | Full, 0.05 | 0.798 | +0.000 | 0.0% | 0.885 |
| MHEALTH | Full, 0.20 | 0.799 | +0.001 | 4.7% | 0.889 |
| MHEALTH | Site-only, 0.10 | 0.796 | -0.001 | 0.0% | 0.963 |
| UCI-HAR | Full, 0.10 | 0.754 | +0.000 | 0.0% | 0.887 |
| UCI-HAR | Full, 0.05 | 0.754 | +0.000 | 0.0% | 0.887 |
| Dataset | min | max | min | max | |
|---|---|---|---|---|---|
| MHEALTH | 0.0799 | 0.1617 | 11451.2 | 23164.9 | 25.88 |
| UCI-HAR | 0.0208 | 0.1460 | 8580.9 | 60116.9 | 18.30 |
| Dataset | Method | F1 | Max F1 | |||
|---|---|---|---|---|---|---|
| MHEALTH | 2D-Fed | -1.12e-04 | 2.04e-03 | 1.53e-04 | +0.0000 | 0.0000 |
| MHEALTH | IPW mean | -1.08e-04 | 1.99e-03 | 1.55e-04 | +0.0000 | 0.0000 |
| MHEALTH | Zero-fill | -2.13e-05 | 9.09e-04 | 5.67e-05 | +0.0031 | 0.0093 |
| UCI-HAR | 2D-Fed | -4.69e-04 | 4.04e-03 | 3.48e-04 | +0.0014 | 0.0056 |
| UCI-HAR | IPW mean | -4.12e-04 | 3.84e-03 | 3.67e-04 | +0.0025 | 0.0036 |
| UCI-HAR | Zero-fill | -4.31e-04 | 3.98e-03 | 3.82e-04 | +0.0010 | 0.0021 |
| Method | 35% absence | 65% absence |
|---|---|---|
| MHEALTH | ||
| Zero-fill (mean imputation) | 0.858 | 0.763 |
| Conditional imputation | 0.875 | 0.776 |
| IPW mean | 0.848 | 0.778 |
| 2D-FedSMKL (IPW, robust) | 0.856 | 0.785 |
| Conditional imputation + robust | 0.884 | 0.816 |
| Dataset | IPW | Conditional imputation | Selected |
|---|---|---|---|
| MHEALTH | 0.8168 | 0.8282 | Conditional imputation |
| UCI-HAR | 0.5549 | 0.5769 | Conditional imputation |
| Dataset | |||||
|---|---|---|---|---|---|
| MHEALTH | 0.8016 | 0.8093 | 0.8169 | 0.8168 | 0.8089 |
| UCI-HAR | 0.5463 | 0.5481 | 0.5512 | 0.5533 | 0.5549 |
| Data | Layout/method | Gap | Target round | MiB | |
|---|---|---|---|---|---|
| MHEALTH | 1x1 FISTA | 1.12e-04 | 2.04e-03 | 188 | 232.8 |
| MHEALTH | 1x1 GD | 4.40e-02 | 6.84e-02 | – | 232.8 |
| MHEALTH | 2x2 FISTA | 1.12e-04 | 2.04e-03 | 188 | 463.9 |
| MHEALTH | 2x2 GD | 4.40e-02 | 6.84e-02 | – | 463.9 |
| MHEALTH | 4x4 FISTA | 1.12e-04 | 2.04e-03 | 188 | 926.0 |
| MHEALTH | 4x4 GD | 4.40e-02 | 6.84e-02 | – | 926.0 |
| Formulation | Data and target | Coordination and guarantee object | Relation to the present study |
|---|---|---|---|
| SimpleMKL [ 20 ] | Kernel combination for supervised prediction; a foundation for block-norm regularization. | Classifier and kernel-weight optimization; the weighted-norm/mixed-norm connection. | Supplies the block-norm foundation. The present target couples this principle to corrected modality scores and site-risk aggregation. |
| FedHD [ 9 ] | Learning from hybrid data, including non-uniform holding patterns. | Local updates and gradient tracking for communication-efficient hybrid optimization. | Provides a hybrid optimization comparison. The present target explicitly couples propensity-corrected scores, group regularization and a specified site-risk aggregate. |
| Zhang et al. [ 10 ] | Sample/feature heterogeneity with global/local model matching. | Feature-extractor/classifier coordination through model matching. | Addresses representation/model compatibility; the present interface assumes feature correspondence and does not perform that matching. |
| 2D-FedSMKL | Aligned row/column partitions; fixed representations and propensity corrections; smooth worst-site decision fusion. | Partial scores, site-risk scalars, norm summaries and block gradients. Exact synchronous reductions recover the common objective and gradient. | Target-specific convexity/curvature and equivalence properties; conditional accelerated convergence and an oracle-propensity binary risk bound. Guarantees concern the stated objective and exact synchronous reductions. |
| Family | Training information and inference handling | Comparison role |
|---|---|---|
| Pooled linear SVM | Central access to all masked feature vectors and labels; zero filling during training and inference. | Statistical reference under centralized access. |
| Complete-case CS/SMKL | Fit on rows with every optional block observed; infer on all masked test rows after zero filling. The public CS implementation sets the MKL penalty to zero. | Joint training-selection/inference strategy; its gap does not isolate loss of training rows alone. |
| FedAvg-CS/FedProx-CS | Horizontal site learners have complete zero-filled feature vectors for their own rows; exchange model parameters. | Horizontal statistical references; they do not enforce separate column custodians. |
| Mean/robust zero-fill/IPW | The same fixed representation and masked rows; cross site aggregation (mean/robust) with test and training score handling (zero-fill/IPW). | Factorial controls for aggregation, availability correction and their interaction. |
| No-MKL | Same corrected inputs and robust site objective as the full method, with . | Control for group-kernel regularization; group weights describe the fitted coefficients rather than causal modality importance. |
| Oracle full SMKL | Central access to complete training and test modalities, without artificial removal. | Infeasible full-information reference, not a deployment competitor. |
Multimodal Federated Learning
Multimodal federated learning trains models across data silos with different data types while keeping raw data local.
See all 56 papers