Federated multimodal models often assume every site has every modality, although hospitals differ in access to EHRs, chest radiographs, and ECGs. We study this setting on a MIMIC-derived respiratory deterioration task with simulated FL clients and introduce FedCoRe (Federated Cross-Modal Representation Completion). FedCoRe learns representation- or logit-space corrections rather than generating synthetic ECGs or CXR images. When a client observes a modality that may be missing at deployment, it evaluates the same example with and without that modality to obtain paired supervision. Only clients with such pairs update the completion module, and validation may retain the unchanged prediction. We freeze the trained multimodal predictor during evaluation so that measured differences come only from completion. Hiding ECG reduced AUROC by about 0.085; paired-example FedAvg restored 0.0415 AUROC, or 49.0% of the lost performance. We therefore report two distinct effects: paired-example FedAvg partially recovers the missing-ECG gap, while validation-selected completion is a task-specific classifier-logit correction rather than literal ECG recovery. For CXR, effect-aware completion recovers 52.8% of the loss in a controlled test where CXR is hidden. Paired-example FedAvg transfers part of this effect, but validation keeps the no-completion baseline for deployment cases whose inputs lack CXR. Thus, FedCoRe should be read as a validation-gated completion/correction framework: it can recover missing-modality signal in supported settings, but it should be deployed only when paired examples and validation evidence support that modality.
Figures & tables
Figure 1: Complete-modality evaluation contrasts with federated deployment, where sites and patients may have different subsets of EHR, CXR, and ECG.
Figure 2: FedCoRe workflow: paired observations train the completion operator, sites without valid pairs receive zero weight, and validation applies completion or leaves predictions unchanged.
Template
Clients
Full-modality clients
Missing-ECG mass
Paired-example structure
Purpose
No full-modality client
4
0
66.3%
ECG-present but no complete site
Negative control
Patient-mixed any-missing
4
Mixed
30.0%
Complete and ECG-present patients within clients
Main heterogeneous setting
Patient-mixed heavy
4
Mixed
36.3%
Heavier missing-ECG patient mix
Stress setting
Table 1: Core FL heterogeneity templates for the missing-ECG frozen-predictor completion test. Missing-ECG mass is expected test-set patient mass under the client mask; paired sources are examples in which ECG is observed and can supervise completion.
Figure 3: Observed gains with the multimodal predictor frozen. Bars report matched target-removal/recovery tests rather than deployment gains; the box separately reports CXR deployment, where validation selected α=0 and retained the baseline.
Scenario
Completion
Seeds
Before
After
Full ref
Ref.-norm.
Δ AUROC [95% CI]
AUPRC
Positive
No full-modality client
FedAvg
7,17,42
0.5781
0.5781
0.6627
0.0%
+0.0000 [0.0000, 0.0000]
0.0375 / 0.0375
0/3
No full-modality client
Val-selected
7,17,42
0.5781
0.5781
0.6627
0.0%
+0.0000 [0.0000, 0.0000]
0.0375 / 0.0375
0/3
Patient-mixed combined
FedAvg
6 rows
0.5781
0.6195
0.6627
49.0%
+0.0415 [0.0023, 0.0778]
0.0375 / 0.0368
6/6
Patient-mixed combined
Val-selected
6 rows
0.5781
0.7235
0.6627
171.8%
+0.1455 [0.0617, 0.2213]
0.0375 / 0.0495
6/6
Table 2: Frozen-predictor missing-ECG test. Both methods select scale on validation; “Val-selected” also selects source. CIs use shared patient-level resamples across the six matched patient-mixed predictions. Full ref is ECG-present replay; reference-normalized gain is (After−Before)/(Fullref−Before) , where values above 100% denote correction beyond the reference.
Seed
Full AUROC
Missing
Completed
Δ AUROC [95% CI]
Gap rec.
AUPRC before/after
α
7
0.6673
0.5237
0.6454
+0.1217 [0.0005, 0.2330]
84.7%
0.0204 / 0.0409
0.50
17
0.6969
0.5933
0.6228
+0.0295 [-0.0520, 0.1104]
28.5%
0.0402 / 0.0331
0.75
23
0.7303
0.6187
0.6187
+0.0000 [0.0000, 0.0000]
0.0%
0.0297 / 0.0297
0.00
31
0.7072
0.5188
0.6594
+0.1405 [0.0301, 0.2464]
74.6%
0.0255 / 0.0345
0.10
42
0.7192
0.5307
0.6739
+0.1431 [-0.0037, 0.2826]
75.9%
0.0327 / 0.0611
0.10
Mean
0.7042
0.5570
0.6440
+0.0870
52.8%
0.0297 / 0.0399
–
Table 3: Frozen-predictor missing-CXR completion test. Validation selects scale and gate threshold; only scale is shown. Delta intervals are paired patient-level bootstrap 95% CIs. Means and gap recovery are unweighted across seeds.
In this paper, we address the problem of multimodal federated learning with missing modality. Existing methods utilize an additional public dataset or perform naive feature synthesis that is based solely on the available modality. To address these limitations, we propose ProMoE-FL, a Prototype-conditioned Mixture-of-Experts framework for robust missing-modality feature synthesis in multimodal federated learning. ProMoE-FL builds a global client-aware prototype bank that captures clinically meaningful modality priors across institutions. Our Mixture of Experts is conditioned on these prototypes and modality indices to enable direction-aware expert routing for dynamically synthesizing missing features. We perform extensive quantitative and qualitative evaluations on four public chest X-ray datasets (MIMIC-CXR, NIH Open-I, PadChest, and CheXpert) and demonstrate that ProMoE-FL consistently outperforms state-of-the-art methods in both homogeneous as well as the more challenging heterogeneous settings.
Aavash Chhetri, Bibek Niroula, Eduard Vazquez +4
NepAl Applied Mathematics and Informatics Institute for research, Nepal · Fogsphere (Redev.AI Ltd), UK · University of Lausanne, Switzerland +4
Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative training, but real-world clinical applications often suffer from within-modality missingness caused by sensor intermittency or irregular sampling. Existing methods implicitly represent unobserved data via architectural alignment or missing embeddings, often failing to recover the true distribution and yielding sub-optimal performance. We propose CondI, a federated framework explicitly addressing this missingness using conditional diffusion models. CondI employs a two-phase training pipeline: first, imputing unobserved temporal components using available multimodal context and conditional embeddings; second, optimizing modality-specific extractors and joint embedding spaces. During inference, imputed raw data pass through trained extractors to generate robust features, providing a holistic representation for downstream tasks. Explicit data imputation ensures models operate on complete semantic structures, significantly enhancing resilience against severe data incompleteness. Experiments on three clinical datasets (PTB-XL, SLEEP-EDF, MIMIC-IV) demonstrate CondI achieves comparable results to state-of-the-art baselines. Code: https://github.com/ZhengWugeng/CondI
Wugeng Zheng, Ziwen Kan, Katie Wang +2
University of Central Florida · Independent Researcher
Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.