MOSAIC: Modality-agnostic Spectral Alignment for Federated Image-level Weakly Supervised Tumor Segmentation under Client-specific Missing Modalities

Tarun Kumar Garg; Vaanathi Sundaresan

eess.IVAug 20, 2026

MOSAIC: Modality-agnostic Spectral Alignment for Federated Image-level Weakly Supervised Tumor Segmentation under Client-specific Missing Modalities

Authors: Tarun Kumar Garg, Vaanathi Sundaresan

Organizations: Department of IISc Mathematics Initiative, Indian Institute of Science, Bengaluru 560012, Karnataka, India · Department of Computational and Data Sciences, Indian Institute of Science, Bengaluru 560012, Karnataka, India

Abstract

Trustworthy multimodal fusion in clinical settings requires handling incomplete and heterogeneous modality subsets across institutions, where privacy constraints prohibit centralized data sharing. Federated learning (FL) mitigates data-sharing constraints but suffers from client-specific missing modalities, where institutions possess incomplete multimodal subsets, degrading fusion quality and segmentation performance. While FL and weak supervision have been studied separately, their joint use with image-level labels under heterogeneous missing modalities remains unaddressed. We propose MOSAIC, the first modality-agnostic federated framework for weakly supervised binary tumor segmentation under client-specific missing modalities. We introduce a client-specific modality-alignment module that fuses available channels into a shared latent space without prior knowledge of modality identity, a spectral prototype alignment loss that reconciles cross-client distribution shift using compact non-invertible frequency-domain statistics, and a dedicated federated refinement network that denoises the resulting CAM pseudo-labels into accurate masks, breaking the accuracy ceiling of weak supervision. Experiments on three multi-institutional brain tumor benchmarks (FeTS2022, BraTS-MEN, and BraTS-SSA) demonstrate significant improvements over all image, box, and point-supervised baselines, approaching fully supervised accuracy using only image-level labels and reaching 0.84 Dice on FeTS2022. Dynamic new client addition enables previously unseen institutions to join an already-trained federation within 0.01-0.04 Dice without retraining. Code is available at https://github.com/Tarun2201/MOSAIC.

Figures & tables

Figure 1: Overview of MOSAIC comprising $K$ clients and a central server. Phase 1: Each client passes its available modalities through a local modality-alignment (MA) module to obtain $Z_{i}^{k}$ . A multi-exit classifier generates CAMs which, together with a projection of $Z_{i}^{k}$ , are processed by an aggregation network to produce pseudo-labels $\hat{y}$ . The SPA loss aligns local prototypes $E_{\mathrm{loc}}$ to the global prototype $E_{\mathrm{global}}$ . Phase 2: An EMA teacher–student framework combines the teacher prediction $p_{\mathrm{tea}}$ , prototype cue $p_{\mathrm{proto}}$ , and trimap target $y_{\mathrm{tri}}$ into a soft CAM that supervises the student via $\mathcal{L}_{\mathrm{FT}}$ . The prototype-alignment loss $\mathcal{L}_{\mathrm{proto}}$ aligns student features $f_{\mathrm{st}}$ to federated prototypes $(\mu_{\mathrm{fg}},\mu_{\mathrm{bg}})$ .

Dataset	Client	Modalities	Train	Val	Test
FeTS2022	1	FLAIR, T1ce	4495 (29)	1395 (9)	1395 (9)
	2	FLAIR, T2	3255 (21)	1085 (7)	1085 (7)
	3	T1ce, T2	3255 (21)	1085 (7)	1085 (7)
	4	FLAIR	3410 (22)	930 (6)	930 (6)
BraTS-MEN	1	FLAIR, T1ce	3720 (24)	1240 (8)	1240 (8)
	2	T1ce	3720 (24)	1240 (8)	1240 (8)

Table 1: Client-wise modality subsets and data splits for the three datasets (FeTS2022, BraTS-MEN, and BraTS-SSA). Entries are slices (subjects in parentheses).

Method	Client 1			Client 2			Client 3			Client 4			Average
Method	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )
a. Fully supervised federated methods
FedAvg	0.87 0.27	0.83 0.29	31.2 97.3	0.90 0.22	0.86 0.25	22.9 79.1	0.86 0.28	0.82 0.29	35.6 104.1	0.89 0.25	0.85 0.27	28.9 93.2	0.88	0.84	29.7
FedBN	0.88 0.25	0.84 0.28	27.9 91.9	0.91 0.21	0.87 0.24	20.2 75.6	0.86 0.29	0.82 0.30	38.7 108.0	0.89 0.24	0.85 0.27	27.2 89.7	0.88	0.85	28.5
FedProx	0.87 0.26	0.83 0.28	31.8 95.9	0.89 0.24	0.85 0.26	23.6 81.4	0.86 0.27	0.83 0.29	31.9 98.4	0.87 0.26	0.83 0.29	30.2 93.9	0.87	0.83	29.4
MOON	0.87 0.26	0.84 0.28	29.0 93.3	0.91 0.21	0.87 0.23	18.8 73.3	0.86 0.28	0.82 0.29	36.4 105.6	0.89 0.25	0.85 0.27	29.3 93.9	0.88	0.85	28.4

Table 2: Segmentation performance on FeTS2022 per client and averaged across clients. Dice (D,

\uparrow

), IoU (J,

\uparrow

), and HD95 (H, mm,

\downarrow

) are reported; subscripts denote per-slice standard deviation. Fully supervised methods serve as upper references. Bold denotes the best weakly supervised result; ∗ marks baselines that MOSAIC significantly outperforms on all metrics (

p<0.01

, two-sided paired Wilcoxon signed-rank test).

Method	Client 1			Client 2			Client 3			Client 4			Average
Method	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )
a. Fully supervised federated methods
FedAvg	0.76 0.40	0.74 0.41	77.3 148.8	0.86 0.33	0.85 0.34	50.9 125.7	0.84 0.34	0.82 0.36	54.6 130.2	0.79 0.38	0.77 0.39	70.6 142.9	0.81	0.79	63.3
FedBN	0.82 0.34	0.80 0.36	48.8 119.5	0.85 0.34	0.83 0.35	52.2 128.1	0.86 0.33	0.84 0.34	49.4 125.2	0.81 0.35	0.79 0.37	59.4 132.1	0.83	0.81	52.4
FedProx	0.82 0.35	0.80 0.36	55.6 129.6	0.85 0.34	0.83 0.35	52.7 126.6	0.85 0.34	0.84 0.35	51.5 127.9	0.79 0.36	0.77 0.38	61.3 133.3	0.83	0.81	55.3
MOON	0.85 0.32	0.83 0.33	44.3 116.4	0.84 0.35	0.83 0.36	54.1 128.6	0.86 0.32	0.85 0.33	47.5 122.9	0.80 0.37	0.78 0.38	63.9 136.6	0.84	0.82	52.5

Table 3: Segmentation performance on BraTS-MEN per client and averaged across clients. Dice (D,

\uparrow

), IoU (J,

\uparrow

), and HD95 (H, mm,

\downarrow

) are reported; subscripts denote per-slice standard deviation. Fully supervised methods (group a) and the centralized image-level model (group b) serve as upper references and are excluded from the comparison. Bold denotes the best federated weakly supervised result; ∗ marks baselines MOSAIC significantly outperforms (

p<0.01

, two-sided paired Wilcoxon; all metrics except HD95 for FedDM).

Method	Client 1			Client 2			Client 3			Average
Method	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )	D ( $\uparrow$ )	J ( $\uparrow$ )	H ( $\downarrow$ )
a. Fully supervised federated methods
FedAvg	0.79 0.35	0.75 0.36	56.8 126.5	0.76 0.35	0.71 0.36	60.3 130.2	0.86 0.27	0.81 0.29	35.9 99.3	0.80	0.75	51.0
FedBN	0.82 0.31	0.78 0.33	45.2 110.6	0.77 0.34	0.72 0.35	57.7 127.4	0.87 0.26	0.83 0.28	32.6 93.2	0.82	0.77	45.2
FedProx	0.84 0.29	0.80 0.31	36.9 100.1	0.74 0.36	0.69 0.37	62.9 128.4	0.84 0.28	0.79 0.31	34.0 93.8	0.81	0.76	44.6
MOON	0.84 0.29	0.80 0.31	38.4 102.1	0.78 0.36	0.74 0.36	66.2 137.3	0.86 0.26	0.81 0.28	36.7 90.5	0.83	0.78	47.1

Table 4: Segmentation performance on BraTS-SSA per client and averaged across clients. Dice (D,

\uparrow

), IoU (J,

\uparrow

), and HD95 (H, mm,

\downarrow

p<0.01

, two-sided paired Wilcoxon).

Figure 2: Visual comparison of MOSAIC with state-of-the-art methods on FeTS2022. A representative test slice for each client shown with predictions overlaid on the input slice. True positives, false positives, and false negatives are shown in yellow red and blue respectively.

Figure 3: Visual comparison of methods on BraTS-MEN. A representative test slice for each client shown with predictions overlaid on the input slice. True positives, false positives, and false negatives are shown in yellow red and blue respectively.

Figure 4: Visual comparison of methods on BraTS-SSA. A representative test slice for each client shown with predictions overlaid on the input slice. True positives, false positives, and false negatives are shown in yellow red and blue respectively.

$\Omega$ (bands)	Dice	IoU	HD95
2	0.81 0.29	0.76 0.32	36.3 100.6
4	0.82 0.29	0.76 0.31	34.7 100.1
6	0.82 0.29	0.77 0.31	35.0 99.0
8	0.84 0.28	0.79 0.30	32.6 95.1
10	0.81 0.29	0.76 0.32	34.8 97.4
12	0.82 0.28	0.77 0.31	32.0 93.8

Table 5: Ablation on the number of radial spectral bands

\Omega

in the SPA loss, on FeTS2022. Macro-averaged Dice (

\uparrow

), IoU (

\uparrow

) and HD95 (mm,

\downarrow

\Omega=8

is our default configuration; bold denotes the best value per column.

Alignment objective	Dice	IoU	HD95
Std ∗	0.65	0.61	67.5
Contrastive ∗	0.70	0.66	51.6
Mean ∗	0.73	0.67	58.2
Histogram KL ∗	0.78	0.73	41.2
SPA (ours)	0.80	0.74	40.6

Table 6: Feature-alignment objectives compared as drop-in replacements for SPA during Phase 1 on FeTS2022. Macro-averaged Dice (

\uparrow

), IoU (

\uparrow

), and HD95 (mm,

\downarrow

) across four clients; ∗ denotes objectives SPA significantly outperforms on all metrics (

p<0.01

, two-sided paired Wilcoxon).

Configuration	Dice	IoU	HD95
Full MOSAIC	0.84 0.28	0.79 0.30	32.6 95.1
w/o trimap denoising	0.82 0.29	0.78 0.31	35.6 99.5
w/o CRF loss $\mathcal{L}_{\mathrm{CRF}}$	0.83 0.29	0.78 0.31	33.8 96.4
w/o prototype alignment $\mathcal{L}_{\mathrm{proto}}$	0.83 0.28	0.78 0.31	33.1 96.4
w/o consistency $\mathcal{L}_{\mathrm{cons}}$	0.83 0.29	0.78 0.31	34.8 98.4
w/o presence $\mathcal{L}_{\mathrm{cls}}$	0.80 0.31	0.75 0.33	39.1 102.1

Table 7: Leave-one-out model and loss component ablation on FeTS2022. Macro-averaged Dice (

\uparrow

), IoU (

\uparrow

), and HD95 (mm,

\downarrow

) over four clients. Each row disables one component while keeping all others fixed; core-only retains only the modality-alignment module and SPA. Bold denotes the best value per column.

	Local-adapt (LA)			Direct-join (DJ)
Mod.	Dice	IoU	HD95	Dice	IoU	HD95
F, T1ce	0.81 0.27	0.76 0.31	21.5 65.1	0.78 0.30	0.72 0.33	32.3 87.0
F, T2	0.80 0.28	0.74 0.32	22.0 67.0	0.80 0.29	0.74 0.32	29.1 82.6
T1ce, T2	0.81 0.30	0.76 0.32	37.6 100.9	0.81 0.31	0.76 0.32	38.9 102.2
F	0.81 0.27	0.75 0.31	22.0 67.0	0.82 0.27	0.76 0.31	20.5 64.0

Table 8: Dynamic client addition on FeTS2022. Segmentation performance of the newly added fifth client, under both local-adapt (LA) and direct-join (DJ) joining strategies. The better of the two strategies for each metric indicated in bold.

Dataset	Client	$H_{\mathrm{fg}}$	$H_{\mathrm{bg}}$	$\mathrm{Dice}_{\mathrm{unc}}$
FeTS2022	1	0.0500	0.0042	0.0164
	2	0.0527	0.0041	0.0207
	3	0.0729	0.0065	0.0229
	4	0.0615	0.0040	0.0200
	Avg.	0.0587	0.0047	0.0198
BraTS-MEN	1	0.0775	0.0030	0.0318

Table 9: Aleatoric uncertainty under TTA on three datasets.

H_{\mathrm{fg}}

and

H_{\mathrm{bg}}

: mean predictive entropy over foreground and background pixels;

\mathrm{Dice}_{\mathrm{unc}}

: standard deviation of per-pass Dice under input perturbations. Lower values indicate more confident, stable predictions.

Figure 5: Qualitative aleatoric uncertainty for a representative slice from each dataset. Each row shows the input image with the region of interest, the predicted tumor mask (red) with its ground truth contour (green), and the pixel-wise aleatoric entropy.

Figure 6: Cross-client modality gap with and without SPA, measured at the modality-alignment output $Z^{k}_{i}$ and the U-Net bottleneck using MMD 2 (left) and Fréchet distance (right). Each bar is the mean over all client pairs; lower values indicates better alignment.

Explore similar work

Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation
Mar 5, 2026Hong Liu, Dong Wei, Qian Dai +3Federated Learning Multimodal Federated Learning
National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, 361005, Fujian, China · Jarvis Research Center, Tencent YouTu Lab, Shenzhen, 518075, Guangdong, China · Department of Computer Science at School of Informatics, Xiamen University, Xiamen, 361005, Fujian, China +1
OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imaging
Mar 23, 2026Meilin Liu, Jiaying Wang, Jing ShanFederated Learning Heterogeneous Federated Learning
School of Software, Shenyang University of Technology
FedMust: Semi-supervised Multi-task Student-Teacher Federated Learning for Multi-organ CT Segmentation
Sep 21, 2026Ashkan Moradi, Bendik Skarre Abrahamsen, Mattijs ElschotMulti-Task Learning Semi-Supervised Medical Image Segmentation
Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway · Central Staff at St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway

Mar 5, 2026J/K move · Enter open · S save

Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation

Hong Liu, Dong Wei, Qian Dai, Xian Wu+2

National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, 361005, Fujian, China · Jarvis Research Center, Tencent YouTu Lab, Shenzhen, 518075, Guangdong, China · Department of Computer Science at School of Informatics, Xiamen University, Xiamen, 361005, Fujian, China+1

Federated Learning Multimodal Federated Learning Heterogeneous Federated Learning Multimodal Fusion Brain Tumor Segmentation

Open paper AskPDF arXiv

Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, some FL participants may possess only a subset of the complete imaging modalities, posing intermodal heterogeneity as a challenge to effectively training a global model on all participants' data. Meanwhile, each participant expects a personalized model tailored to its local data characteristics in FL. This work proposes a new FL framework with federated modality-specific encoders and partially personalized multimodal fusion decoders (FedMEPD) to address the two concurrent issues. Specifically, FedMEPD employs an exclusive encoder for each modality to account for the intermodal heterogeneity. While these encoders are fully federated, the decoders are partially personalized to meet individual needs -- using the discrepancy between global and local parameter updates to dynamically determine which decoder filters are personalized. Implementation-wise, a server with full-modal data employs a fusion decoder to fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation. Moreover, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the model parameters. Conversely, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up for the information loss due to absent modalities. FedMEPD is validated on the BraTS 2018 and 2020 multimodal brain tumor segmentation benchmarks. Results show that it outperforms various up-to-date methods for multimodal and personalized FL, and its novel designs are effective.

Abstract

Explore similar work

Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation

OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imaging

FedMust: Semi-supervised Multi-task Student-Teacher Federated Learning for Multi-organ CT Segmentation