Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

Milad Rostamian; Mohammadreza Mohammadi; Ender Konukoglu; Mohsen Soryani

cs.CVOct 13, 2026

Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

Authors: Milad Rostamian, Mohammadreza Mohammadi, Ender Konukoglu, Mohsen Soryani

Organizations: School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran · Computer Vision Laboratory, ETH Zurich, 8092 Zurich, Switzerland

Abstract

Different MRI modalities reveal different parts of a brain tumor. Segmentation models combine information from these modalities to outline tumor regions. Many models use a separate encoder for each modality. However, exchanging features among these encoders at several stages can increase model complexity. We introduce the Block-Wise Shared Adapter (BWSA), which combines compressed features from the modality encoders in a shared path and returns adapted features to each stream. In its fixed-blending form, BWSA adds the adapted features to the original features. Its trainable-gated variant (TG-BWSA) instead learns separate weights for the original and adapted features at each modality and encoder stage. We evaluate both variants on BraTS 2020 and BraTS 2023 with four segmentation backbones. On the official BraTS 2023 online validation set, BWSA increases average region-wise Dice by 1.3--7.6 percentage points relative to the corresponding unmodified backbones. TG-BWSA increases it by 2.8--7.7 points. Both variants add approximately 0.85--3.73% parameters and reduce average HD95 relative to the unmodified backbones.

Figures & tables

Fig. 1: Representative fusion placements. (a) Early fusion concatenates the modalities at the input. (b) Late fusion combines independently produced predictions. (c) BWSA inserts a shared adaptation path between encoder stages. Its gated extension learns scalar weights for the original and adapted streams of each modality at each stage.

Fig. 2: Overall multimodal framework. Each modality is processed by a separate encoder branch. The adapters connect successive encoder blocks, and their outputs are passed to a shared decoder through skip connections.

Panel of Figure 3 — Fig. 3: TG-BWSA. (a) Compression $\rightarrow$ shared recalibration/refinement $\rightarrow$ gated redistribution. (b) CMCR operating on the concatenated multimodal latent tensor.

Backbone	BraTS 2020 mean Dice			BraTS 2023 mean Dice			TG-BWSA overhead
Backbone	Base	+BWSA	+TG-BWSA	Base	+BWSA	+TG-BWSA	TG-BWSA overhead
RFNet [ 9 ]	0.8623	0.8693	0.8712	0.8941	0.8955	0.8982	3.73%
mmFormer [ 5 ]	0.8635	0.8695	0.8715	0.8976	0.8995	0.9013	0.92%
MEUNet	0.8873	0.8957	0.8974	0.9169	0.9198	0.9240	2.94%
F2Net [ 6 ]	0.8744	—	—	0.9048	—	—	—

TABLE II: Mean local full-modality Dice over WT, TC, and ET across three runs on BraTS 2020 and 2023. The 2D scores are slice-wise and are not directly comparable with the 3D scores. Bold marks the best variant within each backbone row. The last column shows the parameter increase from each unmodified backbone to its TG-BWSA version. Region-wise results and pretrained MEFCT comparisons appear in Tables III – VI and Table VII , respectively.

Model	WT	TC	ET	Mean
RFNet [ 9 ]	0.9110 $\pm$ 0.0032	0.8825 $\pm$ 0.0031	0.7935 $\pm$ 0.0029	0.8623
RFNet*	0.9138 $\pm$ 0.0016	0.8847 $\pm$ 0.0019	0.8094 $\pm$ 0.0017	0.8693
RFNet**	0.9161 $\pm$ 0.0018	0.8861 $\pm$ 0.0020	0.8114 $\pm$ 0.0011	0.8712
mmFormer [ 5 ]	0.9133 $\pm$ 0.0034	0.8798 $\pm$ 0.0030	0.7974 $\pm$ 0.0028	0.8635
mmFormer*	0.9151 $\pm$ 0.0028	0.8829 $\pm$ 0.0023	0.8105 $\pm$ 0.0026	0.8695
mmFormer**	0.9165 $\pm$ 0.0017	0.8847 $\pm$ 0.0025	0.8134 $\pm$ 0.0023	0.8715

TABLE III: BraTS 2020 local-validation Dice for 3D models (mean

\pm

standard deviation over three runs).

Model	WT	TC	ET	Mean
MEUNet	0.8724 $\pm$ 0.0029	0.8925 $\pm$ 0.0037	0.8971 $\pm$ 0.0031	0.8873
MEUNet*	0.8838 $\pm$ 0.0011	0.9019 $\pm$ 0.0015	0.9014 $\pm$ 0.0012	0.8957
MEUNet**	0.8853 $\pm$ 0.0013	0.9041 $\pm$ 0.0011	0.9029 $\pm$ 0.0010	0.8974
F2Net [ 6 ]	0.8213 $\pm$ 0.0086	0.9034 $\pm$ 0.0035	0.8985 $\pm$ 0.0044	0.8744

TABLE IV: BraTS 2020 slice-wise local-validation Dice for 2D models with trainable encoders (mean

\pm

standard deviation over three runs).

Model	WT	TC	ET	Mean
RFNet [ 9 ]	0.9244 $\pm$ 0.0022	0.8976 $\pm$ 0.0023	0.8603 $\pm$ 0.0019	0.8941
RFNet*	0.9261 $\pm$ 0.0015	0.8990 $\pm$ 0.0002	0.8613 $\pm$ 0.0015	0.8955
RFNet**	0.9293 $\pm$ 0.0009	0.9007 $\pm$ 0.0011	0.8647 $\pm$ 0.0008	0.8982
mmFormer [ 5 ]	0.9293 $\pm$ 0.0043	0.9011 $\pm$ 0.0040	0.8624 $\pm$ 0.0042	0.8976
mmFormer*	0.9307 $\pm$ 0.0017	0.9029 $\pm$ 0.0013	0.8648 $\pm$ 0.0010	0.8995
mmFormer**	0.9322 $\pm$ 0.0015	0.9070 $\pm$ 0.0016	0.8647 $\pm$ 0.0013	0.9013

TABLE V: BraTS 2023 local-validation Dice for 3D models (mean

\pm

standard deviation over three runs).

Model	WT	TC	ET	Mean
MEUNet	0.8955 $\pm$ 0.0033	0.9291 $\pm$ 0.0030	0.9260 $\pm$ 0.0034	0.9169
MEUNet*	0.9017 $\pm$ 0.0012	0.9306 $\pm$ 0.0013	0.9271 $\pm$ 0.0010	0.9198
MEUNet**	0.9037 $\pm$ 0.0011	0.9345 $\pm$ 0.0013	0.9338 $\pm$ 0.0012	0.9240
F2Net [ 6 ]	0.8634 $\pm$ 0.0249	0.9271 $\pm$ 0.0032	0.9240 $\pm$ 0.0024	0.9048

TABLE VI: BraTS 2023 slice-wise local-validation Dice for 2D MEUNet (mean

\pm

standard deviation over three runs).

(a) BraTS 2020
Model	WT	TC	ET	Mean
MEFCT	0.8614 $\pm$ 0.0041	0.8792 $\pm$ 0.0035	0.8775 $\pm$ 0.0047	0.8727
MEFCT*	0.8708 $\pm$ 0.0013	0.8802 $\pm$ 0.0010	0.8894 $\pm$ 0.0014	0.8801
MEFCT**	0.8720 $\pm$ 0.0011	0.8871 $\pm$ 0.0009	0.8911 $\pm$ 0.0008	0.8834
MEFCT-LoRA [ 24 ]	0.8694 $\pm$ 0.0014	0.8813 $\pm$ 0.0019	0.8873 $\pm$ 0.0007	0.8793

TABLE VII: Adapter comparison for 2D MEFCT on BraTS 2020 and 2023 local validation sets using slice-wise Dice under partial fine-tuning of pretrained encoders (bottleneck blocks unfrozen). Values are mean

\pm

standard deviation over three runs.

Model	Dice ET	Dice TC	Dice WT	HD95 ET	HD95 TC	HD95 WT	Dice Avg	HD95 Avg
RFNet [ 9 ]	0.7889	0.8201	0.8853	35.4500	26.8100	24.1500	0.8314	28.8033
RFNet + BWSA	0.8313	0.8511	0.9093	20.5500	16.3800	6.1700	0.8639	14.3667
RFNet + TG-BWSA	0.8364	0.8472	0.9127	18.9200	23.3400	6.7700	0.8654	16.3433
mmFormer [ 5 ]	0.6209	0.7025	0.8289	48.1700	33.2500	24.7900	0.7174	35.4033
mmFormer + BWSA	0.7502	0.7745	0.8563	26.9400	28.6200	16.5000	0.7937	24.0200
mmFormer + TG-BWSA	0.7685	0.7751	0.8394	23.5300	22.4800	19.8300	0.7943	21.9467

TABLE VIII: BraTS 2023 online validation results for all models (Dice/HD95). † Pretrained encoders use partial fine-tuning according to the protocol in Sec. IV-B .

Model	L-Dice ET	L-Dice TC	L-Dice WT	L-HD95 ET	L-HD95 TC	L-HD95 WT	Avg L-Dice	Avg L-HD95
Team @s7zuqer	0.866	0.89	0.909	10.594	7.228	11.086	0.888	9.636
Team @simis	0.857	0.876	0.908	13.649	12.825	11.270	0.880	12.581
Team @qian8	0.856	0.868	0.904	14.070	13.778	10.735	0.876	12.861
Faking_it [ 29 ]	0.854	0.858	0.887	17.321	18.007	19.322	0.866	18.217
BiomedMBZ [ 30 ]	0.837	0.859	0.903	25.451	19.185	12.685	0.866	19.107
Shenhai Zheng [ 28 ]	0.823	0.860	0.862	25.602	15.110	29.291	0.848	23.334

TABLE IX: BraTS 2023 online validation lesion-wise Dice and HD95. Comparator values and anonymized team identifiers are transcribed from Zheng et al. [ 28 ] .

Panel of Figure 12 — Fig. 4: Qualitative segmentation comparisons on randomly sampled BraTS 2023 local validation cases. Predictions are shown for the unmodified backbones and their adapter variants. Cyan denotes non-enhancing tumor, green denotes edema, and yellow denotes enhancing tumor.

Backbone	$P_{\text{base}}$	$P_{\text{+TG-BWSA}}$	$\Delta_{\%}$
RFNet	9.0	9.3	3.73
mmFormer	36.7	37.0	0.92
Multi-Enc. U-Net	29.8	30.7	2.94
Multi-Enc. FCT	102.7	103.6	0.85

TABLE X: Parameter counts (millions) for each unmodified backbone and its TG-BWSA version. The reported overhead compares these two models. Counts are rounded to one decimal place; percentages may differ from calculations using the displayed counts.

Model	Latency (s/input)	GPU Memory (GB)	GFLOPs
RFNet [ 9 ]	1.544	1.95	102.8
RFNet**	1.581	2.11	107.0

TABLE XI: Inference cost for one forward pass with an

80^{3}

input on an NVIDIA A6000 GPU.

Fig. 5: Training loss for RFNet and RFNet+TG-BWSA. The dashed line marks activation of the region-fusion loss near epoch 100. The curves describe optimization on the training data and should be interpreted together with the validation results.

Panel of Figure 16 — Fig. 6: Learned gate coefficients for RFNet+TG-BWSA over 300 training epochs. (a) Coefficient trajectories for each modality and encoder stage. (b) Distributions of values recorded across training epochs. (c) Final coefficients after epoch 300. $\alpha$ weights the adapted stream and $\beta$ weights the original stream.

Configuration	WT	TC	ET	Mean
TG-BWSA (learned gates)	0.9293 $\pm$ 0.0009	0.9007 $\pm$ 0.0011	0.8647 $\pm$ 0.0008	0.8982
BWSA (fixed weights)	0.9261 $\pm$ 0.0015	0.8990 $\pm$ 0.0002	0.8613 $\pm$ 0.0015	0.8955
Per-Modality Adapters (PMA)	0.9250 $\pm$ 0.0018	0.8985 $\pm$ 0.0009	0.8605 $\pm$ 0.0012	0.8947
No Residual Path	0.9230 $\pm$ 0.0025	0.9001 $\pm$ 0.0018	0.8601 $\pm$ 0.0020	0.8944
Baseline (RFNet)	0.9244 $\pm$ 0.0022	0.8976 $\pm$ 0.0023	0.8603 $\pm$ 0.0019	0.8941
w/o CMCR (no channel recalibration)	0.9265 $\pm$ 0.0024	0.8964 $\pm$ 0.0020	0.8562 $\pm$ 0.0022	0.8930

TABLE XII: Component-wise evaluation of adapter variants on BraTS 2023 using RFNet (mean

\pm

standard deviation over three runs). Best results are in bold , and second-best results are underlined .

$\alpha_{0}$	$\beta_{0}$	$\lambda$	WT	TC	ET
0.5	1.0	1e-4	0.9293	0.9007	0.8647
0.1	1.0	1e-4	0.9272	0.8991	0.8614
0.5	0.5	1e-4	0.9269	0.8970	0.8598
0.5	1.0	1e-5	0.9280	0.9001	0.8631
0.5	1.0	1e-3	0.9263	0.8964	0.8602

TABLE XIII: Effect of gate initialization and regularization on BraTS 2023 with RFNet.

Region	RFNet	+BWSA	+TG-BWSA
WT	0.754	0.825	0.834
TC	0.531	0.796	0.779
ET	0.430	0.627	0.629

TABLE XIV: Mean Dice across the 14 incomplete-input modality configurations on BraTS 2023. The complete-input case is excluded and reported with the full-modality results. Differences in the text are calculated from unrounded means.

Explore similar work

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
Apr 24, 2026Peibo Song, Xiaotian Xue, Jinshuo Zhang +5Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures
School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab +2
A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities
Apr 22, 2024Ming Kang, Fung Fung Ting, Shier Nee Saw +3Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation
School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia +1
Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation
Mar 20, 2026Lokendra Kumar, Shubham Aggarwal3D Medical Image Segmentation Hyper-Connections

Apr 24, 2026J/K move · Enter open · S save

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

Peibo Song, Xiaotian Xue, Jinshuo Zhang, Zihao Wang+4

School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab+2

Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures Medical Image Segmentation Medical Image Representation Learning Brain Tumor Segmentation

Open paper AskPDF arXiv

Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME

Abstract

Explore similar work

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation