Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME

Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are often missing due to resource constraints, resulting in significant performance degradation for methods that rely on complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Mamba-Transformer hybrid network (MMTSeg) for accurate brain tumor segmentation with missing modalities. We first employ a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodalities to extract complete modality information. We further develop an Unimodal Feature Enhancement (UFE) module to model the semantic relationship between global and local information. Finally, we built a Cross-Modal Fusion (CMF) module to explicitly align the global correlations across modalities, even when some modalities are missing. Complementary features within and across modalities are refined by the Mamba-Transformer hybrid architectures in both the UFE and CMF modules, dynamically capturing long-range dependencies and global semantic information for complex spatial contexts. A boundary-wise loss function is employed as the segmentation loss of the proposed MMTSeg to minimize boundary discrepancies for a distance-based metric. Our ablation study demonstrates the importance of the proposed feature enhancement and fusion modules in the proposed network and the Transformer with Mamba block for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS 2018 and BraTS 2020 datasets demonstrate that the proposed MMTSeg framework outperforms state-of-the-art methods when modalities are missing.

Multimodal MRI provides complementary information for brain tumor segmentation, but high accuracy alone does not reveal whether a model relies on clinically appropriate imaging evidence. Radiologists use different sequences for different findings, such as T1ce for enhancing tumor and FLAIR for peritumoral edema. Yet most 3D segmentation networks fuse MRI sequences through fixed connections, regardless of patient, region, or depth. We frame multimodal segmentation as grounded multimodal reasoning and introduce (i) a model agnostic probing framework combining per region modality ablation, Dominant Sensitivity Enhancement (DSE), and gate trajectory analysis to reveal sequence reliance, and (ii) evidence that adaptive Hyper-Connections (HC) induce clinically aligned modality reliance without modality supervision. Across nnU Net, SwinUNETR, and VT-UNet on BraTS 2021, most HC configurations increase reliance on clinically appropriate sequences. This interpretable pattern recurs across all three backbones, suggesting it arises from adaptive connections rather than a single architecture. HC also improves segmentation accuracy at negligible parameter cost. On validation, the best HC configuration increases mean Dice by 0.43 to 0.81 percentage points across the three backbones. On the held-out test set, it reduces mean HD95 by 1.17 to 1.78 mm (approximately 20 to 27 percent) for nnU-Net and VT UNet, adding at most 0.53 percent parameters. Our framework offers an inspectable measure of clinical grounding and extends naturally to multimodal medical foundation models.