Progressive multi-scale tumor-state memory for incomplete multimodal brain tumor segmentation

Milad Rostamian; Mohammadreza Mohammadi; Mohsen Soryani

eess.IVOct 19, 2026

Progressive multi-scale tumor-state memory for incomplete multimodal brain tumor segmentation

Authors: Milad Rostamian, Mohammadreza Mohammadi, Mohsen Soryani

Organizations: School of Computer Engineering, Iran University of Science and Technology (IUST), Hengam Street, Resalat Square, 13114-16846, Tehran, Iran

Abstract

Brain tumor segmentation uses complementary magnetic resonance imaging (MRI) sequences, but some sequences may be unavailable in practice. Their absence changes the information available for segmenting different tumor regions. Fusion methods can support different input subsets. However, fusion at several scales does not by itself specify how tumor evidence is retained and updated to guide fusion at later stages. We propose a progressive multi-scale tumor-state memory network (PMTM) for segmentation from any non-empty subset of T1ce, T1, FLAIR, and T2. PMTM maintains a compact state for the current subject across the encoder stages. At each stage, we use assignment maps to pool spatial features into ordered evidence vectors for each available modality. We average these vectors across the available modalities, and a learned gate combines this evidence with the previous state. The up- dated state guides residual fusion at that stage and is used in the state update at the next stage. A modality-token mixer also models interactions among the available modalities at each stage. We apply semantic supervision to the assignment maps at all four stages. We evaluate PMTM on BraTS2018, BraTS2020, and BraTS2023 GLI across all 15 non-empty modality subsets. Averaged over the three tumor regions and the 15 subsets, PMTM achieves Dice scores of 75.31%, 77.54%, and 80.47% on BraTS2018, BraTS2020, and BraTS2023 GLI, respectively. These scores are higher than those of RobustSeg, RFNet, and mmFormer in our experiments on each dataset. Component ablations and internal state analysis are performed on BraTS2020.

Figures & tables

Figure 1: Overview of PMTM. Four modality-specific encoders produce multiscale features. At each stage, evidence from the available modalities updates the tumor state, which guides fusion with modality-token interaction. The fused features from all four stages are passed to a shared decoder. Auxiliary decoders supervise each modality branch during training and are discarded at inference. The auxiliary skip connections and token-mixer internals are omitted from the diagram.

Figure 2: Progressive tumor-state update at stage

s

. The state head has an assignment branch and an evidence branch (Eq. ( 1 )). Weighted spatial pooling produces four 64-dimensional vectors per modality. The MEAN block averages corresponding vectors over

m\in S

(Eq. ( 2 )). Slot-specific gates combine this evidence with the previous state (Eq. ( 3 )).

Figure 3: State-guided fusion and modality-token interaction. Panel (a) shows the base path, local gate, residual averaging, scaling, and instance normalization (Eqs. ( 4 ) and ( 6 )). Panel (b) shows descriptor projection, masked global interaction among modality tokens, and the token gates (Eq. ( 5 )). Local and token gates are averaged before weighting residual features.

Dataset	Method	Epochs	Crop	WT	TC	ET	Avg.	PMTM gain
BraTS2018	RobustSeg	1000	80	84.55	69.11	51.28	68.32	+6.99
	RFNet	1000	80	85.68	74.86	57.17	72.57	+2.74
	mmFormer	1000	128	87.50	74.92	57.77	73.40	+1.91
	Ours	1000	128	86.79	77.17	61.95	75.31	–
BraTS2020	RobustSeg	1000	80	84.34	73.60	55.60	71.18	+6.36
	RFNet	1000	80	86.80	78.15	57.44	74.13	+3.41

Table 1: Results on the three BraTS cohorts, averaged over all 15 non-empty modality subsets. Dice values are percentages; PMTM gains are percentage-point differences. Crop denotes the side length, in voxels, of the cubic training input. Bold marks the highest score and underlining the second-highest distinct score within each dataset. Ties receive the same marking.

Panel of Table 2 — Table 2: BraTS2018 Dice (%) for all 15 non-empty modality subsets.

Method	Source	WT $\uparrow$	TC $\uparrow$	ET $\uparrow$	Avg. $\uparrow$
MMANet [ Wei et al., 2023 ]	[ Zou et al., 2026 ]	82.64	76.99	56.39	72.01
PASSION [ Shi et al., 2024 ]	[ Zou et al., 2026 ]	84.28	71.37	53.92	69.86
DMAF-Net [ Lan et al., 2025 ]	[ Zou et al., 2026 ]	84.71	71.98	54.30	70.33
IM-Fuse [ Pipoli et al., 2025 ]	[ Zou et al., 2026 ]	87.08	79.01	61.70	75.93
IM-Fuse + MGML [ Zou et al., 2026 ]	[ Zou et al., 2026 ]	87.55	79.36	62.67	76.53
LS 3 M [ Z. Zhang et al., 2025 ]	[ Z. Zhang et al., 2025 ]	88.22	79.76	63.60	77.19

Table 3: BraTS2020 Dice (%) averaged over all 15 non-empty modality subsets. The Source column identifies experiments conducted in this study and published results. Bold marks the highest score and underlining the second-highest distinct score among all methods shown. Ties receive the same marking.

Panel of Table 4 — Table 4: BraTS2020 Dice (%) for all 15 non-empty modality subsets.

Method	Source	WT $\uparrow$	TC $\uparrow$	ET $\uparrow$	Avg. $\uparrow$
IM-Fuse [ Pipoli et al., 2025 ]	[ Zou et al., 2026 ]	89.69	81.74	69.41	80.28
IM-Fuse + MGML [ Zou et al., 2026 ]	[ Zou et al., 2026 ]	90.47	83.01	70.30	81.26
RobustSeg [ Chen et al., 2019 ]	This study	84.69	73.10	60.69	72.82
RFNet [ Ding et al., 2021 ]	This study	85.68	75.66	61.71	74.35
mmFormer [ Y. Zhang et al., 2022 ]	This study	87.45	79.42	66.21	77.69
Ours	This study	87.74	82.72	70.96	80.47

Table 5: BraTS2023 GLI Dice (%) averaged over all 15 non-empty modality subsets. The Source column identifies experiments conducted in this study and published results. Bold marks the highest score and underlining the second-highest distinct score among all methods shown. Ties receive the same marking.

Panel of Table 6 — Table 6: BraTS2023 GLI Dice (%) for all 15 non-empty modality subsets.

Variant	WT	TC	ET	Avg.
w/o tumor-state path	86.74	78.44	59.02	74.73
Local state memory	87.31	80.49	62.08	76.63
w/o semantic assignment supervision	87.42	78.85	60.29	75.52
w/o modality-token interaction	87.58	81.25	63.01	77.28
Ours	87.63	81.53	63.47	77.54

Table 7: Ablation results on BraTS2020, averaged over 100 subjects and all 15 modality subsets. Local state memory removes cross-stage state propagation. Dice values are percentages. Bold marks the highest score and underlining the second-highest distinct score among the variants.

Figure 4: Assignment probabilities at encoder stages 1–4 for a representative BraTS2020 subject under complete input (axial slice $z=61$ ). The FLAIR image and ground-truth segmentation are shown for reference. White contours mark the boundary of the corresponding ground-truth channel. The ordered assignment channels follow BG, NCR/NET, ED, and ET.

Stage	WT	TC	ET	Avg.	Norm. entropy	Mean max probability	Mean cosine similarity between different state slots	Identity match
1	15.22	8.18	7.36	10.25	0.974	0.334	-0.286	–
2	48.72	32.93	37.12	39.59	0.939	0.416	-0.215	1.000
3	55.85	37.79	25.96	39.87	0.883	0.461	-0.049	1.000
4	70.34	64.40	41.50	58.75	0.729	0.648	0.189	1.000

Table 8: Internal state diagnostics on all 50 subjects in the BraTS2020 validation partition under complete input. Assignment Dice values are percentages. For these Dice columns, bold and underlining mark the highest and second-highest distinct scores across stages. Identity matching uses the corresponding stage 1 slot as the reference.

Figure 5: Segmentation maps for three BraTS2020 test subjects selected for illustration under incomplete and complete input. Cases 346, 046, and 127 refer to BraTS20_Training_346, BraTS20_Training_046, and BraTS20_Training_127. The incomplete subsets are FLAIR+T2 for case 346, T1ce+FLAIR for case 046, and T1+FLAIR for case 127. Red, green, and yellow denote NCR/NET, ED, and ET. Dice values below each prediction are reported on a zero-to-one scale in WT/TC/ET order. The highest displayed values within each input setting are shown in bold, including ties.

Missing-Modality Medical Image Segmentation

See all 10 papers

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities
Apr 22, 2024Ming Kang, Fung Fung Ting, Shier Nee Saw +3Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation
School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia +1
LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities
Sep 6, 2026Haobin Chen, Ao Chang, Rundong Wang +3Missing-Modality Medical Image Segmentation Medical Image Segmentation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · College of Information Science and Technology, Donghua University
Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
Apr 24, 2026Peibo Song, Xiaotian Xue, Jinshuo Zhang +5Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures
School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab +2

Apr 22, 2024J/K move · Enter open · S save

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

Ming Kang, Fung Fung Ting, Shier Nee Saw, Raphaël C. -W. Phan+2

School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia+1

Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation Missing-Modality Medical Image Segmentation Medical Image Segmentation Multimodal Fusion Brain Tumor Segmentation

Open paper AskPDF arXiv

Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are often missing due to resource constraints, resulting in significant performance degradation for methods that rely on complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Mamba-Transformer hybrid network (MMTSeg) for accurate brain tumor segmentation with missing modalities. We first employ a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodalities to extract complete modality information. We further develop an Unimodal Feature Enhancement (UFE) module to model the semantic relationship between global and local information. Finally, we built a Cross-Modal Fusion (CMF) module to explicitly align the global correlations across modalities, even when some modalities are missing. Complementary features within and across modalities are refined by the Mamba-Transformer hybrid architectures in both the UFE and CMF modules, dynamically capturing long-range dependencies and global semantic information for complex spatial contexts. A boundary-wise loss function is employed as the segmentation loss of the proposed MMTSeg to minimize boundary discrepancies for a distance-based metric. Our ablation study demonstrates the importance of the proposed feature enhancement and fusion modules in the proposed network and the Transformer with Mamba block for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS 2018 and BraTS 2020 datasets demonstrate that the proposed MMTSeg framework outperforms state-of-the-art methods when modalities are missing.

Abstract

Missing-Modality Medical Image Segmentation

Explore similar work

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities