Meta-D: Metadata-Aware Architectures for Brain Tumor Analysis and Missing-Modality Segmentation

SangHyuk Kim; Daniel Haehn; Sumientra Rampersad

cs.CVMar 5, 2026

Meta-D: Metadata-Aware Architectures for Brain Tumor Analysis and Missing-Modality Segmentation

Authors: SangHyuk Kim, Daniel Haehn, Sumientra Rampersad

Organizations: University of Massachusetts Boston

Abstract

We present Meta-D, an architecture that explicitly leverages categorical scanner metadata such as MRI sequence and plane orientation to guide feature extraction for brain tumor analysis. We aim to improve the performance of medical image deep learning pipelines by integrating explicit metadata to stabilize feature representations. We first evaluate this in 2D tumor detection, where injecting sequence (e.g., T1, T2) and plane (e.g., axial) metadata dynamically modulates convolutional features, yielding an absolute increase of up to 2.62% in F1-score over image-only baselines. Because metadata grounds feature extraction when data are available, we hypothesize it can serve as a robust anchor when data are missing. We apply this to 3D missing-modality tumor segmentation. Our Transformer Maximizer utilizes metadata-based cross-attention to isolate and route available modalities, ensuring the network focuses on valid slices. This targeted attention improves brain tumor segmentation Dice scores by up to 5.12% under extreme modality scarcity while reducing model parameters by 24.1%.

Figures & tables

Figure 1: An overview of the Meta-D ( $T_{max}$ ) architecture. By querying a fixed 4-modality metadata dictionary instead of $N$ spatial patches, the module handles missing sequences via deterministic masking and reduces attention complexity from $O(N^{2})$ to $O(N)$ .

Dataset	bias-	2D Image	Meta-D (Ours)
	corr.	only	Img + Seq	Img + Plane	Img + Seq + Plane
BraTS 2020	✓	$0.9037\pm 0.0033$	$0.9129\pm 0.0015$	$0.9108\pm 0.0255$	0.9138 $\pm 0.0089$
		$0.9055\pm 0.0209$	$0.9030\pm 0.0130$	$0.9070\pm 0.0028$	0.9099 $\pm 0.0134$
BRISC	✓	$0.6890\pm 0.0226$	$0.6860\pm 0.0622$	$0.6955\pm 0.0118$	0.6989 $\pm 0.0199$
		$0.6896\pm 0.0670$	$0.6676\pm 0.0260$	$0.6321\pm 0.0874$	0.7016 $\pm 0.0483$
BRISC (skull-stripped)	✓	$0.6986\pm 0.0393$	$0.7091\pm 0.0988$	$0.7116\pm 0.0479$	0.7248 $\pm 0.0163$

Table 1: 2D Tumor Detection F1-Score Performance. Models were trained on balanced 5500 slice subsets. Values are presented as mean

\pm

std. on three-fold cross validation. Best results per scenario are bolded, showing that Meta-D with both sequence and plane metadata consistently outperforms all other models.

Figure 2: Grad-CAM visualizations comparing the image-only baseline with our Meta-D architecture. The explicit integration of metadata successfully shifts the spatial attention pathways directly to the tumor margins, resolving contrast ambiguity.

Explore similar work

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities
Apr 22, 2024Ming Kang, Fung Fung Ting, Shier Nee Saw +3Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation
School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia +1
D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities
May 21, 2026Danish Ali, Ajmal Mian, Naveed Akhtar +1Missing-Modality Learning BraTS
The University of Western Australia, Perth, WA 6009, Australia · The University of Melbourne, Melbourne, Parkville VIC 3010, Australia
Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
Apr 24, 2026Peibo Song, Xiaotian Xue, Jinshuo Zhang +5Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures
School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab +2

Apr 22, 2024J/K move · Enter open · S save

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

Ming Kang, Fung Fung Ting, Shier Nee Saw, Raphaël C. -W. Phan+2

School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia+1

Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation Medical Image Segmentation Multimodal Fusion Brain Tumor Segmentation

Open paper AskPDF arXiv

Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are often missing due to resource constraints, resulting in significant performance degradation for methods that rely on complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Mamba-Transformer hybrid network (MMTSeg) for accurate brain tumor segmentation with missing modalities. We first employ a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodalities to extract complete modality information. We further develop an Unimodal Feature Enhancement (UFE) module to model the semantic relationship between global and local information. Finally, we built a Cross-Modal Fusion (CMF) module to explicitly align the global correlations across modalities, even when some modalities are missing. Complementary features within and across modalities are refined by the Mamba-Transformer hybrid architectures in both the UFE and CMF modules, dynamically capturing long-range dependencies and global semantic information for complex spatial contexts. A boundary-wise loss function is employed as the segmentation loss of the proposed MMTSeg to minimize boundary discrepancies for a distance-based metric. Our ablation study demonstrates the importance of the proposed feature enhancement and fusion modules in the proposed network and the Transformer with Mamba block for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS 2018 and BraTS 2020 datasets demonstrate that the proposed MMTSeg framework outperforms state-of-the-art methods when modalities are missing.

Abstract

Explore similar work

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities