Brain Tumor Segmentation

Follow this topic Ask about this topic

Latest papers 76

14 papers in the last four weeks, up 56% on the four weeks before. 0.1% of all new papers.

Mar 24, 2026

An Explainable AI-Driven Framework for Automated Brain Tumor Segmentation Using an Attention-Enhanced U-Net

Computer-aided segmentation of brain tumors from MRI data is of crucial significance to clinical decision-making in diagnosis, treatment planning, and follow-up disease monitoring. Gliomas, owing to their high malignancy and heterogeneity, represent a very challenging task for accurate and reliable segmentation into intra-tumoral sub-regions. Manual segmentation is typically time-consuming and not reliable, which justifies the need for robust automated techniques.This research resolves this problem by leveraging the BraTS 2020 dataset, where we have labeled MRI scans of glioma patients with four significant classes: background/healthy tissue, necrotic/non-enhancing core, edema, and enhancing tumor. In this work, we present a new segmentation technique based on a U-Net model augmented with executed attention gates to focus on the most significant regions of images. To counter class imbalance, we employ manually designed loss functions like Dice Loss and Categorical Dice Loss, in conjunction with standard categorical cross-entropy. Other evaluation metrics, like sensitivity and specificity, were used to measure discriminability of the model between tumor classes. Besides, we introduce Grad-CAM-based explainable AI to enable visualizing attention regions and improve model interpretability, together with a smooth heatmap generation technique through Gaussian filtering. Our approach achieved superior performance with accuracy of 0.9919, Dice coefficient of 0.9901, mean IoU of 0.9873, sensitivity of 0.9908, and specificity of 0.9974. This study demonstrates that the use of attention mechanisms, personalized loss functions, and explainable AI significantly improves highly complex tumor structure segmentation precision in MRI scans, providing a reliable and explainable method for clinical applications.

MD Rashidul Islam, Bakary Gibba

School of Computing and Informatics Albukhary International University, 05200, Alor Setar, Malaysia

Brain Tumor Segmentation Convolutional Neural Networks BraTS Explainable Medical Image Analysis

Mar 23, 2026

PGR-Net: Prior-Guided ROI Reasoning Network for Brain Tumor MRI Segmentation

Brain tumor MRI segmentation is essential for clinical diagnosis and treatment planning, enabling accurate lesion detection and radiotherapy target delineation. However, tumor lesions occupy only a small fraction of the volumetric space, resulting in severe spatial sparsity, while existing segmentation networks often overlook clinically observed spatial priors of tumor occurrence, leading to redundant feature computation over extensive background regions. To address this issue, we propose PGR-Net (Prior-Guided ROI Reasoning Network) - an explicit ROI-aware framework that incorporates a data-driven spatial prior set to capture the distribution and scale characteristics of tumor lesions, providing global guidance for more stable segmentation. Leveraging these priors, PGR-Net introduces a hierarchical Top-K ROI decision mechanism that progressively selects the most confident lesion candidate regions across encoder layers to improve localization precision. We further develop the WinGS-ROI (Windowed Gaussian-Spatial Decay ROI) module, which uses multi-window Gaussian templates with a spatial decay function to produce center-enhanced guidance maps, thus directing feature learning throughout the network. With these ROI features, a windowed RetNet backbone is adopted to enhance localization reliability. Experiments on BraTS-2019/2023 and MSD Task01 show that PGR-Net consistently outperforms existing approaches while using only 8.64M Params, achieving Dice scores of 89.02%, 91.82%, and 89.67% on the Whole Tumor region. Code is available at https://github.com/CNU-MedAI-Lab/PGR-Net.

Jiacheng Lu, Hui Ding, Shiyu Zhang +1

College of Information Engineering, Capital Normal University, Beijing, China · School of Artificial Intelligence, China University of Mining and Technology-Beijing, Beijing, China

Image Segmentation Brain Tumor Segmentation 3D Medical Image Segmentation BraTS Medical Image Localization

Mar 22, 2026

DGRNet: Disagreement-Guided Refinement for Uncertainty-Aware Brain Tumor Segmentation

Accurate brain tumor segmentation from MRI scans is critical for diagnosis and treatment planning. Despite the strong performance of recent deep learning approaches, two fundamental limitations remain: (1) the lack of reliable uncertainty quantification in single-model predictions, which is essential for clinical deployment because the level of uncertainty may impact treatment decision-making, and (2) the under-utilization of rich information in radiology reports that can guide segmentation in ambiguous regions. In this paper, we propose the Disagreement-Guided Refinement Network (DGRNet), a novel framework that addresses both limitations through multi-view disagreement-based uncertainty estimation and text-conditioned refinement. DGRNet generates diverse predictions via four lightweight view-specific adapters attached to a shared encoder-decoder, enabling efficient uncertainty quantification within a single forward pass. Afterward, we build disagreement maps to identify regions of high segmentation uncertainty, which are then selectively refined according to clinical reports. Moreover, we introduce a diversity-preserving training strategy that combines pairwise similarity penalties and gradient isolation to prevent view collapse. The experimental results on the TextBraTS dataset show that DGRNet favorably improves state-of-the-art segmentation accuracy by 2.4% and 11% in main metrics Dice and HD95, respectively, while providing meaningful uncertainty estimates.

Bahram Mohammadi, Yanqiu Wu, Vu Minh Hieu Phan +6

Macquarie University, Sydney, NSW, Australia · Adelaide University, Adelaide, SA, Australia · Flinders University, Adelaide, SA, Australia +1

Medical Image Segmentation Brain Tumor Segmentation Segmentation Uncertainty Estimation Text-Guided Image Segmentation

Mar 22, 2026

Hierarchical Text-Guided Brain Tumor Segmentation via Sub-Region-Aware Prompts

Brain tumor segmentation remains challenging because the three standard sub-regions, i.e., whole tumor (WT), tumor core (TC), and enhancing tumor (ET), often exhibit ambiguous visual boundaries. Integrating radiological description texts with imaging has shown promise. However, most multimodal approaches typically compress a report into a single global text embedding shared across all sub-regions, overlooking their distinct clinical characteristics. We propose TextCSP (text-modulated soft cascade architecture), a hierarchical text-guided framework that builds on the TextBraTS baseline with three novel components: (1) a text-modulated soft cascade decoder that predicts WT->TC->ET in a coarse-to-fine manner consistent with their anatomical containment hierarchy. (2) sub-region-aware prompt tuning, which uses learnable soft prompts with a LoRA-adapted BioBERT encoder to generate specialized text representations tailored for each sub-region; (3) text-semantic channel modulators that convert the aforementioned representations into channel-wise refinement signals, enabling the decoder to emphasize features aligned with clinically described patterns. Experiments on the TextBraTS dataset demonstrate consistent improvements across all sub-regions against state-of-the-art methods by 1.7% and 6% on the main metrics Dice and HD95.

Bahram Mohammadi, Ta Duc Huy, Afrouz Sheikholeslami +8

Macquarie University, Sydney, NSW, Australia · Adelaide University, Adelaide, SA, Australia · Flinders University, Adelaide, SA, Australia +1

Medical Image Segmentation Brain Tumor Segmentation Text-Guided Image Segmentation BraTS Prompt Tuning

Mar 20, 2026

Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

Multimodal MRI provides complementary information for brain tumor segmentation, but high accuracy alone does not reveal whether a model relies on clinically appropriate imaging evidence. Radiologists use different sequences for different findings, such as T1ce for enhancing tumor and FLAIR for peritumoral edema. Yet most 3D segmentation networks fuse MRI sequences through fixed connections, regardless of patient, region, or depth. We frame multimodal segmentation as grounded multimodal reasoning and introduce (i) a model agnostic probing framework combining per region modality ablation, Dominant Sensitivity Enhancement (DSE), and gate trajectory analysis to reveal sequence reliance, and (ii) evidence that adaptive Hyper-Connections (HC) induce clinically aligned modality reliance without modality supervision. Across nnU Net, SwinUNETR, and VT-UNet on BraTS 2021, most HC configurations increase reliance on clinically appropriate sequences. This interpretable pattern recurs across all three backbones, suggesting it arises from adaptive connections rather than a single architecture. HC also improves segmentation accuracy at negligible parameter cost. On validation, the best HC configuration increases mean Dice by 0.43 to 0.81 percentage points across the three backbones. On the held-out test set, it reduces mean HD95 by 1.17 to 1.78 mm (approximately 20 to 27 percent) for nnU-Net and VT UNet, adding at most 0.53 percent parameters. Our framework offers an inspectable measure of clinical grounding and extends naturally to multimodal medical foundation models.

Lokendra Kumar, Shubham Aggarwal

3D Medical Image Segmentation Hyper-Connections Brain Tumor Segmentation Multimodal Fusion BraTS

Mar 18, 2026

LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation

Precise localization and delineation of brain tumors using magnetic resonance imaging (MRI) are essential for planning therapy and guiding surgical decisions. To address this, we propose LoGSAM, a parameter-efficient, detection-driven framework that transforms radiologist dictation into text prompts for foundation-model-based localization and segmentation. Radiologist speech is first transcribed and translated using a pretrained Whisper ASR model, followed by negation-aware clinical NLP to extract tumor-specific textual prompts. These prompts guide text-conditioned tumor localization via a LoRA-adapted vision-language detection model, Grounding DINO (GDINO). The predicted bounding boxes are used as prompts for MedSAM to generate pixel-level tumor masks without any additional fine-tuning. On BRISC 2025, LoGSAM attains a Dice score of 80.32%, reaching 98.6% of a fully fine-tuned GDINO + MedSAM baseline while training fewer than 5% of its parameters, indicating a favorable accuracy/parameter trade-off. In addition, we evaluate the full pipeline using German dictations from a board-certified radiologist on unseen MRI scans, achieving 91.7% case-level class-extraction accuracy. These results highlight the feasibility of constructing a modular speech-to-segmentation pipeline from pretrained foundation models with minimal parameter updates.

Mohammad Robaitul Islam Bhuiyan, Sheethal Bhat, Melika Qahqaie +4

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Siemens Healthineers, Erlangen, Germany

Image Segmentation Medical Imaging Foundation Models Text-Guided Image Segmentation Medical Image Segmentation Medical Image Grounding Brain Tumor Segmentation

Mar 6, 2026

Ensemble Learning with Sparse Hypercolumns

Directly inspired by findings in biological vision, high-dimensional hypercolumns are feature vectors built by concatenating multi-scale activations of convolutional neural networks for a single image pixel location. Together with powerful classifiers, they can be used for image segmentation i.e. pixel classification. However, in practice, there are only very few works dedicated to the use of hypercolumns. One reason is the computational complexity of processing concatenated dense hypercolumns that grows linearly with the size

N

of the training set. In this work, we address this challenge by applying stratified subsampling to the VGG16 based hypercolumns. Furthermore, we investigate the performance of ensemble learning on sparse hypercolumns. Our experiments on a brain tumor dataset show that stacking and voting ensembles deliver competitive performance, but in the extreme low-shot case of

N \leq 20

, a simple Logistic Regression classifier is the most effective method. For 10% stratified subsampling rate, our best average Dice score is 0.66 for

N=20

. This is a statistically significant improvement of 24.53% over the standard multi-scale UNet baseline (

p

-value =

[3.07e-11]

, Wilcoxon signed-rank test), which is less effective due to overfitting.

Julia Dietlmeier, Vayangi Ganepola, Oluwabukola G. Adegboro +3

Insight Research Ireland Centre for Data Analytics, Dublin City University, Glasnevin 9, Dublin, Ireland · Research Ireland Centre for Research Training in Machine Learning (ML-Labs), Dublin City University, Glasnevin 9, Dublin, Ireland · School of Computing, Dublin City University, Glasnevin 9, Dublin, Ireland

Few-Shot Medical Image Segmentation Ensemble Learning Brain Tumor Segmentation

Mar 5, 2026

Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation

Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, some FL participants may possess only a subset of the complete imaging modalities, posing intermodal heterogeneity as a challenge to effectively training a global model on all participants' data. Meanwhile, each participant expects a personalized model tailored to its local data characteristics in FL. This work proposes a new FL framework with federated modality-specific encoders and partially personalized multimodal fusion decoders (FedMEPD) to address the two concurrent issues. Specifically, FedMEPD employs an exclusive encoder for each modality to account for the intermodal heterogeneity. While these encoders are fully federated, the decoders are partially personalized to meet individual needs -- using the discrepancy between global and local parameter updates to dynamically determine which decoder filters are personalized. Implementation-wise, a server with full-modal data employs a fusion decoder to fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation. Moreover, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the model parameters. Conversely, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up for the information loss due to absent modalities. FedMEPD is validated on the BraTS 2018 and 2020 multimodal brain tumor segmentation benchmarks. Results show that it outperforms various up-to-date methods for multimodal and personalized FL, and its novel designs are effective.

Hong Liu, Dong Wei, Qian Dai +3

National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, 361005, Fujian, China · Jarvis Research Center, Tencent YouTu Lab, Shenzhen, 518075, Guangdong, China · Department of Computer Science at School of Informatics, Xiamen University, Xiamen, 361005, Fujian, China +1

Federated Learning Multimodal Federated Learning Heterogeneous Federated Learning Multimodal Fusion Brain Tumor Segmentation

Mar 5, 2026

Meta-D: Metadata-Aware Architectures for Brain Tumor Analysis and Missing-Modality Segmentation

We present Meta-D, an architecture that explicitly leverages categorical scanner metadata such as MRI sequence and plane orientation to guide feature extraction for brain tumor analysis. We aim to improve the performance of medical image deep learning pipelines by integrating explicit metadata to stabilize feature representations. We first evaluate this in 2D tumor detection, where injecting sequence (e.g., T1, T2) and plane (e.g., axial) metadata dynamically modulates convolutional features, yielding an absolute increase of up to 2.62% in F1-score over image-only baselines. Because metadata grounds feature extraction when data are available, we hypothesize it can serve as a robust anchor when data are missing. We apply this to 3D missing-modality tumor segmentation. Our Transformer Maximizer utilizes metadata-based cross-attention to isolate and route available modalities, ensuring the network focuses on valid slices. This targeted attention improves brain tumor segmentation Dice scores by up to 5.12% under extreme modality scarcity while reducing model parameters by 24.1%.

SangHyuk Kim, Daniel Haehn, Sumientra Rampersad

University of Massachusetts Boston

Medical Image Analysis Brain Tumor Segmentation Medical Image Segmentation 3D Medical Image Segmentation

Mar 5, 2026

Comparative Evaluation of Traditional Methods and Deep Learning for Brain Glioma Imaging. Review Paper

Segmentation is crucial for brain gliomas as it delineates the glioma s extent and location, aiding in precise treatment planning and monitoring, thus improving patient outcomes. Accurate segmentation ensures proper identification of the glioma s size and position, transforming images into applicable data for analysis. Classification of brain gliomas is also essential because different types require different treatment approaches. Accurately classifying brain gliomas by size, location, and aggressiveness is essential for personalized prognosis prediction, follow-up care, and monitoring disease progression, ensuring effective diagnosis, treatment, and management. In glioma research, irregular tissues are often observable, but error free and reproducible segmentation is challenging. Many researchers have surveyed brain glioma segmentation, proposing both fully automatic and semi-automatic techniques. The adoption of these methods by radiologists depends on ease of use and supervision, with semi-automatic techniques preferred due to the need for accurate evaluations. This review evaluates effective segmentation and classification techniques post magnetic resonance imaging acquisition, highlighting that convolutional neural network architectures outperform traditional techniques in these tasks.

Kiranmayee Janardhan, Vinay Martin DSa Prabhu, T. Christy Bobby

Department of Electronics and Communications, Ramaiah University of Applied Sciences 4th Phase, Peenya, Bengaluru, Karnataka, India · Department of Radiology, Ramaiah Memorial Hospital and Medical College MSR Nagar, MSRIT Post, Bengaluru, Karnataka, India

Brain Tumor Segmentation Brain Tumor Classification Convolutional Neural Networks

Feb 26, 2026

Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models

Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially -- tumor segmentation (BraTS) and brain age estimation (IXI) -- with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62

\pm

0.07, T2 MAE 0.16

\pm

0.05, with zero forgetting and

<

0.1% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon

p<0.001

). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.

Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen +3

Independent Researcher, USA · Indiana University at Bloomington, United States · Independent Researcher, Taiwan +3

Medical Imaging Foundation Models Brain MRI Brain Tumor Segmentation

Feb 25, 2026

Brain Tumor Segmentation with Special Emphasis on the Non-Enhancing Brain Tumor Compartment

A U-Net based deep learning architecture is designed to segment brain tumors as they appear on various MRI modalities. Special emphasis is lent to the non-enhancing tumor compartment. The latter has not been considered anymore in recent brain tumor segmentation challenges like the MICCAI challenges. However, it is considered to be indicative of the survival time of the patient as well as of areas of further tumor growth. Hence it deems essential to have means to automatically delineate its extension within the tumor.

T. Schaffer, A. Brawanski, S. Wein +2

CIML Group, Biophysics, University of Regensburg, 93040 Regensburg, Germany · Department of Neurosurgery , University Hospital Regensburg, 93042 Regensburg, Germany · Department of Biomedical Imaging, University Hospital Regensburg, 93042 Regensburg, Germany +1

Brain Tumor Segmentation

Feb 24, 2026

Multimodal MRI Report Findings Supervised Brain Lesion Segmentation with Substructures

Report-supervised (RSuper) learning seeks to alleviate the need for dense tumor voxel labels with constraints derived from radiology reports (e.g., volumes, counts, sizes, locations). In MRI studies of brain tumors, however, we often involve multi-parametric scans and substructures. Here, fine-grained modality/parameter-wise reports are usually provided along with global findings and are correlated with different substructures. Moreover, the reports often describe only the largest lesion and provide qualitative or uncertain cues (mild,'' possible''). Classical RSuper losses (e.g., sum volume consistency) can over-constrain or hallucinate unreported findings under such incompleteness, and are unable to utilize these hierarchical findings or exploit the priors of varied lesion types in a merged dataset. We explicitly parse the global quantitative and modality-wise qualitative findings and introduce a unified, one-sided, uncertainty-aware formulation (MS-RSuper) that: (i) aligns modality-specific qualitative cues (e.g., T1c enhancement, FLAIR edema) with their corresponding substructures using existence and absence losses; (ii) enforces one-sided lower-bounds for partial quantitative cues (e.g., largest lesion size, minimal multiplicity); and (iii) adds extra- vs. intra-axial anatomical priors to respect cohort differences. Certainty tokens scale penalties; missing cues are down-weighted. On 1238 report-labeled BraTS-MET/MEN scans, our MS-RSuper largely outperforms both a sparsely-supervised baseline and a naive RSuper method.

Yubin Ge, Yongsong Huang, Xiaofeng Liu

Amazon AWS · Yale University · Tohoku University

Medical Image Segmentation Brain Tumor Segmentation Weakly Supervised Image Segmentation Brain MRI

Feb 23, 2026

HD-TTA: Hypothesis-Driven Test-Time Adaptation for Safer Brain Tumor Segmentation

Standard Test-Time Adaptation (TTA) methods typically treat inference as a blind optimization task, applying generic objectives to all or filtered test samples. In safety-critical medical segmentation, this lack of selectivity often causes the tumor mask to spill into healthy brain tissue or degrades predictions that were already correct. We propose Hypothesis-Driven TTA, a novel framework that reformulates adaptation as a dynamic decision process. Rather than forcing a single optimization trajectory, our method generates intuitive competing geometric hypotheses: compaction (is the prediction noisy? trim artifacts) versus inflation (is the valid tumor under-segmented? safely inflate to recover). It then employs a representation-guided selector to autonomously identify the safest outcome based on intrinsic texture consistency. Additionally, a pre-screening Gatekeeper prevents negative transfer by skipping adaptation on confident cases. We validate this proof-of-concept on a cross-domain binary brain tumor segmentation task, applying a source model trained on adult BraTS gliomas to unseen pediatric and more challenging meningioma target domains. HD-TTA improves safety-oriented outcomes (Hausdorff Distance (HD95) and Precision) over several state-of-the-art representative baselines in the challenging safety regime, reducing the HD95 by approximately 6.4 mm and improving Precision by over 4%, while maintaining comparable Dice scores. These results demonstrate that resolving the safety-adaptation trade-off via explicit hypothesis selection is a viable, robust path for safe clinical model deployment. Code will be made publicly available upon acceptance.

Kartik Jhawar, Lipo Wang

Test-Time Adaptation Medical Image Segmentation Brain Tumor Segmentation

Dec 27, 2025

ReFRM3D: A Radiomics-enhanced Fused Residual Multiparametric 3D Network with Multi-Scale Feature Fusion for Glioma Characterization

Gliomas are among the most aggressive cancers, with complex diagnostic processes. Existing glioma segmentation methods often struggle with high variability in imaging data and inadequate optimization. Furthermore, radiomic analysis is typically applied only after segmentation is finished, limiting its ability to inform the segmentation process itself. To address these challenges, we propose a novel radiomics-enhanced fused residual multiparametric 3D network (ReFRM3D) for brain tumor characterization. The framework is based on a 3D U-Net architecture and features multi-scale feature fusion, hybrid upsampling, and an extended residual skip mechanism. Additionally, we introduce a radiomic conditioning mechanism that extracts texture and intensity descriptors from an intermediate coarse segmentation and re-injects them into the decoder to refine the final output. Experimental results on BraTS2019, BraTS2020, and BraTS2021 show strong performance, with mean DSC values of 93.45%, 93.61%, and 92.06%, respectively, across whole tumor, enhancing tumor, and tumor core regions. Compared with recent models, ReFRM3D improves average DSC by 5.79%, 7.25%, and 0.96% on these datasets. Our model also generalized well on the BraTS-Africa dataset with an average DSC of 87.8%, despite differences in scanner field strength and patient demographics.

Md. Abdur Rahman, Md Noman Hossain, Arefin Ittesafun Abian +4

Applied Artificial Intelligence and Intelligent Systems (AAIINS) Laboratory, Dhaka 1217, Bangladesh · Department of Computer Science and Engineering, United International University, Dhaka, 1212, Bangladesh · Department of Data Science and Artificial Intelligence, Monash University, Clayton, VIC, 3153, Australia +1

BraTS Medical Image Segmentation 3D Medical Image Segmentation Radiomics Brain Tumor Segmentation

Apr 22, 2024

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are often missing due to resource constraints, resulting in significant performance degradation for methods that rely on complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Mamba-Transformer hybrid network (MMTSeg) for accurate brain tumor segmentation with missing modalities. We first employ a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodalities to extract complete modality information. We further develop an Unimodal Feature Enhancement (UFE) module to model the semantic relationship between global and local information. Finally, we built a Cross-Modal Fusion (CMF) module to explicitly align the global correlations across modalities, even when some modalities are missing. Complementary features within and across modalities are refined by the Mamba-Transformer hybrid architectures in both the UFE and CMF modules, dynamically capturing long-range dependencies and global semantic information for complex spatial contexts. A boundary-wise loss function is employed as the segmentation loss of the proposed MMTSeg to minimize boundary discrepancies for a distance-based metric. Our ablation study demonstrates the importance of the proposed feature enhancement and fusion modules in the proposed network and the Transformer with Mamba block for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS 2018 and BraTS 2020 datasets demonstrate that the proposed MMTSeg framework outperforms state-of-the-art methods when modalities are missing.

Ming Kang, Fung Fung Ting, Shier Nee Saw +3

School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia +1

Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation Medical Image Segmentation Multimodal Fusion Brain Tumor Segmentation