Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

Lokendra Kumar; Shubham Aggarwal

cs.CVMar 20, 2026

Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

Authors: Lokendra Kumar, Shubham Aggarwal

Abstract

Multimodal MRI provides complementary information for brain tumor segmentation, but high accuracy alone does not reveal whether a model relies on clinically appropriate imaging evidence. Radiologists use different sequences for different findings, such as T1ce for enhancing tumor and FLAIR for peritumoral edema. Yet most 3D segmentation networks fuse MRI sequences through fixed connections, regardless of patient, region, or depth. We frame multimodal segmentation as grounded multimodal reasoning and introduce (i) a model agnostic probing framework combining per region modality ablation, Dominant Sensitivity Enhancement (DSE), and gate trajectory analysis to reveal sequence reliance, and (ii) evidence that adaptive Hyper-Connections (HC) induce clinically aligned modality reliance without modality supervision. Across nnU Net, SwinUNETR, and VT-UNet on BraTS 2021, most HC configurations increase reliance on clinically appropriate sequences. This interpretable pattern recurs across all three backbones, suggesting it arises from adaptive connections rather than a single architecture. HC also improves segmentation accuracy at negligible parameter cost. On validation, the best HC configuration increases mean Dice by 0.43 to 0.81 percentage points across the three backbones. On the held-out test set, it reduces mean HD95 by 1.17 to 1.78 mm (approximately 20 to 27 percent) for nnU-Net and VT UNet, adding at most 0.53 percent parameters. Our framework offers an inspectable measure of clinical grounding and extends naturally to multimodal medical foundation models.

Figures & tables

Model	Dice ${}_{\text{TC}}$	Dice ${}_{\text{WT}}$	Dice ${}_{\text{ET}}$	Mean Dice
SwinUNETR	$89.62\pm 0.34$	$92.51\pm 0.29$	$88.71\pm 0.24$	$90.28\pm 0.11$
SwinUNETR-DHC $\times$ 2	$90.76\pm 0.26$	$93.12\pm 0.13$	$\mathbf{89.40\pm 0.20}$	$91.05\pm 0.20$
SwinUNETR-DHC $\times$ 4	$\mathbf{90.90\pm 0.47}$	$\mathbf{93.16\pm 0.03}$	$89.37\pm 0.28$	$\mathbf{91.09\pm 0.19}$
SwinUNETR-SHC $\times$ 2	$90.57\pm 0.23$	$93.13\pm 0.11$	$89.35\pm 0.08$	$90.99\pm 0.10$
SwinUNETR-SHC $\times$ 4	$90.77\pm 0.25$	$93.09\pm 0.08$	$89.39\pm 0.06$	$91.03\pm 0.07$
nnU-Net	$89.91\pm 0.23$	$92.10\pm 0.04$	$86.22\pm 0.15$	$89.41\pm 0.10$

Table 1: BraTS 2021 validation set (251 scans). Dice scores are reported as mean

\pm

standard deviation across three random seeds. The best HC configuration improves mean Dice relative to the corresponding baseline for all three backbones.

Model	Dice ${}_{\text{TC}}$	Dice ${}_{\text{WT}}$	Dice ${}_{\text{ET}}$	Mean Dice	HD95 ${}_{\text{TC}}$	HD95 ${}_{\text{WT}}$	HD95 ${}_{\text{ET}}$	Mean HD95
SwinUNETR	$90.79\pm 0.32$	$92.69\pm 0.27$	$87.30\pm 0.25$	$90.26\pm 0.13$	$4.12\pm 0.28$	$5.31\pm 0.42$	$3.89\pm 0.31$	$4.44\pm 0.23$
SwinUNETR-DHC $\times$ 2	$\mathbf{91.23\pm 0.24}$	$92.81\pm 0.16$	$\mathbf{87.36\pm 0.19}$	$\mathbf{90.47\pm 0.16}$	$\mathbf{3.91\pm 0.22}$	$6.15\pm 0.51$	$3.69\pm 0.26$	$4.58\pm 0.27$
SwinUNETR-DHC $\times$ 4	$90.84\pm 0.41$	$\mathbf{92.91\pm 0.09}$	$87.19\pm 0.23$	$90.31\pm 0.18$	$4.01\pm 0.25$	$5.94\pm 0.47$	$\mathbf{3.51\pm 0.24}$	$4.49\pm 0.25$
SwinUNETR-SHC $\times$ 2	$90.63\pm 0.26$	$92.71\pm 0.14$	$87.03\pm 0.12$	$90.12\pm 0.12$	$4.18\pm 0.31$	$6.29\pm 0.56$	$4.19\pm 0.37$	$4.89\pm 0.31$
SwinUNETR-SHC $\times$ 4	$91.04\pm 0.27$	$92.86\pm 0.11$	$87.24\pm 0.10$	$90.38\pm 0.12$	$4.09\pm 0.29$	$\mathbf{5.29\pm 0.39}$	$3.86\pm 0.28$	$\mathbf{4.41\pm 0.22}$
nnU-Net	$89.53\pm 0.25$	$91.48\pm 0.13$	$85.91\pm 0.21$	$88.97\pm 0.14$	$6.64\pm 0.62$	$7.82\pm 0.71$	$5.31\pm 0.48$	$6.59\pm 0.43$

Table 2: BraTS 2021 held-out test set (200 scans). Dice (%, higher is better) and HD95 (mm, lower is better), reported as mean

\pm

standard deviation over three seeds (42, 123, 456). Bold marks the best value per backbone.

Model	Region	$\Delta_{\text{dom}}$	$\Delta_{\text{dom}}$	DSE	$\Delta_{\text{dom}}$	DSE	$\Delta_{\text{dom}}$	DSE	$\Delta_{\text{dom}}$	DSE
		Baseline	SHC $\times$ 2		SHC $\times$ 4		DHC $\times$ 2		DHC $\times$ 4
nnU-Net	TC	0.43	0.50	$+0.07$	0.39	$-0.04$	0.61	$+0.18$	0.43	$+0.00$
	WT	0.31	0.46	$+0.15$	0.35	$+0.04$	0.35	$+0.04$	0.40	$+0.09$
	ET	0.80	0.75	$-0.05$	0.68	$-0.12$	0.84	$+0.04$	0.81	$+0.01$
SwinUNETR	TC	0.50	0.49	$-0.00$	0.51	$+0.01$	0.50	$+0.01$	0.46	$-0.04$
	WT	0.10	0.12	$+0.02$	0.10	$+0.01$	0.13	$+0.03$	0.12	$+0.02$

Table 3: Dominant-modality Dice drop

\Delta_{\text{dom}}

and Dominant Sensitivity Enhancement (DSE) for all backbone–variant pairs. Dominant sequence: T1ce for TC and ET; FLAIR for WT. DSE is relative to the baseline (Eq. 3 ). Positive DSE (bold) indicates increased reliance on the clinically dominant sequence relative to baseline. Values near zero denote no meaningful change.

Method	HC Params (M)	Total Params (M)	$\Delta$ Rate (%)
SwinUNETR-Base	–	21.3406	–
SwinUNETR-DHC $\times$ 2	0.0174	21.3580	$+0.08172$
SwinUNETR-DHC $\times$ 4	0.0235	21.3640	$+0.10991$
SwinUNETR-SHC $\times$ 2	0.000128	21.3407	$+0.00060$
SwinUNETR-SHC $\times$ 4	0.000384	21.3410	$+0.00180$
nnU-Net-Base	–	30.3987	–

Table 4: Parameters added by HC (millions).

\Delta

Rate (%) is the increase over the matching baseline. Any change in reasoning behavior cannot be attributed to capacity: DHC adds

\leq 0.53\%

and SHC

<0.002\%

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Modality ablation (nnU-Net). Dice drop when each MRI sequence is removed at inference, shown per configuration (rows: Baseline, SHC $\times$ 2/ $\times$ 4, DHC $\times$ 2/ $\times$ 4) and tumor region (columns: TC, WT, ET). The dashed line is the mean drop $\bar{\Delta}$ . HC variants increase sensitivity to T1ce (TC/ET) and FLAIR (WT) relative to the baseline.

Panel of Figure 2 — Figure 2: Modality ablation (SwinUNETR). As in Figure 1 , for SwinUNETR. HC variants increase sensitivity to T1ce (TC/ET) and FLAIR (WT).

Panel of Figure 3 — Figure 3: Modality ablation (VT-UNet). As in Figure 1 , for VT-UNet; the same T1ce (TC/ET) and FLAIR (WT) pattern holds, confirming the effect on transformer backbones.

Figure 4: Qualitative results for nnU-Net. Rows: Ground Truth, Baseline, DHC $\times$ 2, DHC $\times$ 4, SHC $\times$ 2, SHC $\times$ 4. Columns: TC (red), WT (blue), ET (green). Dice scores per configuration are shown on the right. HC-based models match ground-truth boundaries more closely, particularly for ET.

Figure 5: Qualitative results for SwinUNETR.

Figure 6: Qualitative results for VT-UNet.

Panel of Figure 7 — Figure 7: Beta evolution atlas nnUNet3D. Top to bottom: DHC- $\times$ 2, DHC- $\times$ 4, SHC- $\times$ 2, SHC- $\times$ 4. A rising $\beta$ indicates increasing reliance on hybrid connections, while a flatter or decreasing trend suggests dominance of skip pathways.

Panel of Figure 8 — Figure 8: Beta evolution atlas SwinUNETR3D. Top to bottom: DHC- $\times$ 2, DHC- $\times$ 4, SHC- $\times$ 2, SHC- $\times$ 4. A rising $\beta$ indicates increasing reliance on hybrid connections, while a flatter or decreasing trend suggests dominance of skip pathways.

Panel of Figure 9 — Figure 9: Beta evolution atlas VTUNet3D. Top to bottom: DHC- $\times$ 2, DHC- $\times$ 4, SHC- $\times$ 2, SHC- $\times$ 4. A rising $\beta$ indicates increasing reliance on hybrid connections, while a flatter or decreasing trend suggests dominance of skip pathways.

Explore similar work

Apr 24, 2026

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME

Peibo Song, Xiaotian Xue, Jinshuo Zhang +5

School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab +2

Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures Medical Image Segmentation Medical Image Representation Learning Brain Tumor Segmentation

Ask PDF

Oct 13, 2026

Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

Different MRI modalities reveal different parts of a brain tumor. Segmentation models combine information from these modalities to outline tumor regions. Many models use a separate encoder for each modality. However, exchanging features among these encoders at several stages can increase model complexity. We introduce the Block-Wise Shared Adapter (BWSA), which combines compressed features from the modality encoders in a shared path and returns adapted features to each stream. In its fixed-blending form, BWSA adds the adapted features to the original features. Its trainable-gated variant (TG-BWSA) instead learns separate weights for the original and adapted features at each modality and encoder stage. We evaluate both variants on BraTS 2020 and BraTS 2023 with four segmentation backbones. On the official BraTS 2023 online validation set, BWSA increases average region-wise Dice by 1.3--7.6 percentage points relative to the corresponding unmodified backbones. TG-BWSA increases it by 2.8--7.7 points. Both variants add approximately 0.85--3.73% parameters and reduce average HD95 relative to the unmodified backbones.

Milad Rostamian, Mohammadreza Mohammadi, Ender Konukoglu +1

School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran · Computer Vision Laboratory, ETH Zurich, 8092 Zurich, Switzerland

Brain Tumor Segmentation Cross-Modality Medical Image Segmentation Multimodal Fusion Parameter-Efficient Fine-Tuning BraTS

Ask PDF

Sep 6, 2026

LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities

Brain tumor segmentation from multimodal MRI relies on complementary evidence across four imaging sequences, yet one or more modalities may be unavailable because of acquisition cost, protocol variation, scan failure, or patient condition. Existing work has explored reconstruction, knowledge transfer, and direct feature fusion, but leaves open whether missing-modality fusion should change with representation level. High-resolution lateral features retain spatial detail, whereas compressed bottleneck features encode semantic and inter-modality context. We therefore hypothesize that fusion should be conditioned jointly on modality availability and feature hierarchy. We propose the Level-Aware Availability-Conditioned Spatial-Semantic Fusion Network (LASSNet), which contains two level-specialized modules. Hierarchical Availability-Conditioned Fusion (HACF) constructs four lateral representations using count-normalized aggregation of available modalities, mask-conditioned channel modulation, and local 3D refinement. Tri-Scale Relational-Spatial Fusion (TriRSF) models relations among available modality descriptors and spatial context across multiple bottleneck resolutions, followed by cross-scale aggregation and availability-conditioned global spatial attention. A shared coarse-to-fine decoder starts from TriRSF semantics and progressively injects HACF features, without reconstructing missing inputs. Across all 15 non-empty modality configurations, LASSNet obtains mean Dice scores of 76.7% and 83.2% over WT, TC, and ET on BraTS2019 and BraTS2023, respectively.

Haobin Chen, Ao Chang, Rundong Wang +3

School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · College of Information Science and Technology, Donghua University

Medical Image Segmentation Brain Tumor Segmentation 3D Medical Image Segmentation Multimodal Fusion

Ask PDF