Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

Lokendra Kumar; Shubham Aggarwal

cs.CVMar 20, 2026

Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

Authors: Lokendra Kumar, Shubham Aggarwal

Abstract

Multimodal MRI provides complementary information for brain tumor segmentation, but high accuracy alone does not reveal whether a model relies on clinically appropriate imaging evidence. Radiologists use different sequences for different findings, such as T1ce for enhancing tumor and FLAIR for peritumoral edema. Yet most 3D segmentation networks fuse MRI sequences through fixed connections, regardless of patient, region, or depth. We frame multimodal segmentation as grounded multimodal reasoning and introduce (i) a model agnostic probing framework combining per region modality ablation, Dominant Sensitivity Enhancement (DSE), and gate trajectory analysis to reveal sequence reliance, and (ii) evidence that adaptive Hyper-Connections (HC) induce clinically aligned modality reliance without modality supervision. Across nnU Net, SwinUNETR, and VT-UNet on BraTS 2021, most HC configurations increase reliance on clinically appropriate sequences. This interpretable pattern recurs across all three backbones, suggesting it arises from adaptive connections rather than a single architecture. HC also improves segmentation accuracy at negligible parameter cost. On validation, the best HC configuration increases mean Dice by 0.43 to 0.81 percentage points across the three backbones. On the held-out test set, it reduces mean HD95 by 1.17 to 1.78 mm (approximately 20 to 27 percent) for nnU-Net and VT UNet, adding at most 0.53 percent parameters. Our framework offers an inspectable measure of clinical grounding and extends naturally to multimodal medical foundation models.

Figures & tables

Model	Dice ${}_{\text{TC}}$	Dice ${}_{\text{WT}}$	Dice ${}_{\text{ET}}$	Mean Dice
SwinUNETR	$89.62\pm 0.34$	$92.51\pm 0.29$	$88.71\pm 0.24$	$90.28\pm 0.11$
SwinUNETR-DHC $\times$ 2	$90.76\pm 0.26$	$93.12\pm 0.13$	$\mathbf{89.40\pm 0.20}$	$91.05\pm 0.20$
SwinUNETR-DHC $\times$ 4	$\mathbf{90.90\pm 0.47}$	$\mathbf{93.16\pm 0.03}$	$89.37\pm 0.28$	$\mathbf{91.09\pm 0.19}$
SwinUNETR-SHC $\times$ 2	$90.57\pm 0.23$	$93.13\pm 0.11$	$89.35\pm 0.08$	$90.99\pm 0.10$
SwinUNETR-SHC $\times$ 4	$90.77\pm 0.25$	$93.09\pm 0.08$	$89.39\pm 0.06$	$91.03\pm 0.07$
nnU-Net	$89.91\pm 0.23$	$92.10\pm 0.04$	$86.22\pm 0.15$	$89.41\pm 0.10$

Table 1: BraTS 2021 validation set (251 scans). Dice scores are reported as mean

\pm

standard deviation across three random seeds. The best HC configuration improves mean Dice relative to the corresponding baseline for all three backbones.

Model	Dice ${}_{\text{TC}}$	Dice ${}_{\text{WT}}$	Dice ${}_{\text{ET}}$	Mean Dice	HD95 ${}_{\text{TC}}$	HD95 ${}_{\text{WT}}$	HD95 ${}_{\text{ET}}$	Mean HD95
SwinUNETR	$90.79\pm 0.32$	$92.69\pm 0.27$	$87.30\pm 0.25$	$90.26\pm 0.13$	$4.12\pm 0.28$	$5.31\pm 0.42$	$3.89\pm 0.31$	$4.44\pm 0.23$
SwinUNETR-DHC $\times$ 2	$\mathbf{91.23\pm 0.24}$	$92.81\pm 0.16$	$\mathbf{87.36\pm 0.19}$	$\mathbf{90.47\pm 0.16}$	$\mathbf{3.91\pm 0.22}$	$6.15\pm 0.51$	$3.69\pm 0.26$	$4.58\pm 0.27$
SwinUNETR-DHC $\times$ 4	$90.84\pm 0.41$	$\mathbf{92.91\pm 0.09}$	$87.19\pm 0.23$	$90.31\pm 0.18$	$4.01\pm 0.25$	$5.94\pm 0.47$	$\mathbf{3.51\pm 0.24}$	$4.49\pm 0.25$
SwinUNETR-SHC $\times$ 2	$90.63\pm 0.26$	$92.71\pm 0.14$	$87.03\pm 0.12$	$90.12\pm 0.12$	$4.18\pm 0.31$	$6.29\pm 0.56$	$4.19\pm 0.37$	$4.89\pm 0.31$
SwinUNETR-SHC $\times$ 4	$91.04\pm 0.27$	$92.86\pm 0.11$	$87.24\pm 0.10$	$90.38\pm 0.12$	$4.09\pm 0.29$	$\mathbf{5.29\pm 0.39}$	$3.86\pm 0.28$	$\mathbf{4.41\pm 0.22}$
nnU-Net	$89.53\pm 0.25$	$91.48\pm 0.13$	$85.91\pm 0.21$	$88.97\pm 0.14$	$6.64\pm 0.62$	$7.82\pm 0.71$	$5.31\pm 0.48$	$6.59\pm 0.43$

Table 2: BraTS 2021 held-out test set (200 scans). Dice (%, higher is better) and HD95 (mm, lower is better), reported as mean

\pm

standard deviation over three seeds (42, 123, 456). Bold marks the best value per backbone.

Model	Region	$\Delta_{\text{dom}}$	$\Delta_{\text{dom}}$	DSE	$\Delta_{\text{dom}}$	DSE	$\Delta_{\text{dom}}$	DSE	$\Delta_{\text{dom}}$	DSE
		Baseline	SHC $\times$ 2		SHC $\times$ 4		DHC $\times$ 2		DHC $\times$ 4
nnU-Net	TC	0.43	0.50	$+0.07$	0.39	$-0.04$	0.61	$+0.18$	0.43	$+0.00$
	WT	0.31	0.46	$+0.15$	0.35	$+0.04$	0.35	$+0.04$	0.40	$+0.09$
	ET	0.80	0.75	$-0.05$	0.68	$-0.12$	0.84	$+0.04$	0.81	$+0.01$
SwinUNETR	TC	0.50	0.49	$-0.00$	0.51	$+0.01$	0.50	$+0.01$	0.46	$-0.04$
	WT	0.10	0.12	$+0.02$	0.10	$+0.01$	0.13	$+0.03$	0.12	$+0.02$

Table 3: Dominant-modality Dice drop

\Delta_{\text{dom}}

and Dominant Sensitivity Enhancement (DSE) for all backbone–variant pairs. Dominant sequence: T1ce for TC and ET; FLAIR for WT. DSE is relative to the baseline (Eq. 3 ). Positive DSE (bold) indicates increased reliance on the clinically dominant sequence relative to baseline. Values near zero denote no meaningful change.

Method	HC Params (M)	Total Params (M)	$\Delta$ Rate (%)
SwinUNETR-Base	–	21.3406	–
SwinUNETR-DHC $\times$ 2	0.0174	21.3580	$+0.08172$
SwinUNETR-DHC $\times$ 4	0.0235	21.3640	$+0.10991$
SwinUNETR-SHC $\times$ 2	0.000128	21.3407	$+0.00060$
SwinUNETR-SHC $\times$ 4	0.000384	21.3410	$+0.00180$
nnU-Net-Base	–	30.3987	–

Table 4: Parameters added by HC (millions).

\Delta

Rate (%) is the increase over the matching baseline. Any change in reasoning behavior cannot be attributed to capacity: DHC adds

\leq 0.53\%

and SHC

<0.002\%

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Modality ablation (nnU-Net). Dice drop when each MRI sequence is removed at inference, shown per configuration (rows: Baseline, SHC $\times$ 2/ $\times$ 4, DHC $\times$ 2/ $\times$ 4) and tumor region (columns: TC, WT, ET). The dashed line is the mean drop $\bar{\Delta}$ . HC variants increase sensitivity to T1ce (TC/ET) and FLAIR (WT) relative to the baseline.

Panel of Figure 2 — Figure 2: Modality ablation (SwinUNETR). As in Figure 1 , for SwinUNETR. HC variants increase sensitivity to T1ce (TC/ET) and FLAIR (WT).

Panel of Figure 3 — Figure 3: Modality ablation (VT-UNet). As in Figure 1 , for VT-UNet; the same T1ce (TC/ET) and FLAIR (WT) pattern holds, confirming the effect on transformer backbones.

Figure 4: Qualitative results for nnU-Net. Rows: Ground Truth, Baseline, DHC $\times$ 2, DHC $\times$ 4, SHC $\times$ 2, SHC $\times$ 4. Columns: TC (red), WT (blue), ET (green). Dice scores per configuration are shown on the right. HC-based models match ground-truth boundaries more closely, particularly for ET.

Figure 5: Qualitative results for SwinUNETR.

Figure 6: Qualitative results for VT-UNet.

Panel of Figure 7 — Figure 7: Beta evolution atlas nnUNet3D. Top to bottom: DHC- $\times$ 2, DHC- $\times$ 4, SHC- $\times$ 2, SHC- $\times$ 4. A rising $\beta$ indicates increasing reliance on hybrid connections, while a flatter or decreasing trend suggests dominance of skip pathways.

Panel of Figure 8 — Figure 8: Beta evolution atlas SwinUNETR3D. Top to bottom: DHC- $\times$ 2, DHC- $\times$ 4, SHC- $\times$ 2, SHC- $\times$ 4. A rising $\beta$ indicates increasing reliance on hybrid connections, while a flatter or decreasing trend suggests dominance of skip pathways.

Panel of Figure 9 — Figure 9: Beta evolution atlas VTUNet3D. Top to bottom: DHC- $\times$ 2, DHC- $\times$ 4, SHC- $\times$ 2, SHC- $\times$ 4. A rising $\beta$ indicates increasing reliance on hybrid connections, while a flatter or decreasing trend suggests dominance of skip pathways.

Explore similar work

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
Apr 24, 2026Peibo Song, Xiaotian Xue, Jinshuo Zhang +5Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures
School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab +2
Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters
Oct 13, 2026Milad Rostamian, Mohammadreza Mohammadi, Ender Konukoglu +1Brain Tumor Segmentation Cross-Modality Medical Image Segmentation
School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran · Computer Vision Laboratory, ETH Zurich, 8092 Zurich, Switzerland
LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities
Sep 6, 2026Haobin Chen, Ao Chang, Rundong Wang +3Medical Image Segmentation Brain Tumor Segmentation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · College of Information Science and Technology, Donghua University

Apr 24, 2026J/K move · Enter open · S save

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

Peibo Song, Xiaotian Xue, Jinshuo Zhang, Zihao Wang+4

School of Mathematics, Shandong University · 2GSFS, The University of Tokyo · 3MedVisAI Lab+2

Cross-Modality Medical Image Segmentation Hybrid CNN-Transformer Architectures Medical Image Segmentation Medical Image Representation Learning Brain Tumor Segmentation

Open paper AskPDF arXiv

Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME

Abstract

Explore similar work

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities