RefineFormer3D: Efficient 3D Medical Image Segmentation via Adaptive Multi-Scale Transformer with Cross Attention Fusion

Kavyansh Tyagi; Vishwas Rathi; Puneet Goyal

eess.IVFeb 18, 2026

RefineFormer3D: Efficient 3D Medical Image Segmentation via Adaptive Multi-Scale Transformer with Cross Attention Fusion

Authors: Kavyansh Tyagi, Vishwas Rathi, Puneet Goyal

Organizations: National Institute of Technology Kurukshetra Department of Electronics and Communication Engineering · National Institute of Technology Kurukshetra Department of Computer Engineering · Indian Institute of Technology Ropar Department of Computer Science and Engineering

Abstract

Accurate and computationally efficient 3D medical image segmentation remains a critical challenge in clinical workflows. Transformer-based architectures often demonstrate superior global contextual modeling but at the expense of excessive parameter counts and memory demands, restricting their clinical deployment. We propose RefineFormer3D, a lightweight hierarchical transformer architecture that balances segmentation accuracy and computational efficiency for volumetric medical imaging. The architecture integrates three key components: (i) GhostConv3D-based patch embedding for efficient feature extraction with minimal redundancy, (ii) MixFFN3D module with low-rank projections and depthwise convolutions for parameter-efficient feature extraction, and (iii) a cross-attention fusion decoder enabling adaptive multi-scale skip connection integration. RefineFormer3D contains only 2.94M parameters, substantially fewer than contemporary transformer-based methods. Extensive experiments on ACDC and BraTS benchmarks demonstrate that RefineFormer3D achieves 93.44% and 85.9% average Dice scores respectively, outperforming or matching state-of-the-art methods while requiring significantly fewer parameters. Furthermore, the model achieves fast inference (8.35 ms per volume on GPU) with low memory requirements, supporting deployment in resource-constrained clinical environments. These results establish RefineFormer3D as an effective and scalable solution for practical 3D medical image segmentation.

Figures & tables

Figure 1: Model efficiency comparison on ACDC dataset ( Bernard et al., 2018 ) . Parameter count versus segmentation performance for RefineFormer3D and existing 3D segmentation models. Blue bars indicate number of parameters; orange-red curve shows average Dice score. RefineFormer3D achieves superior performance with only 2.94M parameters.

Architecture	Params (M)
nnFormer ( Zhou et al., 2023 )	150.5
TransUNet ( Chen et al., 2022 )	96.07
UNETR ( Hatamizadeh et al., 2022 )	92.49
DS-UNETR++ ( Jiang et al., 2025 )	67.7
SwinUNETR ( Hatamizadeh et al., 2021 )	62.83
MS-TCNet ( Ao et al., 2024 )	59.49

Table 1: Comparison of RefineFormer3D with state-of-the-art methods in terms of model size and computational complexity. RefineFormer3D demonstrates notable reduction in parameters while maintaining competitive computational cost.

Figure 2: RefineFormer3D architecture overview. The input volume

\mathbb{R}^{B\times C_{\mathrm{in}}\times D\times H\times W}

is procecssed via 3D patch embedding and encoded through four hierarchical stages. Each encoder stage applies two consecutive transformer blocks which are windowed self attention followed by shifted window attention. The decoder progressively refines features by fusing encoder skip connections through cross attention, where decoder features query encoder representations for selective multi scale aggregation. The final upsampling block performs refinement without skip connections. Deep supervision is applied via auxiliary heads at intermediate decoder outputs.

Method	Params (M)	Avg (%)	RV	Myo	LV
DS-UNETR++ ( Jiang et al., 2025 )	67.7	93.03	92.23	90.82	96.04
PFormer ( Gao et al., 2025 )	46.04	92.33	91.11	89.93	95.88
nnFormer ( Zhou et al., 2023 )	150.5	92.06	90.94	89.58	95.65
GCI-Net ( Qiao et al., 2025 )	13.36	91.43	90.28	89.24	94.77
MS-TCNet ( Ao et al., 2024 )	59.49	91.43	89.43	89.09	95.77
Segformer3D ( Perera et al., 2024 )	4.5	90.96	88.50	88.86	95.53

Table 2: Performance comparison on ACDC dataset. Bold values represent best performance and underlined values indicate second best results. Parameters are reported in millions.

Method	Params	Avg	WT	ET	TC
	(M)	(%)
nnFormer ( Zhou et al., 2023 )	150.5	86.4	91.3	81.8	86.0
GCI-Net ( Qiao et al., 2025 )	13.36	85.88	91.58	85.86	80.20
MS-TCNet ( Ao et al., 2024 )	59.49	85.20	91.20	80.20	84.20
DS-UNETR++ ( Jiang et al., 2025 )	67.7	83.19	91.68	78.58	79.30
Segformer3D ( Perera et al., 2024 )	4.5	82.1	89.9	74.2	82.2

Table 3: Performance comparison on BraTS dataset. Bold values represent best performance and underlined values indicate second-best results. Parameters are reported in millions.

Panel of Figure 3 — Figure 3: Visual comparison of original MRI, ground truth, and predicted segmentation for four representative BraTS cases showing whole tumor (yellow), enhancing tumor (orange), and tumor core (cyan) regions.

Panel of Figure 4 — Figure 4: Visual comparison of original MRI, ground truth, and predicted segmentation for four representative ACDC cardiac cases showing right ventricle (RV), myocardium, and left ventricle (LV).

% Training Set	Dice (%)	Drop in Dice
100%	85.90	–
90%	85.80	0.10
70%	84.86	0.94
50%	82.50	3.40

Table 4: Robustness of RefineFormer3D to reduced training data on BraTS.

Variant	Dice (%)	Params (M)
Full RefineFormer3D	85.90	2.94
w/o MixFFN3D $\rightarrow$ Dense MLP	84.87	2.88
w/o Cross-Attention Fusion $\rightarrow$ Concat + Conv3D	83.22	3.00
GhostConv3D $\rightarrow$ Standard Conv3D	86.21	4.87

Table 5: Ablation results on BraTS. Each variant is trained under identical settings. Dice (%)

\uparrow

; lower

\Delta

indicates performance drop relative to the full model.

Model	Params	FLOPs	Mem †	GPU	CPU
	(M)	(G)	(GB)	(ms)	(ms)
UNETR ( Hatamizadeh et al., 2022 )	92.5	153.5	3.3	82.5	2145
SwinUNETR ( Hatamizadeh et al., 2021 )	62.8	572.4	19.7	228.6	7612
nnFormer ( Zhou et al., 2023 )	149.6	421.5	12.6	148.0	5248
DS-UNETR++ ( Jiang et al., 2025 )	42.6	70.1	2.4	62.4	1498
RefineFormer3D	2.94	191.2	1.5	8.35	296

Table 6: Inference efficiency on BRaTS input (

1\times 4\times 128^{3}

). Times averaged over 200 passes. Env: RTX 5080, Ryzen 9 9950X, PyTorch 2.8, CUDA 12.8.

Explore similar work

SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation
Mar 23, 2026Duy D. Nguyen, Phat T. Tran-Truong3D Medical Image Segmentation 3D ViTs
Department of Software Engineering, Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), 268 Ly Thuong Kiet Street, Dien Hong Ward, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Linh Trung Ward, Ho Chi Minh City, Vietnam
LightMedSeg: Lightweight 3D Medical Image Segmentation with Learned Spatial Anchors
Mar 7, 2026Kavyansh Tyagi, Vishwas Rathi, Puneet GoyalMedical Image Segmentation 3D Medical Image Segmentation
National Institute of Technology Kurukshetra, India · Indian Institute of Technology Ropar, India
GPAFormer: Graph-guided Patch Aggregation Transformer for Efficient 3D Medical Image Segmentation
Apr 8, 2026Chung-Ming Lo, I-Yun Liu, Wei-Yang LinMedical Image Segmentation 3D Medical Image Segmentation
Institute of Artificial Intelligence Innovation, Industry Academia Innovation School, National Yang Ming Chiao Tung University · Department of Computer Science and Information Engineering, National Chung Cheng University

Mar 23, 2026J/K move · Enter open · S save

SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation

Duy D. Nguyen, Phat T. Tran-Truong

Department of Software Engineering, Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), 268 Ly Thuong Kiet Street, Dien Hong Ward, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Linh Trung Ward, Ho Chi Minh City, Vietnam

3D Medical Image Segmentation 3D ViTs Efficient ViTs State Space Models

Open paper AskPDF arXiv

The advent of Transformer and Mamba-based architectures has significantly advanced 3D medical image segmentation by enabling global contextual modeling, a capability traditionally limited in Convolutional Neural Networks (CNNs). However, state-of-the-art Transformer models often entail substantial computational complexity and parameter counts, which is particularly prohibitive for volumetric data and further exacerbated by the limited availability of annotated medical imaging datasets. To address these limitations, this work introduces SegMaFormer, a lightweight hybrid architecture that synergizes Mamba and Transformer modules within a hierarchical volumetric encoder for efficient long-range dependency modeling. The model strategically employs Mamba-based layers in early, high-resolution stages to reduce computational overhead while capturing essential spatial context, and reserves self-attention mechanisms for later, lower-resolution stages to refine feature representation. This design is augmented with generalized rotary position embeddings to enhance spatial awareness. Despite its compact structure, SegMaFormer achieves competitive performance on three public benchmarks (Synapse, BraTS, and ACDC), matching the Dice coefficient of significantly larger models. Empirically, our approach reduces parameters by up to 75x and substantially decreases FLOPs compared to current state-of-the-art models, establishing an efficient and high-performing solution for 3D medical image segmentation.

Abstract

Explore similar work

SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation

LightMedSeg: Lightweight 3D Medical Image Segmentation with Learned Spatial Anchors

GPAFormer: Graph-guided Patch Aggregation Transformer for Efficient 3D Medical Image Segmentation