Accurate medical image segmentation requires effective modeling of both long-range dependencies and fine-grained boundary details. While transformers mitigate the issue of insufficient semantic information arising from the limited receptive field inherent in convolutional neural networks, they introduce new challenges: standard self-attention incurs quadratic computational complexity and often assigns non-negligible attention weights to irrelevant regions, diluting focus on discriminative structures and ultimately compromising segmentation accuracy. Existing attention variants, although effective in reducing computational complexity, fail to suppress redundant computation and inadvertently impair global context modeling. Furthermore, conventional fusion strategies in encoder-decoder architectures, typically based on simple concatenation or summation, can not adaptively integrate high-level semantic information with low-level spatial details. To address these limitations, we propose DCAU-Net, a novel yet efficient segmentation framework with two key ideas. First, a new Differential Cross Attention (DCA) is designed to compute the difference between two independent softmax attention maps to adaptively highlight discriminative structures. By replacing pixel-wise key and value tokens with window-level summary tokens, DCA dramatically reduces computational complexity without sacrificing precision. Second, a Channel-Spatial Feature Fusion (CSFF) strategy is introduced to adaptively recalibrate features from skip connections and up-sampling paths through using sequential channel and spatial attention, effectively suppressing redundant information and amplifying salient cues. Experiments on two public benchmarks demonstrate that DCAU-Net achieves competitive performance with enhanced segmentation accuracy and robustness.
Figures & tables
Fig. 1: Details of the differential cross attention. It performs efficient cross attention between pixel-wise queries and window-level key–value pairs via differential attention, suppressing redundancy and enhancing focus on discriminative structures.
Fig. 2: Details of the DCA Block, consisting of a 3 × 3 depth-wise convolution, a DCA module, and a 2-layer MLP.
Fig. 3: Overall architecture of the proposed DCAU-Net. The network adopts a U-shaped encoder-decoder framework with four hierarchical stages. The encoder integrates DCA blocks, centered on the differential cross attention. Features from the encoder are transferred to the decoder via skip connections and adaptively fused with those from previous decoder layers through CSFF blocks to enhance segmentation accuracy.
Methods
Params (M)
Flops (G)
DSC (%) ↑
HD (mm) ↓
Aorta
Gallbladder
Kidney (L)
Kidney (R)
Liver
Pancreas
Spleen
Stomach
U-Net [ 4 ]
14.80
8.43
76.85
39.70
89.07
69.72
77.77
68.60
93.43
53.98
86.67
75.58
Att-UNet [ 6 ]
34.88
66.64
77.77
36.02
89.55
68.88
77.98
71.11
93.57
58.04
87.30
75.75
TransUNet [ 1 ]
105.28
29.35
77.48
31.69
87.23
63.13
81.87
77.02
94.08
55.86
85.08
75.62
Swin-Unet [ 2 ]
27.17
6.16
79.13
21.55
85.47
66.53
83.28
79.61
94.29
56.58
90.66
76.60
HiFormer [ 12 ]
25.51
8.05
80.39
14.70
86.21
65.69
85.23
79.77
94.61
59.52
90.99
81.08
PVT-CASCADE [ 22 ]
35.28
7.62
81.06
20.23
83.01
70.59
82.23
80.37
94.08
64.43
90.10
83.69
TABLE I: Quantitative results of our approach against other state-of-the-art methods on the Synapse dataset. The symbol ↑ indicates the larger the better. The symbol ↓ denotes the smaller the better. The best result is in blod , and the second best is underlined .
Fig. 4: Qualitative comparisons of our approach against other state-of-the-art methods on the Synapse dataset.
Method
DSC (%) ↑
RV
Myo
LV
R50+U-Net [ 1 ]
87.55
87.10
80.63
94.92
R50+Att-UNet [ 1 ]
86.75
87.58
79.20
93.47
UNETR [ 3 ]
88.61
85.29
86.52
94.02
TransUNet [ 1 ]
89.71
88.86
84.54
95.73
Swin-Unet [ 2 ]
90.00
88.55
85.62
95.83
MISSFormer [ 23 ]
90.86
89.55
88.04
94.99
TABLE II: Quantitative results of different methods on ACDC dataset.
Pre-trained
DSC (%) ↑
HD (mm) ↓
Aorta
Gallbladder
Kidney (L)
Kidney (R)
Liver
Pancreas
Spleen
Stomach
w/o
81.25
17.63
87.68
67.88
86.77
82.31
94.36
59.56
92.02
79.43
w
83.29
15.14
87.91
73.09
88.20
84.05
94.98
63.89
93.26
80.94
TABLE III: Ablation study of pre-trained weights for DCAU-Net on the Synapse dataset.
Attention Type
λ Init Strategy
DSC (%) ↑
HD (mm) ↓
Standard Attention
-
80.79
19.35
Differential Attention
Fixed (0.5)
81.18
19.10
Differential Attention
Fixed (0.8)
81.03
17.94
Differential Attention
Dynamic
81.25
17.63
TABLE IV: Ablation study of the attention type and λ initialization strategy for DCA on the Synapse dataset.
Fusion Strategy
DSC (%) ↑
HD (mm) ↓
Standard U-Net Fusion
79.76
20.50
CSFF (w/o SA)
79.63
23.19
CSFF (w/o CA)
80.83
21.56
CSFF (Full)
81.25
17.63
TABLE V: Ablation study of the CSFF block on the Synapse dataset.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Medical image segmentation requires models that preserve fine anatomical boundaries while remaining practical for clinical deployment. Transformers capture long-range dependencies but incur quadratic attention cost, whereas CNNs are efficient but less effective at global reasoning. Linear attention offers O(N) scaling, but often produces diffuse feature aggregation that weakens boundary-sensitive prediction. We introduce a gated differential linear-attention mixer for medical image segmentation. Its global path, Gated Differential Linear Attention (GDLA), performs differential subtraction between two kernelized attention branches over complementary query/key subspaces to suppress redundant responses, and employs a data-dependent gate for token refinement. A parallel local token-mixing branch with depthwise convolution strengthens neighborhood interactions for better refinement, and the two branches are fused while preserving O(N) complexity. When instantiated in a pretrained Pyramid Vision Transformer (PVT)-based encoder--decoder model, \name achieves state-of-the-art results on the evaluated 2D medical segmentation benchmarks spanning CT, MRI, ultrasound, and dermoscopy, with a favorable accuracy--efficiency trade-off over closely related baselines. The code is publicly available at https://github.com/xmindflow/gdla.
Hongbo Zheng, Afshin Bozorgpour, Dorit Merhof +1
University of Illinois Urbana-Champaign · University of Regensburg
Medical image segmentation relies on the ability of encoder-decoder architectures to translate rich feature representations into accurate pixel-level predictions under challenging conditions such as low contrast, structural ambiguity, and scale variability. While recent advances in large-scale pretraining and transformer-based encoders have substantially improved feature extraction, segmentation accuracy remains constrained by decoder design, particularly in terms of cross-scale alignment, contextual integration, and boundary preservation. In this work, we revisit medical image segmentation from a decoder-centric perspective and propose a context-aware gated decoder that systematically regulates feature fusion and contextual aggregation throughout the decoding process. The proposed decoder integrates lightweight multi-scale channel recalibration, gated skip fusion with spatial competition and a global context aggregation mechanism that injects encoder-wide information into intermediate decoding stages. This design enables effective translation of strong pretrained encoder representations into spatially consistent predictions. Extensive experiments across 11 medical image segmentation benchmarks validate the effectiveness and demonstrate that the proposed approach consistently outperforms strong baselines while remaining computationally practical. Code: https://github.com/saadwazir/MedCAGD
Saad Wazir, Patrick Dominique Vibild, Dinh Phu Tran +2
School of Computing, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea · Department of Energy, Aalborg University, Aalborg, Denmark
Accurate medical image segmentation is an integral part of the medical image analysis pipeline that requires the ability to merge local and global information. While vision transformers are able to capture global interactions using vanilla self-attention, their quadratic computational complexity in the input size remains a struggle for medical image segmentation tasks. Motivated by the dispersion property of vanilla self-attention and recent development of Mamba form of attention, Scalable and Efficient Mamba like Attention (SEMA) utilizes token localization via local window attention to avoid dispersion and maintain focusing, complemented by theoretically consistent arithmetic averaging to capture global aspect of attention. In this work, we present USEMA, a hybrid UNet architecture that merges the local feature extraction ability of convolutional neural networks (CNNs) with SEMA attention. We conduct experiments with USEMA across a variety of modalities and image sizes, demonstrating improved computational efficiency compared to transformer based models using full self-attention, and superior segmentation performance relative to purely convolution and Mamba-based models.