ProSMA-UNet: Decoder Conditioning for Proximal-Sparse Skip Feature Selection
Authors: Chun-Wun Cheng, Yanqi Cheng, Peiyuan Jing, Guang Yang, Javier A. Montoya-Zegarra, Carola-Bibiane Schönlieb, Angelica I. Aviles-Rivero
Organizations: Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK · School of Engineering, Zurich University of Applied Sciences, CH · Bioengineering Department and ImperialX, Imperial College London, UK · Lucerne University Teaching and Research Hospital, CH · Yau Mathematical Sciences Center, Tsinghua University, China
Medical image segmentation commonly relies on U-shaped encoder-decoder architectures such as U-Net, where skip connections preserve fine spatial detail by injecting high-resolution encoder features into the decoder. However, these skip pathways also propagate low-level textures, background clutter, and acquisition noise, allowing irrelevant information to bypass deeper semantic filtering -- an issue that is particularly detrimental in low-contrast clinical imaging. Although attention gates have been introduced to address this limitation, they typically produce dense sigmoid masks that softly reweight features rather than explicitly removing irrelevant activations. We propose ProSMA-UNet (Proximal-Sparse Multi-Scale Attention U-Net), which reformulates skip gating as a decoder-conditioned sparse feature selection problem. ProSMA constructs a multi-scale compatibility field using lightweight depthwise dilated convolutions to capture relevance across local and contextual scales, then enforces explicit sparsity via an ℓ1 proximal operator with learnable per-channel thresholds, yielding a closed-form soft-thresholding gate that can remove noisy responses. To further suppress semantically irrelevant channels, ProSMA incorporates decoder-conditioned channel gating driven by global decoder context. Extensive experiments on challenging 2D and 3D benchmarks demonstrate state-of-the-art performance, with particularly large gains (≈20%) on difficult 3D segmentation tasks. Project page: https://math-ml-x.github.io/ProSMA-UNet/
Figures & tables
Figure 1: ProSMA-UNet motivation and overview. (a) Conventional attention gates generate a dense soft mask (sigmoid reweighting), which can still pass weak but harmful skip activations (noise features) into the decoder. (b) Our ProSMA sparse gating constructs a multi-scale decoder–encoder compatibility field and applies an ℓ1 proximal (soft-thresholding) operator to induce explicit sparsity (exact zeros), enabling direct removal of irrelevant skip responses. (c) ProSMA-UNet integrates the proposed sparse gating module at each skip connection to condition high-resolution feature transfer on decoder context. (d) Overview of ProSMA Spare Gating.
Method
Year
BUSI
GlaS
Kvasir-SEG
IoU ↑
F1 ↑
IoU ↑
F1 ↑
IoU ↑
F1 ↑
U-Net
2015
57.22 ± 4.74
71.91 ± 3.54
86.66 ± 0.91
92.79 ± 0.56
54.58 ± 0.18
79.59 ± 0.08
Att-UNet
2018
55.18 ± 3.61
70.22 ± 2.88
86.84 ± 1.19
92.89 ± 0.65
62.75 ± 0.56
83.36 ± 0.78
U-Net++
2018
57.41 ± 4.77
72.11 ± 3.90
87.07 ± 0.76
92.96 ± 0.44
61.93 ± 0.74
83.33 ± 0.42
U-NeXt
2022
59.06 ± 1.03
73.08 ± 1.32
84.51 ± 0.37
91.55 ± 0.23
49.50 ± 0.48
76.64 ± 0.26
R-UNet
2024
61.00 ± 0.64
74.67 ± 1.24
86.42 ± 0.96
92.63 ± 0.62
60.28 ± 1.03
82.25 ± 0.35
Table 1: Quantitative comparison on three heterogeneous medical imaging benchmarks. Results are reported over three independent runs, demonstrating both superior performance and stable convergence. The best results are indicated by Green .
Table 3
Figure 2: Qualitative comparison of segmentation masks produced by P-UNET and baseline methods across 2D and 3D medical imaging datasets.
U-Net remains a practical baseline for image segmentation because of its simple encoder-decoder structure and skip connections. However, the bottleneck representation is still dominated by a limited set of receptive fields, while decoder features are propagated without explicitly emphasizing the most informative channels and spatial locations. This paper presents MSCA-UNet, a U-Net-based segmentation architecture that combines multi-scale contextual aggregation at the bottleneck with channel-spatial attention refinement in the decoder. The multi-scale module uses parallel atrous convolutions to capture contextual features at different receptive fields, while Convolutional Block Attention Modules (CBAMs) progressively recalibrate decoder features. Under identical experimental settings, the baseline U-Net achieves 96.9% mIoU on a held-out test set. Adding multi-scale context improves mIoU to 97.5%, while attention alone reaches 98.4%. Combining both mechanisms yields 99.1% mIoU, a 2.2 percentage-point improvement over the baseline. Parameter analysis further shows that the attention-only variant adds approximately 0.044M parameters, whereas the multi-scale module contributes most of the additional model capacity. The results support the view that multi-scale context enrichment and attention-based feature refinement provide complementary benefits within a U-Net framework.
Sheng-Wei Chan
Dept. of Electrical Engineering Tamkang University
The U-Net style models have been widely used in many applications. A critical step in these models is to reconstruct the lower-level features using a top-down decoder. This reconstruction requires precise fusion of high-level semantics and low-level details. Existing attention-based fusion methods typically derive attention weights from the top-down decoder features (global) alone or the correlation between the top-down decoder features and the bottom-up encoder features (local), then modulate the encoder features using these weights. In this work, we explore a different paradigm: deriving attention weights from the difference between the two feature streams. To this end, we propose two difference-based gating approaches: Feature-difference gating (FDG), which directly uses the absolute difference between global and local features to generate adaptive gating maps, and Entropy-difference gating (EDG), which measures the representational certainty of each stream via information entropy and uses their signed entropy difference to derive the attention weights. Both methods produce coupled gating maps that simultaneously modulate the global and local features. Experiments on different tasks including medical image segmentation, remote sensing image cloud removal and speech separation showed that both methods outperformed existing attention-based fusion methods, and EDG performed better. The results suggested a new paradigm for multi-scale feature fusion in the U-Net style structures.
Kai Li, Xuechao Zou, Jiashen Fu +3
Chinese Institute for Brain Research (CIBR), Beijing, China. · Department of Computer Science and Technology, Tsinghua University, Beijing, China.
Medical image segmentation relies on the ability of encoder-decoder architectures to translate rich feature representations into accurate pixel-level predictions under challenging conditions such as low contrast, structural ambiguity, and scale variability. While recent advances in large-scale pretraining and transformer-based encoders have substantially improved feature extraction, segmentation accuracy remains constrained by decoder design, particularly in terms of cross-scale alignment, contextual integration, and boundary preservation. In this work, we revisit medical image segmentation from a decoder-centric perspective and propose a context-aware gated decoder that systematically regulates feature fusion and contextual aggregation throughout the decoding process. The proposed decoder integrates lightweight multi-scale channel recalibration, gated skip fusion with spatial competition and a global context aggregation mechanism that injects encoder-wide information into intermediate decoding stages. This design enables effective translation of strong pretrained encoder representations into spatially consistent predictions. Extensive experiments across 11 medical image segmentation benchmarks validate the effectiveness and demonstrate that the proposed approach consistently outperforms strong baselines while remaining computationally practical. Code: https://github.com/saadwazir/MedCAGD
Saad Wazir, Patrick Dominique Vibild, Dinh Phu Tran +2
School of Computing, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea · Department of Energy, Aalborg University, Aalborg, Denmark