XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation

Xinyu Liu; Qing Xu; Zhen Chen

cs.CVMar 28, 2026

XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation

Authors: Xinyu Liu, Qing Xu, Zhen Chen

Organizations: Imperial College London · University of Nottingham · The Hong Kong Polytechnic University

Abstract

In the field of Large Language Models (LLMs), Attention Residuals have recently demonstrated that learned, selective aggregation over all preceding layer outputs can outperform fixed residual connections. We propose Cross-Stage Attention Residuals (XAttnRes), a mechanism that maintains a global feature history pool accumulating both encoder and decoder stage outputs. Through lightweight pseudo-query attention, each stage selectively aggregates from all preceding representations. To bridge the gap between the same-dimensional Transformer layers in LLMs and the multi-scale encoder-decoder stages in segmentation networks, XAttnRes introduces spatial alignment and channel projection steps that handle cross-resolution features with negligible overhead. When added to existing segmentation networks, XAttnRes consistently improves performance across four datasets and three imaging modalities. We further observe that XAttnRes alone, even without skip connections, achieves performance on par with the baseline, suggesting that learned aggregation can recover the inter-stage information flow traditionally provided by predetermined connections.

Figures & tables

Figure 1: Effect of XAttnRes across two backbones (U-Net and EMCAD) and two benchmarks (Synapse multi-organ CT and ColonDB polyp segmentation). Adding XAttnRes alongside existing architectures (“XAttnRes + skip”) consistently improves over the baseline. Removing skip connections (“No Skip”) degrades performance, but XAttnRes alone (“replace”) recovers most of this drop. Dashed lines indicate the baseline.

Figure 2: Architecture overview. (a) Standard U-Net with fixed skip connections between resolution-matched encoder and decoder stages. (b) U-Net with XAttnRes (replace): skip connections are entirely removed. Each stage reads from a causally growing history pool ( $e_{1},\ldots,e_{S}$ for encoder; $e_{1},\ldots,e_{S},d_{1},\ldots$ for decoder) via lightweight pseudo-query attention, and appends its output for subsequent stages. The XAttnRes detail (right) shows how it aligns multi-scale features, computes attention logits via a single learnable vector $\mathbf{w}$ , and outputs a weighted sum of the original values.

Method	Average			Dice Score (%)
Method	DSC $\uparrow$	HD95 $\downarrow$	mIoU $\uparrow$	Aorta	GB	KL	KR	Liver	PC	Spleen	SM
U-Net [ 27 ]	76.25	26.93	65.96	89.93	54.42	78.87	72.87	92.71	59.65	85.98	75.55
Unet3+ [ 14 ]	76.93	26.77	66.02	88.87	61.93	83.06	76.52	93.83	56.21	81.88	73.17
TransUNet [ 5 ]	77.61	26.90	67.32	86.56	60.43	80.54	78.53	94.33	58.47	87.06	75.00
Swin-UNet [ 4 ]	77.58	27.32	66.88	81.76	65.95	82.32	79.22	93.73	53.81	88.04	75.79
SSFormer [ 35 ]	78.01	25.72	67.23	82.78	63.74	80.72	78.11	93.53	61.53	87.07	76.61

Table 1: Comparison on the Synapse multi-organ CT segmentation dataset.

\uparrow

: higher is better.

\downarrow

: lower is better. Best in bold , second underlined .

Method	#Params	#FLOPs	ColonDB	ClinicDB	ISIC 2017	Average
U-Net [ 27 ]	1.06M	2.50G	86.01	92.13	82.13	86.76
UNet++ [ 41 ]	9.16M	34.65G	87.88	92.17	82.98	87.68
Attn U-Net [ 23 ]	34.88M	66.64G	86.46	92.20	83.66	87.44
DeepLabv3+ [ 6 ]	39.76M	14.92G	91.92	93.24	83.84	89.67
PraNet [ 10 ]	32.55M	6.93G	89.16	91.71	83.03	87.97
Unet3+ [ 14 ]	26.97M	199.74G	88.38	92.60	83.34	88.11

Table 2: Comparison on 2D binary segmentation datasets. We report #Params, #FLOPs, and Dice score (%) on ColonDB, ClinicDB (polyp), and ISIC 2017 (skin lesion). Best in bold , second underlined .

Model	Skip	XAttnRes	Synapse		ColonDB
Model	Skip	XAttnRes	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$
U-Net (baseline)	✓	–	76.25	26.93	86.01
U-Net (no skip)	–	–	72.70	31.44	83.28
U-Net + XAttnRes (replace)	–	✓	76.31	26.95	86.07
U-Net + XAttnRes (both)	✓	✓	76.40	26.71	86.22
EMCAD (baseline)	✓	–	83.63	15.68	92.31

Table 3: Ablation: skip connection vs. XAttnRes across two backbones and two benchmarks. “Replace” removes all skip connections and uses only XAttnRes. “Both” retains skip connections and adds XAttnRes on top.

Table 6

Figure 3: Qualitative comparison across four datasets. Each row shows one dataset (Synapse, ColonDB, ClinicDB, ISIC 2017). Columns from left to right: UNet 3+, UCTransNet, U-Net, U-Net + XAttnRes (ours), EMCAD, EMCAD + XAttnRes (ours), and ground truth.

Attention Residual Connections

Attention residual connections use learned, input-dependent routing to combine earlier layer outputs instead of relying only on fixed skips.

See all 23 papers

Cross-Stage Attention Propagation for Efficient Semantic Segmentation
Apr 7, 2026Beoungwoo KangEfficient Semantic Segmentation Semantic Segmentation
Vision Platform Team, Hyundai Mobis, South Korea
Attention Residuals
Mar 16, 2026Kimi Team, Guangyu Chen, Yu Zhang +34Attention Residual Connections Residual Learning
DCAU-Net: Differential Cross Attention and Channel-Spatial Feature Fusion for Medical Image Segmentation
Mar 10, 2026Yanxin Li, Hui Wan, Libin LanMedical Image Segmentation Cross-Attention
College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, China

Apr 7, 2026J/K move · Enter open · S save

Cross-Stage Attention Propagation for Efficient Semantic Segmentation

Beoungwoo Kang

Vision Platform Team, Hyundai Mobis, South Korea

Efficient Semantic Segmentation Semantic Segmentation Multi-Scale Attention

Open paper AskPDF arXiv

Recent lightweight semantic segmentation methods have made significant progress by combining compact backbones with efficient decoder heads. However, most multi-scale decoders compute attention independently at each feature scale, introducing substantial redundancy since the resulting attention distributions across scales are strongly correlated. We propose Cross-Stage Attention Propagation (CSAP), a decoder framework that computes attention at the deepest feature scale and propagates the resulting attention maps to shallower stages, bypassing query-key computation at those stages entirely. This design preserves multi-scale contextual reasoning while substantially reducing the decoder's computational cost. CSAP-Tiny achieves 42.9% mIoU on ADE20K with only 5.5 GFLOPs, 80.5% on Cityscapes with 21.5 GFLOPs, and 40.9% on COCO-Stuff 164K with 5.5 GFLOPs, surpassing SegNeXt-Tiny by +1.8% on ADE20K while requiring 16.8% fewer floating-point operations.

Abstract

Attention Residual Connections

Explore similar work

Cross-Stage Attention Propagation for Efficient Semantic Segmentation

Attention Residuals

DCAU-Net: Differential Cross Attention and Channel-Spatial Feature Fusion for Medical Image Segmentation