Cross-Stage Attention Propagation for Efficient Semantic Segmentation

Beoungwoo Kang

cs.CVApr 7, 2026

Cross-Stage Attention Propagation for Efficient Semantic Segmentation

Authors: Beoungwoo Kang

Organizations: Vision Platform Team, Hyundai Mobis, South Korea

Abstract

Recent lightweight semantic segmentation methods have made significant progress by combining compact backbones with efficient decoder heads. However, most multi-scale decoders compute attention independently at each feature scale, introducing substantial redundancy since the resulting attention distributions across scales are strongly correlated. We propose Cross-Stage Attention Propagation (CSAP), a decoder framework that computes attention at the deepest feature scale and propagates the resulting attention maps to shallower stages, bypassing query-key computation at those stages entirely. This design preserves multi-scale contextual reasoning while substantially reducing the decoder's computational cost. CSAP-Tiny achieves 42.9% mIoU on ADE20K with only 5.5 GFLOPs, 80.5% on Cityscapes with 21.5 GFLOPs, and 40.9% on COCO-Stuff 164K with 5.5 GFLOPs, surpassing SegNeXt-Tiny by +1.8% on ADE20K while requiring 16.8% fewer floating-point operations.

Figures & tables

Figure 1: Performance comparison on ADE20K validation. Our CSAP-Tiny achieves 42.9% mIoU with only 5.5 GFLOPs, outperforming SegNeXt-Tiny (41.1%, 6.6 GFLOPs) with fewer computations and surpassing MetaSeg-T (42.4%, 5.5 GFLOPs) and EDAFormer-T (42.3%, 5.6 GFLOPs).

Figure 2: Overall architecture of the proposed CSAP framework. The hierarchical backbone extracts multi-scale features $\{C_{2},C_{3},C_{4}\}$ at resolutions $\{1/8,1/16,1/32\}$ . Cross-attention is computed at stage 4 with spatially pooled key–value tokens, producing attention maps $A_{4}$ . These maps are propagated to stages 2 and 3, where only value projections are computed while Q–K computation is bypassed.

Figure 3: Detailed structure of the Cross-Stage Attention module. At stage 4, pooled cross-attention computes Q from the original tokens and K, V from spatially pooled context tokens, producing multi-head attention maps $A_{4}$ . The attention maps are reshaped, adaptively pooled to a fixed $8\times 8$ spatial size, and transformed to produce stage-specific attention weights $A_{4\rightarrow 2}$ and $A_{4\rightarrow 3}$ . At stages 2 and 3, only value projections are computed and combined with the propagated attention weights.

Method	Params (M) $\downarrow$	ADE20K		Cityscapes		COCO-Stuff
Method	Params (M) $\downarrow$	GFLOPs $\downarrow$	mIoU(%) $\uparrow$	GFLOPs $\downarrow$	mIoU(%) $\uparrow$	GFLOPs $\downarrow$	mIoU(%) $\uparrow$
TopFormer-B [ 31 ]	5.1	1.8	39.2	5.7	71.2	–	–
SegFormer-B0 [ 27 ]	3.8	8.4	37.4	125.5	76.2	8.4	35.6
FeedFormer-B0 [ 21 ]	4.5	7.8	39.2	107.4	77.9	–	–
VWFormer-B0 [ 28 ]	3.7	5.1	38.9	–	77.2	5.1	36.2
MetaSeg-T [ 12 ]	4.7	5.5	42.4	71.7	80.1	5.5	39.7

Table 1: Comparison with state-of-the-art lightweight semantic segmentation methods on ADE20K, Cityscapes, and COCO-Stuff. Only tiny/small-scale models are compared. GFLOPs are measured at the resolution used for each dataset.

Decoder Design	Params $\downarrow$	GFLOPs $\downarrow$	mIoU(%) $\uparrow$
Hamburger [ 10 ]	4.3M	6.6	41.1
Standard Self-Attn [ 27 ]	5.0M	5.9	43.1
CSAP (Ours)	4.8M	5.5	42.9

Table 2: Effect of attention propagation on ADE20K. Standard computes full Q, K, V attention at all three stages independently. CSAP computes Q, K, V only at stage 4 and propagates the attention maps to stages 2 and 3. Both use the same MSCAN-T backbone.

Source Stage	Params $\downarrow$	GFLOPs $\downarrow$	mIoU(%) $\uparrow$
Stage 2 ( $1/8$ )	5.3M	6.1	42.4
Stage 3 ( $1/16$ )	5.1M	5.7	42.4
Stage 4 ( $1/32$ )	4.8M	5.5	43.0

Table 3: Ablation on the propagation source stage. Full Q, K, V attention is computed at the source stage, and the resulting attention maps are propagated to the remaining two stages. All models use the MSCAN-T backbone on ADE20K.

Figure 4: Attention map visualization comparing standard self-attention (top) and cross-stage attention propagation (bottom). Despite computing Q–K attention only at stage 4, CSAP produces attention patterns highly similar to those of the standard baseline across all three stages, yielding nearly identical predictions. This validates that per-stage Q–K computation is largely redundant.

Figure 5: Qualitative segmentation results on ADE20K validation. From left to right: Input image, SegNeXt-T prediction, and CSAP-T prediction. Our CSAP-T produces more coherent predictions on global regions.

Figure 6: Qualitative segmentation results on Cityscapes validation. From left to right: Input image, SegNeXt-T prediction, and CSAP-T prediction.

Efficient Semantic Segmentation

See all 15 papers

XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation
Mar 28, 2026Xinyu Liu, Qing Xu, Zhen ChenAttention Residual Connections Medical Image Segmentation
Imperial College London · University of Nottingham · The Hong Kong Polytechnic University
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
May 31, 2026Aditya Makineni, Qing TianCross-Architecture Knowledge Distillation Relational Knowledge Distillation
Department of Computer Science University of Alabama at Birmingham Birmingham, US
LightAVSeg: Lightweight Audio-Visual Segmentation
May 9, 2026Qing Zhong, Guodong Ding, Lingqiao Liu +3Audio-Visual Understanding Audio-Visual Alignment
College of Informatics, Huazhong Agricultural University, Wuhan, China · School of Computing, National University of Singapore, Singapore · School of Computer Science, Adelaide University, Australia +2

Mar 28, 2026J/K move · Enter open · S save

XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation

Xinyu Liu, Qing Xu, Zhen Chen

Imperial College London · University of Nottingham · The Hong Kong Polytechnic University

Attention Residual Connections Medical Image Segmentation Multi-Scale Feature Fusion

Open paper AskPDF arXiv

In the field of Large Language Models (LLMs), Attention Residuals have recently demonstrated that learned, selective aggregation over all preceding layer outputs can outperform fixed residual connections. We propose Cross-Stage Attention Residuals (XAttnRes), a mechanism that maintains a global feature history pool accumulating both encoder and decoder stage outputs. Through lightweight pseudo-query attention, each stage selectively aggregates from all preceding representations. To bridge the gap between the same-dimensional Transformer layers in LLMs and the multi-scale encoder-decoder stages in segmentation networks, XAttnRes introduces spatial alignment and channel projection steps that handle cross-resolution features with negligible overhead. When added to existing segmentation networks, XAttnRes consistently improves performance across four datasets and three imaging modalities. We further observe that XAttnRes alone, even without skip connections, achieves performance on par with the baseline, suggesting that learned aggregation can recover the inter-stage information flow traditionally provided by predetermined connections.

Abstract

Efficient Semantic Segmentation

Explore similar work

XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation

SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation

LightAVSeg: Lightweight Audio-Visual Segmentation