Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

Dipan Maity; Suman Mondal; Arindam Roy

cs.LGApr 7, 2026

Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

Authors: Dipan Maity, Suman Mondal, Arindam Roy

Organizations: Student Kolkata, West Bengal, India · Department of Computer Science Yogoda Satsanga Palpara Mahavidyalaya West Bengal, India · Department of Computer Science & Application Prabhat Kumar College Contai West Bengal, India

Abstract

We introduce Gated-SwinRMT, a family of hybrid vision transformers that combine the shifted-window attention of the Swin Transformer with the Manhattan-distance spatial decay of Retentive Networks (RMT), augmented by input-dependent gating. Self-attention is decomposed into consecutive width-wise and height-wise retention passes within each shifted window, where per-head exponential decay masks provide a two-dimensional locality prior without learned positional biases. Two variants are proposed.Gated-SwinRMT-SWAT substitutes softmax with sigmoid activation, implements balanced ALiBi slopes with multiplicative post-activation spatial decay, and gates the value projection via SwiGLU; the Normalized output implicitly suppresses uninformative attention scores. Gated-SwinRMT-Retention retains softmax-normalized retention with an additive log-space decay bias and incorporates an explicit G1 sigmoid gate -- projected from the block input and applied after local context enhancement (LCE) but prior to the output projection~ $W_O$ -- to alleviate the low-rank $W_V \!\cdot\! W_O$ bottleneck and enable input-dependent suppression of attended outputs. We assess both variants on Mini-ImageNet ( $224{\times}224$ , 100 classes) and CIFAR-10 ( $32{\times}32$ , 10 classes) under identical training protocols, utilizing a single GPU due to resource limitations. At ${\approx}77$ -- $79$ ,M parameters, Gated-SwinRMT-SWAT achieves $80.22\%$ and Gated-SwinRMT-Retention $78.20\%$ top-1 test accuracy on Mini-ImageNet, compared with $73.74\%$ for the RMT baseline. On CIFAR-10 -- where small feature maps cause the adaptive windowing mechanism to collapse attention to global scope -- the accuracy advantage compresses from $+6.48$ ,pp to $+0.56$ ,pp.

Figures & tables

Figure 1: Architecture of the proposed Gated-SwinRMT variants. (a) Gated-SwinRMT-SWAT : sigmoid-based normalized attention with SwiGLU-gated values, balanced ALiBi positional bias, and multiplicative spatial decay $\gamma^{|i-j|}$ applied post-sigmoid. (b) Gated-SwinRMT-Retention : softmax-normalized retention with additive log-decay mask $M_{ij}{=}|i{-}j|\log\gamma_{h}$ applied pre-softmax, and a learned G1 sigmoid gate applied after local context enhancement (LCE) and before the output projection $W_{O}$ . Both variants share DWConv $3{\times}3$ positional encoding, LayerScale ( $\gamma_{1},\gamma_{2}$ ) with DropPath, an LCE module on $V$ , adaptive window partitioning with optional cyclic shift, and convolutional patch merging. Best viewed in colour.

Layer	Operation	Kernel	Stride	Activation
1	Conv + BN	$3\times 3$	2	GELU
2	Conv + BN	$3\times 3$	1	GELU
3	Conv + BN	$3\times 3$	2	GELU
4	Conv + BN	$3\times 3$	1	—

Table 1: Patch embedding stem. Channel dim doubles at layers 1 and 3.

Hyper-parameter	Value
Batch size	128
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )
Peak learning rate	$1\times 10^{-4}$
LR schedule	Cosine decay + 5-epoch linear warm-up
Weight decay	$0.05$
Augmentation	RandAugment, Mixup ( $\alpha{=}0.8$ ),

Table 2: Shared training hyper-parameters applied identically to all models on both datasets.

Model	Params	$\boldsymbol{\Delta}$ P	Ep. T	Val Acc	Test Acc	Test Loss
RMT [ 2 ]	77.4M	—	${\sim}240$ s	75.21%	73.74%	1.6526
Gated-SwinRMT-Retention	78.1M	$+$ 0.93%	${\sim}288$ s	79.39%	78.20%	1.5093
Gated-SwinRMT-SWAT	78.6M	$+$ 1.64%	${\sim}297$ s	81.10%	80.22%	1.4573

Table 3: Mini-ImageNet 100-class classification results after 40 epochs. Best result per column in bold .

Model	5	10	15	20	25	30	35	40
RMT	31.1	45.4	55.8	61.3	68.3	72.2	74.6	75.2
Retention	35.4	53.8	65.0	70.1	74.8	77.0	78.6	79.4
SWAT	41.2	60.3	67.7	74.5	77.6	79.0	79.9	81.1

Table 4: Mini-ImageNet validation accuracy (%) at 5-epoch intervals.

Model	Params	$\boldsymbol{\Delta}$ P	Val Acc	Test Acc	Test Loss
RMT [ 2 ]	11.5M	—	85.98%	85.90%	0.8132
Gated-SwinRMT-Retention	15.3M	$+$ 33.9%	86.54%	86.46%	0.8112
Gated-SwinRMT-SWAT	15.3M	$+$ 33.9%	86.76%	86.39%	0.8109

Table 5: CIFAR-10 10-class classification results after 50 epochs (compact

{\approx}11

–

15

M parameter variants). Best result per column in bold .

Model	Ep 10	Ep 20	Ep 30	Ep 40	Ep 50
RMT	70.58	80.44	84.18	85.52	85.98
Retention	70.52	80.58	84.54	86.24	86.54
SWAT	72.34	80.32	83.96	86.16	86.76

Table 6: CIFAR-10 validation accuracy (%) at 10-epoch intervals.

Component(s)	Type	Model pair	Test $\Delta$	Cumulative
DWConv $3{\times}3$ + LCE + G1 gate	Shared	RMT $\to$ Ret.	$+$ 4.46 pp ( $+$ 0.56 pp)	$+$ 4.46 pp
SwiGLU on $V$ + sigmoid SWAT	SWAT-only	Ret. $\to$ SWAT	$+$ 2.02 pp ( $-$ 0.07 pp)	$+$ 6.48 pp
All components	Full	RMT $\to$ SWAT	$\mathbf{+6.48}$ pp ( $+$ 0.49 pp)	$\mathbf{+6.48}$ pp

Table 7: Component-level ablation. Mini-ImageNet deltas are the primary result; CIFAR-10 bypass-regime deltas are shown in grey for comparison.

Components	Type	Model pair	Test $\Delta$	Cumulative
DWConv 3 $\times$ 3 , LCE, G1 gate	Shared	RMT $\to$ Ret.	4.46pp (0.56pp)	4.46pp
SwiGLU on $V$ , sigmoid	SWAT-only	Ret. $\to$ SWAT	2.02pp (-0.07pp)	6.48pp
All components	Full	RMT $\to$ SWAT	6.48pp (0.49pp)	6.48pp

Table 8: Component-level ablation. Mini-ImageNet deltas are the primary result; CIFAR-10 bypass-regime deltas are shown in gray for comparison.

Explore similar work

FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention
Oct 3, 2026Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan +1Vision Transformer Efficient ViTs
Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, USA · Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA · RNET Technologies, Dayton, OH, USA
Rethinking Vision Architectures with Gated Linear Attention and KAN
Sep 18, 2026Ali Mehizel, Oussama KhaldiVisual Representation Learning Vision Transformer
Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria
Gated Spatial Redundancy Projection for Pathology Transformer Attentions
Aug 8, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Cancer Survival Prediction Self-Attention
Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montréal, Canada · Mila - Quebec AI Institute, Montreal, Canada

Oct 3, 2026J/K move · Enter open · S save

FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention

Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan, Shireen Y. Elhabian

Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, USA · Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA · RNET Technologies, Dayton, OH, USA

Vision Transformer Efficient ViTs Attention Mechanisms

Open paper AskPDF arXiv

High-resolution vision backbones have long been forced to trade away local token density to afford larger receptive fields. Hierarchical Swin transformers impose this compromise because standard windowed attention materializes an

M^2\times M^2

score matrix per window, incurring

O(M^4)

memory as windows or token grids grow. Furthermore, Swin adds a learned relative-position bias elementwise to attention scores, requiring full materialization of the score matrix and its gradient. This keeps Swin and SwinV2 trapped in a small-window(

M=8,16

), coarse-token regime with patch size

4\times4

(

p=4

), limiting performance for fine-grained tasks. We introduce FLASHSWIN, which replaces standard windowed attention with a FlashAttention implementation that computes exact softmax attention without materializing the score matrix, reducing per-window memory from

O(M^4)

O(M^2)

. This enables higher token density and larger receptive fields without inflating memory overhead. Training memory is flat across window sizes: at a

32\times32

window, FLASHSWIN-T requires only

12.4

,GB, unchanged from

8\times8

, compared to

70/90

,GB for SwinV2/V1-T. However, applying FlashAttention directly to Swin creates a trade-off: bypassing the score matrix precludes Swin's additive relative-position bias, forfeiting spatial information in exchange for memory efficiency. FLASHSWIN restores position information as window-local learnable 2D RoPE, making large windows and dense token grids both affordable and accurate. At matched scale, FLASHSWIN-T outperforms Swin variants. With dense tokens and wide windows (

p=2,M=32

), the same Tiny model reaches

84.1\%

ImageNet-1K,

44.1

COCO box AP, and

47.28

ADE20K mIoU---gains of

+1.3

+5.1

, and

+1.82

over SwinV2-T at

M=16

, respectively. At fixed

M=32

, halving the patch size yields roughly

3\times

larger gains in boundary quality than in mIoU.

Abstract

Explore similar work

FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention

Rethinking Vision Architectures with Gated Linear Attention and KAN

Gated Spatial Redundancy Projection for Pathology Transformer Attentions