Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

Dipan Maity; Suman Mondal; Arindam Roy

cs.LGApr 7, 2026

Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

Authors: Dipan Maity, Suman Mondal, Arindam Roy

Organizations: Student Kolkata, West Bengal, India · Department of Computer Science Yogoda Satsanga Palpara Mahavidyalaya West Bengal, India · Department of Computer Science & Application Prabhat Kumar College Contai West Bengal, India

Abstract

We introduce Gated-SwinRMT, a family of hybrid vision transformers that combine the shifted-window attention of the Swin Transformer with the Manhattan-distance spatial decay of Retentive Networks (RMT), augmented by input-dependent gating. Self-attention is decomposed into consecutive width-wise and height-wise retention passes within each shifted window, where per-head exponential decay masks provide a two-dimensional locality prior without learned positional biases. Two variants are proposed.Gated-SwinRMT-SWAT substitutes softmax with sigmoid activation, implements balanced ALiBi slopes with multiplicative post-activation spatial decay, and gates the value projection via SwiGLU; the Normalized output implicitly suppresses uninformative attention scores. Gated-SwinRMT-Retention retains softmax-normalized retention with an additive log-space decay bias and incorporates an explicit G1 sigmoid gate -- projected from the block input and applied after local context enhancement (LCE) but prior to the output projection~ $W_O$ -- to alleviate the low-rank $W_V \!\cdot\! W_O$ bottleneck and enable input-dependent suppression of attended outputs. We assess both variants on Mini-ImageNet ( $224{\times}224$ , 100 classes) and CIFAR-10 ( $32{\times}32$ , 10 classes) under identical training protocols, utilizing a single GPU due to resource limitations. At ${\approx}77$ -- $79$ ,M parameters, Gated-SwinRMT-SWAT achieves $80.22\%$ and Gated-SwinRMT-Retention $78.20\%$ top-1 test accuracy on Mini-ImageNet, compared with $73.74\%$ for the RMT baseline. On CIFAR-10 -- where small feature maps cause the adaptive windowing mechanism to collapse attention to global scope -- the accuracy advantage compresses from $+6.48$ ,pp to $+0.56$ ,pp.

Figures & tables

Figure 1: Architecture of the proposed Gated-SwinRMT variants. (a) Gated-SwinRMT-SWAT : sigmoid-based normalized attention with SwiGLU-gated values, balanced ALiBi positional bias, and multiplicative spatial decay $\gamma^{|i-j|}$ applied post-sigmoid. (b) Gated-SwinRMT-Retention : softmax-normalized retention with additive log-decay mask $M_{ij}{=}|i{-}j|\log\gamma_{h}$ applied pre-softmax, and a learned G1 sigmoid gate applied after local context enhancement (LCE) and before the output projection $W_{O}$ . Both variants share DWConv $3{\times}3$ positional encoding, LayerScale ( $\gamma_{1},\gamma_{2}$ ) with DropPath, an LCE module on $V$ , adaptive window partitioning with optional cyclic shift, and convolutional patch merging. Best viewed in colour.

Layer	Operation	Kernel	Stride	Activation
1	Conv + BN	$3\times 3$	2	GELU
2	Conv + BN	$3\times 3$	1	GELU
3	Conv + BN	$3\times 3$	2	GELU
4	Conv + BN	$3\times 3$	1	—

Table 1: Patch embedding stem. Channel dim doubles at layers 1 and 3.

Hyper-parameter	Value
Batch size	128
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )
Peak learning rate	$1\times 10^{-4}$
LR schedule	Cosine decay + 5-epoch linear warm-up
Weight decay	$0.05$
Augmentation	RandAugment, Mixup ( $\alpha{=}0.8$ ),

Table 2: Shared training hyper-parameters applied identically to all models on both datasets.

Model	Params	$\boldsymbol{\Delta}$ P	Ep. T	Val Acc	Test Acc	Test Loss
RMT [ 2 ]	77.4M	—	${\sim}240$ s	75.21%	73.74%	1.6526
Gated-SwinRMT-Retention	78.1M	$+$ 0.93%	${\sim}288$ s	79.39%	78.20%	1.5093
Gated-SwinRMT-SWAT	78.6M	$+$ 1.64%	${\sim}297$ s	81.10%	80.22%	1.4573

Table 3: Mini-ImageNet 100-class classification results after 40 epochs. Best result per column in bold .

Model	5	10	15	20	25	30	35	40
RMT	31.1	45.4	55.8	61.3	68.3	72.2	74.6	75.2
Retention	35.4	53.8	65.0	70.1	74.8	77.0	78.6	79.4
SWAT	41.2	60.3	67.7	74.5	77.6	79.0	79.9	81.1

Table 4: Mini-ImageNet validation accuracy (%) at 5-epoch intervals.

Model	Params	$\boldsymbol{\Delta}$ P	Val Acc	Test Acc	Test Loss
RMT [ 2 ]	11.5M	—	85.98%	85.90%	0.8132
Gated-SwinRMT-Retention	15.3M	$+$ 33.9%	86.54%	86.46%	0.8112
Gated-SwinRMT-SWAT	15.3M	$+$ 33.9%	86.76%	86.39%	0.8109

Table 5: CIFAR-10 10-class classification results after 50 epochs (compact

{\approx}11

–

15

M parameter variants). Best result per column in bold .

Model	Ep 10	Ep 20	Ep 30	Ep 40	Ep 50
RMT	70.58	80.44	84.18	85.52	85.98
Retention	70.52	80.58	84.54	86.24	86.54
SWAT	72.34	80.32	83.96	86.16	86.76

Table 6: CIFAR-10 validation accuracy (%) at 10-epoch intervals.

Component(s)	Type	Model pair	Test $\Delta$	Cumulative
DWConv $3{\times}3$ + LCE + G1 gate	Shared	RMT $\to$ Ret.	$+$ 4.46 pp ( $+$ 0.56 pp)	$+$ 4.46 pp
SwiGLU on $V$ + sigmoid SWAT	SWAT-only	Ret. $\to$ SWAT	$+$ 2.02 pp ( $-$ 0.07 pp)	$+$ 6.48 pp
All components	Full	RMT $\to$ SWAT	$\mathbf{+6.48}$ pp ( $+$ 0.49 pp)	$\mathbf{+6.48}$ pp

Table 7: Component-level ablation. Mini-ImageNet deltas are the primary result; CIFAR-10 bypass-regime deltas are shown in grey for comparison.

Components	Type	Model pair	Test $\Delta$	Cumulative
DWConv 3 $\times$ 3 , LCE, G1 gate	Shared	RMT $\to$ Ret.	4.46pp (0.56pp)	4.46pp
SwiGLU on $V$ , sigmoid	SWAT-only	Ret. $\to$ SWAT	2.02pp (-0.07pp)	6.48pp
All components	Full	RMT $\to$ SWAT	6.48pp (0.49pp)	6.48pp

Table 8: Component-level ablation. Mini-ImageNet deltas are the primary result; CIFAR-10 bypass-regime deltas are shown in gray for comparison.

Explore similar work

Oct 3, 2026

FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention

High-resolution vision backbones have long been forced to trade away local token density to afford larger receptive fields. Hierarchical Swin transformers impose this compromise because standard windowed attention materializes an

M^2\times M^2

score matrix per window, incurring

O(M^4)

memory as windows or token grids grow. Furthermore, Swin adds a learned relative-position bias elementwise to attention scores, requiring full materialization of the score matrix and its gradient. This keeps Swin and SwinV2 trapped in a small-window(

M=8,16

), coarse-token regime with patch size

4\times4

(

p=4

), limiting performance for fine-grained tasks. We introduce FLASHSWIN, which replaces standard windowed attention with a FlashAttention implementation that computes exact softmax attention without materializing the score matrix, reducing per-window memory from

O(M^4)

O(M^2)

. This enables higher token density and larger receptive fields without inflating memory overhead. Training memory is flat across window sizes: at a

32\times32

window, FLASHSWIN-T requires only

12.4

,GB, unchanged from

8\times8

, compared to

70/90

,GB for SwinV2/V1-T. However, applying FlashAttention directly to Swin creates a trade-off: bypassing the score matrix precludes Swin's additive relative-position bias, forfeiting spatial information in exchange for memory efficiency. FLASHSWIN restores position information as window-local learnable 2D RoPE, making large windows and dense token grids both affordable and accurate. At matched scale, FLASHSWIN-T outperforms Swin variants. With dense tokens and wide windows (

p=2,M=32

), the same Tiny model reaches

84.1\%

ImageNet-1K,

44.1

COCO box AP, and

47.28

ADE20K mIoU---gains of

+1.3

+5.1

, and

+1.82

over SwinV2-T at

M=16

, respectively. At fixed

M=32

, halving the patch size yields roughly

3\times

larger gains in boundary quality than in mIoU.

Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan +1

Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, USA · Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA · RNET Technologies, Dayton, OH, USA

Vision Transformer Efficient ViTs Attention Mechanisms

Ask PDF

Sep 18, 2026

Rethinking Vision Architectures with Gated Linear Attention and KAN

Vision Transformers devote most of their parameters to MLPs for channel mixing, but still rely on quadratic multi-head self-attention for token interactions. While linear attention fixes the complexity problem, bringing it down to O(N), it is usually just paired with the same fixed-activation MLP as before. Kolmogorov-Arnold Networks take a different approach, placing learnable univariate functions on the edges instead. However, existing vision KANs either retain standard attention or remove attention entirely, so the two ideas have not been effectively combined. We introduce LKAT (Linear Kolmogorov-Arnold Transformer) to close this gap: an isotropic ViT-style encoder that couples chunk-wise Gated Linear Attention with a two-layer KAN feed-forward block, backed by an I/O-aware fused RBF-KAN kernel to make radial-basis grid functions efficient in practice. Under a shared DeiT-style training recipe, LKAT-B outperforms ViT-B/16, ViT-5-B, and Mixer-B/16 on ImageNet-100, while Tiny, Small, and Base variants scale consistently on CIFAR-10/100. ImageNet-100 pretraining also transfers effectively to CIFAR fine-tuning, suggesting that gated linear attention and KAN-based radial basis functions provide complementary inductive biases for mid-scale visual representation learning. Code: https://github.com/mehizelali/linear-kan-transformer

Ali Mehizel, Oussama Khaldi

Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria

Visual Representation Learning Vision Transformer Gated Attention Efficient ViTs Kolmogorov-Arnold Networks Linear Attention+1

Ask PDF

Aug 8, 2026

Gated Spatial Redundancy Projection for Pathology Transformer Attentions

Transformer models are increasingly used for whole-slide image analysis in computational pathology. Yet, WSIs differ fundamentally from natural images: neighbouring patches often contain highly similar tissue type, stain, texture, and cellular composition. We identify this local spatial redundancy as a pathology-specific failure mode of self-attention, where dominant neighbourhood features can be repeatedly mixed into patch-tokens and weaken subtle diagnostic or prognostic deviations. We propose Gated Spatial Redundancy Projection (Gated SRP), a lightweight drop-in correction module for self-attention layers. For each patch token and attention head, Gated SRP estimates a local redundancy axis from neighbouring value vectors, projects the attention output onto this axis, and applies a learned signed gate to correct the redundancy-aligned component geometrically. Across five TCGA survival cohorts, Gated SRP obtains the highest mean C-index among the compared attention variants in all cohorts, with an average improvement over the base attention, while adding only +0.02% parameters. Across five slide-level classification datasets, it improves the base attention on 12 of 16 reported metrics and achieves the best AUC on three datasets. Code is publicly available at https://github.com/AtlasAnalyticsLab/GatedSRP.

Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1

Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montréal, Canada · Mila - Quebec AI Institute, Montreal, Canada

Cancer Survival Prediction Self-Attention Attention Mechanisms Histopathology Image Classification Whole-Slide Image Analysis

Ask PDF