GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways

Soudeep Ghoshal; Himanshu Buckchash

cs.CVMar 23, 2026

GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways

Authors: Soudeep Ghoshal, Himanshu Buckchash

Organizations: Kalinga Institute of Industrial Technology, Bhubaneswar, India · IMC University of Applied Sciences Krems, Krems, Austria

Abstract

Deep ConvNets suffer from gradient signal degradation as network depth increases, limiting effective feature learning in complex architectures. ResNet addressed this through residual connections, but these fixed short circuits cannot adapt to varying input complexity or selectively emphasize task-relevant features across network hierarchies. This study introduces GradAttn, a variation of the residual approach in CNNs that replaces the fixed residual connections with attention-controlled gradient flow. By extracting multi-scale CNN features at different depths and regulating them through self-attention, GradAttn dynamically weights shallow texture features and deep semantic representations. For representational analysis, we evaluated three GradAttn variants across eight diverse datasets: from natural images and medical imaging to fashion recognition. The results demonstrate that GradAttn outperforms ResNet-18 on five of eight datasets, achieving up to +11.07% accuracy improvement on FashionMNIST while maintaining a comparable network size. Gradient flow analysis reveals that controlled instabilities, introduced by attention, often coincide with improved generalization, challenging the assumption that perfect stability is optimal. Furthermore, positional encoding's effectiveness turned out to be dataset-dependent, with CNN hierarchies frequently encoding sufficient spatial structure. These findings render attention mechanisms as enablers of learnable gradient control, offering a new way for adaptive representation learning in deep neural architectures.

Figures & tables

Figure 1: Proposed attention-controlled gradient flow architecture [GradAttn].

Hyperparameter	Value	Hyperparameter	Value
Optimizer	Adam	Early Stopping Monitor	Val Accuracy
Learning Rate	1 $\times$ 10 -3	Early Stopping Patience	7 epochs
Weight Decay	1 $\times$ 10 -4	Best Weight Restoration	Enabled
Batch Size	128	LR Scheduler	ReduceLROnPlateau
Maximum Epochs	100	LR Patience	3 epochs
Random Seed	42	LR Reduction Factor	0.2

Table 1: Training hyperparameters

Figure 2: Dataset overview and representative samples.

Dataset	Preprocessing Applied (Training)	Normalization (Mean / std)
Tiny ImageNet	None	[0.485, 0.456, 0.406] / [0.229, 0.224, 0.225]
CIFAR-10	RandomHorizontalFlip ( p = 0.5), RandomCrop (32, pad = 4)	[0.4914, 0.4822, 0.4465] / [0.2023, 0.1994, 0.2010]
SVHN	None	[0.4377, 0.4438, 0.4728] / [0.1980, 0.2010, 0.1970]
FashionMNIST	RandomHorizontalFlip ( p = 0.5), RandomRotation (10°), RandomAffine ( translate = 0.1)	[0.2860] /
[0.3530]
TissueMNIST	Resize (64 $\times$ 64), RandomRotation (10°), RandomHorizontalFlip ( p = 0.5), ColorJitter ( b = 0.1, c = 0.1)	[0.485, 0.456, 0.406] / [0.229, 0.224, 0.225]

Table 2: Data preprocessing and augmentation settings per dataset (training split).

Dataset	ResNet-18	No PE	Learnable PE	RoPE
Tiny ImageNet	33.02%	35.89%	34.72%	37.67%
SVHN	97.98%	98.09%	98.09%	98.15%
CIFAR-10	92.10%	88.68%	86.49%	89.16%
FashionMNIST	64.11%	66.70%	75.18%	62.40%
TissueMNIST	69.88%	69.79%	69.88%	69.72%
BloodMNIST	95.97%	96.23%	94.48%	95.59%

Table 3: Top-1 accuracy comparison across datasets.

Model	Top-3 Acc	Top-5 Acc	F1 Score	ECE
Tiny ImageNet
ResNet-18	50.52%	58.62%	0.327	0.344
RoPE	56.45%	64.81%	0.373	0.197
FashionMNIST
ResNet-18	92.21%	98.06%	0.589	0.193
Learnable PE	96.93%	99.30%	0.723	0.121

Table 4: Extended performance metrics.

Model	Precision (M)	Precision (W)	Recall (M)	Recall (W)	F1 (M)	F1 (W)
Tiny ImageNet
ResNet-18	0.328	0.331	0.330	0.330	0.327	0.329
RoPE	0.371	0.374	0.377	0.377	0.371	0.373
FashionMNIST
ResNet-18	0.662	0.662	0.636	0.641	0.585	0.589
Learnable PE	0.760	0.760	0.748	0.752	0.721	0.723

Table 5: Precision, recall, and F1 score across datasets.

Panel of Figure 3 — Figure 3: Training dynamics comparison across Complex Natural Images and Texture Recognition datasets.

Panel of Figure 4 — Figure 4: Training dynamics comparison across Medical datasets.

Dataset	ResNet-18	No PE	Learnable PE	RoPE
Tiny ImageNet	65.77%	16.24%	14.30%	25.38%
SVHN	0.70%	0.94%	1.29%	1.34%
CIFAR-10	6.52%	5.47%	3.16%	6.62%
FashionMNIST	11.57%	10.85%	8.97%	10.96%
TissueMNIST	5.39%	1.31%	1.75%	1.58%
BloodMNIST	0.57%	0.43%	0.63%	0.15%

Table 6: Train-validation accuracy gap at convergence.

Model	GHS			V/E Gradients	Avg Norm	Range	Std
Model	[1 $\times$ 10 -7 , 1 $\times$ 10 2 ]	[1 $\times$ 10 -6 , 1 $\times$ 10 1 ]	[1 $\times$ 10 -5 , 1 $\times$ 10 0 ]	V/E Gradients	Avg Norm	Range	Std
ResNet-18	1.00	1.00	1.00	None	0.153	0.022–0.370	0.106
GradAttn (No PE)	0.80	0.80	0.80	Vanishing (4 layers)	0.085	0.000–0.610	0.106
GradAttn (Learnable PE)	0.80	0.80	0.80	Vanishing (4 layers)	0.078	0.000–0.336	0.070
GradAttn (RoPE)	0.73	0.73	0.73	Vanishing (8 layers)	0.117	0.000–0.957	0.199

Table 7: Gradient flow analysis results on Tiny ImageNet.

Figure 5: Gradient dynamics of GradAttn (RoPE) on Tiny ImageNet dataset.

Dataset	ResNet-18	No PE	Learnable PE	RoPE
Tiny ImageNet	50.90	56.53	58.17	58.59
SVHN	174.61	200.67	205.52	203.86
CIFAR-10	42.03	40.45	40.50	40.84
FashionMNIST	36.33	37.42	37.50	38.94
TissueMNIST	142.00	119.43	152.71	158.90
BloodMNIST	12.52	11.86	12.97	13.28

Table 8: Average training time per epoch (seconds) across datasets.

Model	Tiny ImageNet	SVHN	FashionMNIST
ResNet-18	33.02%	97.98%	64.11%
ResNet-18 + SE	32.01%	97.94%	63.11%
ResNet-18 + CBAM	31.76%	97.99%	66.98%
GradAttn (No PE)	35.89%	98.09%	66.70%
GradAttn (Learnable PE)	34.72%	98.09%	75.18%
GradAttn (RoPE)	37.67%	98.15%	62.40%

Table 9: Comparison with attention-augmented ResNet-18 baselines on selected datasets.

Explore similar work

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Feb 9, 2026Yilang Zhang, Bingcong Li, Niao He +1Residual Learning Neural Network Optimization
ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455
Multi-Gate Residuals
May 22, 2026Zhizhan Zheng, Feiyun Zhang, Shuchun Liu +4Residual Learning
Shanghai Yichuang Information Technology Co.,Ltd. · Independent Researcher · Fudan University
StableGrad: Backward Scale Control without Batch Normalization
May 19, 2026Jose I. Mestre, Alberto Fernández-Hernández, Cristian Pérez-Corral +2Deep Learning Optimization Neural Network Training Dynamics
Universitat Politècnica de València · Universitat Jaume I

Feb 9, 2026J/K move · Enter open · S save

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

Yilang Zhang, Bingcong Li, Niao He, Georgios B. Giannakis

ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455

Residual Learning Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This paper revisits the default mechanism for deepening neural networks, namely residual connections, from an optimization perspective. Rigorous analysis proves that the layout of residual connections can fundamentally shape convergence behavior, and even induces an exponential gap in convergence rates. Prompted by this insight, we introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data. ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (

<1\%

), while enabling more effective utilization of network depth. Extensive numerical tests across pre-training of large language models, diffusion models, and deep ResNets demonstrate consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.

Abstract

Explore similar work

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

Multi-Gate Residuals

StableGrad: Backward Scale Control without Batch Normalization