Understanding the Effect of Visual Attention in CNNs: A Decision-Space Alignment Perspective
Organizations: School of Electrical and Computer Sciences, Indian Institute of Technology Bhubaneswar, 752050, India
Abstract
Visual Attention enhances convolutional neural networks (CNNs) by modulating intermediate features, yet their effects on downstream decisions remain less understood. We present a decision-space framework that models multiplicative attention as an intermediate feature perturbation and derives a first-order prediction of its effect on the target-versus-competitor classification margin. We further introduce the Decision Alignment Score (DAS) to quantify the directional alignment between the perturbation and the local margin gradient. We evaluate the framework on CIFAR-100 using SimAM and CBAM across five popular CNN backbones, ResNet-18, ResNet-20, ResNet-50, ResNet-56, and MobileNetV2. The analysis examines prediction agreement, DAS margin correlations, and downstream activation-state changes. Across most final-attention configurations, predicted and actual margin changes show good agreement, while DAS exhibits positive correlations with actual margin changes. Single-attention experiments further show that prediction agreement and DAS margin correlations depend on attention placement, with stronger associations at later stages.
Figures & tables
| B0 | SimAM ALL | CBAM ALL | ||||
|---|---|---|---|---|---|---|
| Backbone | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 |
| ResNet-18 | 76.88 | 93.56 | 77.33 | 93.85 | 76.19 | 93.74 |
| ResNet-20 | 68.19 | 91.04 | 68.07 | 90.97 | 68.34 | 91.25 |
| ResNet-50 | 75.96 | 93.73 | 79.11 | 94.84 | 77.82 | 94.40 |
| ResNet-56 | 70.54 | 91.91 | 71.44 | 92.01 | 71.81 | 92.03 |
| MobileNetV2 | 70.04 | 92.54 | 70.07 | 92.55 | 71.26 | 93.01 |
| SimAM | CBAM | |||
|---|---|---|---|---|
| Configuration | Top-1 | Top-5 | Top-1 | Top-5 |
| B0 | 76.88 | 93.56 | 76.88 | 93.56 |
| S1 | 77.99 | 93.92 | 76.73 | 93.65 |
| S2 | 76.66 | 93.54 | 76.52 | 93.80 |
| S3 | 77.01 | 93.47 | 76.52 | 93.45 |
| S4 | 76.39 | 93.56 | 76.45 | 93.47 |
| Margin prediction | DAS / activation | |||||||
| Backbone/Module | MAE | RMSE | SA | AC | ||||
| SimAM | ||||||||
| ResNet-18 | 1.00 | 1.00 | 0.02 | 0.03 | 99.90 | 0.97 | 0.98 | 9.39 |
| ResNet-20 | 0.99 | 0.99 | 0.15 | 0.19 | 96.31 | 0.98 | 0.99 | 17.64 |
| ResNet-50 | 0.92 | 0.90 | 1.91 | 2.20 | 93.62 | 0.93 | 0.91 | 18.76 |
| ResNet-56 | 1.00 | 1.00 | 0.03 | 0.04 | 99.38 | 0.98 | 0.99 | 7.73 |
Visual Attention
Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.
See all 66 papersEnd-to-end Feature Alignment: A Simple CNN with Intrinsic Class Attribution
Grad-CAM for Visualizing Attention Regions of PCA and SVM Layers in Convolutional Neural Networks
PCA-Grad-CAM'', a method for visualizing attention regions in PCA feature vectors, and SVM-Grad-CAM'', a method for visualizing attention regions in an SVM classifier layer. Solving a closed-form Jacobian problem comprising partial derivatives from the last convolutional layer to the PCA and/or SVM layers is necessary to complete the proposed methods analytically. In this paper, we present the exact closed-form Jacobian and visualization results of the proposed methods applied to several major datasets. In addition, the insertion and deletion metrics, which are major evaluation metrics in explainable AI, were applied to PCA- and SVM-Grad-CAM. The results suggest that the proposed method can successfully visualize the attention regions of PCA and SVM.