Understanding the Effect of Visual Attention in CNNs: A Decision-Space Alignment Perspective
Organizations: School of Electrical and Computer Sciences, Indian Institute of Technology Bhubaneswar, 752050, India
Abstract
Visual Attention enhances convolutional neural networks (CNNs) by modulating intermediate features, yet their effects on downstream decisions remain less understood. We present a decision-space framework that models multiplicative attention as an intermediate feature perturbation and derives a first-order prediction of its effect on the target-versus-competitor classification margin. We further introduce the Decision Alignment Score (DAS) to quantify the directional alignment between the perturbation and the local margin gradient. We evaluate the framework on CIFAR-100 using SimAM and CBAM across five popular CNN backbones, ResNet-18, ResNet-20, ResNet-50, ResNet-56, and MobileNetV2. The analysis examines prediction agreement, DAS margin correlations, and downstream activation-state changes. Across most final-attention configurations, predicted and actual margin changes show good agreement, while DAS exhibits positive correlations with actual margin changes. Single-attention experiments further show that prediction agreement and DAS margin correlations depend on attention placement, with stronger associations at later stages.
Figures & tables
| B0 | SimAM ALL | CBAM ALL | ||||
|---|---|---|---|---|---|---|
| Backbone | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 |
| ResNet-18 | 76.88 | 93.56 | 77.33 | 93.85 | 76.19 | 93.74 |
| ResNet-20 | 68.19 | 91.04 | 68.07 | 90.97 | 68.34 | 91.25 |
| ResNet-50 | 75.96 | 93.73 | 79.11 | 94.84 | 77.82 | 94.40 |
| ResNet-56 | 70.54 | 91.91 | 71.44 | 92.01 | 71.81 | 92.03 |
| MobileNetV2 | 70.04 | 92.54 | 70.07 | 92.55 | 71.26 | 93.01 |
| SimAM | CBAM | |||
|---|---|---|---|---|
| Configuration | Top-1 | Top-5 | Top-1 | Top-5 |
| B0 | 76.88 | 93.56 | 76.88 | 93.56 |
| S1 | 77.99 | 93.92 | 76.73 | 93.65 |
| S2 | 76.66 | 93.54 | 76.52 | 93.80 |
| S3 | 77.01 | 93.47 | 76.52 | 93.45 |
| S4 | 76.39 | 93.56 | 76.45 | 93.47 |
| Margin prediction | DAS / activation | |||||||
| Backbone/Module | MAE | RMSE | SA | AC | ||||
| SimAM | ||||||||
| ResNet-18 | 1.00 | 1.00 | 0.02 | 0.03 | 99.90 | 0.97 | 0.98 | 9.39 |
| ResNet-20 | 0.99 | 0.99 | 0.15 | 0.19 | 96.31 | 0.98 | 0.99 | 17.64 |
| ResNet-50 | 0.92 | 0.90 | 1.91 | 2.20 | 93.62 | 0.93 | 0.91 | 18.76 |
| ResNet-56 | 1.00 | 1.00 | 0.03 | 0.04 | 99.38 | 0.98 | 0.99 | 7.73 |
Visual Attention
Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.
See all 66 papers