Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation

Marco Stefani; Edoardo Parpaiola; Matteo Bino; Gianluca Caregnato; Francesco Roncolato; Daniel Fusaro; Loris Nanni; Alberto Pretto

cs.CVOct 8, 2026

Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation

Authors: Marco Stefani, Edoardo Parpaiola, Matteo Bino, Gianluca Caregnato, Francesco Roncolato, Daniel Fusaro, Loris Nanni, Alberto Pretto

Organizations: Department of Information Engineering (DEI), University of Padova, Padova, Italy

Abstract

High-risk applications of deep learning require precise characterization of model decision-making. We conduct a systematic explainability evaluation across Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Vision Mamba (ViM). Unlike predominantly qualitative studies, we quantify the alignment between model attention and human visual perception using eye-tracking ground truth. We formalize this comparison via a standardized normalization pipeline mapping post-hoc attribution maps and gaze heatmaps onto spatial probability distributions. Alignment is evaluated across fine-grained classification, medical imaging, and emotion recognition using architecture-specific XAI methods. We measure fidelity using Jensen-Shannon similarity, Chi-square similarity, and Pearson correlation, three complementary estimators capturing support overlap, mass discrepancy, and structural co-variation. Our results reveal distinct attention dynamics: CNNs tend to exhibit a more global focus, with spatially more diffuse saliency maps that correlate well with human perception in general contexts, while ViTs demonstrate fine-grained, patch-level attention, often focused on discriminative microfeatures. The Mamba architecture shows biased behavior: it achieves strong explainability in medical tasks but exhibits divergent intensity distributions in natural images; our analytical characterization of the metrics makes this dissociation explicit, since near-unit support overlap is attained simultaneously with a vanishing linear correlation. We further analyze the effect of backbone freezing and ensemble strategies on model interpretability. The results highlight how architectural and training choices influence the alignment between model saliency and human visual attention.

Figures & tables

Dataset	Task	Classes	Training/Test Nums	Reference	Gaze Download Link
CUB-200-2011	Birds Classification	200	5995/5794	[ 57 ]	[ 58 ]
CXR	Chest X-ray Classification	3	976/107	[ 53 ]	[ 59 ]
KDEF	Facial Emotion Classification	7	3920/980	[ 56 ]	[ 13 ]

Table 1: Dataset details.

Panel of Figure 1 — Figure 1: Qualitative comparison of the datasets. CUB : (a) Brown Creeper, (b) Cardinal, (c) Painted Bunting, (d) Yellow Breasted Chat, (e) Pelagic Cormorant, (f) Eastern Towhee. CXR : (g) No acute cardiopulmonary process, (h) Stable mild pulmonary edema and moderate cardiomegaly, (i) Mild pulmonary edema, (j) Mild pulmonary vascular congestion and cardiomegaly, (k) No pneumonia. There is mild asymmetric pulmonary edema in the left lung, (l) Mild to moderate pulmonary edema. KDEF : (m) Happy Female, (n) Happy Man, (o) Neutral Female, (p) Neutral Male, (q) Sad Female, (r) Sad Male.

Symbol	Meaning
$\Omega$ , $\|\Omega\|$	pixel lattice $\{1..H\}\times\{1..W\}$ and its size
$\mathcal{X}$ , $\mathcal{Y}$ , $K$	input space, label set, number of classes
$\Delta^{m-1}$	probability simplex in $\mathbb{R}^{m}$
$\mathcal{D}$ , $\mathcal{D}_{\mathrm{tr}}$ , $\mathcal{D}_{\mathrm{te}}$	dataset and its train/test partition
$f_{\theta}$ , $z_{c}$ , $\hat{y}$	classifier, class logit, predicted label
$\Phi$	attribution operator (Def. 1 )

Table 2: Summary of notation.

Paper	Model	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	AUC (%)
[ 68 ]	InceptionV3	$65.56$	$45-93$	$57-95$	$40-91$	$-$
[ 68 ]	MobileNet	$74.60$	$65-81$	$37-96$	$71-94$	$-$
[ 69 ]	ResNet50	$84.05$	$-$	$-$	$-$	$-$
[ 70 ]	TransFG	$\mathbf{91.70}$	$-$	$-$	$-$	$-$
Ours	Resnet50	$84.48$	$85.00$	$84.57$	$84.45$	$99.70$
Ours	ViT-base	$87.59$	$\mathbf{88.19}$	$\mathbf{87.71}$	$\mathbf{87.46}$	$\mathbf{99.76}$

Table 3: Comparison with existing works on CUB-200-2011 dataset.

Paper	Model	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	AUC (%)
[ 71 ]	GazeMTL trained by [ 72 ]	$78.50$	$78.60$	$78.10$	$77.90$	$88.70$
[ 73 ]	IAA trained by [ 72 ]	$78.50$	$78.00$	$77.40$	$77.60$	$90.00$
[ 65 ]	EffNet+GG-CAM trained by [ 72 ]	$77.57$	$77.00$	$77.20$	$77.00$	$88.80$
[ 72 ]	GazeGNN	$\mathbf{83.18}$	$\mathbf{83.90}$	$\mathbf{82.10}$	$\mathbf{82.30}$	$\mathbf{92.30}$
Ours	Resnet50	$78.50$	$78.04$	$76.77$	$77.03$	$89.75$
Ours	ViT-base	$81.31$	$81.16$	$82.09$	$81.27$	$89.89$

Table 4: Comparison with existing works on CXR dataset.

Paper	Model	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	AUC (%)
[ 74 ]	CNN	$82.38$	$-$	$-$	$-$	$-$
[ 75 ]	Shallow CNN	$89.4$	$-$	$-$	$-$	$-$
[ 67 ]	AlexNet	$94.0$	$93.2$	$92.6$	$92.5$	$-$
[ 67 ]	VGG16	$96.2$	$95.7$	$95.1$	$95.1$	$-$
[ 67 ]	DenseNet121	$98.5$	$98.2$	$97.8$	$97.8$	$-$
[ 67 ]	EfficientNet-B0	$98.7$	$98.4$	$97.9$	$98.0$	$-$

Table 5: Comparison with existing works on KDEF dataset.

Panel of Figure 2 — Figure 2: Qualitative comparison of the explainability outputs of a Vision Transformer with a frozen backbone during training (a) and (c) versus a non-frozen backbone (b) and (d).

Model	Dataset	Method	Jensen–Shannon	Chi-square	PCC
ResNet50	CUB-200-2011	GradCAM	$\mathbf{0.6067\pm 0.0810}$	$\mathbf{0.4617\pm 0.2050}$	$0.7386\pm 0.1390$
ResNet50	CUB-200-2011	ScoreCAM	$0.5620\pm 0.0900$	$0.3413\pm 0.2471$	$0.6822\pm 0.1450$
ResNet50	CUB-200-2011	AblationCAM	$0.6005\pm 0.0790$	$0.4477\pm 0.2050$	$\mathbf{0.7413\pm 0.1360}$
ViT-base	CUB-200-2011	ViTLRP	$0.5342\pm 0.0670$	$0.2724\pm 0.2000$	$0.5959\pm 0.1570$
ViT-base-F	CUB-200-2011	ViTLRP	$0.5224\pm 0.0650$	$0.2386\pm 0.1960$	$0.5625\pm 0.1560$
ViM-base	CUB-200-2011	MambaLRP	$0.3423\pm 0.0430$	$-0.3579\pm 0.1583$	$0.1350\pm 0.2641$

Table 6: Explainability evaluation on train sets. KDEF-FGD is absent because its gaze annotations (120 images) were allocated entirely to the test set by construction; no gaze ground truth is available for the training split.

Model	Dataset	Method	Jensen–Shannon	Chi-square	PCC
ResNet50	CUB-200-2011	GradCAM	$\mathbf{0.6123\pm 0.0840}$	$\mathbf{0.4732\pm 0.2150}$	$0.7362\pm 0.1470$
ResNet50	CUB-200-2011	ScoreCAM	$0.5712\pm 0.0870$	$0.3675\pm 0.2380$	$0.6935\pm 0.1460$
ResNet50	CUB-200-2011	AblationCAM	$0.6005\pm 0.0800$	$0.4578\pm 0.2090$	$\mathbf{0.7402\pm 0.1400}$
ViT-base	CUB-200-2011	ViTLRP	$0.5391\pm 0.0670$	$0.2867\pm 0.1960$	$0.6047\pm 0.1530$
ViT-base-F	CUB-200-2011	ViTLRP	$0.5239\pm 0.0670$	$0.2434\pm 0.1910$	$0.5669\pm 0.1540$
ViM-base	CUB-200-2011	MambaLRP	$0.3410\pm 0.0430$	$-0.3626\pm 0.1779$	$0.1315\pm 0.2719$

Table 7: Explainability evaluation on test sets.

Model	Model	Dataset	Jensen–Shannon	Chi-square	PCC
ResNet50	ViT-base	CUB-200-2011	$0.6411\pm 0.0528$	$0.5544\pm 0.1270$	$0.7362\pm 0.0986$
ResNet50	ViM-base	CUB-200-2011	$0.4703\pm 0.0655$	$0.0492\pm 0.2018$	$0.1558\pm 0.2659$
ViT-base	ViM-base	CUB-200-2011	$0.4447\pm 0.0801$	$-0.0344\pm 0.2498$	$0.1625\pm 0.2785$
ResNet50	ViT-base	CXR	$0.3868\pm 0.0716$	$-0.1782\pm 0.2446$	$0.2192\pm 0.1709$
ResNet50	ViM-base	CXR	$0.5404\pm 0.1153$	$0.2652\pm 0.3083$	$-0.0138\pm 0.083$
ViT-base	ViM-base	CXR	$0.3941\pm 0.0688$	$-0.1691\pm 0.2286$	$-0.0035\pm 0.1091$

Table 8: Explainability Comparison between Models on test sets.

Panel of Figure 3 — Figure 3: Quantitative comparison of XAI metrics. Visual representation of the fidelity scores across different attribution methods and architectures. Raw values in Table 7 .

Panel of Table 9 — Table 9: Qualitative Comparison of Explainability Methods.

Explore similar work

Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs
Aug 3, 2026Sathiyamohan Nishankar, Nethmi Pathirana, Pubudu Sanjeewani +2Transformer Interpretability Feature Attribution
aFaculty of Computing, Sabaragamuwa University of Sri Lanka, Sri Lanka · bFaculty of Engineering, University of Moratuwa, Katubedda, Sri Lanka · cSchool of Computing Technologies, RMIT University, Melbourne, Australia +3
Cognitive Alignment At No Cost: Inducing Human Attention Biases For Interpretable Vision Transformers
Apr 21, 2026Ethan KnightsTransformer Interpretability Visual Attention
Cambridge, UK
How Can One Choose the Best CAM-Based Explainability Method for a CNN Model?
May 3, 2026Daniel da Silva Costa, Pedro Nuno de Souza Moura, Adriana C. F. AlvimFeature Attribution Saliency Map Evaluation
Postgraduate Program in Informatics, Federal University of the State of Rio de Janeiro (UNIRIO), Rio de Janeiro, RJ, 22290-240, Brazil

Aug 3, 2026J/K move · Enter open · S save

Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs

Sathiyamohan Nishankar, Nethmi Pathirana, Pubudu Sanjeewani, Asanka Perera+1

aFaculty of Computing, Sabaragamuwa University of Sri Lanka, Sri Lanka · bFaculty of Engineering, University of Moratuwa, Katubedda, Sri Lanka · cSchool of Computing Technologies, RMIT University, Melbourne, Australia+3

Transformer Interpretability Feature Attribution Vision Transformer Explainability Evaluation

Open paper AskPDF arXiv

Most evidence on the effectiveness of explainable artificial intelligence (XAI) attribution methods has been established on convolutional neural networks (CNNs), with limited investigation into whether these conclusions generalize to the diverse Vision Transformer (ViT) architectures that now dominate computer vision. This paper presents a controlled benchmark that evaluates attribution quality across five dimensions: faithfulness, localization, robustness, complexity, and computational cost. A standardized framework assesses 13 attribution methods from four algorithmic families on eight representative backbones spanning CNNs, isotropic ViTs, hierarchical transformers, hybrid architectures, and linear-attention transformers. The results show that attribution performance is strongly architecture-dependent and that rankings established on CNNs do not reliably transfer to transformer-based models. CAM-based methods achieve the highest scores under the conventional bounding-box localization metric on CNNs and most ViTs but perform poorly on linear-attention architectures. Pixel-level dense-mask evaluation further reveals that these gains largely reflect metric saturation rather than accurate localization. CAM-based methods also exhibit limited robustness on global-attention transformers, whereas attention rollout provides consistently stable explanations with poor localization. Furthermore, faithfulness correlation offers limited discrimination between attribution methods, highlighting the limitations of single-metric evaluation. These findings challenge prevailing conclusions on attribution performance and demonstrate the need for architecture-aware, multi-dimensional evaluation. The open-source code for the evaluation framework and benchmark results is available at https://github.com/Nishan-Charlie/VIT_XAI_Bench.

Abstract

Explore similar work

Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs

Cognitive Alignment At No Cost: Inducing Human Attention Biases For Interpretable Vision Transformers

How Can One Choose the Best CAM-Based Explainability Method for a CNN Model?