Learn to Rank: Visual Attribution by Learning Importance Ranking

David Schinagl; Christian Fruhwirth-Reisinger; Alexander Prutsch; Samuel Schulter; Horst Possegger

cs.CVApr 7, 2026

Learn to Rank: Visual Attribution by Learning Importance Ranking

Authors: David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch, Samuel Schulter, Horst Possegger

Organizations: Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA

Abstract

Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models. Code and pretrained models are available at https://github.com/dschinagl/AHA.

Figures & tables

Panel of Figure 1 — Figure 1: Qualitative comparison of attribution maps produced by our method (AHA) and recent ViT attribution methods for a ViT-B/16 classifier [ 25 ] on ImageNet [ 22 ] samples. The green / red labels mark correct / incorrect predictions (ground truth in parentheses). While existing approaches operate on the patch representation and produce coarse, blocky attribution maps, our method generates dense, pixel-level attributions that align closely with object boundaries. Note that in this case no additional backprop refinement steps were applied at inference (denoted $T=0$ ).

Panel of Figure 2 — Figure 2: Soft deletion masks. Given an input image $\mathbf{I}$ and its attribution map $\mathbf{A}$ , the soft top- $k$ masks $\mathbf{m}_{\mathbf{A}}^{\text{soft},(k)}$ progressively replace the most important regions with the reference image. Unlike hard binary masks, the soft relaxation ( Sec. 3.1 ) assigns continuous importance values to each region, enabling gradient-based optimization of the deletion and insertion metrics. Note the varying grid resolution $G$ across examples top vs. bottom ( Sec. 3.2 ) which prevents overfitting to a fixed spatial partitioning.

	Method	Deletion $\downarrow$	Insertion $\uparrow$	Ins. - Del. $\uparrow$	Positive $\downarrow$	Negative $\uparrow$	Neg. - Pos. $\uparrow$	ADP $\downarrow$	PIC $\uparrow$	Runtime
Predicted Class	Grad-CAM [ 55 ]	0.2858	0.4982	0.2124	0.3346	0.5788	0.2442	84.3339	4.3577	0.014s
	IG [ 64 ]	0.1857	0.5544	0.3686	0.2248	0.6347	0.4099	22.9340	33.4353	0.251s
	Trans-Att. [ 71 ]	0.1654	0.5833	0.4178	0.2022	0.6654	0.4633	25.6994	30.8467	0.254s
	Bi-Att. [ 17 ]	0.1600	0.5958	0.4358	0.1961	0.6789	0.4828	28.4572	28.2614	0.254s
	TIS [ 27 ]	0.1420	0.6381	0.4961	0.1731	0.7222	0.5492	11.2479	45.5658	1.167s
	ViT-CX [ 69 ]	0.1750	0.5714	0.3964	0.2149	0.6478	0.4328	12.5472	43.4971	0.860s

Table 1: Quantitative comparison of attribution methods on ImageNet using a ViT-B/16 classifier. All metrics are averaged over three reference image modes

\mathbf{I}_{0}

(black, mean, blur). Top-3 results per metric are highlighted (dark to light).

	Method	Deletion $\downarrow$	Insertion $\uparrow$	Ins. - Del. $\uparrow$	Positive $\downarrow$	Negative $\uparrow$	Neg. - Pos. $\uparrow$	ADP $\downarrow$	PIC $\uparrow$	Runtime
Predicted Class	Grad-CAM [ 55 ]	0.2337	0.5294	0.2956	0.2930	0.6328	0.3398	66.9314	8.5072	0.010s
	IG [ 64 ]	0.1897	0.5542	0.3645	0.2426	0.6567	0.4141	14.2831	39.0279	0.157s
	Trans-Att. [ 71 ]	0.2024	0.5507	0.3483	0.2567	0.6534	0.3967	17.1553	34.9780	0.157s
	Bi-Att. [ 17 ]	0.1887	0.5726	0.3839	0.2416	0.6782	0.4366	17.6613	34.3086	0.158s
	TIS [ 27 ]	0.1702	0.6094	0.4392	0.2144	0.7181	0.5037	9.7104	48.3937	0.416s
	ViT-CX [ 69 ]	0.1982	0.5604	0.3622	0.2525	0.6622	0.4098	9.3272	48.9950	0.763s

Table 2: Quantitative comparison of attribution methods on ImageNet using a ViT-B/32 classifier. All metrics are averaged over three reference image modes

\mathbf{I}_{0}

(black, mean, blur). Top-3 results per metric are highlighted (dark to light).

Panel of Figure 3 — Figure 3: Effect of (a) test-time refinement steps $T$ and (b) training perturbation steps $S$ on the Insertion AUC. Results are shown for the predicted class ( Pred. ) and the ground-truth class ( GT ) on the ImageNet validation set (ViT-B/16).

Panel of Figure 4 — Figure 4: Exemplary test-time refinement. Attribution maps produced by AHA without ( $T{=}0$ ) and with ( $T{=}5$ ) test-time refinement for the predicted class using ViT-B/16. Refinement progressively sharpens the heatmaps, concentrating attribution on the relevant object parts while suppressing diffuse background activations.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	$S{=}16$ (train $\sim$ $4.5$ h)		$S{=}8$ (train $\sim$ $2$ h)
Baseline	$T{=}0$	$T{=}3$	$T{=}0$	$T{=}3$
ViT-CX	19.19k	51.76k	8.53k	23.00k
TIS	14.07k	26.13k	6.25k	11.61k
MDA	0 1.19k	0 1.24k	0.53k	0 0.55k

Appendix

Table 3: Training cost amortization. Break-even point (number of generated attributions) after which the total runtime of AHA, including the one-time training, falls below that of the baseline. Results are shown for ViT-B/16 with

S{=}16

and

S{=}8

perturbation steps during training and with

T{=}0

T{=}3

test-time refinement steps at inference. Lower is better.

Panel of Figure 5 — Figure 5: Out-of-distribution shortcut detection. A ViT-B/16 is fine-tuned with a white circle injected into every sunglasses training image. Using an AHA explainer trained on the clean model, test-time refinement progressively reveals the learned shortcut: at $T{=}0$ the attribution highlights generic class features, while increasing $T$ reveals the white circle as important feature for the fine-tuned model’s decision.

	Method	Deletion $\downarrow$	Insertion $\uparrow$	Ins. - Del. $\uparrow$	Positive $\downarrow$	Negative $\uparrow$	Neg. - Pos. $\uparrow$	ADP $\downarrow$	PIC $\uparrow$	Runtime
Pred. Class	ScoreCAM [ 68 ]	0.3779	0.5425	0.1646	0.5348	0.7367	0.2019	56.0312	5.8985	0.821s
	AblationCAM [ 23 ]	0.3340	0.5890	0.2550	0.4753	0.7877	0.3124	48.9473	4.0993	0.742s
	GradCAM++ [ 13 ]	0.3159	0.6008	0.2848	0.4635	0.8231	0.3597	50.9349	4.3820	0.019s
	HiResCAM [ 26 ]	0.2748	0.6267	0.3520	0.3908	0.8314	0.4406	41.5140	5.4472	0.019s
	GradCAM [ 55 ]	0.2748	0.6267	0.3520	0.3908	0.8314	0.4406	41.5133	5.4453	0.004s
	AHA (Ours)	0.1741	0.5911	0.4170	0.2408	0.7616	0.5208	23.6857	6.0192	0.005s

Appendix

Table 4: Quantitative comparison of attribution methods on ImageNet using a ConvNeXt Small classifier [ 42 ] . All metrics are averaged over three reference image modes

\mathbf{I}_{0}

(black, mean, blur). Top-3 results per metric are highlighted (dark to light).

Method	Deletion $\downarrow$	Insertion $\uparrow$	Insertion - Deletion $\uparrow$	Runtime
Grad-CAM [ 55 ]	0.241	0.737	0.496	0.014s
IG [ 64 ]	0.222	0.741	0.518	0.251s
ViT-CX [ 69 ]	0.236	0.722	0.486	0.860s
Trans-Att. [ 71 ]	0.228	0.748	0.520	0.254s
Bi-Att. [ 17 ]	0.218	0.760	0.542	0.254s
TIS [ 27 ]	0.196	0.761	0.565	1.167s

Appendix

Table 5: Quantitative comparison using the MDA evaluation framework [ 66 ] for a ViT-B/16 classifier on their filtered ImageNet subset (5,000 images). Top-3 results per metric are highlighted (dark to light).

	MAS			ROAD
Method	Ins. $\uparrow$	Del. $\downarrow$	Diff. $\uparrow$	LeRF $\uparrow$	MoRF $\downarrow$	Diff. $\uparrow$
ViT-CX [ 69 ]	0.582	0.408	0.174	0.664	0.201	0.463
TIS [ 27 ]	0.615	0.385	0.230	0.712	0.169	0.543
MDA [ 66 ]	0.775	0.279	0.497	0.687	0.189	0.498
AHA ( $T{=}0$ )	0.761	0.290	0.471	0.715	0.160	0.555

Appendix

Table 6: Complementary faithfulness metrics. Magnitude-based MAS [ 67 ] and debiased ROAD [ 53 ] on the

5{,}000

-image MDA subset for the ViT-B/16 classifier. Top-3 results per metric are highlighted (dark to light).

Method	Mean Avg. Precision $\uparrow$	Intersection-over-Union $\uparrow$	F1 Score $\uparrow$
ViT-CX [ 69 ]	0.678	0.535	0.428
TIS [ 27 ]	0.705	0.586	0.460
MDA [ 66 ]	0.796	0.702	0.487
AHA ( $T{=}0$ )	0.790	0.667	0.483

Appendix

Table 7: ImageNet segmentation. Localization performance on the ImageNet segmentation benchmark [ 31 ] for the ViT-B/32 classifier, following the MDA [ 66 ] setting. Top-3 results per metric are highlighted (dark to light).

	Method	Optimized for		Evaluated on	RISE		MAS
	Method	Optimized for		Evaluated on	Diff. $\uparrow$	$\Delta$	Diff. $\uparrow$	$\Delta$
ViT-B/16	MDA	B/16	$\rightarrow$	B/16	0.624	–	0.497	–
	MDA	B/32	$\rightarrow$	B/16	0.473	-24.2%	0.336	-32.4%
	$\text{AHA }_{T=0}$	B/16	$\rightarrow$	B/16	0.715	–	0.471	–
	$\text{AHA }_{T=0}$	B/32	$\rightarrow$	B/16	0.650	-9.1%	0.417	-11.5%
	$\text{AHA }_{T=3}$	B/32	$\rightarrow$	B/16	0.739	+3.3%	0.427	-9.3%

Appendix

Table 8: Cross-architecture transfer. RISE [ 48 ] and MAS [ 67 ] faithfulness metrics ( Insertion

-

Deletion difference) on correctly classified ImageNet samples. Gray rows are the native reference, where the explainer is optimized for the same architecture it is evaluated on. In the remaining transfer rows, an explainer trained for a different model was used.

Grid size $G$	RISE Diff. $\uparrow$	MAS Diff. $\uparrow$	Temp. $\tau$	RISE Diff. $\uparrow$	MAS Diff. $\uparrow$
$G{\sim}\mathcal{U}(7,28)$ (used)	0.715	0.471	$\tau{=}1.0$ (used)	0.715	0.471
Fixed $G{=}7$	0.671	0.427	$\tau{=}0.5$	0.703	0.462
Fixed $G{=}14$	0.698	0.453	$\tau{=}1.5$	0.710	0.469
Fixed $G{=}28$	0.710	0.470	$\tau{=}2.0$	0.708	0.467

Appendix

Table 9: Hyperparameter ablations. Effect of the training grid size

G

and the sorting temperature

\tau

on faithfulness (RISE and MAS Insertion

-

Deletion difference, higher is better) for the ViT-B/16 classifier on correctly classified samples. The configuration used throughout the paper is marked (used) and shown in bold.

		Method	Deletion $\downarrow$	Insertion $\uparrow$	Ins. - Del. $\uparrow$	Positive $\downarrow$	Negative $\uparrow$	Neg. - Pos. $\uparrow$	ADP $\downarrow$	PIC $\uparrow$	Runtime
$\mathbf{I}_{0}\equiv$ Black Image	Predicted Class	Grad-CAM [ 55 ]	0.2542	0.4562	0.2020	0.3024	0.5336	0.2312	90.2340	2.6849	0.014s
		IG [ 64 ]	0.1555	0.5063	0.3508	0.1899	0.5857	0.3958	26.4955	30.3434	0.251s
		Trans-Att. [ 71 ]	0.1402	0.5380	0.3978	0.1726	0.6189	0.4463	29.7100	27.5834	0.254s
		Bi-Att. [ 17 ]	0.1355	0.5533	0.4178	0.1669	0.6348	0.4678	32.5737	25.3995	0.254s
		TIS [ 27 ]	0.1205	0.5976	0.4771	0.1474	0.6807	0.5332	14.1945	42.2192	1.167s
		ViT-CX [ 69 ]	0.1488	0.5269	0.3781	0.1845	0.6019	0.4174	17.1795	37.5492	0.860s

Appendix

Table 10: Quantitative comparison of attribution methods on ImageNet using a ViT-B/16 classifier [ 25 ] for all reference image modes

\mathbf{I}_{0}

(black image, mean-colored image, blurred original image). Top-3 results per metric are highlighted (dark to light).

		Method	Deletion $\downarrow$	Insertion $\uparrow$	Ins. - Del. $\uparrow$	Positive $\downarrow$	Negative $\uparrow$	Neg. - Pos. $\uparrow$	ADP $\downarrow$	PIC $\uparrow$	Runtime
$\mathbf{I}_{0}\equiv$ Black Image	Predicted Class	Grad-CAM [ 55 ]	0.1865	0.4705	0.2840	0.2365	0.5688	0.3323	70.1550	7.6858	0.010s
		IG [ 64 ]	0.1483	0.4963	0.3480	0.1905	0.5940	0.4035	16.2898	37.6572	0.157s
		Trans-Att. [ 71 ]	0.1623	0.4923	0.3300	0.2064	0.5902	0.3838	19.2848	33.7333	0.157s
		Bi-Att. [ 17 ]	0.1497	0.5206	0.3709	0.1914	0.6214	0.4300	19.6633	33.5833	0.158s
		TIS [ 27 ]	0.1368	0.5555	0.4186	0.1727	0.6607	0.4880	11.6713	46.1231	0.416s
		ViT-CX [ 69 ]	0.1601	0.5023	0.3422	0.2049	0.5993	0.3944	12.4364	44.0451	0.763s

Appendix

Table 11: Quantitative comparison of attribution methods on ImageNet using a ViT-B/32 classifier [ 25 ] for all reference image modes

\mathbf{I}_{0}

(black image, mean-colored image, blurred original image). Top-3 results per metric are highlighted (dark to light).

		Method	Deletion $\downarrow$	Insertion $\uparrow$	Ins. - Del. $\uparrow$	Positive $\downarrow$	Negative $\uparrow$	Neg. - Pos. $\uparrow$	ADP $\downarrow$	PIC $\uparrow$	Runtime
$\mathbf{I}_{0}\equiv$ Black Image	Pred. Class	ScoreCAM [ 68 ]	0.3405	0.5154	0.1750	0.5082	0.7185	0.2103	56.3717	6.7919	0.821s
		AblationCAM [ 23 ]	0.2930	0.5587	0.2657	0.4337	0.7617	0.3279	51.8343	3.8980	0.742s
		GradCAM++ [ 13 ]	0.2832	0.5677	0.2845	0.4419	0.8028	0.3608	55.9903	3.7580	0.019s
		HiResCAM [ 26 ]	0.2374	0.5948	0.3574	0.3498	0.8067	0.4570	45.7085	4.7639	0.019s
		GradCAM [ 55 ]	0.2374	0.5948	0.3574	0.3498	0.8068	0.4570	45.7078	4.7600	0.004s
		AHA (Ours)	0.1316	0.5567	0.4251	0.1669	0.6937	0.5268	24.2854	6.6999	0.005s

Appendix

Table 12: Quantitative comparison of attribution methods on ImageNet using a ConvNeXt Small classifier [ 42 ] for all reference image modes

\mathbf{I}_{0}

(black image, mean-colored image, blurred original image). Top-3 results per metric are highlighted (dark to light).

Panel of Figure 7 — Figure 7: Qualitative comparison of attribution maps for a ViT-B/16 classifier [ 25 ] on ImageNet [ 22 ] samples. The green / red labels mark correct / incorrect predictions (ground truth in parentheses).

Panel of Figure 8 — Figure 8: Qualitative comparison of attribution maps for a ViT-B/32 classifier [ 25 ] on ImageNet [ 22 ] samples. The green / red labels mark correct / incorrect predictions (ground truth in parentheses).

Panel of Figure 9 — Figure 9: Qualitative comparison of attribution maps for a ConvNeXt Small classifier [ 42 ] on ImageNet [ 22 ] samples. The green / red labels mark correct / incorrect predictions (ground truth in parentheses).

Explore similar work

What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers
Mar 2, 2026Qin Su, Tie LuoTransformer Interpretability Explainable Image Classification
Department of Electrical and Computer Engineering University of Kentucky, Lexington, KY, USA
Decision-Aware Attention Propagation for Vision Transformer Explainability
Apr 20, 2026Sehyeong Jo, Gangjae Jang, Haesol ParkTransformer Interpretability Vision Transformer
Aarhus University · University of Colorado Boulder · KIST
Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs
Aug 3, 2026Sathiyamohan Nishankar, Nethmi Pathirana, Pubudu Sanjeewani +2Transformer Interpretability Feature Attribution
aFaculty of Computing, Sabaragamuwa University of Sri Lanka, Sri Lanka · bFaculty of Engineering, University of Moratuwa, Katubedda, Sri Lanka · cSchool of Computing Technologies, RMIT University, Melbourne, Australia +3

Mar 2, 2026J/K move · Enter open · S save

What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers

Qin Su, Tie Luo

Department of Electrical and Computer Engineering University of Kentucky, Lexington, KY, USA

Transformer Interpretability Explainable Image Classification Feature Attribution Vision Transformer

Open paper AskPDF arXiv

Vision Transformers (ViTs) achieve strong performance in visual recognition, yet their decision-making remains difficult to interpret. We propose BiCAM, a bidirectional class activation mapping method that captures both supportive (positive) and suppressive (negative) contributions to model predictions. Unlike prior CAM-based approaches that discard negative signals, BiCAM preserves signed attributions to produce more complete and contrastive explanations. BiCAM further introduces a Positive-to-Negative Ratio (PNR) that summarizes attribution balance and enables lightweight detection of adversarial examples without retraining. Across ImageNet, VOC, and COCO, BiCAM improves localization and faithfulness while remaining computationally efficient. It generalizes to multiple ViT variants, including DeiT and Swin. These results suggest the importance of modeling both supportive and suppressive evidence for interpreting transformer-based vision models.

Abstract

Explore similar work

What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers

Decision-Aware Attention Propagation for Vision Transformer Explainability

Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs