Attribution Upsampling should Redistribute, Not Interpolate

Vincenzo Buono; Peyman Sheikholharam Mashhadi; Mahmoud Rahat; Prayag Tiwari; Stefan Byttner

cs.CVMar 17, 2026

Attribution Upsampling should Redistribute, Not Interpolate

Authors: Vincenzo Buono, Peyman Sheikholharam Mashhadi, Mahmoud Rahat, Prayag Tiwari, Stefan Byttner

Organizations: Halmstad University, Halmstad, Sweden

Abstract

Attribution methods in explainable AI rely on upsampling techniques that were designed for natural images, not saliency maps. Standard bilinear and bicubic interpolation systematically corrupts attribution signals through aliasing, ringing, and boundary bleeding, producing spurious high-importance regions that misrepresent model reasoning. We identify that the core issue is treating attribution upsampling as an interpolation problem that operates in isolation from the model's reasoning, rather than a mass redistribution problem where model-derived semantic boundaries must govern how importance flows. We present Universal Semantic-Aware Upsampling (USU), a principled method that reformulates upsampling through ratio-form mass redistribution operators, provably preserving attribution mass and relative importance ordering. Extending the axiomatic tradition of feature attribution to upsampling, we formalize four desiderata for faithful upsampling and prove that interpolation structurally violates three of them. These same three force any redistribution operator into a ratio form; the fourth selects the unique potential within this family, yielding USU. Controlled experiments on models with known attribution priors verify USU's formal guarantees; evaluation across ImageNet, CIFAR-10, and CUB-200 confirms consistent faithfulness improvements and qualitatively superior, semantically coherent explanations.

Figures & tables

Figure 1: USU replaces interpolation with semantically guided mass redistribution. Left: The pipeline decomposes upsampling into score-potential computation, emergent boundary detection, and ratio-form mass redistribution. Right: Standard interpolation (bilinear, bicubic) bleeds attribution across object boundaries; USU-IWMR produces sharp, semantically coherent saliency maps that faithfully reflect model reasoning.

Figure 2: Upsampling corrupts saliency maps through reconstruction errors. Classical saliency upsampling kernels can introduce reconstruction errors, leading to incorrect visual explanations. RRL-constrained [ 50 ] model on synthetic shapes; see the supplemental material for setup and extended results.

Figure 3: Identical saliency for distinct inputs. Each row shows (left) the input, (middle) the raw heatmap, (right) the upsampled saliency map, and (rightmost) the ground-truth attribution priors. Despite distinct ground truths, the upsampled explanations converge to a single pattern: interpolation cannot distinguish what the model attends to from what it ignores. RRL-constrained [ 50 ] model on synthetic shapes; see the supplemental material for setup and extended results.

Figure 4: Qualitative comparison. Top row: synthetic patterns with known ground-truth attribution priors. Bottom row: ImageNet examples with GradCAM attributions. Interpolation methods (bilinear, bicubic, Lanczos) produce ringing artifacts, boundary bleeding, and aliasing; USU preserves semantic boundaries and concentrates attribution within model-relevant regions.

Table 5

Dataset	Model	Bilinear	Bicubic	Lanczos	USU	USU ${}_{\text{IWMR}}$
	VGG16	$6.91\times 10^{6}$	$7.82\times 10^{6}$	$8.14\times 10^{6}$	$\mathbf{1.14\times 10^{5}}$	$\underline{1.28\times 10^{5}}$
ImageNet	ResNet50	$4.87\times 10^{7}$	$5.41\times 10^{7}$	$5.63\times 10^{7}$	$\mathbf{2.21\times 10^{5}}$	$\underline{2.45\times 10^{5}}$
	ViT-B-16	$2.19\times 10^{8}$	$2.48\times 10^{8}$	$2.59\times 10^{8}$	$\mathbf{7.02\times 10^{5}}$	$\underline{7.85\times 10^{5}}$
	VGG16	$3.12\times 10^{7}$	$3.55\times 10^{7}$	$3.71\times 10^{7}$	$\underline{7.12\times 10^{4}}$	$\mathbf{6.47\times 10^{4}}$
CIFAR-10	ResNet50	$8.95\times 10^{7}$	$9.87\times 10^{7}$	$1.02\times 10^{8}$	$\mathbf{2.21\times 10^{5}}$	$\underline{2.38\times 10^{5}}$
	ViT-B-16	$2.57\times 10^{8}$	$2.89\times 10^{8}$	$3.01\times 10^{8}$	$\underline{2.15\times 10^{4}}$	$\mathbf{1.99\times 10^{4}}$

Table 3: Infidelity (

\downarrow

) across datasets and models. USU achieves one to four orders of magnitude improvement over interpolation baselines. Best in bold , second-best underlined .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Complete USU pipeline. Given input $x$ , the pipeline proceeds in three stages. Scoring (bottom right): masked perturbation evaluates segment importance, producing the score matrix $\Phi$ . Hierarchical refinement (bottom left): the depth fusion loop convolves $\Phi$ with the diagonal difference kernel to compute the H-map, selects heterogeneous segments for re-segmentation, and merges coarse and fine scores so that segment boundaries emerge naturally from the model’s own reasoning; convergence is governed by the comparator threshold $\mathcal{L}$ . Redistribution (top): the coarse attribution $A(x)$ is projected onto the refined segmentation and redistributed via pixel-level weighting with the tensor potential, yielding $A^{\uparrow}$ .

Figure 6: Hierarchical boundary refinement. At each depth level, the score matrix $\Phi^{d}$ captures per-segment importance, the H-map detects score transitions at segment boundaries, and the adaptive merge combines coarse and fine attributions. Only boundary segments (high $|H_{\mathrm{map}}|$ ) are refined further, analogous to splitting impure nodes in a decision tree ( Table 4 ).

Decision Tree	USU Refinement	Formal Guarantee
Root node	Coarsest partition $\mathcal{S}^{0}$ over $\Omega$	Partitions pixel grid; refinement is reflexive and transitive ( Definition 1 )
Node impurity	Score heterogeneity $\max_{x\in\sigma_{p}}\|H_{\mathrm{map}}(x)\|$	Linear in $\Phi^{d}$ ; zero on constant regions ( Propositions 3 and 4 )
Split criterion	Boundary threshold $\|H_{\mathrm{map}}\|>\theta$	Monotone in $\theta$ : lowering $\theta$ weakly increases boundary set ( Proposition 5 )
Child nodes	Finer sub-segments $\sigma_{q}^{d+1}\subseteq\sigma_{p}^{d}$	Refinement predicate: every child contained in parent ( Definition 1 )
Leaf value	Merged score $\Phi_{\mathrm{merged}}(x)$ via adaptive $\alpha(x)$	Convex combination; bounded in $[0,1]$ ; identity on equal inputs ( Propositions 8 , 9 and 10 )
Stopping rule	$\\|\Phi^{d}{-}\Phi^{d-1}\\|_{F}\leq\mathcal{L}$ or $d=D$	Symmetric and self-terminating ( Propositions 6 and 7 )

Appendix

Table 4: Decision-tree analogy. Each stage of USU’s hierarchical refinement mirrors a decision-tree operation, with a corresponding formal guarantee verified in this supplement.

Figure 7: Compression error types. From top to bottom: ground-truth attribution, input image, upsampled saliency map, and raw low-resolution heatmap. Left column shows $\beta$ -errors where true attributions disappear due to compression (signal loss). Right column shows $\alpha$ -errors where spurious attributions emerge despite absence in ground truth (false signal generation). Both error types corrupt explanation fidelity.

Figure 8: Extended qualitative comparison. Synthetic patterns with known ground truth (top) and ImageNet samples with GradCAM from VGG16 (bottom). Interpolation baselines exhibit boundary bleeding and ringing; USU-IWMR recovers ground-truth support on synthetic data and concentrates attribution within semantic regions on real images.

Model	Method	IoU $\uparrow$	Conc. $\uparrow$	PG $\uparrow$
RRL-MLP	Nearest	0.77	0.52	0.76
	Bilinear	0.81	0.56	0.80
	Bicubic	0.84	0.55	0.79
	USU	0.88	0.79	1.00
	IWMR-USU	0.91	0.84	1.00
	Oracle-USU	0.97	0.90	1.00

Appendix

Table 5: Full synthetic results: shape classification. Best IoU (

\uparrow

), attribution concentration (

\uparrow

), and pointing game accuracy (

\uparrow

) across all methods and model families. Boldface marks the best non-oracle result per model; shading highlights our methods.

Metric	Dir.	Description
Infidelity	$\downarrow$	Expected squared error between attribution-weighted perturbations and model output changes.
IROF	$\uparrow$	Fraction of features removed before the predicted class changes (Iterative Removal of Features).
Monotonicity	$\uparrow$	Spearman correlation between attribution rank and feature deletion impact.
PixelFlipping	$\downarrow$	Area under the perturbation curve when removing pixels in attribution order.
ROAD	$\downarrow$	Remove-and-debias evaluation; measures explanation quality under debiased perturbation.
Sparseness	$\uparrow$	Gini coefficient of the attribution map; higher values indicate more concentrated explanations.

Appendix

Table 6: Evaluation metrics. All metrics are computed via Quantus. Direction indicates whether higher (

\uparrow

) or lower (

\downarrow

) values are better.

Magnitude	$\boldsymbol{\|d\|}$
Small	$0.2$ – $0.5$
Medium	$0.5$ – $0.8$
Large	$>0.8$

Appendix

Table 7: Effect size analysis. (a) Cohen’s

d

interpretation thresholds. (b) Observed Cohen’s

d

for USU vs. bilinear on Infidelity. Shading marks the active category: all configurations exhibit large effects.

USU
	lower is better
Attribution Source	ImageNet $\downarrow$	CIFAR-10 $\downarrow$	CUB-200 $\downarrow$
GradCAM	$1{,}384\to 3.15$	$1{,}349\to 0.230$	$879.9\to 0.932$
HiResCAM	$337.4\to 0.853$	$265.4\to 0.051$	$176.2\to 0.197$
GradCAM++	$1{,}041\to 2.47$	$1{,}045\to 0.187$	$715.1\to 0.773$
XGradCAM	$1{,}318\to 2.83$	$1{,}637\to 0.281$	$711.7\to 0.683$

Appendix

Table 8: Attribution method generalization. Raw infidelity values (bilinear

\to

variant), averaged across VGG16, ResNet50, and ViT-B-16.

ImageNet
		lower is better			higher is better
Model	Method	Infid. $\downarrow$	PxFlip $\downarrow$	ROAD $\downarrow$	IROF $\uparrow$	Mono. $\uparrow$	Sparse. $\uparrow$
VGG16	Nearest	$7.15\times 10^{6}$	0.62	0.58	0.41	0.18	0.31
	Bilinear	$6.91\times 10^{6}$	0.60	0.56	0.43	0.19	0.33
	Bicubic	$7.82\times 10^{6}$	0.61	0.57	0.42	0.18	0.32
	Lanczos-3	$8.14\times 10^{6}$	0.62	0.58	0.41	0.18	0.31

Appendix

Table 9: Full ImageNet and CIFAR-10 results. Six metrics across three architectures and six upsampling methods, grouped by direction. Boldface marks the best per model and metric; shading highlights our methods.

		lower is better			higher is better
Model	Method	Infid. $\downarrow$	PxFlip $\downarrow$	ROAD $\downarrow$	IROF $\uparrow$	Mono. $\uparrow$	Sparse. $\uparrow$
VGG16	Nearest	$5.68\times 10^{6}$	0.63	0.59	0.40	0.18	0.30
	Bilinear	$5.44\times 10^{6}$	0.61	0.57	0.42	0.19	0.32
	Bicubic	$6.12\times 10^{6}$	0.62	0.58	0.41	0.18	0.31
	Lanczos-3	$6.38\times 10^{6}$	0.63	0.59	0.40	0.18	0.30
	USU	$1.83\times 10^{3}$	0.27	0.26	0.59	0.32	0.71

Appendix

Table 10: Full CUB-200-2011 results. Same format as Table 9 .

Figure 9: Interpolation artifacts vs. USU reconstruction on synthetic data. Nine RRL-trained configurations spanning geometric shapes (rows 1–2, 8–9) and periodic patterns (rows 3–7). From left: ground-truth attribution $e(x)$ , input sample, coarse $7{\times}7$ heatmap $\hat{e}(x)$ , bilinear, bicubic, and Lanczos upsampling, and USU (ours). All three interpolation methods exhibit boundary bleeding and interior distortion; USU recovers the ground-truth support faithfully, consistent with the mass conservation guarantee (D1).

Feature Attribution

Feature attribution estimates which input features or internal components contribute to a model’s predictions.

See all 231 papers

Frequency-Aware Model Parameter Explorer: A new attribution method for improving explainability
Sep 25, 2025Ali Yavari, Alireza Mohamadi, Elham Beydaghi +2Feature Attribution Perturbation-Based Feature Attribution
Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria · Independent · Dept. of Computer Engineering, Sharif University of Technology, Tehran, Iran +1
Learn to Rank: Visual Attribution by Learning Importance Ranking
Apr 7, 2026David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2Feature Attribution Explainable Image Classification
Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA
When Attribution Patching Lies: Diagnosis and a Second-Order Correction
Jun 5, 2026Luyang Zhang, Jialu WangGradient-Based Attribution LLM Interpretability
1Carnegie Mellon University · University of California, Santa Cruz

Sep 25, 2025J/K move · Enter open · S save

Frequency-Aware Model Parameter Explorer: A new attribution method for improving explainability

Ali Yavari, Alireza Mohamadi, Elham Beydaghi, Philipp Seeböck+1

Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria · Independent · Dept. of Computer Engineering, Sharif University of Technology, Tehran, Iran+1

Feature Attribution Perturbation-Based Feature Attribution Explainable Image Classification Saliency Map Evaluation

Open paper AskPDF arXiv

State-of-the-art attribution methods such as AttEXplore rely on adversarial sample generation that applies an all-pass filter across the frequency spectrum: they perturb all frequencies of the input image alike, from the low frequencies that carry coarse shape to the high frequencies that carry edges and textures. However, models rely on low and high frequencies to different degrees, and such an all-pass perturbation cannot adapt to the model being explained, which limits the faithfulness of the resulting attribution maps. We propose FAMPE (Frequency-Aware Model Parameter Explorer), an attribution method that overcomes this limitation by perturbing the low- and high-frequency bands of the input independently, so that the perturbation can be shifted toward the band that matters for the model. The boundary between the two bands is set for each image automatically from its spectral energy. On ImageNet with five convolutional neural network and Vision Transformer architectures, FAMPE significantly improves the Insertion Score over the strongest baseline, AttEXplore, on four of them, by up to 15.4%, at essentially the same computational cost. How the perturbed frequency bands are weighted matters: the best and the worst balance between them differ by 19-36% in Insertion Score, and the best balance depends on the architecture. Choosing the balance for each image with a probe that needs no ground truth raises the Insertion Score significantly further on all five architectures, reaching up to 24.4% above AttEXplore. Controlling which frequency band is perturbed is therefore a simple, low-cost way to improve attribution faithfulness.

Abstract

Feature Attribution

Explore similar work

Frequency-Aware Model Parameter Explorer: A new attribution method for improving explainability

Learn to Rank: Visual Attribution by Learning Importance Ranking

When Attribution Patching Lies: Diagnosis and a Second-Order Correction