Frequency-Aware Model Parameter Explorer: A new attribution method for improving explainability
Authors: Ali Yavari, Alireza Mohamadi, Elham Beydaghi, Philipp Seeböck, Rainer A. Leitgeb
Organizations: Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria · Independent · Dept. of Computer Engineering, Sharif University of Technology, Tehran, Iran · MANO Group, CIR Lab, Dept. of Biomedical Imaging and Image-Guided Therapy, Comprehensive Center for AI in Medicine, Medical University of Vienna, Vienna, Austria
State-of-the-art attribution methods such as AttEXplore rely on adversarial sample generation that applies an all-pass filter across the frequency spectrum: they perturb all frequencies of the input image alike, from the low frequencies that carry coarse shape to the high frequencies that carry edges and textures. However, models rely on low and high frequencies to different degrees, and such an all-pass perturbation cannot adapt to the model being explained, which limits the faithfulness of the resulting attribution maps. We propose FAMPE (Frequency-Aware Model Parameter Explorer), an attribution method that overcomes this limitation by perturbing the low- and high-frequency bands of the input independently, so that the perturbation can be shifted toward the band that matters for the model. The boundary between the two bands is set for each image automatically from its spectral energy. On ImageNet with five convolutional neural network and Vision Transformer architectures, FAMPE significantly improves the Insertion Score over the strongest baseline, AttEXplore, on four of them, by up to 15.4%, at essentially the same computational cost. How the perturbed frequency bands are weighted matters: the best and the worst balance between them differ by 19-36% in Insertion Score, and the best balance depends on the architecture. Choosing the balance for each image with a probe that needs no ground truth raises the Insertion Score significantly further on all five architectures, reaching up to 24.4% above AttEXplore. Controlling which frequency band is perturbed is therefore a simple, low-cost way to improve attribution faithfulness.
Figures & tables
Figure 1: Comparison between frequency-based SOTA attribution methods and our proposed approach. Top: Existing methods such as AttEXplore apply a single all-pass multiplicative perturbation to the whole spectrum, so every frequency band is perturbed alike and no band can be weighted separately. Bottom: FAMPE splits the spectrum at an energy-defined cutoff and perturbs the low-frequency band (capturing the object’s overall shape, e.g., the dog’s silhouette) and the high-frequency band (capturing edges and fine details) with two independently sampled fields weighted by α and 1−α . We use a centered Fourier spectrum because the cutoff is defined as a radius from the DC component; the same construction could be built on a DCT. In this example the resulting map is more closely aligned with the object; the population-level comparison is given in Section 4.2 .
Figure 2: Overview of FAMPE’s frequency-aware adversarial sample generation. Stage 1: input image (a) is transformed via FFT to obtain the centered Fourier spectrum (b). Stage 2: the spectrum is independently modulated by a low-frequency Gaussian mask scaled by α (c, top) and a high-frequency Gaussian mask scaled by 1−α (d, bottom), each combined with multiplicative Gaussian noise N(1,σ) . Stage 3: modulated spectra are summed and mapped back via IFFT, yielding the reconstructed perturbed image (e). The parameter α controls the relative perturbation per spectral band.
Method
Inception-v3
ResNet-50
VGG-16
MaxViT-T
ViT-B/16
Mean
Ins.
Del.
Ins.
Del.
Ins.
Del.
Ins.
Del.
Ins.
Del.
Ins.
Del.
IG
0.1990
0.1650
0.1853
0.1402
0.1360
0.1123
0.3342
0.2893
0.2547
0.2197
0.2219
0.1853
DeepLIFT
0.1975
0.1769
0.1820
0.1637
0.1415
0.1170
0.3360
0.2890
0.2560
0.2200
0.2226
0.1933
GIG
0.2076
0.1483
0.1912
0.1202
0.1409
0.0916
0.3506
0.2606
0.2544
0.1967
0.2289
0.1635
AGI
0.3839
0.1169
0.3100
0.0975
0.2487
0.0774
0.2928
0.3011
0.4350
0.1861
0.3341
0.1558
AttEXplore
0.5316
0.1027
0.3448
0.1068
0.2795
0.0899
0.3829
0.2488
0.4766
0.1572
0.4031
0.1411
Table 1: Performance of our method, FAMPE, compared to five competing approaches across five deep learning architectures (n=1000 images per cell). Evaluation uses Insertion Score ( ↑ better) and Deletion Score ( ↓ better); bold marks the best score in each column, excluding the oracle upper bound. α∗ is the best fixed α per architecture: 0.4 (Inception-v3), 0.7 (ResNet-50), 0.8 (VGG-16), 0.1 (MaxViT-T), 0.8 (ViT-B/16). Selected α is a deployable per-image choice made by a cheap 2-step/5-probe insertion probe with no access to ground truth, at 2.87× the cost of AttEXplore; oracle α is a non-deployable upper bound — the per-image maximum Insertion and minimum Deletion achievable in the full 11-point sweep , each chosen independently, so the two may come from different α for the same image; unobtainable without the full evaluation score itself — reported only to bound the remaining headroom. FAMPE (all-pass) is an ablation with the frequency split disabled ( α=1 , low-pass mask forced to all-ones) and reproduces AttEXplore on every architecture and both metrics, indicating that the gains are not explained by other differences between the two implementations.
Figure 3: Qualitative comparison of attribution maps from FAMPE, AGI, and AttEXplore on three ImageNet samples of varying spectral complexity: (A) a strawberry on MaxViT-T (low-frequency content, small cf ), (B) an African hunting dog on MaxViT-T (mixed mid- to high-frequency content, intermediate cf ), and (C) a ruddy turnstone on Inception-v3 (high-frequency content, large cf ). For each panel, AGI and AttEXplore maps are followed by 11 FAMPE maps with α from 0 (purely high-frequency perturbation) to 1 (purely low-frequency perturbation). On these two architectures ( α∗=0.1 and 0.4 ), lower- α FAMPE maps place mass more tightly on the target object than the baselines.
Figure 4: Joint distribution of the per-image cutoff cf and the best- α value (maximizing Insertion Score) across 1,000 ImageNet samples on MaxViT-T; cf is recomputed with the estimator FAMPE uses at attribution time, on the preprocessed image at the network’s input size. The central heatmap shows image counts per (cf,α) bin (darker = higher density), with marginal histograms of cutoff radii (top) and best- α values (right). Both marginals are skewed — most images have a small cf , and most are best served by a low α — but the two are close to independent (upper-left quadrant: 410 images observed vs. 429 expected; Cramér’s V=0.12 ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Attribution maps for low-frequency images (small cf ). Each panel shows the input image, the AGI and AttEXplore maps, and the FAMPE maps for α=0,0.1,…,1 .
Figure 2: Attribution maps for middle-frequency images (intermediate cf ). Layout as in Fig. 1 .
Figure 3: Attribution maps for high-frequency images (large cf ), part 1. Layout as in Fig. 1 .
Figure 4: Attribution maps for high-frequency images (large cf ), part 2. Layout as in Fig. 1 .
Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models. Code and pretrained models are available at https://github.com/dschinagl/AHA.
David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2
Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA
Attribution methods in explainable AI rely on upsampling techniques that were designed for natural images, not saliency maps. Standard bilinear and bicubic interpolation systematically corrupts attribution signals through aliasing, ringing, and boundary bleeding, producing spurious high-importance regions that misrepresent model reasoning. We identify that the core issue is treating attribution upsampling as an interpolation problem that operates in isolation from the model's reasoning, rather than a mass redistribution problem where model-derived semantic boundaries must govern how importance flows. We present Universal Semantic-Aware Upsampling (USU), a principled method that reformulates upsampling through ratio-form mass redistribution operators, provably preserving attribution mass and relative importance ordering. Extending the axiomatic tradition of feature attribution to upsampling, we formalize four desiderata for faithful upsampling and prove that interpolation structurally violates three of them. These same three force any redistribution operator into a ratio form; the fourth selects the unique potential within this family, yielding USU. Controlled experiments on models with known attribution priors verify USU's formal guarantees; evaluation across ImageNet, CIFAR-10, and CUB-200 confirms consistent faithfulness improvements and qualitatively superior, semantically coherent explanations.
Vincenzo Buono, Peyman Sheikholharam Mashhadi, Mahmoud Rahat +2
Post-hoc feature attribution methods are widely deployed in safety-critical vision systems, yet their stability under realistic input perturbations remains poorly characterized. Existing metrics evaluate explanations primarily under additive noise, collapse stability to a single scalar, and fail to condition on prediction preservation, conflating explanation fragility with model sensitivity. We introduce the Feature Attribution Stability Suite (FASS), a benchmark that enforces prediction-invariance filtering, decomposes stability into three complementary metrics: structural similarity, rank correlation, and top-k Jaccard overlap-and evaluates across geometric, photometric, and compression perturbations. Evaluating four attribution methods (Integrated Gradients, GradientSHAP, Grad-CAM, LIME) across four architectures and three datasets-ImageNet-1K, MS COCO, and CIFAR-10, FASS shows that stability estimates depend critically on perturbation family and prediction-invariance filtering. Geometric perturbations expose substantially greater attribution instability than photometric changes, and without conditioning on prediction preservation, up to 99% of evaluated pairs involve changed predictions. Under this controlled evaluation, we observe consistent method-level trends, with Grad-CAM achieving the highest stability across datasets.