Regularizing Attention Scores with Bootstrapping

Neo Christopher Chung; Maxim Laletin

cs.CVApr 1, 2026

Regularizing Attention Scores with Bootstrapping

Authors: Neo Christopher Chung, Maxim Laletin

Organizations: University of Warsaw · Samsung AI Center, Warsaw

Abstract

Vision transformers (ViT) rely on attention mechanism to weigh input features, and therefore attention scores have naturally been considered as explanations for its decision-making process. However, attention scores are almost always non-zero, resulting in noisy and diffused attention maps and limiting interpretability. Can we quantify uncertainty measures of attention scores and obtain regularized attention scores? To this end, we consider attention scores of ViT in a statistical framework where independent noise would lead to insignificant yet non-zero scores. Leveraging statistical learning techniques, we introduce the bootstrapping for attention scores which generates a baseline distribution of attention scores by resampling input features. Such a bootstrap distribution is then used to estimate significances and posterior probabilities of attention scores. In natural and medical images, the proposed Attention Regularization approach demonstrates a straightforward removal of spurious attention arising from noise, drastically improving shrinkage and sparsity. Quantitative evaluations are conducted using both simulation and real-world datasets. Our study highlights bootstrapping as a practical regularization tool when using attention scores as explanations for ViT. Code available: https://github.com/ncchung/AttentionRegularization

Figures & tables

Figure 1: Histogram of

z

-statistics from the observed attention score sample corresponding to one of the images we use in our analysis and from the derived bootstrap sample.

Panel of Figure 2 — Figure 2: Example of a perturbed image ( n02102040_821.JPEG ) with the attention map before and after regularization via different shrinkage methods: $p$ -thresholding and $l$ -thresholding with thresholds set at the 10-th percentile for $p$ -values and LFDR respectively.

Panel of Figure 3 — Figure 3: Regularization efficiency in ROI for various images from all the categories of the Imagenette validation subset expressed in terms of: a) mean percentile of scores in ROI w.r.t. the whole image; b) percentage of non-zero attention scores in ROI. Each dot denotes the results of using a method for a perturbed image. Thresholding values $p_{\rm th}=0.3$ and $l_{\rm th}=0.3$ . The blue dashed line indicates no regularization.

Figure 4: The suppression factor

D

for different categories of the Imagenette validation subset and different shrinkage methods. Lower

D

corresponds to better regularization.

Panel of Figure 5 — Figure 5: Sensitivity vs. specificity curves for 5 random images from each of the categories of the Imagenette validation subset (denoted with different colors) regularized via $p$ -thresholding ( left ) and $l$ -thresholding ( right ) for 50 values of the corresponding threshold spanning logarithmically from $0$ to $1$ . The dots correspond to the $\pi_{0}$ -threshold value for each curve. The black solid line shows the median of all the curves.

Panel of Figure 6 — Figure 6: Results of the proposed attention regularization on the lung cancer medical images (malignant case 21). The top row shows the histograms of LFDRs, p-values, and z-scores. In the bottom row, regularized attention maps with various $p$ -thresholds are shown alongside the CT scan and the original attention map. In the rightmost, a $\pi_{0}$ threshold is automatically estimated.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure S1: Examples of perturbed images with the attention maps before and after regularization via different shrinkage methods: $p$ -thresholding and $l$ -thresholding with thresholds set at the 10-th percentile for $p$ -values and LFDR respectively, and $\pi_{0}$ -thresholding with $p$ -values. The names of the images, the coordinates of the noise patch (bottom left corner of the patch) and the mean $z$ in ROI are as follows (from top to bottom): n03445777_3192.JPEG , $(200,150)$ , $\langle z\rangle=0.98$ ; n02979186_9450.JPEG , $(100,300)$ , $\langle z\rangle=1.03$ ; n03028079_8361.JPEG , $(58,44)$ , $\langle z\rangle=0.68$ . The noise patch in the image and attention maps is highlighted with a red frame.

Panel of Figure 8 — Figure S2: Example of a perturbed image ( n03000684_5970.JPEG ) with the attention map before and after regularization via $p$ -thresholding (middle row) and $l$ -thresholding (bottom row) with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image. The noise patch in the image and attention maps is highlighted with a red frame $(44,300)$ . The mean $z$ -score in ROI is $\langle z\rangle=0.54$ .

Panel of Figure 9 — Figure S3: Example of a perturbed image ( n03417042_3300.JPEG ) with the attention map before and after regularization via $p$ -thresholding (middle row) and $l$ -thresholding (bottom row) with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image. The noise patch in the image and attention maps is highlighted with a red frame $(50,300)$ . The mean $z$ -score in ROI is $\langle z\rangle=0.88$ .

Panel of Figure 10 — Figure S4: Example of a perturbed image ( n01440764_1310 ) with the attention map before and after regularization via $p$ -thresholding (middle row) and $l$ -thresholding (bottom row) with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image. The noise patch in the image and attention maps is highlighted with a red frame $(300,150)$ . The mean $z$ -score in ROI is $\langle z\rangle=0.8$ .

Panel of Figure 11 — Figure S5: Example of a perturbed image ( n03888257_6331.JPEG ) with the attention map before and after regularization via $p$ -thresholding (middle row) and $l$ -thresholding (bottom row) with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image. The noise patch in the image and attention maps is highlighted with a red frame $(300,300)$ . The mean $z$ -score in ROI is $\langle z\rangle=0.71$ .

Panel of Figure 12 — Figure S6: Mean z-scores in ROI vs. mean percentile of attention scores in ROI w.r.t. the rest of the image before regularization for mean $|z|\leq 1$ (left) and for all of our noise simulation cases (right). Each point corresponds to one perturbed image with color denoting the category of the image from the Imagenette validation dataset.

Panel of Figure 13 — Figure S7: sRSMDs in ROI for $p$ -values vs. the mean percentile in ROI. Each point corresponds to one perturbed image and the color denotes the category of this image. Gray dashed line corresponds to a perfect uniformity of scores, while the red and magenta dashed lines correspond to the values being concentrated around 0 and 1 respectively. See text for more details.

Panel of Figure 14 — Figure S8: Regularization efficiency in ROI for various images from a particular category of the Imagenette validation subset (n02102040) without $z$ -filtering expressed in terms of: a) mean percentile of scores in ROI w.r.t. the whole image before and after regularization; b) percentage of non-zero attention scores in ROI before and after regularization. Each dot denotes the results of using one method for one perturbed image: $p$ -thresholding (red), $l$ -thresholding (yellow), $\pi_{0}$ -thresholding (green). The threshold values for this example are $p_{\rm th}=0.3$ and $l_{\rm th}=0.3$ . The blue dashed line corresponds to the same metric values before and after regularization.

Panel of Figure 15 — Figure S9: Regularization efficiency in ROI for various images from a particular category of the Imagenette validation subset (n03028079) without $z$ -filtering expressed in terms of: a) mean percentile of scores in ROI w.r.t. the whole image before and after regularization; b) percentage of non-zero attention scores in ROI before and after regularization. Each dot denotes the results of using one method for one perturbed image: $p$ -thresholding (red), $l$ -thresholding (yellow), $\pi_{0}$ -thresholding (green). The threshold values for this example are $p_{\rm th}=0.3$ and $l_{\rm th}=0.3$ . The blue dashed line corresponds to the same metric values before and after regularization.

Figure S10: The average suppression factor

D

for different categories of the Imagenette validation subset without

z

-filtering and different shrinkage methods. The lower is

D

the better average regularization is achieved with

D=1

being the case when all the points in Fig. 3(a) located on the dashed blue curve. The relative uncertainty of the results are shown with the black lines.

Panel of Figure 17 — Figure S11: The average suppression factor $D$ for different values of the $B$ hyperparameter ( left ) and for different values of the $\omega$ hyperparameter ( right ) for all the images from the Imagenette validation subset (with the mean $|z|$ in ROI $\leq 1$ ) and different shrinkage methods. The relative uncertainty of the results are shown with the black lines. See text for more details.

Panel of Figure 18 — Figure S12: Attention map before and after regularization via different shrinkage methods ( $p$ -thresholding, $l$ -thresholding and $\pi_{0}$ -thresholding) for the same image as in Fig. 2 ( n02102040_821.JPEG ), but for $B=10$ (top row); Histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image (bottom row). The noise patch in the image and attention maps is highlighted with a red frame.

Panel of Figure 19 — Figure S13: Attention map before and after regularization via different shrinkage methods ( $p$ -thresholding, $l$ -thresholding and $\pi_{0}$ -thresholding) for the same image as in Fig. 2 ( n02102040_821.JPEG ), but for standard deviation of $=4$ (top row); Histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image (bottom row). The noise patch in the image and attention maps is highlighted with a red frame.

Panel of Figure 20 — Figure S14: Examples of bootstrap images generated from the original image ( n02102040_821.JPEG , subfigure a ) via parametric (gaussian) bootstrap ( b ) and non-parametric bootstrap ( c ).

Panel of Figure 21 — Figure S15: Attention map before and after regularization via different shrinkage methods ( $p$ -thresholding, $l$ -thresholding and $\pi_{0}$ -thresholding) for an image ( n02102040_821.JPEG ) perturbed with diffuse gaussian noise, (top row); Histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image (bottom row). The noise patches in the image and attention maps are highlighted with a red frame. See text for more details.

Figure S16: The average suppression factor

D

for different categories of images from the Imagenette validation subset perturbed with diffuse gaussian noise (with the mean

|z|

in ROI

\leq 1

) and different shrinkage methods. The relative uncertainty of the results are shown with the black lines. See text for more details.

Panel of Figure 23 — Figure S17: Attention map before and after regularization via different shrinkage methods ( $p$ -thresholding, $l$ -thresholding and $\pi_{0}$ -thresholding) for the same image as in Fig. 2 ( n02102040_821.JPEG ), but processed with a DINOv2 ViT with patch size $14\times 14$ (top row); Histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image (bottom row). The noise patch in the image and attention maps is highlighted with a red frame.

Panel of Figure 24 — Figure S18: Example of a perturbed image ( n03000684_5970.JPEG ) with the attention map before and after regularization with the DINO v2 ViT (patch size $14\times 14$ ) via $p$ -thresholding (middle row) and $l$ -thresholding (bottom row) with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values in ROI, LFDR values in ROI and z-scores of the observed attention scores (green), bootstrap attention scores (gray) and the attention scores observed in ROI (red) corresponding to the image. The noise patch in the image and attention maps is highlighted with a red frame $(44,300)$ .

Figure S19: The average suppression factor

D

for different categories of images from the Imagenette validation subset processed with the DINO-v2 ViT model with patch size 14 and for different shrinkage methods (without

z

-filtering). The relative uncertainty of the results are shown with the black lines.

Panel of Figure 26 — Figure S20: Regularization efficiency in for various images from the IQ-OTH/NCCD dataset expressed in terms of: a) mean attention scores before and after regularization scaled via min-max normalization b) percentage of non-zero attention scores before and after regularization. Each dot denotes the results of using one method for one perturbed image: $p$ -thresholding (red), $l$ -thresholding (yellow), $\pi_{0}$ -thresholding (green). For each point we use the corresponding median of the $p$ -value and LFDR distributions as the respective regularization threshold. The blue dashed line corresponds to the same metric values before and after regularization.

Panel of Figure 27 — Figure S21: The examples of applying bootstrap regularization to different images from the IQ-OTH/NCCD dataset from top to bottom: malignant case (23 and 90), benign case (28 and 44) and normal case (9 and 18). The original image is displayed on the left of each row and is succeeded by the attention map before regularization and the attentions maps after: $p$ -thresholding, $l$ -thresholding and $\pi_{0}$ -thresholding regularizations. For $p$ - and $l$ -thresholds we use the median values of the respective statistic distributions.

Panel of Figure 28 — Figure S22: Results of applying $p$ -thresholding (second row) and $l$ -thresholding (third row) regularizations to the malignant case 21 image with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values, LFDR values and z-scores of the observed attention scores (green) and bootstrap attention scores (gray). The two bottom rows display the attention scores after regularization vs. before regularization for the regularized attention maps considered above, for $p$ -thresholding (red) and $l$ -thresholding (gold). The blue dashed line corresponds to the same attention score values before and after regularization.

Panel of Figure 29 — Figure S23: Results of applying $p$ -thresholding (second row) and $l$ -thresholding (third row) regularizations to the malignant case 73 image with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values, LFDR values and z-scores of the observed attention scores (green) and bootstrap attention scores (gray). The two bottom rows display the attention scores after regularization vs. before regularization for the regularized attention maps considered above, for $p$ -thresholding (red) and $l$ -thresholding (gold). The blue dashed line corresponds to the same attention score values before and after regularization.

Panel of Figure 30 — Figure S24: Results of applying $p$ -thresholding (second row) and $l$ -thresholding (third row) regularizations to the benign case 5 image with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values, LFDR values and z-scores of the observed attention scores (green) and bootstrap attention scores (gray). The two bottom rows display the attention scores after regularization vs. before regularization for the regularized attention maps considered above, for $p$ -thresholding (red) and $l$ -thresholding (gold). The blue dashed line corresponds to the same attention score values before and after regularization.

Panel of Figure 31 — Figure S25: Results of applying $p$ -thresholding (second row) and $l$ -thresholding (third row) regularizations to the normal case 17 image with the respective thresholds varying from $0.2$ to $0.6$ . The last attention map in each of the rows to the right corresponds to $\pi_{0}$ -thresholding. The top row shows the histograms of p-values, LFDR values and z-scores of the observed attention scores (green) and bootstrap attention scores (gray). The two bottom rows display the attention scores after regularization vs. before regularization for the regularized attention maps considered above, for $p$ -thresholding (red) and $l$ -thresholding (gold). The blue dashed line corresponds to the same attention score values before and after regularization.

Explore similar work

HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives
Sep 1, 2026Sathiyamohan Nishankar, Pubudu Sanjeewani, Asanka Perera +1Transformer Interpretability Vision Transformer
Faculty of Engineering, University of Peradeniya, Sri Lanka · School of Computing Technologies, RMIT University, Melbourne, Australia · School of Engineering & Digital Technologies, University of Southern Queensland, Brisbane, Australia +1
Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations
Jul 26, 2026Xianghao Jiao, Ruoyu Chen, Wei Wang +6Feature Attribution Image Classification
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · University of Chinese Academy of Sciences, Beijing, China · School of Automation, Nanjing University of Information Science and Technology +2
What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers
Mar 2, 2026Qin Su, Tie LuoTransformer Interpretability Explainable Image Classification
Department of Electrical and Computer Engineering University of Kentucky, Lexington, KY, USA

Sep 1, 2026J/K move · Enter open · S save

HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives

Sathiyamohan Nishankar, Pubudu Sanjeewani, Asanka Perera, Selvarajah Thuseethan

Faculty of Engineering, University of Peradeniya, Sri Lanka · School of Computing Technologies, RMIT University, Melbourne, Australia · School of Engineering & Digital Technologies, University of Southern Queensland, Brisbane, Australia+1

Transformer Interpretability Vision Transformer Layer-Wise Relevance Propagation

Open paper AskPDF arXiv

Vision Transformer (ViT) design has become increasingly diverse, with backbones combining convolutional stems, windowed, linear, or multi-axis attention, patch merging, and spatial reduction in various configurations. This diversity poses challenges for existing attribution methods, whose assumptions often do not hold across ViT variants: Grad-CAM requires a terminal spatial feature map, attention rollout assumes global softmax attention, and layer-wise relevance propagation (LRP) requires module-specific rules. To the best of our knowledge, no existing method provides a unified attribution framework across this architectural space. We show that this architectural diversity can be captured by a simpler underlying structure. The attention and resolution-reduction operators in current ViTs can be decomposed into four operation types: linear maps, bilinear mixing, normalization or gating, and reindexing. Each operation admits a relevance rule that satisfies conservation. Based on these rules, HiLRP supports new backbones by construction rather than by architecture-specific derivation, and its attribution maps decompose the prediction rather than relying on heuristic assumptions. We prove conservation and conditional equivariance and verify both to machine precision. Across 14 attribution methods and 10 architectures, we find that no prior method remains reliable across ViT families, while Faithfulness Correlation becomes uninformative for backbones robust to spatial masking. HiLRP alone preserves conservation across windowed, spatial-reduction, multi-axis, and linear-attention models, where naive extensions can produce zero or inflated relevance. It also localizes attribution failures in class activation mapping, achieving 0.97 Pointing compared with 0.55 for competing methods on EfficientViT.

Abstract

Explore similar work

HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives

Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations

What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers