cs.LGNov 3, 2026
SaveExploiting Biases in Attention for Transformers
Organizations: LTCI, Télécom Paris, Institut Polytechnique de Paris, France · University of Turin, Italy · CNR-ISASI, Italy
Abstract
Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phenomenon becomes increasingly pronounced with network depth. This can be interpreted as saying that tokens have an absolute importance for the attention process, independently of queries. Inspired by this, we propose a reformulation of the attention formula that makes this phenomenon explicit by adding an input-dependent bias to the query-key dot product. We show that this new term gives each token a scalar value indicating its anticipated influence on other tokens during the attention process, and can thus serve as a proxy for pruning attention maps. Across multiple transformer model sizes, this allows us to substantially reduce the computational cost of attention, in terms of both FLOPs and inference latency, while retaining most of the original classification accuracy. We publicly released the code and models at https://github.com/EIDOSLAB/AttentionWithBias.
Figures & tables
Figure 1: Bias-based column pruning. The learned bias provides a query-independent importance score for each token, allowing low-bias columns to be removed before computing keys and values. Attention is then evaluated only over the retained set , reducing its size from to .
Figure 2: Bias-based pruning pipeline. The importance score is computed first (shaded) and selects the kept set . All queries are kept, but keys and values are computed only for the kept tokens, so the pruned columns are never materialized.
Figure 3: Attention maps at different depths of the pretrained DeiT-S model. Vertical lines naturally emerge in the last layers.
Figure 4: Decomposition of the attention logits at different network depths.
Figure 5: Learned bias versus the per-column mean of the query-key logits , at three depths. The correlation grows with depth ( reported per panel).
Figure 6: Layer-wise Pearson correlation between the learned bias and the mean attention received by each column.
| Model | Pruning schedule | Accuracy | FLOPs (G) | Latency (ms) | |
| Top-1 (%) | Top-5 (%) | ||||
| DeiT-Ti | - | ||||
| DeiT-Ti b | ✗ | ||||
| DeiT-Ti b | |||||
| DeiT-S | - | ||||
| DeiT-S b | ✗ | ||||
Table 1: Accuracy and efficiency of bias-based attention pruning on ImageNet-1k. End-to-end inference latency, including bias computation, top- selection, index gathering, and all remaining model operations, is measured with the math attention backend on a single image on one NVIDIA A40 GPU.
| Model | Pruning schedule | COCO accuracy | Backbone only | Full detector | |||
| AP | AP | GFLOPs | Latency (ms) | GFLOPs | Latency (ms) | ||
| DeiT-S | – | 43.3 | 38.7 | 485.60 | 1171.68 | ||
| DeiT-S b | 42.7 | 38.1 | 331.17 | 1017.25 | |||
Table 2: Object detection and instance segmentation on COCO val2017 . Both models use Mask R-CNN with a DeiT-S backbone and the same ViTDet-style training recipe. GFLOPs and end-to-end latency are reported for the isolated backbone and the complete detector. Latency is measured in full precision with the math attention backend on one NVIDIA A40 GPU using a single image.
Figure 7: Accuracy-efficiency trade-off for ToME [ 2 ] , our bias-based column pruning (star), and their combination on DeiT-S. The dashed line denotes the original DeiT-S accuracy.
| Method | Pruning | Head | Top-1 | FLOPs | Latency |
| schedule | pruning | (%) | (G) | (ms) | |
| DeiT-S b (ours) | – | ||||
| Structured pruning | – | ✗ | |||
| Structured pruning | – | ✓ |
Table 3: Comparison with -norm structured pruning on DeiT-S. Latency is measured with the math attention backend at resolution on one NVIDIA A40 GPU.
| Training pruning | Inference pruning | Accuracy | |
| Top-1 (%) | Top-5 (%) | ||
| ✗ | ✗ | ||
| ✗ | ✓ | ||
| ✓ | ✗ | ||
| ✓ | ✓ | ||
Table 4: Effect of applying top- pruning during training and inference on ImageNet-1k. All configurations use DeiT-S b and the keep-ratio schedule whenever pruning is enabled.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | |
| Epochs | 300 |
| Batch size | 1 024 |
| Optimizer | AdamW |
| Learning rate | |
| LR decay | cosine |
| Warmup epochs | 5 |
Table 5: Training recipe used for the ImageNet100 experiments. RandAugment “2/9” denotes 2 operations at magnitude 9, following the torchvision implementation. Mixup ( ) and Cutmix ( ) intensities follow the PyTorch convention: is the parameter of the symmetric Beta distribution from which the mixing coefficient is sampled.
| Model | Pruning schedule | Accuracy | FLOPs (G) | Latency (ms) | |
| Top-1 (%) | Top-5 (%) | ||||
| DeiT-Ti | - | ||||
| DeiT-Ti b | |||||
| DeiT-Ti b | |||||
| DeiT-Ti b | |||||
Table 6: Accuracy and efficiency of bias-based attention pruning on ImageNet-1k with different pruning schedules. Inference latency is measured with the math attention backend on a single image on one NVIDIA A40 GPU.
| Method | Selection rule | Pruning ratio (%) | Accuracy | |
| Top-1 (%) | Top-5 (%) | |||
| No pruning | – | 0 | 80.04 | 93.30 |
| Probability threshold | 49.55 | 79.66 | 93.18 | |
| Fixed top- | 50.00 | 79.70 | 92.30 | |
| Layer-wise top- | 50.00 | 80.48 | 93.18 | |
Table 7: Ablation of the column-selection strategy on ImageNet100 using ViT-S b . The pruning ratio denotes the fraction of removed attention columns.
| Model | Probability threshold | Pruning ratio (%) | Accuracy | |
| Top-1 (%) | Top-5 (%) | |||
| ViT-Ti | – | 0 | 79.08 | 92.90 |
| ViT-Ti b | ✗ | 0 | 78.40 | 92.82 |
| ViT-Ti b | 48.00 | 78.56 | 93.14 | |
| ViT-S | – | 0 | 80.12 | 92.54 |
| ViT-S b | ✗ | 0 | 80.04 | 93.30 |
Table 8: Results of the probability-threshold pruning strategy on ImageNet100. The pruning ratio denotes the fraction of removed attention columns and is measured at the end of training.
| Model | Top- value | Pruning ratio (%) | Accuracy | |
| Top-1 (%) | Top-5 (%) | |||
| ViT-Ti | – | 0 | 79.08 | 92.90 |
| ViT-Ti b | ✗ | 0 | 78.40 | 92.82 |
| ViT-Ti b | 50 | 78.74 | 92.94 | |
| ViT-Ti b | 50 | 78.68 | 92.80 | |
| ViT-S | – | 0 | 80.12 | 92.54 |
Table 9: Results of the top- pruning strategy on ImageNet100. The pruning ratio denotes the fraction of removed attention columns. The top- value is expressed as the fraction of tokens that are kept.
| Model | Query-key-free blocks | Accuracy | |
| Top-1 (%) | Top-5 (%) | ||
| ViT-S | – | ||
| ViT-S b | ✗ | ||
| ViT-S b | 4 | ||
| ViT-S b | 6 | ||
| ViT-S b | 8 | ||
Table 10: ImageNet100 accuracy when replacing the query–key term with the broadcast bias in the last transformer blocks. “Query-key-free blocks” denote the number of final blocks using only .
| math backend | efficient backend | |||
| Model | Original | Bias-pruned | Original | Bias-pruned |
| DeiT-Ti | ||||
| DeiT-S | ||||
| DeiT-B | ||||
| ViT-L (MAE) | ||||
Table 11: Inference latency (ms) of the original and bias-pruned models using the math and efficient attention backends. Measurements use a single image on one NVIDIA A40 GPU. Values in parentheses report the speedup of the pruned model over its corresponding original model.
| Model | Original | Bias-pruned |
| DeiT-Ti | ||
| DeiT-S | ||
| DeiT-B | ||
| ViT-L (MAE) |
Table 12: CPU inference latency (ms) of the original and bias-pruned models. Measurements use one image on an AMD EPYC 7402 processor with 12 CPU threads and the math attention backend. We report the mean and standard deviation over runs after warm-up iterations. Parentheses indicate the speedup over the corresponding original model.
Figure 8: Global versus query-dependent column selection in the original pretrained DeiT-S, averaged over all attention heads and ImageNet-1K validation images. (a–c) Retained attention mass at different keep ratios. (d) Oracle gap across all evaluated keep ratios.
Figure 9: Layer-wise Pearson correlation between the learned bias and the pre-softmax per-column mean . The correlation grows steadily with depth, reaching in the final layer, confirming that the bias serves as an effective and cheap surrogate of column importance.
Figure 10: Attention maps for DeiT-Ti at layers (columns), head 2, across several ImageNet validation images (rows). In all cases the attention evolves from a diagonal, locally-concentrated pattern in early layers to strongly column-sparse maps in the final blocks, where a small number of tokens attract the attention of virtually all queries.
Figure 11: Attention maps for DeiT-S at layers (columns), head 4, across several ImageNet validation images (rows). The transition from diagonal to vertical attention structure is visible and consistent across all images, confirming that column sparsity is a systematic property of the attention mechanism rather than an image-specific artifact.
Figure 12: Attention maps for DeiT-B at layers (columns), head 7, across several ImageNet validation images (rows). The same depth-wise evolution observed in DeiT-Ti and DeiT-S is reproduced here, showing that column sparsity is not specific to a particular model width.
Figure 13: Attention maps for the MAE-pretrained ViT-L at layers (columns), head 11, across several ImageNet validation images (rows). Despite its greater depth and different pretraining strategy, ViT-L exhibits the same transition from predominantly diagonal attention in early blocks to increasingly column-sparse patterns in deeper blocks.
Figure 14: Decomposition of the attention logits ( , , ) for DeiT-Ti b at layers , head 2. In the early layers, dominates with a strong diagonal pattern. With increasing depth the bias progressively captures the vertical structures, until in the final layers it accounts for most of the visible structure in the combined logits .
Figure 15: Decomposition of the attention logits for DeiT-S b at layers , head 2. The bias term increasingly dominates the combined logits with depth, consistently with Fig. 4 and Fig. 5 of the main paper.
Figure 16: Decomposition of the attention logits for DeiT-B b at layers , head 2. The depth-wise takeover of the bias term is visible across all three model sizes, confirming that the learned bias captures a universal property of deep vision-transformer attention.
Figure 17: Decomposition of the attention logits ( , , and ) for MAE-pretrained ViT-L b at layers , head 2. The model is fine-tuned with the bias term but without top- pruning. As depth increases, the bias progressively captures the vertical, query-independent structure of the combined logits, reproducing the behavior observed for DeiT.
Figure 18: Scatter plots of the learned bias versus for DeiT-Ti b at layers (columns), head 2, across multiple validation images (rows). The Pearson correlation reported per panel grows steadily with depth and remains consistent across images, confirming that the bias provides a reliable surrogate across different input images of column importance.
Figure 19: Scatter plots for DeiT-S b at layers , head 2, across multiple validation images. The correlation reaches in the final layer, independently of the input image, showing that the bias approximates the query-averaged column importance with high fidelity in the deeper blocks.
Figure 20: Scatter plots for DeiT-B b at layers , head 2, across multiple validation images. The depth-wise growth of is reproduced in the larger model, confirming that the relationship between the bias and the query-averaged column importance is a property of the attention mechanism rather than of a specific model scale.
Figure 21: Learned bias versus the per-column mean for MAE-pretrained ViT-L b at layers (columns), head 2, across multiple ImageNet validation images (rows). The Pearson correlation reported in each panel becomes stronger with depth, showing that the bias also learns a reliable proxy for query-averaged column importance in ViT-L.
More in this topic
Transformer Attention
Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.
See all 331 papers