Pretrained transformers spend a large share of their inference compute inside self-attention, yet the per-head query-key interaction is structurally low-rank by construction. We ask a diagnostic question: across layers and architectures, which heads actually use the full rank budget on real data, and which do not? We propose SVD-Attention, a post-hoc method that decomposes the per-head interaction matrix Mh and scores every singular mode by both its spectral magnitude and its activation variance across tokens on a small calibration set. The score keeps the modes the head can amplify and that real data actually activates, and discards the rest; the resulting factorization is exact for the retained modes and needs no finetuning. Across seven pretrained models and five task types, the diagnostic finds a task-dependent pattern: redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. It is also predictive: truncating the layers it flags costs at most 1.75 F1, while truncating the same number of layers elsewhere in the same model costs up to 50 F1. SVD-Attention is an analysis tool rather than a compression method: acting on it reduces inference FLOPs by 0.8-1.5% on its own and by 8-11% when composed with weight pruning, with 0-4% change in task quality.
Figures & tables
Figure 1: Conceptual representation of our proposed SVD-Attention is shown in (a). The SVD-Attention method is outlined in (b).
(a) ViT-B, ImageNet-1K, layers {6,7,8} (Top-1 %)
Method
ϵ=0.1
ϵ=0.2
ϵ=0.3
Singular value only
77.84
77.13
76.12
SVD-Attention (data-aware)
77.94
77.62
76.86
Gap
+0.10
+0.49
+0.74
Table 1: Ablation of the ranking rule, holding the unit of decomposition and the layer selection fixed. (a) On ViT-B the data-aware score wins at every ϵ and the gap widens as more modes are discarded. (b) On RoBERTa-base the singular-value-only rule keeps roughly ten more modes per head and still loses up to 39 F1.
Model
Layers
ϵ
Metric
Δ FLOPs
DistilBERT on SST-5 ( Accuracy % )
Baseline
—
—
49.95
—
WANDA-only
—
—
49.82
- 9.65%
Ours
{0,1,2}
0.1
50.95
- 0.83%
Ours + WANDA
{0,1,2}
0.1
51.09
- 10.39%
RoBERTa-base on NewsQA ( F1 )
Table 2: Main results across seven pretrained transformers and five task types; settings whose metrics do not fit this format are reported in Tab. 3 . Ours applies SVD-Attention (data-aware SVD truncation); Ours + WANDA applies WANDA pruning at sparsity 0.1 on top. Metric measures task quality in units written in bold . Δ FLOPs is the % reduction in inference FLOPs relative to baseline.
(a) Extractive QA at ϵ=0.3 (F1)
Model / dataset
Baseline
Flagged
Non-flagged
RoBERTa / SQuAD 2.0
83.01
82.85{8..11}
52.23{4..7}
RoBERTa / NewsQA
70.81
69.06{8..11}
58.85{0..3}
MiniLM / SQuAD 2.0
77.77
77.12{9..11}
69.98{6..8}
MiniLM / NewsQA
56.03
55.58{9..11}
05.94{3..5}
Table 3: Flagged against non-flagged layer groups at matched ϵ , for the settings whose metrics do not fit the format of Tab. 2 . Flagged is the layer group the diagnostic selects; Non-flagged truncates the same number of layers elsewhere in the same model. Layer sets in braces.
Layers
Top-1 (%)
SVD Δ FLOPs
+ WANDA
{0,1,2}
77.22
−2.47%
−11.67%
{3,4,5}
77.92
−1.09%
−10.43%
{6,7,8}
77.94
−0.76%
−10.13%
{9,10,11}
71.63
−0.62%
−10.00%
{0,…,5}
71.07
−3.56%
−12.66%
{6,…,11}
67.60
−1.38%
−10.69%
Table 4: ViT-B layer-wise sweep on ImageNet-1K at ϵ=0.1 . Truncating middle layers {6,7,8} preserves Top-1 within 3.14 points of the baseline (81.08). Pushing truncation to layers {9,10,11} or wider drops accuracy sharply.
Layers
Acc. (%)
SVD Δ FLOPs
+ WANDA
{0,1}
50.63
−1.79%
−11.26%
{2,3}
49.64
−1.53%
−11.03%
{0,1,2}
50.95
−0.83%
−10.39%
{3,4,5}
50.62
−1.57%
−11.06%
{0,…,5}
50.50
−4.15%
−13.38%
Table 5: DistilBERT layer-wise sweep on SST-5 at ϵ=0.1 . Several layer choices land slightly above the baseline (49.95), including the flagged group {0,1,2} ; we read this as quality being preserved rather than improved.
Layers
Top-1 (%)
SVD Δ FLOPs
+ WANDA
{0,…,3}
84.68
−1.70%
−11.06%
{4,…,7}
84.68
−1.29%
−10.65%
{8,…,11}
84.64
−0.62%
−10.05%
{12,…,15}
84.24
−0.58%
−10.01%
{16,…,19}
83.41
−0.50%
−9.94%
{20,…,23}
80.88
−0.54%
−9.98%
Table 6: Swin-B layer-wise sweep on ImageNet-1K at ϵ=0.1 . Accuracy is well preserved in the first three blocks (layers 0-11) and degrades monotonically as truncation moves to later blocks.
Layers
mIoU
SVD Δ FLOPs
+ WANDA
{0,…,3}
44.29
−2.00%
−8.82%
{4,…,7}
45.38
−1.55%
−8.40%
{8,…,11}
45.73
−1.52%
−8.37%
{12,…,15}
45.95
−1.70%
−8.54%
{16,…,19}
46.02
−1.54%
−8.40%
{20,…,23}
45.99
−1.01%
−7.91%
Table 7: DPT-Large layer-wise sweep on ADE20K at ϵ=0.1 . mIoU rises monotonically as the truncation window moves later, the opposite of ViT-B and Swin-B; late layers {16,…,19} preserve baseline mIoU.
RoBERTa-base, ϵ=0.3 (F1)
Layers truncated
calib 1%
calib 15%
{8,…,11} (flagged redundant)
56.25
56.03
{0,…,3}
51.55
53.34
{4,…,7} (flagged critical)
25.87
25.49
MiniLM, ϵ=0.6 (F1)
{9,10,11} (flagged redundant)
37.45
Table 8: Robustness to distribution shift. The diagnostic is calibrated on SQuAD 2.0 and evaluated on NewsQA. On MiniLM the flagged layers hold up even when 60% of the importance mass is discarded, while truncating the critical layers {6,7,8} costs over 42 F1.
Layers truncated
Accuracy (mean ± std)
Min-Max
{0,1} (flagged)
45.73±0.06
45.70 - 45.84
{2,3}
44.49±0.16
44.21 - 44.57
{4,5}
44.25±0.12
44.07 - 44.39
Table 9: Robustness to the calibration draw. DistilBERT on SST-5 at ϵ=0.9 , mean over five independent draws of the same size.
Calib. size (share)
{0,…,3}
{4,…,7}
{8,…,11}
100 ( 0.08% )
76.87
56.95
83.10
1,303 ( 1% )
77.06
52.23
82.85
6,516 ( 5% )
75.23
51.76
82.86
13,032 ( 10% )
75.41
53.51
82.97
19,548 ( 15% )
75.58
53.57
82.95
Table 10: Robustness to calibration size. RoBERTa-base on SQuAD 2.0 at ϵ=0.3 (F1, baseline 83.01 ). Bold column is the layer group the diagnostic flags as redundant.
ϵ=0.1
ϵ=0.2
ϵ=0.3
F1
83.24
83.09
82.85
Mean retained rank rh
49
38
29
Table 11: RoBERTa-base on SQuAD 2.0, flagged layers {8,…,11} (baseline F1 83.01 , dh=64 ). The quality-versus- ϵ frontier is smooth: more than half of the average head’s rank budget is discarded at ϵ=0.3 for a loss of 0.16 F1.
Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1