While representation similarity across independently trained language models is well-documented, how internal mechanics such as attention behave across models remains far less characterized. Inspired by this gap, we examine the structure of post-softmax attention weights by marginalizing over query positions, mapping them into a joint token-head "marginal attention space". Evaluating across 60+ diverse LLMs, we find that different properties emerge when reducing this space along its token and head axes. When reduced token-wise, marginal attention yields a text-intrinsic signal robustly conserved across models. To explain this property, we empirically connect marginal attention to the input-output Jacobian of the network, and prove theoretically that under a smoothness assumption, models with similar next-token distributions are guaranteed to have similar input-output Jacobian statistics. When reduced head-wise, it forms a model-private signature conserved across documents. Practically, this provides a natural way to estimate a per-head budget for key-value (KV) cache eviction, effectively decoupling model-specific budget allocation from text-intrinsic token scoring. On standard eviction benchmarks, a per-head budget precomputed offline on pretraining text, combined with a training-free token score, shows competitive performance with methods that recompute the budget on every document or train it per target. Code available at https://github.com/Flegyas/marginal-attention
Figures & tables
Figure 1: Marginal attention on its two axes. Averaged over query positions, the attention of Qwen2.5-0.5B on one document is a matrix M of 336 heads by tokens. Its per-token reduction (green) agrees across 67 models on the same document ( ρˉ=0.87 ), and its per-head reduction (lavender) on one document matches the model’s average over its other documents ( ρˉ=0.95 ).
Figure 2: Per-token marginal attention across models. Spearman correlation of two profiles on one document; 67 models, 850 documents. Left: models against the consensus of the others (top) and all 2211 pairs (bottom); dashed line the agreement between 10 pretraining seeds of pythia-410m. Right: pairs grouped by pair class, most related on the left.
Figure 3: Per-head marginal attention within and between models. Spearman correlation between per-head profiles; 67 models, 170 documents.
Figure 4: What the per-token marginal attention tracks. Spearman correlation per document. Left: with the per-token marginal influence of Qwen2.5-0.5B over 85 documents. Right: word properties regressing the unweighted ( ω≡1 ) consensus profile over 550 documents.
Figure 5: Within a tokenizer group, pairs with a smaller logit gap δ agree more on per-token marginal influence (left) and per-token marginal attention (right, ω≡1 ). Colour: pair class; marker: tokenizer; lines: least-squares fits; bars: 95% intervals. Estimator in Section E.1 .
cost
Qwen3-8B
Llama-3.1-8B
phi-4
Qwen3-32B
Mistral-7B
offline
online
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
4 ×
8 ×
16 ×
AdaKV + EA
88.4
72.8
46.5
75.7
45.3
15.2
68.7
44.0
13.8
81.9
63.6
36.4
67.2
34.0
19.9
AdaKV + KVzap
△△
94.0
89.2
49.9
94.5
89.3
63.1
87.8
77.0
51.6
95.4
90.0
50.3
88.3
70.2
42.1
KVzip
∙∙
95.0
92.3
59.4
95.3
92.8
74.2
91.5
89.6
68.8
95.6
95.6
91.8
91.2
89.2
72.2
LU-KV + EA
△
90.6
75.2
58.4
95.1
91.8
67.0
92.4
87.9
69.5
90.2
72.6
38.2
91.2
85.1
61.7
HM + EA
△
95.0
91.8
73.3
95.1
92.3
66.3
91.5
87.8
68.5
95.2
93.2
79.1
90.5
84.3
65.3
Table 1: Eviction scores. RULER 4096 score over its 13 tasks and documents at 4 × , 8 × and 16 × compression. Cost: △ offline work per target, ∙ online work per document ( Table 11 ). Shade: rank within a column, darker is better; Bold: best method; the uncompressed row is not ranked. Italics: our run, where no budget curve or surrogate is published for the target.
Appendix figures & tables60 assets
Supplementary material from the paper’s appendix.
Appendix
document set
models
documents
consensus of the other models
between two models
mix1000, all domains
67
1000
0.8443 ± 0.0042
0.7253 ± 0.0065
mix1000, natural domains
67
850
0.8673 ± 0.0032
0.7607 ± 0.0051
mix1000, generated domains
67
150
0.6114 ± 0.0136
0.4093 ± 0.0162
mix10k, all domains
12
9792
0.7977 ± 0.0021
0.6787 ± 0.0028
mix10k, natural domains
12
8292
0.8357 ± 0.0012
0.7277 ± 0.0018
mix10k, generated domains
12
1500
0.4990 ± 0.0064
0.3362 ± 0.0060
Appendix
Table 2: Per-token agreement on every document set. Each row applies the same estimator to the value-weighted per-token marginal attention on one set of documents: mix1000 (20 kinds of text), its natural and generated subsets, and mix10k . ± is half the 95% bootstrap interval over documents.
Figure 6: Agreement is high in every natural domain; its link to length varies. Left: per-document per-token agreement, tick the median, dashed line the mean. Right: Spearman correlation of length with agreement, 95% intervals. Setting in Section A.1 .
Figure 7: Document length varies about threefold across domains, from PhilPapers and Gutenberg (PG-19) down to NIH ExPorter and PubMed Abstracts. Dots are documents, ticks medians, bars interquartile ranges. Setting in Section A.1 .
Figure 8: Models agree less on generated text than on natural text. Per-document per-token agreement of each model with the consensus of the others, averaged over models; tick the median, open markers generated domains. Rows scored as in Table 2 .
vs the other 66
two models
whole census
agreement
0.867 ± 0.003
0.761 ± 0.005
different document (control)
0.015
0.014
10 pythia-410m seeds (ceiling)
0.942
0.900
pairs
mean
median
5th to 95th
pair class
base vs its instruct
12
0.968
0.976
[0.924, 0.993]
vision vs its text model
3
0.941
0.944
[0.928, 0.951]
Appendix
Table 3: Per-token census summary. 67 models, 850 natural documents. Upper: agreement with the other 66 models and between two models, with the different-document control and the pythia-410m seed ceiling. Lower: agreement between two models by pair class. On the first row, ± is half the 95% bootstrap interval over documents.
model
B
ref.
model
B
ref.
01-ai/Yi-6B
6.1
( 01.AI et al., 2024 )
meta-llama/Llama-3.1-8B
8.0
( Grattafiori et al., 2024 )
01-ai/Yi-6B-Chat
6.1
( 01.AI et al., 2024 )
meta-llama/Llama-3.1-8B-Instruct
8.0
( Grattafiori et al., 2024 )
01-ai/Yi-9B
8.8
( 01.AI et al., 2024 )
microsoft/phi-4
14.7
( Abdin et al., 2024 )
allenai/OLMo-2-0425-1B
1.5
( Walsh et al., 2025 )
mistralai/Mistral-7B-Instruct-v0.1
7.2
( Jiang et al., 2023 )
allenai/OLMo-2-0425-1B-Instruct
1.5
( Walsh et al., 2025 )
mistralai/Mistral-7B-Instruct-v0.2
7.2
( Jiang et al., 2023 )
allenai/OLMoE-1B-7B-0924
6.9
( Muennighoff et al., 2025 )
mistralai/Mistral-7B-Instruct-v0.3
7.2
( Jiang et al., 2023 )
Appendix
Table 4: Every model used. Repository id, parameter count in billions (B) and the report the model card names (ref.); alphabetical by repository, down the left block and then down the right. 69 models: 67 form both the per-token census and the per-head analyses, and Qwen3-32B and Mistral-7B-Instruct-v0.3 are used only as eviction targets. Eviction targets: Qwen3-8B, Llama-3.1-8B-Instruct, phi-4, Qwen3-32B, Mistral-7B-Instruct-v0.3; guide: Qwen2.5-0.5B-Instruct. The Pythia pretraining-seed pool (10 seeds at each of 5 sizes) is not listed.
characters on the axis
axis (%)
documents
vs the other 66
two models
all characters
100.0
850
0.8673
0.7607
punctuation deleted
90.8
850
0.8705
0.7664
punctuation and whitespace deleted
87.6
850
0.8734
0.7711
punctuation only
5.0
848
0.8648
0.7601
whitespace only
2.3
645
0.7336
0.6169
a different document (control)
100.0
850
0.0155
0.0139
Appendix
Table 5: Deleting punctuation and whitespace from the character axis does not lower agreement. Each row deletes one class of characters from the axis, or keeps only that class, and compares the per-token marginal attention of 67 models on 850 natural documents: one model with the consensus of the other 66, and two models with each other. Whitespace only is the one low row; it keeps a character only where every model’s token is whitespace, which leaves 2.3% of the axis and no character at all in 205 of the 850 documents. The two rows under the rule are references: a different document, and the unweighted variant ( ω≡1 ) on all characters.
comparison
n
write per head
write per layer
output norm
document split halves
67
0.992
0.992
0.992 (66)
leave-one-document-out
67
0.953
0.978
0.934 (66)
base vs derived model
17
0.990
0.990
0.987
same family
55
0.783
0.783
0.884
same architecture, different family
108
0.820
0.820
0.841
different architecture
2031
0.373
0.373
0.436 (1965)
Appendix
Table 6: Every head-wise statistic is stable inside one model and much lower between models of different architecture. Rows are the comparisons of Figure 3 ; entries are the mean Spearman correlation between per-head profiles over n models or model pairs (in parentheses where the output norm has fewer), on 170 documents. Setting in Appendix C .
Figure 9: Per-head Jacobian blocks are near-orthogonal. Pythia-410M over training: the norm of the head-summed value-channel block divided by the sum of the per-head norms, relative to the same ratio for perfectly orthogonal heads (dotted, 1).
Figure 10: Per-head Jacobian blocks are typically orthogonal across depth. The same cancellation coefficient of Figure 9 (the norm of the head-summed value-channel block divided by the sum of the per-head norms, relative to the same ratio for idealized orthogonal heads) per relative-depth bin (0 at the input, 1 at the output), one line per model; dotted: idealized orthogonal heads.
model
block
κ
orth.
ratio
cross
rank
op.
op. orth.
Qwen2.5-0.5B
value
0.387
0.423
0.915
− 0.112
0.978
0.343
0.298
value + key
0.395
0.425
0.930
− 0.103
0.979
0.357
0.315
value + key + query
0.376
0.401
0.935
− 0.094
0.977
–
–
TinyLlama-1.1B
value
0.216
0.258
0.834
− 0.386
0.960
0.166
0.141
value + key
0.226
0.265
0.852
− 0.373
0.959
0.218
0.189
value + key + query
0.221
0.258
0.858
− 0.360
0.954
–
–
Appendix
Table 7: Cancellation coefficient per model and block version. We compute cancellation coefficients for the value channel alone, value channel plus the key channel, and with the query term on the diagonal: κ , head-summed block norm over the sum of per-head norms; its value for orthogonal heads; their ratio; the cross-term share (0 for orthogonal heads); the Spearman rank agreement over source positions between the sum-of-norms and norm-of-sum profiles; and the operator-norm ratio with its orthogonal-subspaces reference on 512 pairs per layer. Pythia-410M is shown at both initialization (step 0) and at the end of training.
vs all other models
vs other families
map
agreement
different document
agreement
different document
two models
unweighted profile
0.8612
0.0334
0.8076
0.0321
0.7652
influence, gradient length
0.7722
0.0336
0.6768
0.0316
0.6295
influence, substitution
0.7759
0.0328
0.6828
0.0308
0.6349
influence, gradient × input
0.6568
0.0344
0.5797
0.0318
0.4771
Appendix
Table 8: Marginal influence is shared across models, marginal attention more so. 16 models from 6 families, 100 documents. Spearman correlation per document of one model’s map with the consensus of the other models, with its own family left out of the consensus, and between two models, each beside a different-document reference. Rows: the unweighted per-token marginal attention and three row-gradient estimates of the per-token marginal influence ( Section E.1 ): gradient length, substitution sensitivity, gradient times input.
Figure 11: Agreement with other models rises as the loss falls. Pythia-410M over training: agreement of its per-token marginal attention with four trained models (left axis) and loss in bits per character (right axis); dash-dotted line: a bigram model’s loss. Setting in Section E.3 .
Figure 12: Per-token marginal influence of models without attention. Spearman correlation with the consensus per-token marginal attention of the census; one model per row; dashed: references. Setting in Section E.4 .
target
key-value heads
1 vs 199
5 vs 195
20 vs 180
100 vs 100
top decile
prefill queries
Qwen3-8B
288
0.9828 (0.9704)
0.9952
0.9985
0.9995
1.0000
0.9624
Llama-3.1-8B
256
0.9830 (0.9744)
0.9951
0.9985
0.9994
0.9259
0.9394
phi-4
400
0.9836 (0.9706)
0.9954
0.9986
0.9992
0.9512
0.9681
Qwen3-32B
512
0.9815 (0.9512)
0.9952
0.9986
0.9993
1.0000
0.9656
Mistral-7B
256
0.9829 (0.9729)
0.9954
0.9984
0.9994
1.0000
0.9666
Appendix
Table 9: Head Mass tables from disjoint sets of pretraining documents agree at a median Spearman of 0.98 or higher. Spearman correlation over each target’s key-value heads between the per-head marginal attention ( Equation 2 ) of two disjoint sets of the 200 pretraining documents, read on KVzip’s reconstruction queries at 8 × . 1 vs 199 is the median (5th percentile) over documents, 5 vs 195 and 20 vs 180 the median of 100 draws, and 100 vs 100 the even documents compared with the odd ones. The top decile column is the Jaccard overlap of the top tenth of heads between even and odd documents, and the prefill queries column is the 1 vs 199 median on the document’s own queries.
Figure 13: Agreement of Head Mass tables from k and 200−k documents. One panel per compression ratio, one row per target, one column per k . Each cell is the median, over 100 random draws of k of the 200 pretraining documents, of the Spearman correlation over the target’s key-value heads between the per-head marginal attention of the drawn documents and of the rest.
target
LU-KV
KVzip+
Proportional
Prefill attention
key-value heads
Qwen3-8B
0.81
0.92
0.96
0.13
288
Llama-3.1-8B
0.80
0.91
0.95
0.38
256
phi-4
0.81
0.92
0.97
0.30
400
Qwen3-32B
0.80
0.91
0.98
0.17
512
Mistral-7B
0.81
0.93
0.96
0.42
256
mean over targets
0.81
0.92
0.96
0.28
–
Appendix
Table 10: LU-KV, KVzip+ and Proportional agree with Head Mass, prefill attention doesn’t. Spearman correlation with the Head Mass table over each target’s (layer, key-value head) grid at 8 × , for the tables of Section F.2 . The last row is the mean over targets.
Figure 14: Per-head budgets at 8 × , Qwen3-8B. One cell per key-value head, layer 0 at the bottom; colour is the fraction of the head’s cache kept (log scale, uniform 0.125). Head Mass averages 200 pretraining documents; LU-KV is its budget curve at compression 0.88, hence a mean of 0.12; KVzip+ computes its global cut on each document at run time; the panel is the fraction it keeps in each head averaged over 12 held-out documents; Proportional is Equation 4 ; Prefill attention applies the Head Mass recipe to the document’s own prefill queries. ρ is the Spearman correlation with Head Mass.
Figure 15: Per-head budgets at 8 × , Llama-3.1-8B. Layout as in Figure 14 .
Figure 16: Per-head budgets at 8 × , phi-4. Layout as in Figure 14 .
Figure 17: Per-head budgets at 8 × , Qwen3-32B. Layout as in Figure 14 .
Figure 18: Per-head budgets at 8 × , Mistral-7B. Layout as in Figure 14 .
offline, per target
online, per document
offline time, Mistral-7B
AdaKV + EA
none
none
none
AdaKV + KVzap
△△
surrogate training on KVzip’s score
surrogate call, not counted
4 h 14 m, one H100
KVzip
∙∙
none
one extra target forward
none
LU-KV + EA
△
per-head budget curve calibration
none
published curve
HM + EA
△
per-head budget table, 200 pretraining documents
none
34 min
HM + Guide
△∙
the same table and a guide map
one Qwen2.5-0.5B-Instruct forward
36 min
Appendix
Table 11: HM + EA needs one offline budget table per target and no work per document. Work each method needs beyond ordinary serving. △ marks work done once per target and ∙ work per document; two marks denote training a surrogate or an extra target forward. HM is Head Mass and EA is Expected Attention. The last column is the wall time of the offline step on Mistral-7B-Instruct-v0.3 (cluster job walltime).
RULER, context 4096
RULER, context 8192
Target
4 ×
8 ×
16 ×
8 ×
16 ×
Qwen3-8B
94.8 ( − 0.2 )
91.9 (0.2)
73.1 ( − 0.2)
89.1 (0.1)
69.5 (0.4)
Llama-3.1-8B
95.0 ( − 0.1)
90.8 ( − 1.5 )
65.1 ( − 1.2 )
86.3 ( − 1.7 )
60.7 ( − 2.1 )
phi-4
90.8 ( − 0.6 )
84.8 ( − 3.0 )
66.8 ( − 1.7 )
82.8 ( − 2.1 )
67.3 ( − 1.1 )
Qwen3-32B
95.0 ( − 0.2 )
92.0 ( − 1.2 )
67.6 ( − 11.5 )
90.9 ( − 1.1 )
66.5 ( − 11.4 )
Mistral-7B
89.3 ( − 1.2 )
83.2 ( − 1.1 )
63.2 ( − 2.2 )
79.7 ( − 0.7 )
57.8 ( − 3.3 )
Appendix
Table 12: With EA as the scorer, the proportional per-head budget ( Equation 4 ) is within half a point of HM + EA at every setting only on Qwen3-8B; elsewhere it trails by up to 3.3 points, except on Qwen3-32B at 16 × , where it trails by more than 11. Each cell is the RULER score of the proportional budget with EA; in parentheses, its paired difference from HM + EA, bold where the 95% interval excludes zero. Both budgets come from KVzip’s reconstruction pass on the same pretraining documents.
4 ×
8 ×
16 ×
AdaKV + EA
88.4
72.8
46.5
HM + EA
95.0
91.8
73.3
HM + KVzap
94.9
92.0
73.0
Proportional (reconstruction) + EA
94.8
91.9
73.1
Proportional (reading) + EA
61.1
53.0
37.7
paired difference, with its 95% interval
Appendix
Table 13: On Qwen3-8B, the proportional per-head budget ( Equation 4 ) read on the reconstruction queries is within 0.25 points of HM + EA; read on the document’s own queries it scores below AdaKV + EA at every compression ratio. Over heads, the Spearman correlation between the reading-pass budget and Head Mass is − 0.11, − 0.15 and − 0.10 at 4 × , 8 × and 16 × . The upper rows are RULER-4096 scores (mean over 13 tasks, 6500 documents); the lower rows are paired differences with 95% bootstrap intervals.
cost
Qwen3-8B
Llama-3.1-8B
phi-4
Qwen3-32B
Mistral-7B
offline
online
8 ×
16 ×
8 ×
16 ×
8 ×
16 ×
8 ×
16 ×
8 ×
16 ×
AdaKV + EA
71.7
50.9
41.2
15.8
41.7
13.6
64.1
38.4
33.5
20.5
AdaKV + KVzap
△△
86.6
46.5
86.7
60.5
70.8
43.9
88.5
50.3
67.0
40.2
KVzip
∙∙
89.8
55.6
90.1
73.6
88.0
64.2
95.0
92.1
82.5
66.3
LU-KV + EA
△
72.9
55.7
87.7
62.4
86.5
68.8
73.3
38.6
78.8
58.3
HM + EA
△
89.0
69.1
88.0
62.8
84.9
68.4
92.0
77.9
80.5
61.1
Appendix
Table 14: At context 8192, HM + EA scores above AdaKV + EA in all 10 columns. Mean string-match score over the 13 RULER tasks at context 8192, 6500 documents and one run per cell, for five targets at 8 × and 16 × compression. Paired 95% bootstrap half-widths of the differences from HM + Guide are 0.33 to 1.09 points. Methods, shading, cost symbols and italics as in Table 1 ; the uncompressed row is not ranked.
cost
Qwen3-8B
Llama-3.1-8B
Qwen3-32B
Mistral-7B
offline
online
8 ×
16 ×
8 ×
16 ×
8 ×
16 ×
8 ×
16 ×
AdaKV + EA
42.1
33.1
36.4
28.4
46.0
39.2
31.0
27.6
AdaKV + KVzap
△△
34.0
23.9
42.0
34.4
36.8
19.0
28.5
20.0
KVzip
∙∙
44.7
28.0
44.5
33.8
50.7
45.2
43.8
38.2
LU-KV + EA
△
46.3
41.1
45.8
43.0
48.0
43.4
42.6
38.8
HM + EA
△
46.2
39.9
45.6
40.5
49.9
44.8
42.3
39.5
Appendix
Table 15: On LongBench, LU-KV + EA and HM + EA differ by at most 2.5 points over eight columns. Unweighted mean over the 16 English LongBench subsets at their full official split (3750 documents), each scored by its official metric, one run per cell, for four targets at 8 × and 16 × compression; phi-4 was not run. No interval is given, since the mean mixes metrics. Methods, shading, cost symbols and italics as in Table 1 ; the uncompressed row is not ranked.
Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.
Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.
Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen. Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels.
Hengshuai Yao, Xing Chen, Ahmed Murtadha +8
Sapient Intelligence · University of Alberta · Tsinghua University
How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the selected set size and measuring the increase in negative log-likelihood (NLL), we estimate the effective attention set size needed to stay within a chosen loss tolerance. Relatively small selected sets can keep NLL close to the full-attention baseline, although the required size varies across models. Attention-based selection substantially outperforms random selection. Selected sets exhibit geometric structure, although geometric separation alone does not establish that model loss is preserved. Extending context while evaluating the same prediction targets increases the required set size, while its fraction of context decreases over the tested range. Experiments with a fixed supporting fact show that additional background pushes its tokens down the attention ranking and reduces their attention mass. Renormalizing the retained weights can substantially reduce the required set size, showing that it also depends on how selected representations are combined. Conditional theoretical models explain how competition and attention-mass retention can produce growing set sizes without more distinct information to retrieve. These results provide a way to measure effective attention set size in language models and investigate its dependence on context, competition, and aggregation.