Emergent Structure in the Marginal Attention Space of Language Models

Valentino Maiorca; Walter Nelson; Francesco Locatello

cs.CLOct 2, 2026

Emergent Structure in the Marginal Attention Space of Language Models

Authors: Valentino Maiorca, Walter Nelson, Francesco Locatello

Organizations: Institute of Science and Technology Austria (ISTA)

Abstract

While representation similarity across independently trained language models is well-documented, how internal mechanics such as attention behave across models remains far less characterized. Inspired by this gap, we examine the structure of post-softmax attention weights by marginalizing over query positions, mapping them into a joint token-head "marginal attention space". Evaluating across 60+ diverse LLMs, we find that different properties emerge when reducing this space along its token and head axes. When reduced token-wise, marginal attention yields a text-intrinsic signal robustly conserved across models. To explain this property, we empirically connect marginal attention to the input-output Jacobian of the network, and prove theoretically that under a smoothness assumption, models with similar next-token distributions are guaranteed to have similar input-output Jacobian statistics. When reduced head-wise, it forms a model-private signature conserved across documents. Practically, this provides a natural way to estimate a per-head budget for key-value (KV) cache eviction, effectively decoupling model-specific budget allocation from text-intrinsic token scoring. On standard eviction benchmarks, a per-head budget precomputed offline on pretraining text, combined with a training-free token score, shows competitive performance with methods that recompute the budget on every document or train it per target. Code available at https://github.com/Flegyas/marginal-attention

Figures & tables

Figure 1: Marginal attention on its two axes. Averaged over query positions, the attention of Qwen2.5-0.5B on one document is a matrix $\mathbf{M}$ of $336$ heads by tokens. Its per-token reduction (green) agrees across 67 models on the same document ( $\bar{\rho}=0.87$ ), and its per-head reduction (lavender) on one document matches the model’s average over its other documents ( $\bar{\rho}=0.95$ ).

Figure 2: Per-token marginal attention across models. Spearman correlation of two profiles on one document; 67 models, 850 documents. Left: models against the consensus of the others (top) and all 2211 pairs (bottom); dashed line the agreement between 10 pretraining seeds of pythia-410m. Right: pairs grouped by pair class, most related on the left.

Figure 3: Per-head marginal attention within and between models. Spearman correlation between per-head profiles; 67 models, 170 documents.

Figure 4: What the per-token marginal attention tracks. Spearman correlation per document. Left: with the per-token marginal influence of Qwen2.5-0.5B over 85 documents. Right: word properties regressing the unweighted (

\omega\equiv 1

) consensus profile over 550 documents.

Figure 5: Within a tokenizer group, pairs with a smaller logit gap

\delta

agree more on per-token marginal influence (left) and per-token marginal attention (right,

\omega\equiv 1

). Colour: pair class; marker: tokenizer; lines: least-squares fits; bars: 95% intervals. Estimator in Section E.1 .

	cost		Qwen3-8B			Llama-3.1-8B			phi-4			Qwen3-32B			Mistral-7B
	offline	online	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$	4 $\times$	8 $\times$	16 $\times$
AdaKV + EA			88.4	72.8	46.5	75.7	45.3	15.2	68.7	44.0	13.8	81.9	63.6	36.4	67.2	34.0	19.9
AdaKV + KVzap	$\triangle\triangle$		94.0	89.2	49.9	94.5	89.3	63.1	87.8	77.0	51.6	95.4	90.0	50.3	88.3	70.2	42.1
KVzip		$\bullet\bullet$	95.0	92.3	59.4	95.3	92.8	74.2	91.5	89.6	68.8	95.6	95.6	91.8	91.2	89.2	72.2
LU-KV + EA	$\triangle$		90.6	75.2	58.4	95.1	91.8	67.0	92.4	87.9	69.5	90.2	72.6	38.2	91.2	85.1	61.7
HM + EA	$\triangle$		95.0	91.8	73.3	95.1	92.3	66.3	91.5	87.8	68.5	95.2	93.2	79.1	90.5	84.3	65.3

Table 1: Eviction scores. RULER 4096 score over its 13 tasks and documents at 4

\times

, 8

\times

and 16

\times

compression. Cost:

\triangle

offline work per target,

\bullet

online work per document ( Table 11 ). Shade: rank within a column, darker is better; Bold: best method; the uncompressed row is not ranked. Italics: our run, where no budget curve or surrogate is published for the target.

Appendix figures & tables60 assets

Supplementary material from the paper’s appendix.

Appendix

document set	models	documents	consensus of the other models	between two models
mix1000, all domains	67	1000	0.8443 $\pm$ 0.0042	0.7253 $\pm$ 0.0065
mix1000, natural domains	67	850	0.8673 $\pm$ 0.0032	0.7607 $\pm$ 0.0051
mix1000, generated domains	67	150	0.6114 $\pm$ 0.0136	0.4093 $\pm$ 0.0162
mix10k, all domains	12	9792	0.7977 $\pm$ 0.0021	0.6787 $\pm$ 0.0028
mix10k, natural domains	12	8292	0.8357 $\pm$ 0.0012	0.7277 $\pm$ 0.0018
mix10k, generated domains	12	1500	0.4990 $\pm$ 0.0064	0.3362 $\pm$ 0.0060

Appendix

Table 2: Per-token agreement on every document set. Each row applies the same estimator to the value-weighted per-token marginal attention on one set of documents: mix1000 (20 kinds of text), its natural and generated subsets, and mix10k .

\pm

is half the 95% bootstrap interval over documents.

Figure 6: Agreement is high in every natural domain; its link to length varies. Left: per-document per-token agreement, tick the median, dashed line the mean. Right: Spearman correlation of length with agreement, 95% intervals. Setting in Section A.1 .

Figure 7: Document length varies about threefold across domains, from PhilPapers and Gutenberg (PG-19) down to NIH ExPorter and PubMed Abstracts. Dots are documents, ticks medians, bars interquartile ranges. Setting in Section A.1 .

Figure 8: Models agree less on generated text than on natural text. Per-document per-token agreement of each model with the consensus of the others, averaged over models; tick the median, open markers generated domains. Rows scored as in Table 2 .

		vs the other 66	two models
whole census	agreement	0.867 $\pm$ 0.003	0.761 $\pm$ 0.005
	different document (control)	0.015	0.014
	10 pythia-410m seeds (ceiling)	0.942	0.900
		pairs	mean	median	5th to 95th
pair class	base vs its instruct	12	0.968	0.976	[0.924, 0.993]
	vision vs its text model	3	0.941	0.944	[0.928, 0.951]

Appendix

Table 3: Per-token census summary. 67 models, 850 natural documents. Upper: agreement with the other 66 models and between two models, with the different-document control and the pythia-410m seed ceiling. Lower: agreement between two models by pair class. On the first row,

\pm

is half the 95% bootstrap interval over documents.

model	B	ref.	model	B	ref.
01-ai/Yi-6B	6.1	( 01.AI et al., 2024 )	meta-llama/Llama-3.1-8B	8.0	( Grattafiori et al., 2024 )
01-ai/Yi-6B-Chat	6.1	( 01.AI et al., 2024 )	meta-llama/Llama-3.1-8B-Instruct	8.0	( Grattafiori et al., 2024 )
01-ai/Yi-9B	8.8	( 01.AI et al., 2024 )	microsoft/phi-4	14.7	( Abdin et al., 2024 )
allenai/OLMo-2-0425-1B	1.5	( Walsh et al., 2025 )	mistralai/Mistral-7B-Instruct-v0.1	7.2	( Jiang et al., 2023 )
allenai/OLMo-2-0425-1B-Instruct	1.5	( Walsh et al., 2025 )	mistralai/Mistral-7B-Instruct-v0.2	7.2	( Jiang et al., 2023 )
allenai/OLMoE-1B-7B-0924	6.9	( Muennighoff et al., 2025 )	mistralai/Mistral-7B-Instruct-v0.3	7.2	( Jiang et al., 2023 )

Appendix

Table 4: Every model used. Repository id, parameter count in billions (B) and the report the model card names (ref.); alphabetical by repository, down the left block and then down the right. 69 models: 67 form both the per-token census and the per-head analyses, and Qwen3-32B and Mistral-7B-Instruct-v0.3 are used only as eviction targets. Eviction targets: Qwen3-8B, Llama-3.1-8B-Instruct, phi-4, Qwen3-32B, Mistral-7B-Instruct-v0.3; guide: Qwen2.5-0.5B-Instruct. The Pythia pretraining-seed pool (10 seeds at each of 5 sizes) is not listed.

characters on the axis	axis (%)	documents	vs the other 66	two models
all characters	100.0	850	0.8673	0.7607
punctuation deleted	90.8	850	0.8705	0.7664
punctuation and whitespace deleted	87.6	850	0.8734	0.7711
punctuation only	5.0	848	0.8648	0.7601
whitespace only	2.3	645	0.7336	0.6169
a different document (control)	100.0	850	0.0155	0.0139

Appendix

Table 5: Deleting punctuation and whitespace from the character axis does not lower agreement. Each row deletes one class of characters from the axis, or keeps only that class, and compares the per-token marginal attention of 67 models on 850 natural documents: one model with the consensus of the other 66, and two models with each other. Whitespace only is the one low row; it keeps a character only where every model’s token is whitespace, which leaves 2.3% of the axis and no character at all in 205 of the 850 documents. The two rows under the rule are references: a different document, and the unweighted variant (

\omega\equiv 1

) on all characters.

comparison	$n$	write per head	write per layer	output norm
document split halves	67	0.992	0.992	0.992 (66)
leave-one-document-out	67	0.953	0.978	0.934 (66)
base vs derived model	17	0.990	0.990	0.987
same family	55	0.783	0.783	0.884
same architecture, different family	108	0.820	0.820	0.841
different architecture	2031	0.373	0.373	0.436 (1965)

Appendix

Table 6: Every head-wise statistic is stable inside one model and much lower between models of different architecture. Rows are the comparisons of Figure 3 ; entries are the mean Spearman correlation between per-head profiles over

n

models or model pairs (in parentheses where the output norm has fewer), on 170 documents. Setting in Appendix C .

Figure 9: Per-head Jacobian blocks are near-orthogonal. Pythia-410M over training: the norm of the head-summed value-channel block divided by the sum of the per-head norms, relative to the same ratio for perfectly orthogonal heads (dotted, 1).

Figure 10: Per-head Jacobian blocks are typically orthogonal across depth. The same cancellation coefficient of Figure 9 (the norm of the head-summed value-channel block divided by the sum of the per-head norms, relative to the same ratio for idealized orthogonal heads) per relative-depth bin (0 at the input, 1 at the output), one line per model; dotted: idealized orthogonal heads.

model	block	$\kappa$	orth.	ratio	cross	rank	op.	op. orth.
Qwen2.5-0.5B	value	0.387	0.423	0.915	$-$ 0.112	0.978	0.343	0.298
	value + key	0.395	0.425	0.930	$-$ 0.103	0.979	0.357	0.315
	value + key + query	0.376	0.401	0.935	$-$ 0.094	0.977	–	–
TinyLlama-1.1B	value	0.216	0.258	0.834	$-$ 0.386	0.960	0.166	0.141
	value + key	0.226	0.265	0.852	$-$ 0.373	0.959	0.218	0.189
	value + key + query	0.221	0.258	0.858	$-$ 0.360	0.954	–	–

Appendix

Table 7: Cancellation coefficient per model and block version. We compute cancellation coefficients for the value channel alone, value channel plus the key channel, and with the query term on the diagonal:

\kappa

, head-summed block norm over the sum of per-head norms; its value for orthogonal heads; their ratio; the cross-term share (0 for orthogonal heads); the Spearman rank agreement over source positions between the sum-of-norms and norm-of-sum profiles; and the operator-norm ratio with its orthogonal-subspaces reference on 512 pairs per layer. Pythia-410M is shown at both initialization (step 0) and at the end of training.

map	agreement	different document	agreement	different document	two models
	vs all other models		vs other families
unweighted profile	0.8612	0.0334	0.8076	0.0321	0.7652
influence, gradient length	0.7722	0.0336	0.6768	0.0316	0.6295
influence, substitution	0.7759	0.0328	0.6828	0.0308	0.6349
influence, gradient $\times$ input	0.6568	0.0344	0.5797	0.0318	0.4771

Appendix

Table 8: Marginal influence is shared across models, marginal attention more so. 16 models from 6 families, 100 documents. Spearman correlation per document of one model’s map with the consensus of the other models, with its own family left out of the consensus, and between two models, each beside a different-document reference. Rows: the unweighted per-token marginal attention and three row-gradient estimates of the per-token marginal influence ( Section E.1 ): gradient length, substitution sensitivity, gradient times input.

Figure 11: Agreement with other models rises as the loss falls. Pythia-410M over training: agreement of its per-token marginal attention with four trained models (left axis) and loss in bits per character (right axis); dash-dotted line: a bigram model’s loss. Setting in Section E.3 .

Figure 12: Per-token marginal influence of models without attention. Spearman correlation with the consensus per-token marginal attention of the census; one model per row; dashed: references. Setting in Section E.4 .

target	key-value heads	1 vs 199	5 vs 195	20 vs 180	100 vs 100	top decile	prefill queries
Qwen3-8B	288	0.9828 (0.9704)	0.9952	0.9985	0.9995	1.0000	0.9624
Llama-3.1-8B	256	0.9830 (0.9744)	0.9951	0.9985	0.9994	0.9259	0.9394
phi-4	400	0.9836 (0.9706)	0.9954	0.9986	0.9992	0.9512	0.9681
Qwen3-32B	512	0.9815 (0.9512)	0.9952	0.9986	0.9993	1.0000	0.9656
Mistral-7B	256	0.9829 (0.9729)	0.9954	0.9984	0.9994	1.0000	0.9666

Appendix

Table 9: Head Mass tables from disjoint sets of pretraining documents agree at a median Spearman of 0.98 or higher. Spearman correlation over each target’s key-value heads between the per-head marginal attention ( Equation 2 ) of two disjoint sets of the 200 pretraining documents, read on KVzip’s reconstruction queries at 8

\times

. 1 vs 199 is the median (5th percentile) over documents, 5 vs 195 and 20 vs 180 the median of 100 draws, and 100 vs 100 the even documents compared with the odd ones. The top decile column is the Jaccard overlap of the top tenth of heads between even and odd documents, and the prefill queries column is the 1 vs 199 median on the document’s own queries.

Figure 13: Agreement of Head Mass tables from $k$ and ${200-k}$ documents. One panel per compression ratio, one row per target, one column per $k$ . Each cell is the median, over 100 random draws of $k$ of the 200 pretraining documents, of the Spearman correlation over the target’s key-value heads between the per-head marginal attention of the drawn documents and of the rest.

target	LU-KV	KVzip+	Proportional	Prefill attention	key-value heads
Qwen3-8B	0.81	0.92	0.96	0.13	288
Llama-3.1-8B	0.80	0.91	0.95	0.38	256
phi-4	0.81	0.92	0.97	0.30	400
Qwen3-32B	0.80	0.91	0.98	0.17	512
Mistral-7B	0.81	0.93	0.96	0.42	256
mean over targets	0.81	0.92	0.96	0.28	–

Appendix

Table 10: LU-KV, KVzip+ and Proportional agree with Head Mass, prefill attention doesn’t. Spearman correlation with the Head Mass table over each target’s (layer, key-value head) grid at 8

\times

, for the tables of Section F.2 . The last row is the mean over targets.

Figure 14: Per-head budgets at 8 $\times$ , Qwen3-8B. One cell per key-value head, layer 0 at the bottom; colour is the fraction of the head’s cache kept (log scale, uniform 0.125). Head Mass averages 200 pretraining documents; LU-KV is its budget curve at compression 0.88, hence a mean of 0.12; KVzip+ computes its global cut on each document at run time; the panel is the fraction it keeps in each head averaged over 12 held-out documents; Proportional is Equation 4 ; Prefill attention applies the Head Mass recipe to the document’s own prefill queries. $\rho$ is the Spearman correlation with Head Mass.

Figure 15: Per-head budgets at 8 $\times$ , Llama-3.1-8B. Layout as in Figure 14 .

Figure 16: Per-head budgets at 8 $\times$ , phi-4. Layout as in Figure 14 .

Figure 17: Per-head budgets at 8 $\times$ , Qwen3-32B. Layout as in Figure 14 .

Figure 18: Per-head budgets at 8 $\times$ , Mistral-7B. Layout as in Figure 14 .

		offline, per target	online, per document	offline time, Mistral-7B
AdaKV + EA		none	none	none
AdaKV + KVzap	$\triangle\triangle$	surrogate training on KVzip’s score	surrogate call, not counted	4 h 14 m, one H100
KVzip	$\bullet\bullet$	none	one extra target forward	none
LU-KV + EA	$\triangle$	per-head budget curve calibration	none	published curve
HM + EA	$\triangle$	per-head budget table, 200 pretraining documents	none	34 min
HM + Guide	$\triangle$ $\bullet$	the same table and a guide map	one Qwen2.5-0.5B-Instruct forward	36 min

Appendix

Table 11: HM + EA needs one offline budget table per target and no work per document. Work each method needs beyond ordinary serving.

\triangle

marks work done once per target and

\bullet

work per document; two marks denote training a surrogate or an extra target forward. HM is Head Mass and EA is Expected Attention. The last column is the wall time of the offline step on Mistral-7B-Instruct-v0.3 (cluster job walltime).

	RULER, context 4096			RULER, context 8192
Target	4 $\times$	8 $\times$	16 $\times$	8 $\times$	16 $\times$
Qwen3-8B	94.8 ( $-$ 0.2 )	91.9 (0.2)	73.1 ( $-$ 0.2)	89.1 (0.1)	69.5 (0.4)
Llama-3.1-8B	95.0 ( $-$ 0.1)	90.8 ( $-$ 1.5 )	65.1 ( $-$ 1.2 )	86.3 ( $-$ 1.7 )	60.7 ( $-$ 2.1 )
phi-4	90.8 ( $-$ 0.6 )	84.8 ( $-$ 3.0 )	66.8 ( $-$ 1.7 )	82.8 ( $-$ 2.1 )	67.3 ( $-$ 1.1 )
Qwen3-32B	95.0 ( $-$ 0.2 )	92.0 ( $-$ 1.2 )	67.6 ( $-$ 11.5 )	90.9 ( $-$ 1.1 )	66.5 ( $-$ 11.4 )
Mistral-7B	89.3 ( $-$ 1.2 )	83.2 ( $-$ 1.1 )	63.2 ( $-$ 2.2 )	79.7 ( $-$ 0.7 )	57.8 ( $-$ 3.3 )

Appendix

Table 12: With EA as the scorer, the proportional per-head budget ( Equation 4 ) is within half a point of HM + EA at every setting only on Qwen3-8B; elsewhere it trails by up to 3.3 points, except on Qwen3-32B at 16

\times

, where it trails by more than 11. Each cell is the RULER score of the proportional budget with EA; in parentheses, its paired difference from HM + EA, bold where the 95% interval excludes zero. Both budgets come from KVzip’s reconstruction pass on the same pretraining documents.

paired difference, with its 95% interval
	4 $\times$	8 $\times$	16 $\times$
AdaKV + EA	88.4	72.8	46.5
HM + EA	95.0	91.8	73.3
HM + KVzap	94.9	92.0	73.0
Proportional (reconstruction) + EA	94.8	91.9	73.1
Proportional (reading) + EA	61.1	53.0	37.7

Appendix

Table 13: On Qwen3-8B, the proportional per-head budget ( Equation 4 ) read on the reconstruction queries is within 0.25 points of HM + EA; read on the document’s own queries it scores below AdaKV + EA at every compression ratio. Over heads, the Spearman correlation between the reading-pass budget and Head Mass is

-

0.11,

-

0.15 and

-

0.10 at 4

\times

, 8

\times

and 16

\times

. The upper rows are RULER-4096 scores (mean over 13 tasks, 6500 documents); the lower rows are paired differences with 95% bootstrap intervals.

	cost		Qwen3-8B		Llama-3.1-8B		phi-4		Qwen3-32B		Mistral-7B
	offline	online	8 $\times$	16 $\times$	8 $\times$	16 $\times$	8 $\times$	16 $\times$	8 $\times$	16 $\times$	8 $\times$	16 $\times$
AdaKV + EA			71.7	50.9	41.2	15.8	41.7	13.6	64.1	38.4	33.5	20.5
AdaKV + KVzap	$\triangle\triangle$		86.6	46.5	86.7	60.5	70.8	43.9	88.5	50.3	67.0	40.2
KVzip		$\bullet\bullet$	89.8	55.6	90.1	73.6	88.0	64.2	95.0	92.1	82.5	66.3
LU-KV + EA	$\triangle$		72.9	55.7	87.7	62.4	86.5	68.8	73.3	38.6	78.8	58.3
HM + EA	$\triangle$		89.0	69.1	88.0	62.8	84.9	68.4	92.0	77.9	80.5	61.1

Appendix

Table 14: At context 8192, HM + EA scores above AdaKV + EA in all 10 columns. Mean string-match score over the 13 RULER tasks at context 8192, 6500 documents and one run per cell, for five targets at 8

\times

and 16

\times

compression. Paired 95% bootstrap half-widths of the differences from HM + Guide are 0.33 to 1.09 points. Methods, shading, cost symbols and italics as in Table 1 ; the uncompressed row is not ranked.

	cost		Qwen3-8B		Llama-3.1-8B		Qwen3-32B		Mistral-7B
	offline	online	8 $\times$	16 $\times$	8 $\times$	16 $\times$	8 $\times$	16 $\times$	8 $\times$	16 $\times$
AdaKV + EA			42.1	33.1	36.4	28.4	46.0	39.2	31.0	27.6
AdaKV + KVzap	$\triangle\triangle$		34.0	23.9	42.0	34.4	36.8	19.0	28.5	20.0
KVzip		$\bullet\bullet$	44.7	28.0	44.5	33.8	50.7	45.2	43.8	38.2
LU-KV + EA	$\triangle$		46.3	41.1	45.8	43.0	48.0	43.4	42.6	38.8
HM + EA	$\triangle$		46.2	39.9	45.6	40.5	49.9	44.8	42.3	39.5

Appendix

Table 15: On LongBench, LU-KV + EA and HM + EA differ by at most 2.5 points over eight columns. Unweighted mean over the 16 English LongBench subsets at their full official split (3750 documents), each scored by its official metric, one run per cell, for four targets at 8

\times

and 16

\times

compression; phi-4 was not run. No interval is given, since the mean mixes metrics. Methods, shading, cost symbols and italics as in Table 1 ; the uncompressed row is not ranked.

Figure 19: Per-task scores, RULER-4096, 4

\times

, Qwen3-8B.

Figure 20: Per-task scores, RULER-4096, 4

\times

, Llama-3.1-8B.

Figure 21: Per-task scores, RULER-4096, 4

\times

, phi-4.

Figure 22: Per-task scores, RULER-4096, 4

\times

, Qwen3-32B.

Figure 23: Per-task scores, RULER-4096, 4

\times

, Mistral-7B.

Figure 24: Per-task scores, RULER-4096, 8

\times

, Qwen3-8B.

Figure 25: Per-task scores, RULER-4096, 8

\times

, Llama-3.1-8B.

Figure 26: Per-task scores, RULER-4096, 8

\times

, phi-4.

Figure 27: Per-task scores, RULER-4096, 8

\times

, Qwen3-32B.

Figure 28: Per-task scores, RULER-4096, 8

\times

, Mistral-7B.

Figure 29: Per-task scores, RULER-4096, 16

\times

, Qwen3-8B.

Figure 30: Per-task scores, RULER-4096, 16

\times

, Llama-3.1-8B.

Figure 31: Per-task scores, RULER-4096, 16

\times

, phi-4.

Figure 32: Per-task scores, RULER-4096, 16

\times

, Qwen3-32B.

Figure 33: Per-task scores, RULER-4096, 16

\times

, Mistral-7B.

Figure 34: Per-task scores, RULER-8192, 8

\times

, Qwen3-8B.

Figure 35: Per-task scores, RULER-8192, 8

\times

, Llama-3.1-8B.

Figure 36: Per-task scores, RULER-8192, 8

\times

, phi-4.

Figure 37: Per-task scores, RULER-8192, 8

\times

, Qwen3-32B.

Figure 38: Per-task scores, RULER-8192, 8

\times

, Mistral-7B.

Figure 39: Per-task scores, RULER-8192, 16

\times

, Qwen3-8B.

Figure 40: Per-task scores, RULER-8192, 16

\times

, Llama-3.1-8B.

Figure 41: Per-task scores, RULER-8192, 16

\times

, phi-4.

Figure 42: Per-task scores, RULER-8192, 16

\times

, Qwen3-32B.

Figure 43: Per-task scores, RULER-8192, 16

\times

, Mistral-7B.

Figure 44: Per-subset scores, LongBench, 8

\times

, Qwen3-8B.

Figure 45: Per-subset scores, LongBench, 8

\times

, Llama-3.1-8B.

Figure 46: Per-subset scores, LongBench, 8

\times

, Qwen3-32B.

Figure 47: Per-subset scores, LongBench, 8

\times

, Mistral-7B.

Figure 48: Per-subset scores, LongBench, 16

\times

, Qwen3-8B.

Figure 49: Per-subset scores, LongBench, 16

\times

, Llama-3.1-8B.

Figure 50: Per-subset scores, LongBench, 16

\times

, Qwen3-32B.

Figure 51: Per-subset scores, LongBench, 16

\times

, Mistral-7B.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 513 papers

Memory Attention
Sep 23, 2026Jiale KangMemory-Augmented Language Models LLM Inference Acceleration
Why Attend to Everything? Focus is the Key
Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention Mechanisms Efficient Attention
Sapient Intelligence · University of Alberta · Tsinghua University
Retrieval Capacity of Self-Attention Under Competition
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-Attention Long-Context Language Modeling
University of Luxembourg, Luxembourg

Sep 23, 2026J/K move · Enter open · S save

Memory Attention

Jiale Kang

Memory-Augmented Language Models LLM Inference Acceleration Attention Mechanisms

Open paper AskPDF arXiv

Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.

Abstract

Attention Mechanisms

Explore similar work

Memory Attention

Why Attend to Everything? Focus is the Key

Retrieval Capacity of Self-Attention Under Competition