Attention Matching (AM) compacts a KV cache by keeping some keys per head, refitting their values and adding a bias to each kept key's logit. FlashAttention and vLLM's paged caches have no input for it. We fold the bias into the keys along one closed-form direction per head, fitted on context queries, giving a plain cache with AM's size and head dimension and the model's own query path; the output error is bounded by how far each query's projection on that direction is from 1. With equal per-head budgets, unmodified vLLM serves folded caches through a KV connector, matching Hugging Face's answers on all 51 Qwen3-4B questions and its greedy output on 180 of 200 Qwen3-14B LongBench samples. With equal budgets on QuALITY (Qwen3-1.7B, 4B, 8B, Llama-3.1-8B) and LongHealth (three Qwen3 models), the fold is within noise of AM's accuracy in 13 of 14 settings, with lower KL to the full cache than dropping the bias in all 12 where both ran. Under the published per-head budgets, on Qwen3-4B, its Instruct-2507 variant and Llama-3.1-8B, it is within noise of AM in 9 of 10, no longer beats dropping the bias in KL, and the two each win once in accuracy.
Figures & tables
Figure 1: The fold moves each retained key along one shared direction u by βjd , so a query near the hyperplane q⋅u=1 (shaded) sees the logit shift βj(q⋅u)≈βj . Left: AM, a scalar bias βj per retained key kj . Right: the fold. For the orange open circle, a query far from the hyperplane, AM still adds βj while the fold scales it by q⋅u .
Figure 2: On the median head the default fold stays within about 5% of AM at every distance (the mean, worse far out, is in Appendix A.12 ). Median relative attention-output error over heads on Pride and Prejudice , on 512-query windows starting Δ tokens after the compacted context. Thin lines: interquartile range over heads for keep (dotted) and AM (solid). Shaded: median AM error above median keep error.
ratio
worse >5%
worst 10%
AM < keep (% heads)
model
near
far
far (%)
far (%)
Δ=0
24 k
Qwen3-1.7B
1.005
0.998
11
90
81
18
Qwen3-4B
1.011
1.021
35
92
85
17
Qwen3-8B
1.011
1.022
40
96
82
17
Table 1: On Pride and Prejudice the default fold tracks AM closely on the median head, and a tenth of the far (head, window) pairs carries most of its excess error. Ratio: median of efold/eAM over (head, window) pairs, pooled over near or far windows. Worse >5% : share of far pairs with efold/eAM>1.05 . Worst 10%: share of the total excess max(efold−eAM,0) in the worst tenth of far pairs. AM < keep: share of key-value heads on which AM’s error is below keep’s, in the window at that Δ .
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
Llama-3.1-8B
AM fit
AM-style
ours
ours
ours
budget
10%
5%
10%
5%
10%
5%
10%
5%
full cache
55.8
69.1
73.0
71.3
accuracy (%)
keep
45.5
42.3
62.2
58.8
70.2
62.5
65.7
62.4
drop
–
–
63.3
57.9
68.4
61.2
67.4
61.4
Table 2: On QuALITY no fold − AM accuracy difference in these settings is beyond noise. Fold − AM and drop − fold: paired differences with their 95% bootstrap intervals over the 30 articles below them, accuracy differences in points. Bold: beyond noise. KL > 1: questions with KL above 1 under drop and under the fold. 534 questions. Each column’s AM fit was chosen on the test set for AM’s lower KL (Setup). Drop was not run with the AM-style fit on QuALITY, so the Qwen3-1.7B columns have none, and Table 13 gives drop under our fit.
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
budget
10%
5%
10%
5%
10%
5%
full cache
68.3
77.4
80.4
accuracy (%)
keep
36.2
30.4
46.7
39.7
44.7
37.2
drop
38.9
33.9
54.3
45.2
51.3
42.5
AM
39.9
31.9
54.5
43.5
55.8
45.0
Table 3: On LongHealth the fold keeps AM’s accuracy gain over keep on Qwen3-4B and 8B and loses it on Qwen3-1.7B at 10%. Our AM fit throughout. Layout as in Table 2 , with intervals over the 20 patients (398 questions).
full
fold 10%
fold 5%
prompt tokens (mean)
6,533
781
462
KV per token (KiB)
144
144
144
capacity
4
31
44
decode (tokens/s)
29
200
297
same letter as HF
51/51
51/51
51/51
same 128 tokens as HF
46/51
42/51
44/51
Table 4: A folded Qwen3-4B cache served by unmodified vLLM 0.18.1 (Triton backend, fp16, one Tesla T4, 51 QuALITY questions, equal budgets per head). Capacity: concurrent requests that fit the KV pool with 256 generated tokens each. Decode: tokens/s at the largest batch that fits. Agreement with Hugging Face (HF) on the same cache: answer letter, and exact match of 128 greedy tokens. Correct letter: 51 questions are too few to measure accuracy, which Section 5 does.
Qasper
MF-en
HotpotQA
2WikiMQA
mean
full cache (HF)
43.8
47.5
60.5
41.7
48.4
AM (HF)
24.1
30.7
37.8
30.4
30.8
drop (HF)
20.0
29.8
37.4
33.1
30.1
fold (HF)
28.8
29.9
29.1
27.0
28.7
fold (vLLM)
28.0
30.2
28.9
29.5
29.2
fold − AM
+4.7
−0.8
−8.8
−3.3
−2.0
Table 5: Qwen3-14B with real weights on four LongBench tasks at 10%: the folded cache served by unmodified vLLM 0.18.1 scores as it does in Hugging Face (HF), and its mean is within noise of AM’s. F1 ×100 with the official scorer, 50 samples per task, AM-style fit, equal budgets per head; bf16, one A100-80GB. Fold − AM: paired difference in the HF runs with its 95% bootstrap interval over samples.
capacity
decode (tokens/s)
model
context (max)
full
10%
5%
full
10%
5%
Qwen3-32B
6,969
7
45
64
135 (7)
872 (45)
1,232 (64)
Qwen3-32B
13,754 ( ×2 )
3
28
45
59 (3)
534 (28)
864 (45)
Qwen3-14B
6,969
39
255
256
766 (39)
3,442 (128)
3,831 (128)
Qwen3-14B
27,324 ( ×4 )
10
90
158
196 (8)
1,000 (32)
1,182 (32)
Table 6: Folded caches of larger models served by unmodified vLLM 0.18.1 (default FlashAttention backend, bf16, one A100-80GB, random weights of the real architectures, 51 QuALITY questions). Capacity: concurrent requests that fit the KV pool with 256 generated tokens each. Decode: the highest rate measured, in tokens/s, at the batch in parentheses. At 5% the 14B model reaches vLLM’s limit of 256 concurrent sequences, and at 10% its pool holds 255, one short of it. The ×2 and ×4 rows repeat each article to lengthen the context. 14B peak rates are measured at batch ≤128 and are lower bounds. Fold columns at the native length are injected caches; at the longer lengths they are plain prompts of the fold’s length, which decode at the same rate.
method
per-entry logit term
fold applies
AM [ 33 ]
β=logw , NNLS fit, ∣β∣≤3
yes (this paper)
KV-Surgeon, ARC-KV, TAM, online AM [ 23 , 21 , 16 , 17 ]
Table 7: Logit terms of compaction and merging methods. Static: one scalar per cache entry, the same for every query, which the fold handles. Fold applies: structurally, since the fold has been tested only on AM, one capped log-count merge and two of KV-Surgeon’s compactors.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
10%
5%
acc
agree
KL
acc
agree
KL
Qwen3-4B, QuALITY; full cache 69.1%
keep
69.5
84.5
1.18
63.3
77.5
1.93
drop
68.7
86.0
1.14
64.4
79.4
1.65
AM
67.8
86.7
0.90
65.2
81.1
1.65
fold
68.5
86.5
0.96
64.8
80.9
1.80
Appendix
Table 8: The fold under each model’s published per-head budgets: level with AM in nine of the ten cells, and below it on Llama-3.1-8B LongHealth at 5%. AM-style fit with the published NNLS weighting, bf16; Qwen3-4B QuALITY on a Tesla T4, the rest on an A100. Below each paired difference, its 95% bootstrap interval over the 30 articles (534 questions) or 20 patients (398); accuracy and agreement in %, differences in points. Bold: beyond noise.
10%
5%
acc
agree
KL
acc
agree
KL
Llama-3.1-8B-Instruct, QuALITY; full cache 71.3%
keep
69.3
85.0
0.19
67.6
82.8
0.28
drop
68.9
86.7
0.16
66.9
81.8
0.28
AM
69.3
89.0
0.12
66.7
84.6
0.22
fold
68.9
88.4
0.14
66.7
83.0
0.23
Appendix
Table 8: (continued) The fold under the published per-head budgets on Llama-3.1-8B-Instruct, QuALITY; layout as above.
accuracy (%)
fold − AM
run
keep
drop
AM
fold
acc
KL
as in the main runs ( ∣β∣≤3 )
65
63
62
57
− 5 [ − 9, − 1]
+0.18 [+0.14, +0.23]
folded keys in fp32
65
–
62
58
− 4 [ − 7, − 1]
+0.19 [+0.14, +0.24]
AM’s values
65
–
61
64
+3 [ − 1, +5]
− 0.01 [ − 0.05, +0.03]
self-study fit queries
69
65
64
64
+0 [ − 4, +4]
+0.03 [ − 0.03, +0.11]
∣β∣≤1
65
63
71
70
− 1 [ − 6, +5]
− 0.04 [ − 0.09, +0.01]
Appendix
Table 9: Diagnosing the fold’s loss on Llama-3.1-8B LongHealth at 5%. Upper panel: patients 0–4 (100 questions, so accuracies are whole points), full-cache accuracy 79%. Lower panel: all 20 patients (398 questions), full-cache accuracy 80.2%; the ∣β∣≤3 row is the run of Table 8 , and the ∣β∣≤1 row ran on another A100 with identical full-cache, keep and drop log-probs. Published per-head budgets, AM-style fit with the published NNLS weighting, A100, bf16. AM’s values: AM and the fold both keep AM’s original values instead of refitting them. Fold − AM: paired difference with its 95% bootstrap interval over the patients. Bold: beyond noise.
10%
5%
acc
agree
KL
acc
agree
KL
QuALITY
merge-drop
69.1
84.3
1.15
62.9
76.4
1.95
merge-log
67.8
86.0
0.97
64.2
80.0
1.72
merge-fold
67.2
85.4
0.98
64.4
80.9
1.74
merge-refit
68.5
86.3
0.90
66.3
81.1
1.50
Appendix
Table 10: Log-count merging on Qwen3-4B: folding the count bias into the keys reproduces merge-log within noise on both benchmarks. Published per-head budgets, AM-style fit, cosine gate 0.9 on post-RoPE keys, at most 16 keys per cluster; bf16, QuALITY on a Tesla T4 and LongHealth on an A100. Differences: paired, with their 95% bootstrap intervals over the 30 articles or 20 patients; accuracy and agreement in %, differences in points. Bold: beyond noise. QuALITY: 534 questions, full-cache accuracy 69.1%; LongHealth: 398 questions, 77.4%. Keep, on the same keys, is in Table 8 .
accuracy (%)
unparsed
base compactor
base
fold
fold − base
gained/lost
base/fold
KV-Surgeon-fast
68.5
71.8
+3.3 [ − 1.7, +8.2]
14/8
5/3
KVMerger, fitted β
68.0
65.2
− 2.8 [ − 6.8, +1.1]
11/16
5/11
Appendix
Table 11: Folding the bias of two compactors inside KV-Surgeon’s code and harness, Qwen3-4B QuALITY at 5%: neither difference is beyond noise. KV-Surgeon’s per-head budgets, answers generated with thinking on and parsed by its code, unparsed answers counted wrong; A100, bf16. Fold − base: paired difference in points with its 95% bootstrap interval over the 10 articles. Gained/lost: questions right only with the fold / only without it. 181 questions.
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
10%
5%
10%
5%
10%
5%
KL(full∥⋅)
AM: no refit − refit
− 0.26
− 0.02
+0.37
+0.35
+0.09
+0.01
[ − 0.68, +0.14]
[ − 0.42, +0.39]
[+0.12, +0.64]
[+0.09, +0.59]
[ − 0.11, +0.30]
[ − 0.27, +0.28]
fold: no refit − refit
− 0.53
− 0.21
+0.43
+0.44
+0.36
+0.67
[ − 0.97, − 0.08]
[ − 0.67, +0.24]
[+0.09, +0.78]
[+0.13, +0.75]
[+0.08, +0.67]
[+0.33, +1.00]
Appendix
Table 12: On 4B and 8B the value refit and the covariance term in u each lower the fold’s KL. On 1.7B under our fit both raise it. Each row is a variant minus the cache it modifies, paired within one ablation run, with its 95% bootstrap interval over the 30 articles below it. QuALITY, our AM fit. Mean dir. : u=μ/∥μ∥2 instead of ( 6 ). Absolute values for the default caches are in Tables 2 and 13 .
accuracy (%)
KL(full∥⋅)
model
budget
keep
drop
AM
fold
keep
drop
AM
fold
Δ acc
Δ KL
Qwen3-1.7B
20%
50.4
45.7
48.1
46.6
1.74
3.00
2.07
2.41
− 1.5
+0.34
[ − 4.4, +1.1]
[+0.13, +0.54]
10%
48.3
45.3
43.1
47.2
2.82
3.62
2.57
2.88
+4.1
+0.31
[+1.7, +6.8]
[+0.08, +0.53]
5%
45.7
42.3
40.8
38.4
3.59
4.47
2.69
2.89
− 2.4
+0.20
Appendix
Table 13: QuALITY under our AM fit, on all four models at 20% and 2% and on Qwen3-1.7B also at 10% and 5%: the fold’s KL is below AM’s beyond noise only on Qwen3-8B at 2%, above it on Qwen3-1.7B at 20% and 10%, and level elsewhere. Δ : fold minus AM, with the 95% bootstrap interval over the 30 articles below it; accuracy in %, Δ acc in points. 534 questions; full-cache accuracy 55.6, 69.1, 73.0 and 71.3% (55.6 here and 55.8 in Table 2 : run-to-run variation, Appendix A.10 ).
KL(full∥⋅)
Qwen3
fit
budget
keep
AM
fold
Δ KL
Δ acc (points)
1.7B
AM-style + SS
10%
3.78
2.11
2.26
+0.14 [ − 0.00, +0.30]
+0.9 [ − 0.7, +2.4]
5%
4.97
2.44
2.65
+0.21 [+0.07, +0.37]
− 0.9 [ − 3.4, +1.7]
4B
AM-style
10%
2.10
2.40
2.38
− 0.02 [ − 0.20, +0.17]
− 0.6 [ − 2.5, +1.5]
5%
2.88
3.20
3.21
+0.01 [ − 0.16, +0.16]
+1.1 [ − 0.8, +3.0]
8B
AM-style
10%
1.48
1.35
1.18
− 0.16 [ − 0.30, − 0.03]
+0.4 [ − 1.5, +2.2]
Appendix
Table 14: QuALITY under the AM-style fit, on Qwen3-4B and 8B and with self-study queries on 1.7B: the fold’s KL is lower than AM’s on 8B, at the edge of noise, level on 4B, and higher on 1.7B with self-study queries, beyond noise at 5%. SS (self-study) adds the queries of AM’s four self-study prompts to Q . Keep differs from Table 2 because the kept keys depend on the fit’s ranking and, with SS, on Q . Δ : fold minus AM with 95% paired bootstrap interval over the 30 articles.
accuracy (%)
KL(full∥⋅)
model
budget
keep
drop
AM
fold
keep
drop
AM
fold
Δ acc
Δ KL
Qwen3-1.7B
10%
44.5
39.7
40.7
37.7
3.62
3.78
3.23
3.36
− 3.0
+0.13
[ − 5.5, − 0.5]
[ − 0.07, +0.35]
5%
35.9
35.2
32.4
34.4
4.95
4.72
4.04
4.25
+2.0
+0.21
[ − 1.0, +5.5]
[ − 0.02, +0.46]
Qwen3-4B
10%
48.5
45.7
46.5
47.5
4.25
4.96
4.79
4.88
+1.0
+0.09
Appendix
Table 15: LongHealth under the AM-style fit (our fit: Table 3 ): AM’s KL is higher than under our fit at every size and budget, and its accuracy is at or below keep’s in every cell. Columns as in Table 13 ; intervals over the 20 patients, 398 questions. Full-cache accuracy 68.3, 77.4 and 80.4%.
fold − AM
drop − fold
benchmark
model
10%
5%
10%
5%
QuALITY
Qwen3-1.7B
+0.7 [ − 1.7, +3.3]
+0.6 [ − 1.7, +3.1]
–
–
Qwen3-4B
− 1.3 [ − 3.3, +0.7]
− 1.1 [ − 3.4, +1.1]
− 3.6 [ − 6.0, − 1.1]
− 3.0 [ − 6.2, +0.0]
Qwen3-8B
− 0.7 [ − 3.2, +1.7]
− 0.6 [ − 2.5, +1.3]
− 0.6 [ − 2.4, +1.1]
− 1.3 [ − 4.9, +2.1]
Llama-3.1-8B
+0.6 [ − 1.3, +2.6]
+0.0 [ − 1.5, +1.5]
− 0.4 [ − 3.0, +2.4]
− 0.7 [ − 4.9, +3.0]
LongHealth
Qwen3-1.7B
− 7.0 [ − 12.2, − 2.0]
− 1.5 [ − 4.8, +1.8]
+3.3 [ − 0.5, +6.8]
− 1.0 [ − 5.1, +3.3]
Appendix
Table 16: Agreement with the full cache’s top option: paired differences in points, with 95% bootstrap intervals over contexts. AM fit as in Tables 2 and 3 ; drop was not run with the AM-style fit on QuALITY.
fold
AM
stock run
model
acc (%)
same (%)
acc (%)
same (%)
fold − AM KL
drop − fold KL
Qwen3-1.7B
47.2 / 46.4
97.9
43.1 / 43.6
95.9
+0.28 [ +0.06 , +0.49 ]
+0.71 [ +0.42 , +1.03 ]
Qwen3-4B
62.7 / 63.3
98.7
63.7 / 63.9
99.6
−0.01 [ −0.17 , +0.15 ]
+0.68 [ +0.44 , +0.96 ]
Qwen3-8B
69.1 / 69.3
98.9
68.4 / 68.9
99.1
−0.21 [ −0.39 , −0.05 ]
+0.59 [ +0.40 , +0.81 ]
Llama-3.1-8B
67.6 / 67.2
99.4
66.1 / 66.5
99.1
−0.00 [ −0.02 , +0.01 ]
+0.08 [ +0.04 , +0.12 ]
Appendix
Table 17: QuALITY at 10%, our fit: the fp32 explicit-softmax path of the main text against stock PyTorch kernels in bf16 (flash without a bias, masked SDPA for AM). Accuracy and KL are fp32 / stock. Same: share of questions with the same top option in both runs. Last two columns: fold − AM KL and drop − fold KL in the stock run, with 95% bootstrap intervals.
decode ( qlen=1 )
prefill ( qlen=512 )
B=1
B=8
B=1
B=8
L=512 , 1024, 2048, 4096 entries
biased
1.06 (1.02–1.29)
1.58 (1.09–1.90)
1.27 (1.16–1.50)
1.26 (1.16–1.39)
Flex
1.00 (1.00–1.23)
1.29 (1.00–1.45)
1.16 (1.00–1.43)
1.16 (1.07–1.24)
Flex +β / Flex
1.05 (1.00–1.29)
1.19 (1.09–1.33)
1.08 (1.06–1.31)
1.08 (1.06–1.30)
flash ( μ s)
13–22
19–92
52–344
130–1503
Appendix
Table 18: With launch cost removed by CUDA graphs, AM’s bias costs little in single-sequence decode and most at batch-8 decode. Top: 512–4096 cached entries, which covers our compacted caches. Bottom: 16k–32k. CUDA-graphed per-call times, one run per model. Cells: time over the fastest plain time (flash, cuDNN or FlexAttention) of the same model and setting, median over the four models and the L values of the panel, min–max in parentheses. Biased: fastest implementation that accepts the bias. Flex: FlexAttention without a bias. Flex +β /Flex: FlexAttention with the bias over FlexAttention without it. Flash: per-call time of plain flash SDPA in μ s, min–max.
The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a retention-preserving hierarchy for mixed-format paged attention. Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3; every live-request page remains addressable. Format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax merge, enabling direct heterogeneous decode without a cache-sized dense shadow. Across separate, configuration-bound Qwen3.6-27B profiles on a single 96-GB NVIDIA RTX PRO 6000 Blackwell GPU, deployment accounting reports 18.3 KiB of attention KV per live token, corresponding to 3.50x compression relative to BF16 and 1.75x relative to FP8. A materializing quality profile matches its dense control on 16K RULER needle-in-a-haystack tasks. On the same 503-question LongBench v2 set, measured deltas are -0.80, -0.60, and -0.40 percentage points at 16K, 32K, and 64K. A separate single-pair direct-decode canary with two 59,008-token requests measures 3.625x active-KV compression and 0.9821x throughput relative to its control, routes all 16 full-attention layers without fallback, and retains no dense shadow. These results establish a practical mixed-format path for compressing long-context state without evicting live-request KV pages.
Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest. On schema-dense input streams such as nested JSON, this score acts as a non-stationary filter that disproportionately retains noise: a non-content sink role (delimiters or whitespace) carries an order of magnitude more energy than any content role, and structural KEY tokens are over-retained at roughly 1.8x the rate of the answer-carrying VALUE tokens, collapsing exact-match accuracy from 88% to 0% at a 5% budget as the signal-to-noise ratio of the retained state degrades. A counterfactual experiment establishes that suppressing KEY tokens is the best deployable filter. Our retraining-free, role-conditional allocation over SnapKV's windowed score, governed by a single tuned hyperparameter, closes 63-98% of the H2O gap at sub-20% budgets and, at higher budgets, modestly matches or exceeds full-cache accuracy -- a small, seed-sensitive denoising effect (borderline significant at B=0.50; not distinguishable from zero at B=0.30 over four seeds). A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains open.
Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable reasoning. To avoid this issue, we turn to the pre-RoPE space, where we observe that Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions -- Q/K concentration. We show that this concentration causes queries to preferentially attend to keys at specific distances (e.g., nearest keys), with the centers determining which distances are preferred via a trigonometric series. Based on this, we propose TriAttention to estimate key importance by leveraging these centers. Via the trigonometric series, we use the distance preference characterized by these centers to score keys according to their positions, and also leverage Q/K norms as an additional signal for importance estimation. On AIME25 with 32K-token generation, TriAttention matches Full Attention reasoning accuracy while achieving 2.5x higher throughput or 10.7x KV memory reduction, whereas leading baselines achieve only about half the accuracy at the same efficiency. TriAttention enables OpenClaw deployment on a single consumer GPU, where long context would otherwise cause out-of-memory with Full Attention.