Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.
Figures & tables
Figure 1: Accuracy vs. Compaction Time Trade-off (Qwen3-4B; QuALITY). We compare downstream QA accuracy ( n=894 ) after compaction, plotted against the average wall-clock time required to compact a context (seconds, log-scale) using a single H100 GPU at a fixed 50 × compaction ratio. Our attention-matching (AM) methods trace a speed–quality tradeoff and form the Pareto frontier, outperforming prior token-selection baselines and exceeding the performance of Cartridges ( Eyuboglu et al., 2025 ) while being 2 orders of magnitude faster; additional Cartridges training may further improve its results.
Figure 2: KV compaction via Attention Matching. We replace the full KV cache (K,V) with a smaller cache (Ck,Cv) chosen so that attention outputs are approximately preserved when the compacted prefix is concatenated with arbitrary fixed or future tokens (Kfixed,Vfixed) .
Figure 3: Head sensitivity curves in Qwen3-4B. We fix all KV heads to a baseline compaction ratio of 0.05× and vary the compaction ratio of a single head. We report the change in loss relative to the baseline (lower is better) as a function of the varied head’s compaction ratio. Curves are averaged over 10 QuALITY articles; shaded regions denote ± 1 standard error of the mean across articles. Some heads (e.g., L0H0 ) are largely insensitive to additional capacity, whereas others (e.g., L15H2 ) benefit substantially from storing more KV pairs.
Figure 4: Accuracy vs. compaction ratio across methods. We compare AM-OMP and AM-HighestAttnKeys against Cartridges, summarization, and four prior methods. Evaluations are conducted on QuALITY and LongHealth using Qwen3-4B, Llama3.1-8B, and Gemma3-12B. Attention Matching (AM) consistently outperforms other approaches across compaction ratios, while matching Cartridges’ performance at ultra-high compaction.
Method
0%
1%
2%
5%
10%
20%
100%
Qwen3-4B
Baseline (no ctx / orig.)
0.104
0.444
Summarization
0.219
0.283
H2O+
0.188
0.243
0.310
0.350
0.390
KVMerger
0.171
0.215
0.260
0.303
0.356
KVzip
0.127
0.124
0.220
0.411
0.438
Table 1: QASPER, average token-level F1 ( 562 contexts, 2,010 questions).
Method
0%
1%
2%
5%
10%
20%
100%
Baseline (no ctx / orig.)
17.0
72.3
Summarization
23.4
29.8
44.7
H2O+
10.6
21.3
36.2
44.7
61.7
KVMerger
19.2
21.3
34.0
44.7
48.9
KVzip
17.0
21.3
36.2
48.9
66.0
AM-HighestAttnKeys
61.7
59.6
66.0
70.2
68.1
Table 2: LongBench v2 accuracy (%) , Qwen3-4B-Instruct-2507, 47 -context subset (correct with full context, incorrect without; max 97 k, avg 42 k tokens).
Stage
Method
Time (s)
context-prefill
7
Query generation
repeat-prefill
8
self-study
139
Highest attention
3
Key selection
OMP
565
OMP-fast ( k=4 , refit interval=2)
104
Table 3: Wall-clock compaction time breakdown on a 60k-token LongHealth context with Gemma-3-12B on a single H200 GPU .
Method
Cache Size (%)
Accuracy (%)
Full Context
100.00
71.1
Summarize
4.80
55.2
Summarize × (0.2× AM-OMP)
0.92
55.7
Summarize × (0.1× AM-OMP)
0.46
55.0
Summarize × (0.05× AM-OMP)
0.21
49.2
Table 4: Summarization plus Attention Matching. Applying AM-OMP on top of a summary enables up to ∼ 200 × total compaction with accuracy comparable to summarization alone.
Figure 5: Leave-one-out experiments. We ablate our main AM-OMP method and measure average log-perplexity of generations decoded from the original cache, since this is lower in variance and well-correlated with downstream performance (Appendix F.2 ). In “no biases,” we first compute OMP as usual and then zero out β , keeping the keys selected by OMP.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Context
Self-Study Tokens
H100-hours
LongHealth-5
60k
100–150M
∼15
Quality
7k
25–30M
∼5
LongHealth-10 (Original)
120k
300M
-
Appendix
Table 5: Training compute for Cartridges. We sample approximately the same number of self-study tokens per context token as in the original Cartridges implementation.
Method
Queries
Key selection
β
Values
Nonuniform
H2O+
context-prefill
HighestAttnKeys
none
direct
none
SnapKV (agnostic)
mock-question
HighestAttnKeys
none
direct
none
PyramidKV (agnostic)
mock-question
HighestAttnKeys
none
direct
linearly decreasing
KVzip-uniform
repeat-prefill
HighestAttnKeys
none
direct
none
KVzip
repeat-prefill
GlobalHighestAttnKeys
none
direct
global top- k
KVMerger
none
cosine-similarity merge
none
merged
none
Appendix
Table 6: Baseline configurations. “Mock-question” denotes a single synthetic question generated via self-study to instantiate originally question-aware methods in a question-agnostic setting. “Direct” values indicate that retained keys keep their original values (i.e., no value fitting with least squares); “merged” values denote Gaussian-kernel weighted averaging of clustered tokens (KVMerger).
Name
Prompt
summarize
Summarize the following text: {article_text} Summary:
summarize_indepth
Summarize the following text in depth: {article_text} Summary:
summarize_keypoints_questions
Summarize the following text, providing all key points that might be necessary to answer comprehension questions: {article_text} Summary:
summarize_concise
Summarize the following text concisely: {article_text} Summary:
summarize_very_concise
Summarize the following text very concisely: {article_text} Summary:
Appendix
Table 7: Summarization prompts.
Name
Prompt
3-question
Write 3 questions that test understanding of different parts of the context. Answer with just the 3 questions and options (do not say the correct answer), each one separated with 2 newlines.
summarize
Summarize the main points of the context.
structure_json
Structure the information in JSON form and include all important details like dates, times, names, and numerical values.
aggregate
Aggregate all the key facts mentioned in the context.
Appendix
Table 8: Prompt templates used in self-study.
Figure 6: Reference query sampling comparison. We compare eight variants for sampling queries. Self-study-based methods perform best, especially at the greatest compaction ratios, with repeat and context-prefill close behind. Subsampling reference queries preserves performance.
Figure 7: Head proportions induced by global highest-attention selection. We plot the fraction of total selected keys attributed to each of the 288 KV heads ( 36 layers, 8 heads per layer) in Qwen3-4B. Top: proportions across 10 QuALITY articles at a fixed global compaction ratio of 0.05 . Bottom: average proportions across multiple global compaction ratios. The low variance across articles and target ratios indicates stable head importance patterns.
Figure 8: More Head sensitivity curves. Analogous to Figure 3 , showing loss change as a function of per-head compaction ratio for four selected heads. Left: Qwen3-4B with LongHealth exhibits similar patterns to Figure 3 . Right: Llama-3.1-8B-Instruct also shows head-dependent sensitivity patterns: some heads are largely insensitive to KV capacity, while others benefit substantially from retaining more keys.
Algorithm 4 Nonuniform KV Allocation via Greedy Swaps
Figure 9: Optimized head budgets. We compare the head proportions learned by our nonuniform allocation algorithm to those induced by global highest-attention key selection. The optimized schedule has similar structure as the global highest attention scores schedule. The optimized schedule tends to assign more budget to later layers and less to earlier layers, contrary to PyramidKV ( Cai et al., 2025 ) .
Figure 10: Effect of aggregation function on key selection. We plot accuracy versus compaction ratio for four baseline methods and our Highest Attention Keys method on a subset of 20 QuALITY articles using Qwen3-4B. All methods select the top- t keys based on aggregated attention scores over a shared set of reference queries.
Figure 11: Reconstruction loss vs. downstream accuracy. Left: Loss-based analogue of Figure 1 , instead reporting log-perplexity on original-cache generations. Right: Downstream multiple-choice accuracy plotted against this loss across various methods and compaction ratios.
Phys Len
Eff Len
AIME (/30)
Notes
2048
2048
1.25
4096
4096
7.75
8192
8192
13
With online compaction
2048
4096
8
≤ 2 compactions
2048
8192
13
≤ 6 compactions
Appendix
Table 9: AIME performance (pass@1, averaged over 4 runs) with and without mid-trajectory compaction. Effective sequence length is the total number of decoded tokens, while physical sequence length is the maximum physical size of the KV cache. Each compaction reduces the size of the entire KV cache (including previously compacted portions) by 50% , so ≤ 2 and ≤ 6 compactions correspond to approximately 2× and 4× more decoded tokens, respectively. For implementation simplicity, we do not enable non-uniform compaction or on-policy queries. We use AM-HighestAttnKeys-fast. We believe more tuned methods (e.g. with higher compaction ratios, compacting only the most recently generated tokens, etc.) may improve results.
Method
0%
1%
2%
5%
10%
20%
100%
Baseline (no ctx / orig.)
36.8
71.1
KVMerger
43.4
49.4
55.1
60.9
65.5
AM-HighestAttnKeys
60.4
65.0
70.1
71.3
71.0
Appendix
Table 10: QuALITY accuracy (%) with KVMerger , Qwen3-4B, 50 contexts. KVMerger tracks our token-eviction baselines and is well below Attention Matching.
Method
0%
5%
12.5%
100%
Llama-3.1-8B-Instruct
Baseline (no ctx / orig.)
2.6
95.7
PyramidKV
27.2
TOVA
46.4
KeyDiff
55.7
Finch (query-aware)
66.7
Appendix
Table 11: RULER accuracy (%) over 6,500 kvpress tasks. Even our fastest variant outperforms KVzip and all leaderboard methods at every evaluated ratio.
The KV cache is the memory bottleneck of long-horizon language model deployment. Practically, a deployable compactor must be lightweight enough to call during inference, expressive enough to preserve context under constraint, and reusable across a trajectory. Existing compaction methods satisfy only part of this requirement: selection methods are lightweight but subset-bound, while synthesis methods are expressive but rely on per-context optimization. Here we introduce Still, a small per-layer Perceiver trained once against a frozen base model that produces compact keys and values in a single forward pass. On Qwen and Gemma models, Still occupies the favorable side of the speed--quality frontier across compression ratios from 8× to 200× and context lengths from 8k to 128k. On the long-context RULER grid, Still exceeds the strongest baseline by 8--22 points. The same compact cache also supports free-form summarization, preserving most of the full-context gain on HELMET and winning a pairwise LongBench summarization comparison against KV-Distill. Because compaction is a forward pass, Still can be applied iteratively, entering a long-horizon regime unavailable to per-context methods. We show that amortization makes long-context cache compaction tractable, and synthesis makes its compact state useful at extreme compression.
Charles O'Neill, Alex Sandomirsky, Harry Partridge +2
Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce Latent Compaction, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32× compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural network and optimizing it with test-time training can preserve retrieval accuracy under aggressive compression. The code is released at https://www.github.com/THUNLP/latent-compaction
Yingfa Chen, Xingyu Shen, Chaojun Xiao +1
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China
Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task performance with compact KV caches. However, iterative anchor search dominates the compaction cost of OMP-based Attention Matching. This motivates our selective amortization principle of learning a reusable anchor-selection policy across contexts while retaining context-specific reconstruction. In this work, we propose ARC-KV, a novel reconstruction-based KV cache compaction method that follows this principle. To this end, we first train a value-aware indexer to select real-key anchors in a single scoring pass. ARC-KV then applies convex-hull-constrained key merging and fits an attention-mass bias and compact values against the full cache. At inference time, ARC-KV builds the compact cache once per context using the frozen indexer and reuses it for all subsequent queries. Extensive experiments demonstrate that ARC-KV outperforms reported compaction methods in most settings across QuALITY, RULER, and LongBench on Llama-3.1-8B-Instruct. In particular, at 10% KV retention on QuALITY, ARC-KV improves accuracy from 0.6409 to 0.6474 over Attention Matching while reducing compaction time by a factor of 25.73, from 959.8 s to 37.3 s.