Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce Latent Compaction, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32× compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural network and optimizing it with test-time training can preserve retrieval accuracy under aggressive compression. The code is released at https://www.github.com/THUNLP/latent-compaction
Figures & tables
Figure 1: Latent Compaction and baselines across compression ratios and context lengths on RULER and Phonebook Lookup with Qwen3-1.7B.
Figure 2: Latent Compaction training pipeline. Preservers, anchors, and diversifiers are initialized from the original cache. Stage 1 freezes preservers and fits anchors and diversifiers to full-cache attention outputs. Stage 2 refines all entries through end-to-end distillation.
Figure 3: Comparison of compression methods on LongBench.
Figure 4: Phonebook Lookup results for additional Qwen3 model sizes and MiniCPM5 models. Each model uses its own document-specific cache optimization.
Figure 5: Phonebook Lookup (4K, 8K, and 16K context lengths) accuracy of Latent Compaction at 16× compression ratio with different fractions of slots allocated to preservers, and different anchor fractions within the combined anchor and diversifier budget.
↪ Replace diversifiers with Gaussian random vectors
27.50
17.50
14.25
19.75
↪ Replace diversifiers with first document KVs
65.50
42.50
22.50
43.50
↪ Replace anchors with KVZip+
69.25
58.75
43.50
57.17
Table 1: Initialization ablations on Phonebook Lookup with Qwen3-1.7B at 16× compression.
Method
4K
8K
16K
Avg.
Latent Compaction
87.75
83.75
70.50
80.67
↪ Remove stage 1
13.75
13.25
14.75
13.92
↪ Remove frozen preserver KVs in stage 1
83.25
60.75
43.50
62.50
↪ Preserver KVs are trainable in stage 1
43.75
32.25
24.25
33.42
↪ Remove logZ MSE loss term
42.50
38.00
29.00
36.50
Table 2: Stage-1 ablation results on Phonebook Lookup with Qwen3-1.7B at 16× compression.
Method
4K
8K
16K
Avg.
Latent Compaction
87.75
83.75
70.50
80.67
↪ Remove stage 2
81.00
68.25
62.75
70.67
↪ KL-divergence only
83.00
83.50
65.25
77.25
↪ Remove tail-probability term in KL divergence
82.00
81.75
61.00
75.25
↪ Full-vocabulary distillation
87.75
84.25
70.00
80.67
Table 3: Stage-2 ablations on Phonebook Lookup with Qwen3-1.7B at 16× compression. The default groups the teacher distribution into its top 20 tokens and the remaining mass.
Figure 6: Phonebook Lookup accuracy of Latent Compaction of varying training durations. For clarity, dotted lines only show the accuracy at step 0 and step 1,000 of stage 2.
Method
4K
8K
16K
Avg.
Latent Compaction
43.75
32.25
24.25
33.42
↪ Remove attention biases (bcmp)
43.00
29.25
24.00
32.08
↪ Equal mix: replace Eq. 5 with 21(Acmp+Anew)
0.25
0.00
0.25
0.17
Table 4: Architectural ablations on Phonebook Lookup at 16× compression.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qwen3-1.7B training losses on 16K Phonebook Lookup at 16× compression. Stage 1 shows the normalized attention-fitting objective (Equation 8 ); stage 2 shows grouped KL and next-token cross-entropy (Equation 10 ). Each stage runs for 1,000 updates. Thin lines show per-step means; thick lines show 25-step moving averages.
Figure 9: RULER NIAH-Single-1: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 10: RULER NIAH-Single-2: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 11: RULER NIAH-Single-3: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 12: RULER NIAH-Multi-Key-1: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 13: RULER NIAH-Multi-Key-2: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 14: RULER NIAH-Multi-Key-3: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 15: RULER NIAH-Multi-Query: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 16: RULER NIAH-Multi-Value: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 17: RULER VT: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 18: RULER CWE: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 19: RULER FWE: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 20: RULER QA-1: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 21: RULER QA-2: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.
The KV cache is the memory bottleneck of long-horizon language model deployment. Practically, a deployable compactor must be lightweight enough to call during inference, expressive enough to preserve context under constraint, and reusable across a trajectory. Existing compaction methods satisfy only part of this requirement: selection methods are lightweight but subset-bound, while synthesis methods are expressive but rely on per-context optimization. Here we introduce Still, a small per-layer Perceiver trained once against a frozen base model that produces compact keys and values in a single forward pass. On Qwen and Gemma models, Still occupies the favorable side of the speed--quality frontier across compression ratios from 8× to 200× and context lengths from 8k to 128k. On the long-context RULER grid, Still exceeds the strongest baseline by 8--22 points. The same compact cache also supports free-form summarization, preserving most of the full-context gain on HELMET and winning a pairwise LongBench summarization comparison against KV-Distill. Because compaction is a forward pass, Still can be applied iteratively, entering a long-horizon regime unavailable to per-context methods. We show that amortization makes long-context cache compaction tractable, and synthesis makes its compact state useful at extreme compression.
Charles O'Neill, Alex Sandomirsky, Harry Partridge +2
Long-context language models are limited by the memory footprint of the key-value (KV) cache. Existing training-free KV compression methods usually rank tokens by one importance signal -- attention, recency, layer-wise allocation, or key distinctiveness -- which becomes brittle when useful context is globally distinctive, locally episodic, or immediately relevant. We introduce NestedKV, a key-only KV cache compression method inspired by the Continuum Memory System in Nested Learning. NestedKV maintains global, block-level, and sliding-window key anchors, scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner using head-adaptive mixing and surprise-gated token routing. The score is paired with adaptive per-head budgets and requires no training or LLM modification. Across RULER (4k--32k), LooGLE, LongBench, LongBench-E, InfiniteBench, and MMLU-Pro on Qwen3 and Llama-3.2 models, NestedKV is strongest when the retained cache is small. On Qwen3-4B, it improves over KeyDiff by up to 19.10 points on RULER and 19.29 on LongBench at r=0.75; at r=0.95, it retains 37.32 on LongBench versus 17.55 for KeyDiff.
Hong Chen, Xiang Liu, Yubo Gao +5
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University