Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce Latent Compaction, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32× compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural network and optimizing it with test-time training can preserve retrieval accuracy under aggressive compression. The code is released at https://www.github.com/THUNLP/latent-compaction
Figures & tables
Figure 1: Latent Compaction and baselines across compression ratios and context lengths on RULER and Phonebook Lookup with Qwen3-1.7B.
Figure 2: Latent Compaction training pipeline. Preservers, anchors, and diversifiers are initialized from the original cache. Stage 1 freezes preservers and fits anchors and diversifiers to full-cache attention outputs. Stage 2 refines all entries through end-to-end distillation.
Figure 3: Comparison of compression methods on LongBench.
Figure 4: Phonebook Lookup results for additional Qwen3 model sizes and MiniCPM5 models. Each model uses its own document-specific cache optimization.
Figure 5: Phonebook Lookup (4K, 8K, and 16K context lengths) accuracy of Latent Compaction at 16× compression ratio with different fractions of slots allocated to preservers, and different anchor fractions within the combined anchor and diversifier budget.
↪ Replace diversifiers with Gaussian random vectors
27.50
17.50
14.25
19.75
↪ Replace diversifiers with first document KVs
65.50
42.50
22.50
43.50
↪ Replace anchors with KVZip+
69.25
58.75
43.50
57.17
Table 1: Initialization ablations on Phonebook Lookup with Qwen3-1.7B at 16× compression.
Method
4K
8K
16K
Avg.
Latent Compaction
87.75
83.75
70.50
80.67
↪ Remove stage 1
13.75
13.25
14.75
13.92
↪ Remove frozen preserver KVs in stage 1
83.25
60.75
43.50
62.50
↪ Preserver KVs are trainable in stage 1
43.75
32.25
24.25
33.42
↪ Remove logZ MSE loss term
42.50
38.00
29.00
36.50
Table 2: Stage-1 ablation results on Phonebook Lookup with Qwen3-1.7B at 16× compression.
Method
4K
8K
16K
Avg.
Latent Compaction
87.75
83.75
70.50
80.67
↪ Remove stage 2
81.00
68.25
62.75
70.67
↪ KL-divergence only
83.00
83.50
65.25
77.25
↪ Remove tail-probability term in KL divergence
82.00
81.75
61.00
75.25
↪ Full-vocabulary distillation
87.75
84.25
70.00
80.67
Table 3: Stage-2 ablations on Phonebook Lookup with Qwen3-1.7B at 16× compression. The default groups the teacher distribution into its top 20 tokens and the remaining mass.
Figure 6: Phonebook Lookup accuracy of Latent Compaction of varying training durations. For clarity, dotted lines only show the accuracy at step 0 and step 1,000 of stage 2.
Method
4K
8K
16K
Avg.
Latent Compaction
43.75
32.25
24.25
33.42
↪ Remove attention biases (bcmp)
43.00
29.25
24.00
32.08
↪ Equal mix: replace Eq. 5 with 21(Acmp+Anew)
0.25
0.00
0.25
0.17
Table 4: Architectural ablations on Phonebook Lookup at 16× compression.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qwen3-1.7B training losses on 16K Phonebook Lookup at 16× compression. Stage 1 shows the normalized attention-fitting objective (Equation 8 ); stage 2 shows grouped KL and next-token cross-entropy (Equation 10 ). Each stage runs for 1,000 updates. Thin lines show per-step means; thick lines show 25-step moving averages.
Figure 9: RULER NIAH-Single-1: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 10: RULER NIAH-Single-2: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 11: RULER NIAH-Single-3: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 12: RULER NIAH-Multi-Key-1: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 13: RULER NIAH-Multi-Key-2: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 14: RULER NIAH-Multi-Key-3: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 15: RULER NIAH-Multi-Query: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 16: RULER NIAH-Multi-Value: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 17: RULER VT: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 18: RULER CWE: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 19: RULER FWE: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 20: RULER QA-1: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).
Figure 21: RULER QA-2: accuracy versus compression ratio at 16K context (left), and versus context length at 16× (middle) and 32× compression (right).