Reasoning vision-language models (VLMs) impose substantial KV cache demands from both large multimodal contexts and growing reasoning traces. Low-rank KV compression has emerged as a promising paradigm that stores low-dimensional latents of KV states and can be combined with quantization. However, existing low-rank methods often induce growing predictive policy drift from the dense model when compressing reasoning KV states, resulting in reasoning failures. We identify two key causes of this degradation: calibration-state mismatch between static contexts and on-policy reasoning states, and unequal error sensitivity across KV states. To address these challenges, we propose TraceKV, a training-free low-rank KV compression framework that unifies on-policy calibration, sensitivity-aware SVD, and global rank allocation under a single policy-preserving principle, substantially reducing policy-KL divergence during reasoning. Experiments across multiple VLMs and visual reasoning benchmarks demonstrate superior accuracy--memory trade-offs, achieving near-lossless reasoning accuracy with 5.64× KV cache compression when combined with quantization. Optimized low-rank attention kernels further deliver up to 1.83× end-to-end decoding speedup. Code will be available at https://github.com/AK-DREAM/TraceKV.
Figures & tables
Figure 1: Comparison of compression paradigms and policy-KL divergence during reasoning. Left: Static, reconstruction-centric low-rank KV compression versus our on-policy, sensitivity-aware paradigm. Right: Next-token policy-KL divergence between dense and compressed models along dense-model-generated reasoning traces, evaluated on MMMU dataset using Qwen2.5-VL-7B-Instruct.
Figure 2: Analysis of KV states on 32 MMMU samples (rows). (a) KV reconstruction errors at 50% rank ratio under input-context calibration, and (b) KV state sensitivities estimated using Fisher information, averaged over all KV projections. Token positions are aligned at the input-generation boundary (yellow line).
Figure 3: Overview of TraceKV. Left: The dense VLM generates on-policy traces to construct calibration data covering both multimodal contexts and reasoning states. Top: Monte Carlo sampling of next-token predictions provides score gradients for estimating policy-KL sensitivities. Bottom: KV-state sensitivities guide weighted SVD to obtain singular components, which are further evaluated by their policy-KL importance and selected under a shared global rank budget to construct the final low-rank projections.
Figure 4: Visualization of policy-KL divergence and reconstruction error under different compression strategies. Evaluated using Qwen2.5-VL-7B-Instruct at 50% rank ratio on MMMU.
Figure 5: Decoding speedup on Qwen2.5-VL-7B (batch size 16). At 32K, dense model runs out of memory; dashed lines and hollow markers denote speedups estimated by linear extrapolation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Monte Carlo estimation stability on Qwen2.5-VL-7B-Instruct. (a) Mean relative Frobenius error of sensitivity-weighted covariance matrices. (b) Global Spearman correlation of singular-component importance scores. Results show mean ± standard deviation over three random seeds, using Kref=64 as reference.
Figure 7: Visualization of KV-state sensitivity for a ScienceQA example on Qwen2.5-VL-7B-Instruct, measured at the value projection of layer 14.
Figure 8: Visualization of global rank allocation on Qwen2.5-VL-7B-Instruct at 50% rank retention.
Figure 9: Per-step decoding latency breakdown on Qwen2.5-VL-7B-Instruct with batch size 16 and 16K context length. Reported latencies may differ slightly from end-to-end measurements due to the overhead of kernel-level profiling.
Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to 5.8× memory reduction while nearly preserving full cache accuracy and improving throughput by over 4.3×.
Long chain-of-thought reasoning substantially increases KV-cache memory during autoregressive decoding, as every generated token introduces new key and value states and causes the cache to grow linearly with decoding length. Existing KV-cache compression methods typically control this growth through token eviction, but irreversible deletion can remove historical states that later reasoning may need to revisit. SVD-based low-rank compression provides an alternative by retaining all positions with a more compact representation. However, extending it from a fixed prompt cache to online decoding is non-trivial. Through our investigation, we find that if the basis is updated for new tokens while old tokens keep their coordinates in the old basis, the stored history drifts substantially. Based on this observation, we propose iS-KV, an online low-rank KV-cache compression method for long-horizon reasoning. iS-KV keeps a recent window exact while incrementally folding older states into bounded-rank representations. As the low-rank basis evolves, it synchronizes historical coordinates with the updated basis to maintain representation consistency. On DeepSeek-R1-Distill-Llama-8B, iS-KV achieves 82.6% accuracy at 4.06-fold persistent-KV compression, close to the original model's 83.6%. On Qwen3-8B, it achieves 89.2% accuracy at 5.64-fold compression. Under matched memory budgets, iS-KV consistently outperforms token-eviction baselines.
Yiren Zhao, Guanghui Song, Tianrui Qin +3
The Hong Kong University of Science and Technology (Guangzhou) · Guangdong OPPO Mobile Telecommunications Corp., Ltd. · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences +2
Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compression addresses this problem by reducing the storage cost of previous tokens. Among existing approaches, low-rank compression is particularly attractive because it represents every token in reduced dimensions. Previous low-rank methods typically derive fixed projection spaces from model weights, construct fixed spaces from calibration activations, or construct a shared basis over a broad cache region. Such representations may not capture detailed but important information. We partition each per-head KV cache into fixed-length logical pages and observe substantial low-rank structure within individual pages. Based on this observation, we propose PuzzleKV, a training- and calibration-free method that treats each completed page as an independent compression unit. PuzzleKV decomposes pages within each layer and KV head, computes attention directly over dense and factorized pages, and incrementally compresses newly eligible pages during autoregressive decoding. Experiments across models, context lengths, and benchmarks demonstrate the effectiveness of PuzzleKV under matched storage budgets. At approximately 60% of the original KV cache storage, PuzzleKV achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings, with substantial gains over Global SVD on RULER and competitive performance on LongBench. To achieve a more aggressive compression ratio, PuzzleKV can be further combined with quantization while retaining more than 93% of Full KV performance using only 18.7% of the original storage.