VFold: Symmetry-Aware Cross-Layer Value Cache Compression

Neha Verma; Sungwon Kim; Kenton Murray; Kevin Duh

cs.CLOct 8, 2026

VFold: Symmetry-Aware Cross-Layer Value Cache Compression

Authors: Neha Verma, Sungwon Kim, Kenton Murray, Kevin Duh

Organizations: Johns Hopkins University · George Mason University

Abstract

While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However, most existing techniques necessitate architectural changes to LLMs and incur substantial overhead. In this work, we propose a symmetry-aware value cache merging strategy that reduces cache memory while avoiding both harmful performance degradation and architectural overhead during decoding. Furthermore, we show that this approach can be exploited alongside existing cache compression techniques, composing with high-ratio quantization or key cache pruning to reach compression ratios that neither method reaches alone, with minimal additional cost. Ultimately, our findings reveal a major source of underutilized capacity in the value cache, offering a simple yet highly effective direction for scaling context windows under memory constraints.

Figures & tables

Figure 1: Overview of VFold . In the full cache setting (left), individual value caches are stored for layers

L

and

L+1

. Each layer’s new value vector for the

t^{\text{th}}

token is appended to its respective cache. In the VFold setting (right), we first offline fit and absorb a function-preserving linear map

T

and its inverse into

W_{V}

and

W_{O}

(not shown), respectively (step 0). The resulting value vectors are used in attention (step 1) unperturbed, and then merged into a single value vector (step 2), that is then appended to a shared inter-layer cache (step 3), resulting in memory savings.

Panel of Figure 2 — Figure 2: Adjacent KV layer similarities for Llama-3.1-8B-Instruct on GSM8K show near-zero cosine but high CKA similarity.

Setting	Acc.
Full cache	80.7%
Avg. keys	2.7%
Avg. values	79.5%

Table 1: GSM8K accuracy when averaging keys vs. values across adjacent layers.

Figure 3: Similarity between combined caches and original value caches. We construct the aligned average as

V_{\mathrm{merge}}=\frac{1}{2}\left(V_{\ell}+F(V_{\ell+1})\right)

and compare it with

V_{\ell}

, for both

F=I

(unaligned) and CCA. We also map the merged cache back to the coordinate system of layer

\ell+1

and compare

F^{-1}(V_{\mathrm{merge}})

with

V_{\ell+1}

. The aligned average achieves cosine similarities close to

0.85

on held-out data.

Figure 4: In VFold , we apply two main invertible maps to prepare for cache merging.

P

permutes the ordering of the KV heads, and propagates to all relevant attention weight matrices.

T

, block-diagonally shaped, transforms each head to be more similar to its corresponding head in a reference layer.

Panel of Table 2 — Table 2: RULER (16K) results for KV cache compression methods at 25% total reduction, matching VFold ’s 50% value cache reduction. Best average results are bolded, second-best are underlined.

Explore similar work

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
Sep 29, 2026Joao Monteiro, Louis Béthune, Anastasiia Filippova +3LLM Inference Acceleration Low-Rank Compression
Apple
S $^4$ R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Aug 1, 2026Jialong Han, You Wu, Kewei TuKV Caching Long-Context Language Model Inference
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging
AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology

Sep 29, 2026, cs.AIJ/K move · Enter open · S save

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

Joao Monteiro, Louis Béthune, Anastasiia Filippova, Sonia Laguna+2

Apple

LLM Inference Acceleration Low-Rank Compression KV-Cache Compression KV-Cache Quantization

Open paper AskPDF arXiv

As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negatively, since decoding is memory-bound and decode cost grows with cache size. Recent work alleviates this bottleneck by discarding the least relevant tokens. Eviction introduces a tension, since a one-off decision to discard content may prove detrimental later. Instead, we focus on alternative choices that can lead to cache compression without evicting tokens. We achieve this by learning a selector that is able to produce, based on context, a per-layer cache configuration towards an overall compression budget. The selector operates along three axes: sharing one cache across layers (depth), caching at fewer bits (precision), or truncating the low-rank latent cache representations (rank). We call the resulting method KV-Kaizen, for the many small per-layer choices it compounds. We observe that these interventions taken independently and uniformly over all layers limit achievable compression because they degrade accuracy. Crucially, composing them locally and adaptively to the context can instead preserve accuracy while achieving large memory savings. At inference, the selector runs once, before pre-fill. In evaluations on instruction following and reasoning tasks, our selectors reach the Pareto frontier of accuracy against cache size, against learning-free and post-hoc baselines. On long-context tasks, KV-Kaizen improves on eviction and can be composed with it, reaching a 32x smaller decode-time cache on a 14B model while preserving accuracy. A 4x cache size reduction incurs no accuracy degradation from 7B parameters up, and a compressed model is more accurate than a smaller uncompressed one with the same cache size. Together, these findings support pre-training large models and compressing them only afterwards.

Abstract

Explore similar work

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

S4^44R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching

AnchorKV: Anchor-Residual KV Cache Compression

S $^4$ R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching