Latent KV cache compression reduces memory footprint by mapping historical keys and values into compact representations. However, generic compressed key states require reconstruction to native dimension before applying RoPE and computing attention scores, adding sequence-scaling overhead to the decode path. We characterize when a latent KV representation can serve directly as the execution state of pretrained RoPE. For RoPE-based transformers, exact static linear execution of the rotary key path requires the latent-to-native map to preserve the pretrained RoPE action through an intertwining relation, restricting exact mixing to coordinates that share the same RoPE frequency. Guided by this characterization, we build GaugeRoPE, a training-free executable latent KV representation. It combines a RoPE-compatible carrier that preserves the rotary action with an approximate position-independent residual key--value (RKV) correction for complementary key and value information, both evaluated without per-token dense historical K/V reconstruction. On Mistral-7B, GaugeRoPE achieves 5.85× KV compression while maintaining quality close to the full-cache baseline. Additional MHA results validate the construction beyond GQA. GaugeRoPE achieves up to 2.48× the decode throughput of the vanilla uncompressed baseline through direct latent execution.
Figure 2: GaugeRoPE overview. GaugeRoPE represents each historical token with two latent components: a RoPE-compatible carrier zs for the rotary key path and a position-independent RKV code bs for complementary residual-key and value information. The two paths produce complementary attention scores that are combined before softmax, while values are aggregated in latent space and decoded once, avoiding per-token historical native-K/V reconstruction.
Method
Comp. Ratio
NIAH
VT
CWE–FWE
QA
Avg.
Full
1.00×
95.00
98.80
93.07
68.00
90.84
Palu-50%
2.00×
91.19
92.80
75.50
55.00
83.33
ReCalKV-50%
2.00×
88.44
92.80
74.83
55.00
81.54
Gauge-50%
2.00×
91.13
90.80
82.60
66.00
85.92
KIVI (2 bits)
5.06×
85.63
97.60
91.83
67.00
84.64
Palu-50% (6 bits)
5.20×
90.13
93.20
76.87
55.00
82.92
Table 1: Mistral-7B-Instruct-v0.3 RULER category macros. For GaugeRoPE, 8/4 bits denotes 8-bit carrier and 4-bit RKV storage. NIAH averages the eight needle-in-a-haystack tasks; CWE–FWE and QA average two tasks each; Avg. averages all 13 tasks.
Method
Comp. Ratio
Single-doc QA
Multi-doc QA
Summ.
Few-shot
Synthetic
Code
Avg.
Full
1.00×
40.28
36.91
29.57
47.29
51.00
58.68
42.59
Palu-50%
2.00×
39.00
36.50
26.39
49.37
28.00
49.13
38.00
ReCalKV-50%
2.00×
37.55
30.66
26.50
44.28
36.00
49.48
36.74
Gauge-50%
2.00×
32.23
32.07
28.35
35.11
41.00
54.83
35.93
KIVI (2 bits)
5.05×
39.86
37.90
28.78
44.04
51.00
57.31
41.77
Palu-50% (6 bits)
5.20×
39.64
36.83
26.80
49.82
25.00
48.42
37.88
Table 2: Mistral-7B-Instruct-v0.3 LongBench results on the same 800 examples: 16 English tasks with 50 examples each and a 32K input cap. Configurations match Table 1 ; Avg. is the task-equal macro.
Figure 3: Compatible capacity and coefficient freedom. Left: same-frequency Gram energy captured by real and complex carriers. Right: attention KL on identical normalized Full states at a fixed budget of 1024 real scalars per token/layer.
Figure 4: Decoding at batch size 4 with BF16 caches. Left: full-model throughput normalized to Full; Palu/ReCalKV run out of memory during initialization at 64K. Right: attention latency averaged over 32 layers, measured one layer at a time.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Cached scalars
Native recent tokens
Comp. Ratio
RULER
LongBench
1024
0
2.00×
85.92
36.46
1024
256
1.94×
86.69
38.84
Gauge-50% (8/4 bits)
Native recent tokens
Comp. Ratio (RULER / LB)
RULER
LongBench
0
5.85× / 5.85×
85.56
35.73
128
5.42× / 5.41×
86.31
38.63
Appendix
Table 4: Recent-token ablation. BF16 uses the original 3,750-example LongBench evaluation; 8/4 bits uses the main-table samples and mean measured compression, including native tokens and metadata.
Metric
Real R2
Complex R2
Spectral capture (%)
79.27
83.06
Carrier-only KL
0.472319
0.406088
Hybrid KL
0.080179
0.072916
Hybrid Top-8
0.813745
0.825609
Hybrid output rel-L2
0.267827
0.251463
RULER (13 tasks)
6.05
85.92
Appendix
Table 5: Equal-rank real/complex control at the same cache size.
Residual score
Attention KL ↓
Top-8 ↑
Output rel-L2 ↓
A: position independent
0.072916
0.825609
0.251463
B: RoPE restored
0.025022
0.912959
0.153077
C: exact reference
1.55×10−9
0.999999
9.39×10−7
Appendix
Table 6: Residual-RoPE attribution. Fixed codes and Full values; the protocol matches Appendix D.1 .
Seed
R1
Real R2
Complex R2
95% CI for gain
53
0.00
0.00
78.00
[69.75,85.75]
54
0.00
0.00
84.75
[77.62,91.25]
55
0.00
0.00
80.00
[72.25,87.12]
Appendix
Table 7: Calibration-sample sensitivity at 1024 real scalars per token/layer. Four-task RULER scores; 95% CIs for Complex R2 minus either control use 20,000 within-task paired bootstrap resamples.
Backbone
Full
α=1/16
α=1/8
α=1/4
CodeLlama
79.57
58.80
67.78
69.50
Appendix
Table 8: RULER macros for the fixed-budget MHA rank sweep.
Model / batch / context
Throughput (tok/s)
Speedup
Decode peak (GiB)
Mistral-7B / 8 / 16K
39.69 / 91.77
2.31×
31.67 / 24.42
Mistral-7B / 16 / 16K
41.76 / 105.00
2.51×
49.82 / 34.18
Qwen2.5-14B-Instruct / 4 / 32K
11.91 / 26.92
2.26×
54.29 / 43.21
Appendix
Table 9: Larger batches and models. Full/Gauge medians over three requests after one warmup.
Model
Method
8K
16K
32K
Mistral
Full
92.50
88.75
90.63
Gauge
86.56
82.19
80.31
Qwen
Full
96.88
95.94
93.13
TransMLA
76.25
37.19
11.88
Gauge
90.00
83.75
78.75
Appendix
Table 10: NIAH length scaling. Four tasks, 20 examples each. Latent budgets: 1024 real scalars per token/layer for Mistral-7B and 512 for Qwen2.5-7B.
Head-dimension compression can reduce the memory required by the key-value (KV) cache, but existing methods often reconstruct full-dimensional representations during inference, adding significant decoding FLOPs overhead. We formalize the design considerations for converting pretrained attention into a natively reduced-dimensional form that reduces both KV-cache storage and attention computation using a calibration set. Specifically, we consider reductions that (1) apply a shared projection to queries and keys so that they remain in a common reduced feature space, (2) derive this projection jointly from query and key activations to better preserve their interaction, and (3) are compatible with the pretrained RoPE transformation. To obtain a reconstruction-free realization of this reduction, we restrict it to coordinate selection and theoretically show that RoPE compatibility then holds if and only if complete rotary pairs are retained. Building on this characterization, we introduce RACE, a post-training method that selects rotary pairs using their joint query-key activation energy. We theoretically show that this criterion minimizes a separable, calibration-dependent upper bound on post-RoPE query-key product approximation error. The resulting reductions are folded into the model weights and require neither runtime reconstruction nor specialized kernels. Across three model families and different compression levels, RACE outperforms structured pruning and remains competitive with reconstruction-based methods on language-modeling, zero-shot, and long-context tasks. At 50% KV-cache retention and 256K context, RACE reduces total inference FLOPs by 47.8%, yielding 1.79x higher throughput, 43.2% lower time-to-first-token (TTFT), and 28.1% lower allocated GPU memory in comparison to an uncompressed model.
Irene Tenison, Abdulrahman Alabdulkareem, Daniel Karl I. Weidele +2
Larger models and longer contexts make the KV cache a memory and bandwidth bottleneck. We introduce Block-GTQ, a framework that uses the two-dimensional frequency blocks of rotary position embeddings (RoPE) to allocate cached-key precision. A blockwise logit-error bound motivates a label-free query-key energy score, which we combine with codec-dependent rate penalties. Under equal-cost budgets and diminishing marginal gains, largest-marginal-gain allocation exactly minimizes the separable integer proxy. We instantiate Block-GTQ with TurboQuant-MSE (TQ-MSE), grouping equal-rate blocks to support attention over packed caches without materializing a decoded fp16 KV cache. Across ten model configurations, K-only diagnostics reduce layer-averaged logit mean absolute error by about 32-80% relative to uniform TQ-MSE, improving all 367 layers at both 2- and 3-bit average logical key rates. At K2V2 (2 average logical bits per K/V coordinate) on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task needle-in-a-haystack (NIAH) average from 70.6 to 97.4 and LongBench-EN from 33.7 to 44.9; K2V2 gains also hold on the tested 32B and 70B models. On DeepSeek-R1-Distill-Qwen-7B at K3V2, without fp16 sink or recent-token buffers, it achieves 49.6/38.8% avg@8 on AIME 2024/2025 versus fp16's 54.2/37.9%, where uniform TQ-MSE scores zero on both. On Qwen2.5-3B-Instruct, packed K3V3 serving on one H800 reduces resident KV footprint by a factor of 3.24 and retains perplexity close to fp16 at 128K, while uniform TQ-MSE degrades severely.
Fengfeng Liang, Yuechen Zhang, Jiaya Jia
Hong Kong University of Science and Technology · The Chinese University of Hong Kong · MiMo, Xiaomi Corporation
Large language models (LLMs) now serve as the backbone of agents that reason over many turns, call tools, and pursue long-horizon tasks. To avoid recomputation, a model stores the attention keys and values of all past tokens in a key-value (KV) cache. Each layer maintains its own cache, so KV-cache memory can become the main bottleneck for GPU serving. Most compression methods operate within a single layer, by quantizing the cache entries, reducing the channel dimension, or evicting less important tokens. All three lose substantial accuracy at high compression ratios. Cross-layer methods share one compressed cache across adjacent layers to go beyond this limit. However, these methods also degrade sharply at high compression ratios. We find that the KV-cache representations of adjacent layers are quite different, whereas the input hidden states of adjacent layers are highly similar. We also find that, under the same memory budget, reducing the dimension of this shared latent loses far more accuracy than quantizing it. Based on these findings, we propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. At inference time, each layer reads its own keys and values from this latent through a layer-specific projection. We keep the latent at full dimension and compress it in two other ways: BellowsKV-Quant stores the latent at low precision, which is nearly lossless because the latent has no dominant outlier channels; BellowsKV-Evict makes one shared eviction decision per block, because the layers rank token importance very similarly. Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6x compression and 97-98% at 13-14x, where the strongest cross-layer method retains at most 84% at 16x. Code: https://github.com/MorrisZJ/BellowsKV