Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels
Organizations: San José State University
Abstract
In the global attention layers of Gemma 4 31B, the value is the normalized key projection, and the key is the value scaled by a fixed weight, with rotary position encoding on one quarter of its dimensions. The Gemma 4 report states that this design reduces the global KV cache by up to 37.5%, but the open serving engines that we checked still store a full key and a full value: 1,024 values per KV head and token. We store one 640-value row instead, exact in real arithmetic, and an unchanged paged attention kernel reads it through two overlapping strided views as its key cache and its value cache. In a vLLM prototype with the 16-bit 31B model, the outputs are bit-identical to an emulation of the same arithmetic on the stock cache, 2 of 432 final answers differ from the stock engine (within its run-to-run noise), and the steady decode KV memory per request falls by a factor of 1.34 to 1.40 at 32K to 64K tokens. On a prefill/decode pair of two A100 GPUs with NIXL transfer and CPU offload, a decode engine with a full 15 GiB pool ran 1.38 times the stock decode tokens per step for 16K-token prompts (repeated to 0.06%) and 1.24 times for 8K-token prompts; a model of the vLLM block pool predicted both ratios to within 1% (the engine was 0.7% to 0.9% higher) and matched the change between them. The eager decode throughput rose by 1.30 and 1.11 times, and on this path the compact engine gave the same output tokens as the emulation for 8 prompts at batch 1. The run-time prototype makes an eager step 4% to 9% slower at equal batch, and the method does not yet stack with an 8-bit KV cache at the default scale.
Figures & tables
| 32K tokens | 64K tokens | |
|---|---|---|
| stock / compact | stock / compact | |
| Steady decode footprint (blocks) | 1,349 / 1,008 (1.338 ) | 2,373 / 1,690 (1.404 ) |
| Steady decode footprint (GiB, all layers) | 3.29 / 2.46 | 5.79 / 4.13 |
| Prefill peak (blocks) | 2,602 / 2,262 (1.150 ) | 3,626 / 2,944 (1.232 ) |
| Largest batch with no preemption | 3 / 4 | 2 / 2 |
| Decode tokens/s at that batch | 56.0 / 69.7 (1.24 ) | 36.3 / 34.8 (0.96 ) |
| decode tokens per work step | model prediction of | ||||
|---|---|---|---|---|---|
| Run | stock | compact | seed 0 | seeds 0–4 (frozen band) | |
| D3H, first run | 5.458, 5.459 | 7.509 | 1.376 | 1.364 | 1.308–1.364 (1.295–1.378) |
| D3H, repeat run | 7.514 | 1.377 | |||
| D3Q | 8.192 | 10.133 | 1.237 | 1.229 | 1.169–1.229 (1.158–1.241) |
| Model, seed 0 | 5.454 / 8.246 | 7.438 / 10.130 | |||
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Run | GPU | Outcome | Used in this paper as | What it measured |
|---|---|---|---|---|
| E-49 | H100 | PIVOT | context (App. D ) | Ceiling of the decode-step saving, 16-bit KV (6.65% and 11.33%) |
| E-50 | H100 | GO | context (App. D ) | The same on an FP8 KV cache near the pool limit (12.1% and 15.0%) |
| E-51 | H100 | INVALID | exploratory (App. D ) | A compact FP8 Triton kernel against its standard twin |
| E-52 | H100 | INVALID | exploratory (App. D ) | The two-view read in Triton and stock FP8 kernels |
| E-53 | H100 | FAIL | result (App. D ) | The FP8 compact row at KV scale 1.0, real activations |
| E-54 | H100 | PASS | result (App. D ) | The FP8 compact row with tied, calibrated scales |
KV-Cache Management
Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.
See all 216 papers