Long-running agents retain KV caches across turns and sessions for hours or days. Compressing these persistent caches requires reducing storage while controlling reconstruction error. We present AlignPred, a codec that combines local alignment with predictive coding under a configurable KV reconstruction-error budget. Local orthogonal transforms align related heads or layers into a common coordinate frame, allowing AlignPred to combine token-wise prediction with same-token prediction across heads or layers. Residuals are quantized on an error-bounded lattice and entropy-coded, without learned predictors, encoder-side mode search, or large joint transforms. Across three attention mechanisms, AlignPred requires less storage than the evaluated baselines at comparable measured reconstruction error. Against the evaluated CPU codec implementations, it decodes at least 1.6 times faster while matching or improving encode time. AlignPred achieves near-native task scores at substantially compressed operating points on LongBench-Pro at 128K context. AlignPred's SGLang HiCache integration reduces persistent target-KV storage by 42-72% relative to native HiCache, with comparable latency.
Figure 2: Alignment of related KV blocks. (a) Rotation registers same-token vectors across heads or layers. (b) Token rows share channels. Alignment also reduces cross-block differences.
Figure 3: After alignment, high-order-bit MI is higher across neighboring heads than across neighboring tokens on average in Qwen3.5-9B .
Figure 4: Predictive residual coding for three aligned cache blocks: token-wise prediction initializes the first block, and cross-block prediction uses reconstructed blocks to predict subsequent blocks. Residuals undergo error-bounded quantization and lossless entropy coding.
Figure 5: Error-bounded residual quantization. (a) Small residuals occupy 43.8% of KV cache storage but contribute only 3.6% of residual energy. (b) Quantization merges nearby values within the error bound.
Figure 6: Overlap of background KV compression with foreground inference, and pipelined CPU–GPU restoration. Each CPU compression job starts after its corresponding GPU step. Timeline lengths are schematic and illustrate overlap rather than measured stage durations. L3 denotes HiCache’s persistent file tier.
Baseline
Storage
Encode
Decode
KVCodec
1.10–1.30
5.3–6.8
5.4–6.3
KVTC-style
1.13–1.18
1.3–1.6
3.3–3.7
Rotation + SZ3
1.07–1.10
1.02–1.04
1.6–2.1
Table 1: Cost of the evaluated CPU codec implementations relative to Ours at the FP8-near operating points in Table 7 , range over the three models. Above 1 favors Ours.
Figure 7: Storage size versus reconstruction error on five InfiniteBench categories, including KVCodec, KVTC-style, Rotation + SZ3 auto, and unrotated SZ3 auto. Error axes are logarithmic. Dotted lines mark FP8 error. Axis ranges vary by model.
Figure 8: LongBench-Pro task score versus storage rate (bpv). KIVI and TurboQuant-MSE provide GPU-resident quantization references.
Figure 10
Figure 10: Persistent target-KV file size and L3-hit restore TTFT versus native HiCache, after a restart with empty L1/L2. Batch size 1. Axis ranges vary by model.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Illustrative dialogue motivating error-controlled KV compression. A configurable reconstruction-error budget supports different application requirements and provides a numerical reference for deployment validation.
Model
Compressed cache
d
Streams/object
Blocks/group
Qwen3.5-9B
Full-attention K/V
256
16
4 heads each
GLM-5.3-Flash
Post-LN latent
512
3
4, 4, 3 layers
DeepSeek-V4-Flash
C4 CSA latent
512
6
1+5×4 layers
Appendix
Table 3: Cache geometry for Figure 7 .
Item
Setting
Input caches
All methods cover the same cache positions and retain positional encoding where applicable, keeping their input representations consistent.
FP8 reference
E4M3FN, scale 1. FP16 inputs clamped to [−448,448] .
KVCodec
Adapted software-codec reference: unrotated token layout, lossless x265, veryfast preset, three B-frames with adaptive selection, and at most 256 positions per GOP.
KVTC-style
Adapted transform-coding reference: joint PCA, quantization, and Deflate. DeepSeek uses three-layer groups. Shared PCA parameters are excluded from bpv.
SZ3 auto
Official SZ3 auto on the same caches and frozen means as Ours. Rotation + SZ3 applies fitted rotations, and unrotated SZ3 auto does not. Figure 7 , Table 7 , Table 11 , and Table 12 .
Residual-only reference
Component-ablation control (Table 9 , first row): error-lattice quantization plus rANS, with alignment and prediction disabled.
Appendix
Table 4: Offline RD baseline and operating-point settings.
Method
Primary setting
Role here
CacheGen
Remote KV streaming
Related persistent/remote KV system
KVTC
Persistent KV storage
Adapted transform-coding reference
KVCodec
Remote KV reuse
Adapted software-codec reference
LLM.265
Video codec as tensor codec
Related codec-style KV compression
SZ3 auto
Error-bounded array compression
Generic error-controlled baseline
KIVI, TurboQuant
GPU-resident KV quantization
Task quality at different storage sizes
Appendix
Table 5: Nearby methods by primary setting and experimental role.
AlignPred
Full-width joint PCA
Transform matrices
m matrices, d×d
One matrix, md×md
Parameter storage
O(md2)
O(m2d2)
Transform work
O(Nmd2)
O(Nm2d2)
Decomposition size
d×d
md×md
Decomposition work
O(md3) per GPA iteration
O(m3d3)
Appendix
Table 6: Local transforms versus full-width joint PCA for m blocks of width d and N cache positions.
Figure 12: Same INT8 same-bit MI diagnostic as Figure 3 on GLM MLA and DeepSeek CSA leftover-group layers.
Model
Ours
KVCodec
KVTC-style
Rotation + SZ3
Qwen3.5-9B
4.347 (2.67)
5.642 (2.51)
4.931 (2.58)
4.775 (2.60)
GLM-5.3-Flash
5.025 (2.66)
6.035 (2.66)
5.909 (3.07)
5.395 (2.59)
DeepSeek-V4-Flash
5.049 (2.66)
5.547 (2.66)
5.974 (3.03)
5.404 (2.60)
Appendix
Table 7: Representative operating points near the FP8 reconstruction-error reference, reported as bpv (NRMSE, %). Figure 7 is the primary rate–distortion comparison. SZ3 uses auto mode on the same locally aligned caches.
Table 11: Rotation+SZ3 auto points for Figure 7 , on the same locally aligned caches and frozen means/rotations as Ours. m=1 is the Table 7 column.
Qwen3.5-9B
GLM-5.3-Flash
DeepSeek-V4-Flash
m
bpv
Error (%)
m
bpv
Error (%)
m
bpv
Error (%)
3
3.481
7.80
8
2.668
20.72
8
2.544
20.47
2
4.058
5.20
4
3.632
10.36
4
3.511
10.30
1.5
4.477
3.90
2
4.628
5.18
2
4.504
5.17
1.25
4.742
3.25
1
5.650
2.59
1
5.524
2.59
1
5.069
2.60
0.5
6.649
1.30
0.5
6.535
1.30
Appendix
Table 12: Unrotated SZ3 auto points for Figure 7 , on the same caches and frozen means as Ours, without fitted rotations. m=1 matches Table 11 .
Figure 13: Reconstruction-error companion to Figure 8 : the same LongBench-Pro compression settings, plotted as mean compressed-region NRMSE (%) versus storage rate (bpv, including tails and metadata). Figure 8 uses task score on the vertical axis. This figure uses numerical fidelity. Error axes are logarithmic.
Figure 14: Per-task official LongBench-Pro scores for Native, FP8, and Ours at error multipliers m=1,4,8 , i.e. the AlignPred settings highlighted in Figure 8 . Each panel is one model.
Model
Multiplier
Auxiliary
Specific
Specific total
Qwen3.5-9B
1
4.347
4.132
4.216
4
2.398
2.210
2.294
8
1.525
1.369
1.453
GLM-5.3-Flash
1
5.025
4.901
5.071
4
3.037
2.920
3.090
8
2.100
1.995
2.165
Appendix
Table 13: Five-category calibration storage (bpv). “Specific total” includes one dedicated parameter bank per document.
λv≥3
λ2
0
0.25
0.5
0.75
1
0
4.644
4.547
4.473
4.437
4.452
0.25
4.599
4.497
4.419
4.381
4.397
0.5
4.572
4.467
4.387
4.347
4.363
0.75
4.570
4.464
4.383
4.342
4.358
1
4.594
4.491
4.411
4.371
4.386
Appendix
Table 14: Post-hoc blending-weight sensitivity on Qwen at m=1 (bpv ↓ ). Bold marks the pre-specified default, fixed on independent development data before evaluation. Results are pooled over five held-out InfiniteBench documents at matched reconstruction error.
λ2
λv≥3
m=1
m=4
m=8
0
0
4.644
2.665
1.740
0.5
0.75
4.347
2.398
1.525
0.75
0.75
4.342
2.399
1.533
1
1
4.386
2.451
1.595
Appendix
Table 15: Selected blending-weight settings at three error multipliers (bpv ↓ ). Bold marks the pre-specified default. No weights are tuned on these evaluation documents.
Model
Partition
Views
bpv
Qwen3.5-9B
4 heads / layer
4
4.347
2 adjacent layers
8
4.304⋆
4 adjacent layers
16
4.360
8 layers, K or V
32
4.529
GLM-5.3-Flash
4/4/3 layers
4
5.025
5/6 layers
6
5.031
Appendix
Table 16: Held-out grouping sensitivity at m=1 (bpv ↓ ). Bold is the paper default, and ⋆ is the evaluation minimum. Pooled over five InfiniteBench documents at the frozen FP8 error.
Model
Compressed path
Qwen
Four 64-token pages per object, separate K/V streams, and 32 objects per decode batch. Aligned host staging feeds layerwise GPU inverse transforms and FP16 page scatter.
GLM
Post-LN MLA latent, grouped 4/4/3 layers. Rank-0 decoding with 32-object batches, two host/GPU slots, GPU inverse transforms, and ordered BF16 broadcasts.
DeepSeek
Pre-FP8 C4 CSA values, grouped 1+5×4 layers. Rank-0 decoding with eight-object batches, two host/GPU slots, inverse transforms, broadcasts, and native FP8 writeback.
Common
In-memory indices, object read-ahead, page mappings, and event-guarded buffer reuse. Raw tails and native sidecar paths are retained.
B distinct InfiniteBench book documents per model, the first B of one fixed prompt bank, identical across arms and lengths. Each prefix is followed by 64 appended IDs. Greedy generation of exactly 256 tokens, ignoring EOS.
Phases
Each arm writes its own L3 files online during the first requests’ chunked prefill. The writer drains and stops. A fresh reader then serves the same B requests concurrently from L3.
Timing
TTFT: request submission to first token, averaged over the B requests. Persistence completion: first submission to the L3 write of the batch’s last target-KV page. Reading, decoding, H2D, inverse transforms, communication, and native writeback are included.
Graph/warmup
Prefill graphs disabled. Full decode graphs for batch sizes 1 – B . Unrelated full-chunk and 64-token warmups. For B>1 , readers also run k -request EXTEND batches ( k=2,…,B ) over an unrelated cached prefix. Startup and warmup excluded.
Appendix
Table 18: Serving experiment matrix and controls.
Figure 15: First-request write TTFT with no prior L3 hit, while online writes populate L3. Batch size 1. Axis ranges vary by model. Storage and L3-hit restore TTFT are in Figure 10 .
Model
CPU wall (s)
Δ TTFT (s)
Persist before TTFT
Qwen
8.690
0.180
87.5%
GLM
2.130
0.263
100.0%
DeepSeek
2.803
0.044
100.0%
Appendix
Table 19: CPU compression overlap at 64K, B=1 .
Model
Write TTFT (s)
Rehit TTFT (ms)
Stored KV (MiB)
Native
Ours
Native
Ours
Native
Ours
Qwen
9.105
9.286
843.3
469.5
2048.0
577.1
GLM
6.390
6.653
628.0
502.1
704.0
227.7
DeepSeek
7.176
7.221
316.4
302.0
192.0
110.8
Appendix
Table 20: 64K online write then L3 rehit, B=1 .
Model
Ctx
Write TTFT (ms)
Rehit TTFT (ms)
Stored KV (MiB)
Native
Ours
Native
Ours
Native
Ours
Qwen
32K
4082.0
4204.3
547.3
307.1
1024.0
287.6
64K
9105.3
9285.6
843.3
469.5
2048.0
577.1
128K
22294.3
22491.2
1615.5
856.0
4096.0
1161.9
GLM
32K
3268.9
3356.3
479.7
391.9
352.0
114.0
64K
6390.0
6653.2
628.0
502.1
704.0
227.7
Appendix
Table 21: Online write then L3 rehit, B=1 . Storage covers the target KV portion only. Figure 10 plots L3-hit restore TTFT. Figure 15 plots first-request write TTFT with no L3 hit. Persistence completion is in the B=1 column of Table 22 (b).
Figure 16: Mean L3-hit restore TTFT of B concurrent requests. † : GPU KV capacity admits only five requests in both arms. Axis ranges vary by panel.
Model
Ctx
B=1
B=2
B=4
B=8
N
O
N
O
N
O
N
O
Qwen
32K
547
307
658
413
928
621
1441
996
64K
843
470
1123
680
1616
1045
2619
1747
128K
1615
856
2086
1251
3021
1963
6320 †
6431 †
GLM
32K
480
392
521
429
764
771
1270
1157
64K
628
502
708
576
1140
1087
1741
1776
Appendix
Table 22: B concurrent requests. (a) Mean L3-hit restore TTFT (ms). (b) Write-phase persistence completion (s), first submission to the L3 write of the batch’s last target-KV page. N: native HiCache, O: Ours 1× . † : GPU KV capacity admits only five requests in both arms.
Model
Ours
KVCodec
KVTC-style
Rotation + SZ3
Qwen3.5-9B
Encode
1266
7468
2079
1308
Decode
564
3321
1918
926
GLM-5.3-Flash
Encode
676
3589
888
704
Decode
231
1239
759
493
DeepSeek-V4-Flash
Encode
976
6675
1401
994
Decode
338
2126
1245
682
Appendix
Table 23: Same-host CPU encode and decode time (ms) at the Table 7 operating points (median of three passes over 160 objects).
KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache logits induced by their removal and evaluating the resulting KL to the full-cache next-token distribution. Using pre-eviction forward statistics, the method avoids running separate masked forward passes for each candidate. Across diverse architectures and both prefill-time and generation-time compression, our method delivers substantial gains in downstream task quality over lightweight attention-based heuristics at matched retained-KV budgets, with the largest gains under aggressive compression. It achieves these gains with additional compression-time computation while retaining an end-to-end speedup over full-cache inference in our evaluated settings.
Doo Hwan Hwang, Junyoung Jang, Junho Na +2
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea
The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attention mechanisms. We prove that, under a white-noise quantization model, the expected attention-aware distortion decomposes into additive key and value contributions that factor across tokens and channels. Building on transform coding and reverse water-filling, which are classical tools from signal processing and rate-distortion theory, we introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion. On Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, evaluated across LongBench, RULER, GSM8K, MMLU-Pro, and MATH-500, our method achieves near-lossless accuracy at approximately 5.8× compression, whereas each baseline degrades in at least some settings.
Hannah Laus, Claudio Mayrink Verdun, Hao Wang +2
Technical University of Darmstadt, the Technical University of Munich and the Munich Center for Machine Learning · Massachusetts Institute of Technology · Red Hat AI Innovation Team and the MIT-IBM Watson AI Lab +1
The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by 20× without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.
Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2
Department of Computer Science, Technion - Israel Institute of Technology