AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding

Xu Yang; Jiapeng Zhang; Zhiwei Zuo; Yuxin Chen; Feiqiang Sun; Chengguang Xu; Feng Jin; Zhuo Tang

cs.PFOct 13, 2026

AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding

Authors: Xu Yang, Jiapeng Zhang, Zhiwei Zuo, Yuxin Chen, Feiqiang Sun, Chengguang Xu, Feng Jin, Zhuo Tang

Organizations: Hunan University · Central China Normal University · Tencent

Abstract

Long-running agents retain KV caches across turns and sessions for hours or days. Compressing these persistent caches requires reducing storage while controlling reconstruction error. We present AlignPred, a codec that combines local alignment with predictive coding under a configurable KV reconstruction-error budget. Local orthogonal transforms align related heads or layers into a common coordinate frame, allowing AlignPred to combine token-wise prediction with same-token prediction across heads or layers. Residuals are quantized on an error-bounded lattice and entropy-coded, without learned predictors, encoder-side mode search, or large joint transforms. Across three attention mechanisms, AlignPred requires less storage than the evaluated baselines at comparable measured reconstruction error. Against the evaluated CPU codec implementations, it decodes at least 1.6 times faster while matching or improving encode time. AlignPred achieves near-native task scores at substantially compressed operating points on LongBench-Pro at 128K context. AlignPred's SGLang HiCache integration reduces persistent target-KV storage by 42-72% relative to native HiCache, with comparable latency.

Figures & tables

Figure 1: Long-lived KV caches motivate compact storage.

Figure 2: Alignment of related KV blocks. (a) Rotation registers same-token vectors across heads or layers. (b) Token rows share channels. Alignment also reduces cross-block differences.

Figure 3: After alignment, high-order-bit MI is higher across neighboring heads than across neighboring tokens on average in Qwen3.5-9B .

Figure 4: Predictive residual coding for three aligned cache blocks: token-wise prediction initializes the first block, and cross-block prediction uses reconstructed blocks to predict subsequent blocks. Residuals undergo error-bounded quantization and lossless entropy coding.

Figure 5: Error-bounded residual quantization. (a) Small residuals occupy 43.8% of KV cache storage but contribute only 3.6% of residual energy. (b) Quantization merges nearby values within the error bound.

Figure 6: Overlap of background KV compression with foreground inference, and pipelined CPU–GPU restoration. Each CPU compression job starts after its corresponding GPU step. Timeline lengths are schematic and illustrate overlap rather than measured stage durations. L3 denotes HiCache’s persistent file tier.

Baseline	Storage	Encode	Decode
KVCodec	1.10–1.30	5.3–6.8	5.4–6.3
KVTC-style	1.13–1.18	1.3–1.6	3.3–3.7
Rotation + SZ3	1.07–1.10	1.02–1.04	1.6–2.1

Table 1: Cost of the evaluated CPU codec implementations relative to Ours at the FP8-near operating points in Table 7 , range over the three models. Above

1

favors Ours.

Figure 7: Storage size versus reconstruction error on five InfiniteBench categories, including KVCodec, KVTC-style, Rotation + SZ3 auto, and unrotated SZ3 auto. Error axes are logarithmic. Dotted lines mark FP8 error. Axis ranges vary by model.

Figure 8: LongBench-Pro task score versus storage rate (bpv). KIVI and TurboQuant-MSE provide GPU-resident quantization references.

Figure 10: Persistent target-KV file size and L3-hit restore TTFT versus native HiCache, after a restart with empty L1/L2. Batch size 1. Axis ranges vary by model.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Illustrative dialogue motivating error-controlled KV compression. A configurable reconstruction-error budget supports different application requirements and provides a numerical reference for deployment validation.

Model	Compressed cache	$d$	Streams/object	Blocks/group
Qwen3.5-9B	Full-attention K/V	256	16	4 heads each
GLM-5.3-Flash	Post-LN latent	512	3	4, 4, 3 layers
DeepSeek-V4-Flash	C4 CSA latent	512	6	$1+5\times 4$ layers

Appendix

Table 3: Cache geometry for Figure 7 .

Item	Setting
Input caches	All methods cover the same cache positions and retain positional encoding where applicable, keeping their input representations consistent.
FP8 reference	E4M3FN, scale 1. FP16 inputs clamped to $[-448,448]$ .
KVCodec	Adapted software-codec reference: unrotated token layout, lossless x265, veryfast preset, three B-frames with adaptive selection, and at most 256 positions per GOP.
KVTC-style	Adapted transform-coding reference: joint PCA, quantization, and Deflate. DeepSeek uses three-layer groups. Shared PCA parameters are excluded from bpv.
SZ3 auto	Official SZ3 auto on the same caches and frozen means as Ours. Rotation + SZ3 applies fitted rotations, and unrotated SZ3 auto does not. Figure 7 , Table 7 , Table 11 , and Table 12 .
Residual-only reference	Component-ablation control (Table 9 , first row): error-lattice quantization plus rANS, with alignment and prediction disabled.

Appendix

Table 4: Offline RD baseline and operating-point settings.

Method	Primary setting	Role here
CacheGen	Remote KV streaming	Related persistent/remote KV system
KVTC	Persistent KV storage	Adapted transform-coding reference
KVCodec	Remote KV reuse	Adapted software-codec reference
LLM.265	Video codec as tensor codec	Related codec-style KV compression
SZ3 auto	Error-bounded array compression	Generic error-controlled baseline
KIVI, TurboQuant	GPU-resident KV quantization	Task quality at different storage sizes

Appendix

Table 5: Nearby methods by primary setting and experimental role.

	AlignPred	Full-width joint PCA
Transform matrices	$m$ matrices, $d\times d$	One matrix, $md\times md$
Parameter storage	$O(md^{2})$	$O(m^{2}d^{2})$
Transform work	$O(Nmd^{2})$	$O(Nm^{2}d^{2})$
Decomposition size	$d\times d$	$md\times md$
Decomposition work	$O(md^{3})$ per GPA iteration	$O(m^{3}d^{3})$

Appendix

Table 6: Local transforms versus full-width joint PCA for

m

blocks of width

d

and

N

cache positions.

Figure 12: Same INT8 same-bit MI diagnostic as Figure 3 on GLM MLA and DeepSeek CSA leftover-group layers.

Model	Ours	KVCodec	KVTC-style	Rotation + SZ3
Qwen3.5-9B	4.347 (2.67)	5.642 (2.51)	4.931 (2.58)	4.775 (2.60)
GLM-5.3-Flash	5.025 (2.66)	6.035 (2.66)	5.909 (3.07)	5.395 (2.59)
DeepSeek-V4-Flash	5.049 (2.66)	5.547 (2.66)	5.974 (3.03)	5.404 (2.60)

Appendix

Table 7: Representative operating points near the FP8 reconstruction-error reference, reported as bpv (NRMSE, %). Figure 7 is the primary rate–distortion comparison. SZ3 uses auto mode on the same locally aligned caches.

Ours			KVCodec			KVTC-style
$m$	bpv	Error (%)	$m$	bpv	Error (%)	$b$	bpv	Error (%)
4	2.398	10.61	8	2.224	21.05	3.5	2.632	16.52
3	2.787	7.98	6.452	2.499	17.13	4	3.186	9.58
2	3.352	5.33	6	2.593	15.96	4.5	3.610	5.91
1.5	3.761	4.00	4	3.137	10.69	6	4.931	2.58
1.25	4.024	3.33	3	3.536	8.04	7	5.760	2.13

Appendix

Table 8: Qwen3.5-9B: numerical rate–distortion results for Figure 7 . FP8 NRMSE: 2.65%.

Ours			KVCodec			KVTC-style
$m$	bpv	Error (%)	$m$	bpv	Error (%)	$b$	bpv	Error (%)
6	2.479	15.96	12	2.199	32.03	3.5	2.775	24.64
4	3.037	10.65	9.537	2.494	25.46	4	3.301	16.70
3	3.443	7.99	8	2.730	21.37	4.5	3.804	10.75
2	4.022	5.33	6	3.136	16.02	6	5.087	3.96
1.5	4.437	4.00	4	3.733	10.69	7	5.909	3.07

Appendix

Table 9: GLM-5.3-Flash: numerical rate–distortion results for Figure 7 . FP8 NRMSE: 2.66%.

Ours			KVCodec			KVTC-style
$m$	bpv	Error (%)	$m$	bpv	Error (%)	$b$	bpv	Error (%)
8	2.128	21.00	8	2.412	20.96	3.5	2.864	27.21
6.366	2.429	16.81	6	2.794	15.76	4	3.413	18.67
6	2.509	15.86	4	3.361	10.53	4.5	3.886	11.24
4	3.065	10.62	3	3.783	7.92	6	5.111	4.22
3	3.468	7.97	2	4.406	5.29	7	5.974	3.03

Appendix

Table 10: DeepSeek-V4-Flash: numerical rate–distortion results for Figure 7 . FP8 NRMSE: 2.67%.

Qwen3.5-9B			GLM-5.3-Flash			DeepSeek-V4-Flash
$m$	bpv	Error (%)	$m$	bpv	Error (%)	$m$	bpv	Error (%)
3	3.193	7.79	8	2.427	20.71	8	2.439	20.65
2	3.769	5.20	4	3.389	10.36	4	3.400	10.37
1.5	4.187	3.90	2	4.378	5.18	2	4.386	5.19
1.25	4.450	3.25	1	5.395	2.59	1	5.404	2.60
1	4.775	2.60	0.5	6.402	1.30	0.5	6.406	1.30

Appendix

Table 11: Rotation+SZ3 auto points for Figure 7 , on the same locally aligned caches and frozen means/rotations as Ours.

m=1

is the Table 7 column.

Qwen3.5-9B			GLM-5.3-Flash			DeepSeek-V4-Flash
$m$	bpv	Error (%)	$m$	bpv	Error (%)	$m$	bpv	Error (%)
3	3.481	7.80	8	2.668	20.72	8	2.544	20.47
2	4.058	5.20	4	3.632	10.36	4	3.511	10.30
1.5	4.477	3.90	2	4.628	5.18	2	4.504	5.17
1.25	4.742	3.25	1	5.650	2.59	1	5.524	2.59
1	5.069	2.60	0.5	6.649	1.30	0.5	6.535	1.30

Appendix

Table 12: Unrotated SZ3 auto points for Figure 7 , on the same caches and frozen means as Ours, without fitted rotations.

m=1

matches Table 11 .

Figure 13: Reconstruction-error companion to Figure 8 : the same LongBench-Pro compression settings, plotted as mean compressed-region NRMSE (%) versus storage rate (bpv, including tails and metadata). Figure 8 uses task score on the vertical axis. This figure uses numerical fidelity. Error axes are logarithmic.

Figure 14: Per-task official LongBench-Pro scores for Native, FP8, and Ours at error multipliers

m=1,4,8

, i.e. the AlignPred settings highlighted in Figure 8 . Each panel is one model.

Model	Multiplier	Auxiliary	Specific	Specific total
Qwen3.5-9B	1	4.347	4.132	4.216
	4	2.398	2.210	2.294
	8	1.525	1.369	1.453
GLM-5.3-Flash	1	5.025	4.901	5.071
	4	3.037	2.920	3.090
	8	2.100	1.995	2.165

Appendix

Table 13: Five-category calibration storage (bpv). “Specific total” includes one dedicated parameter bank per document.

	$\lambda_{v\geq 3}$
$\lambda_{2}$	$0$	$0.25$	$0.5$	$0.75$	$1$
$0$	4.644	4.547	4.473	4.437	4.452
$0.25$	4.599	4.497	4.419	4.381	4.397
$0.5$	4.572	4.467	4.387	4.347	4.363
$0.75$	4.570	4.464	4.383	4.342	4.358
$1$	4.594	4.491	4.411	4.371	4.386

Appendix

Table 14: Post-hoc blending-weight sensitivity on Qwen at

m=1

(bpv

\downarrow

). Bold marks the pre-specified default, fixed on independent development data before evaluation. Results are pooled over five held-out InfiniteBench documents at matched reconstruction error.

$\lambda_{2}$	$\lambda_{v\geq 3}$	$m=1$	$m=4$	$m=8$
$0$	$0$	4.644	2.665	1.740
$0.5$	$0.75$	4.347	2.398	1.525
$0.75$	$0.75$	4.342	2.399	1.533
$1$	$1$	4.386	2.451	1.595

Appendix

Table 15: Selected blending-weight settings at three error multipliers (bpv

\downarrow

). Bold marks the pre-specified default. No weights are tuned on these evaluation documents.

Model	Partition	Views	bpv
Qwen3.5-9B	4 heads / layer	4	4.347
	2 adjacent layers	8	$4.304^{\star}$
	4 adjacent layers	16	4.360
	8 layers, K or V	32	4.529
GLM-5.3-Flash	$4/4/3$ layers	4	5.025
	$5/6$ layers	6	5.031

Appendix

Table 16: Held-out grouping sensitivity at

m=1

(bpv

\downarrow

). Bold is the paper default, and

\star

is the evaluation minimum. Pooled over five InfiniteBench documents at the frozen FP8 error.

Model	Compressed path
Qwen	Four 64-token pages per object, separate K/V streams, and 32 objects per decode batch. Aligned host staging feeds layerwise GPU inverse transforms and FP16 page scatter.
GLM	Post-LN MLA latent, grouped 4/4/3 layers. Rank-0 decoding with 32-object batches, two host/GPU slots, GPU inverse transforms, and ordered BF16 broadcasts.
DeepSeek	Pre-FP8 C4 CSA values, grouped $1+5\times 4$ layers. Rank-0 decoding with eight-object batches, two host/GPU slots, inverse transforms, broadcasts, and native FP8 writeback.
Common	In-memory indices, object read-ahead, page mappings, and event-guarded buffer reuse. Raw tails and native sidecar paths are retained.

Appendix

Table 17: Serving restoration paths.

Item	Setting
Models	Qwen3.5-9B: FP16, TP1. GLM-5.3-Flash: BF16, TP8. DeepSeek-V4-Flash: BF16 compute, native mixed FP8 cache, TP8.
Matrix	Three models $\times$ 32,768/65,536/131,072-token prefixes $\times$ $B=1/2/4/8$ concurrent requests $\times$ native HiCache/Ours $1\times$ .
Input/output	$B$ distinct InfiniteBench book documents per model, the first $B$ of one fixed prompt bank, identical across arms and lengths. Each prefix is followed by 64 appended IDs. Greedy generation of exactly 256 tokens, ignoring EOS.
Phases	Each arm writes its own L3 files online during the first requests’ chunked prefill. The writer drains and stops. A fresh reader then serves the same $B$ requests concurrently from L3.
Timing	TTFT: request submission to first token, averaged over the $B$ requests. Persistence completion: first submission to the L3 write of the batch’s last target-KV page. Reading, decoding, H2D, inverse transforms, communication, and native writeback are included.
Graph/warmup	Prefill graphs disabled. Full decode graphs for batch sizes $1$ – $B$ . Unrelated full-chunk and 64-token warmups. For $B>1$ , readers also run $k$ -request EXTEND batches ( $k=2,\dots,B$ ) over an unrelated cached prefix. Startup and warmup excluded.

Appendix

Table 18: Serving experiment matrix and controls.

Figure 15: First-request write TTFT with no prior L3 hit, while online writes populate L3. Batch size 1. Axis ranges vary by model. Storage and L3-hit restore TTFT are in Figure 10 .

Model	CPU wall (s)	$\Delta$ TTFT (s)	Persist before TTFT
Qwen	8.690	0.180	87.5%
GLM	2.130	0.263	100.0%
DeepSeek	2.803	0.044	100.0%

Appendix

Table 19: CPU compression overlap at 64K,

B=1

Model	Write TTFT (s)		Rehit TTFT (ms)		Stored KV (MiB)
	Native	Ours	Native	Ours	Native	Ours
Qwen	9.105	9.286	843.3	469.5	2048.0	577.1
GLM	6.390	6.653	628.0	502.1	704.0	227.7
DeepSeek	7.176	7.221	316.4	302.0	192.0	110.8

Appendix

Table 20: 64K online write then L3 rehit,

B=1

Model	Ctx	Write TTFT (ms)		Rehit TTFT (ms)		Stored KV (MiB)
		Native	Ours	Native	Ours	Native	Ours
Qwen	32K	4082.0	4204.3	547.3	307.1	1024.0	287.6
	64K	9105.3	9285.6	843.3	469.5	2048.0	577.1
	128K	22294.3	22491.2	1615.5	856.0	4096.0	1161.9
GLM	32K	3268.9	3356.3	479.7	391.9	352.0	114.0
	64K	6390.0	6653.2	628.0	502.1	704.0	227.7

Appendix

Table 21: Online write then L3 rehit,

B=1

. Storage covers the target KV portion only. Figure 10 plots L3-hit restore TTFT. Figure 15 plots first-request write TTFT with no L3 hit. Persistence completion is in the

B=1

column of Table 22 (b).

Figure 16: Mean L3-hit restore TTFT of

B

concurrent requests.

\dagger

: GPU KV capacity admits only five requests in both arms. Axis ranges vary by panel.

Model	Ctx	$B=1$		$B=2$		$B=4$		$B=8$
		N	O	N	O	N	O	N	O
Qwen	32K	547	307	658	413	928	621	1441	996
	64K	843	470	1123	680	1616	1045	2619	1747
	128K	1615	856	2086	1251	3021	1963	6320 †	6431 †
GLM	32K	480	392	521	429	764	771	1270	1157
	64K	628	502	708	576	1140	1087	1741	1776

Appendix

Table 22:

B

concurrent requests. (a) Mean L3-hit restore TTFT (ms). (b) Write-phase persistence completion (s), first submission to the L3 write of the batch’s last target-KV page. N: native HiCache, O: Ours

1\times

\dagger

: GPU KV capacity admits only five requests in both arms.

Model		Ours	KVCodec	KVTC-style	Rotation + SZ3
Qwen3.5-9B	Encode	1266	7468	2079	1308
	Decode	564	3321	1918	926
GLM-5.3-Flash	Encode	676	3589	888	704
	Decode	231	1239	759	493
DeepSeek-V4-Flash	Encode	976	6675	1401	994
	Decode	338	2126	1245	682

Appendix

Table 23: Same-host CPU encode and decode time (ms) at the Table 7 operating points (median of three passes over 160 objects).

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 257 papers

Behavior-Preserving KV Cache Compression
Oct 5, 2026Doo Hwan Hwang, Junyoung Jang, Junho Na +2KV-Cache Eviction LLM Inference Acceleration
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea
KV Cache Compression Through the Lens of Transform Coding
Aug 14, 2026Hannah Laus, Claudio Mayrink Verdun, Hao Wang +2KV-Cache Compression KV-Cache Quantization
Technical University of Darmstadt, the Technical University of Munich and the Munich Center for Machine Learning · Massachusetts Institute of Technology · Red Hat AI Innovation Team and the MIT-IBM Watson AI Lab +1
AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology

Oct 5, 2026J/K move · Enter open · S save

Behavior-Preserving KV Cache Compression

Doo Hwan Hwang, Junyoung Jang, Junho Na, Hosung Lim+1

Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea

KV-Cache Eviction LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache logits induced by their removal and evaluating the resulting KL to the full-cache next-token distribution. Using pre-eviction forward statistics, the method avoids running separate masked forward passes for each candidate. Across diverse architectures and both prefill-time and generation-time compression, our method delivers substantial gains in downstream task quality over lightweight attention-based heuristics at matched retained-KV budgets, with the largest gains under aggressive compression. It achieves these gains with additional compression-time computation while retaining an end-to-end speedup over full-cache inference in our evaluated settings.

Abstract

KV-Cache Compression

Explore similar work

Behavior-Preserving KV Cache Compression

KV Cache Compression Through the Lens of Transform Coding

AnchorKV: Anchor-Residual KV Cache Compression