Long-context large language model (LLM) inference is important for modern intelligent applications but remains constrained by peak memory during prompt prefill. Existing KV cache compression reduces decoding memory but typically requires full-prompt prefill to preserve global context, retaining a high construction peak. Online chunk-level eviction lowers memory earlier but selects before each layer has observed the complete prompt, potentially degrading cache construction and decoding. We present TideKV, an on-GPU cache-construction architecture using layer-major delayed eviction. TideKV processes all prompt chunks within each Transformer layer, retains full-prompt KV only for the active layer, and evicts at layer end. This schedule preserves full-prompt-aware selection while reducing activation and temporary memory. Experiments across multiple computing platforms show that TideKV preserves the inference quality and final cache size of the evaluated compression methods. At 64K on A100, it reduces the construction-time transient footprint by 83.89%. On RTX 4090, it increases time to first token by only 1.06% at 64K and enables 128K prefill. TideKV also completes 128K on Jetson, whereas matched post-prefill compression runs out of memory on both resource-constrained platforms. These results show that delayed layer-end eviction substantially expands feasible context lengths without sacrificing cache utility.
Figures & tables
Figure 1: Four schedules for constructing a compressed KV cache during prefill. (a) Full prefill without intermediate cache release retains uncompressed KV across all layers until the prompt has been processed, preserving full-prompt visibility at a high construction peak. (b) Chunked prefill without eviction processes the prompt chunk by chunk, yet every chunk’s KV stays resident, so memory still grows with prompt length. (c) Chunked prefill with online eviction evicts after each chunk to cap memory, but it decides what to keep before the full prompt is seen, which can degrade quality through myopic eviction. (d) TideKV (ours) streams the prompt chunk by chunk in a layer-major order and defers eviction to each layer’s boundary, so every layer scores against the full prompt while full-prompt KV is confined to one active layer and temporary activations to one chunk, alongside the hidden-state bridge and retained caches.
Symbol
Meaning
N
number of Transformer layers
B
batch size
S
prompt (sequence) length
C
chunk size used during prefill
Hq,Hkv
number of query heads, KV heads
D
head dimension
Table 1: Notation.
Figure 2: Layer-major delayed eviction and the corresponding tensor lifetimes. Left: full prompt KV is resident only for the active layer l ; completed layers hold compressed caches and future layers hold none. Right: chunk j attends to the accumulated uncompressed KV of chunks 1,…,j , while output hidden chunks form the bridge to layer l+1 . After all m chunks complete, TideKV gathers the retained KV and absolute-position metadata into C^[l] and releases the full KV. Hidden-state write/read and retained-KV gather are the principal added data movements; the layer-end commit is the synchronization boundary before layer l+1 .
Figure 3: Memory-controlled two-pass blockwise scoring. Instead of materializing the dense W×S observation-by-prefix score tensor, the prefix keys are split into blocks. Pass 1 scans the blocks to build the numerically stable log-sum-exp normalizers (mq,Zq) , keeping only W -length state. Pass 2 rescans to form normalized weights, pools them into per-key scores, and keeps the top Kl−∣Rmand∣ nonmandatory entries together with the mandatory recent/sink set before gathering the retained (K^,V^) . The temporary footprint Temp=O(Wbs+Kl) is independent of S .
Figure 4: Logical architecture of the TideKV reference implementation. The layer-major scheduler coordinates the layer-end compressor and cross-layer cache manager to transform each layer’s full KV into a compact retained cache at the layer boundary. After all layers complete, the decode adapter exposes the assembled initial cache to generation. The memory accountant records component residency and allocator/process-level peaks.
Table 2: Principal experimental configuration. Platform-specific software is held fixed within every matched comparison.
Single-Doc. QA
Multi-Doc. QA
Summarization
Few-shot Learning
Synthetic
Code
NrtvQA
Qasper
MF-en
HotpotQA
2WikiMQA
Musique
GovReport
QMSum
MultiNews
TREC
TriviaQA
SAMSum
PCount
PRe
Lcc
RB-P
Ave. Score
Full Cache
29.69
44.92
55.78
54.68
47.26
30.89
35.09
23.55
27.23
73.0
90.23
44.22
10.38
99.5
64.58
58.97
49.37
Llama-3.1-8B-Instruct, Budget=256
snap
26.9
31.93
52.54
54.55
43.2
28.41
22.19
23.81
22.22
55.6
85.27
40.94
6.75
99
61.69
54.93
44.37
tidekv-snap
26.83
32.99
53.25
54.99
43.6
28.72
21.91
23.75
22.55
55.5
86.72
41.3
5.28
99.5
62.32
55.67
44.68
pyramidkv
29.17
36.12
52.84
54.72
43.8
29.87
23.13
22.87
22.6
54.5
76.42
41.64
6.78
99.5
60.54
51.97
44.15
Table 3: Detailed results of Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2 on LongBench. Bold scores indicate the higher average within each selector pair.
Method
Budget = 256
Budget = 1024
Budget = 4096
4K
16K
32K
64K
128K
Avg
4K
16K
32K
64K
128K
Avg
4K
16K
32K
64K
128K
Avg
SnapKV
68.47
65.68
62.97
60.20
43.31
60.13
88.56
78.78
74.46
68.96
58.92
73.94
95.18
89.50
81.12
76.72
65.90
81.68
TideKV-Snap
69.70
67.88
64.62
59.95
42.94
61.02
88.25
78.57
74.28
69.14
59.07
73.86
95.00
90.47
82.07
78.07
68.51
82.82
PyramidKV
61.64
61.35
58.61
53.93
39.24
54.95
80.58
73.66
70.97
65.72
56.09
69.40
95.18
87.71
77.34
71.73
63.29
79.05
TideKV-PyramidKV
61.70
61.11
59.48
54.29
40.68
55.45
80.21
73.50
70.09
65.60
56.17
69.11
95.00
85.30
77.66
71.66
63.35
78.59
AdaKV
70.05
68.21
63.98
59.38
41.43
60.61
88.50
79.90
75.42
68.85
58.46
74.23
95.09
90.05
81.93
77.25
65.70
82.00
Table 4: RULER performance (%) of Llama-3.1-8B-Instruct on A100 under different physical KV budgets and context lengths. Avg equally weights the five displayed lengths. SnapKV and PyramidKV denote selector-matched controls. Higher is better.
Figure 5: Matched construction-memory accounting on A100. (a) At 64K, both methods reach the same 15.466-GiB post-prefill allocation and retain the same 512-MiB physical cache; TideKV-Snap reduces the transient construction footprint from 7.281 to 1.173 GiB. Diamonds mark the NVML process peaks. (b,c) At 128K, the allocated-peak reduction is identical under Native and Async, while allocator retention changes the reserved and process-level realization. Percentages denote reductions from SnapKV to TideKV-Snap.
Backend
Context
Alloc.
Reserved
NVML
FA2
8K
3.74%
5.41%
5.26%
FA2
16K
8.08%
11.08%
10.79%
FA2
32K
15.53%
21.38%
20.89%
FA2
64K
26.85%
24.59%
24.13%
SDPA
8K
3.55%
4.81%
4.68%
SDPA
16K
7.04%
9.97%
9.70%
Table 5: Backend and persistent-workload robustness on A100. Positive values in panel (a) denote reductions from SnapKV to TideKV-Snap; negative values denote increases. Panel (b) reports 150 requests per configuration (600 total).
Platform
Context
SnapKV peak
TideKV-Snap peak
RTX 4090
32K
20.027
16.771
RTX 4090
64K
23.447
17.240
RTX 4090
128K
OOM
18.529
Jetson AGX Orin
32K
19.099
16.131
Jetson AGX Orin
64K
OOM
16.631
Jetson AGX Orin
128K
OOM
17.632
Table 6: First-token capacity on resource-constrained platforms. Entries are peak memory in GiB: NVML process memory on the RTX 4090 and CUDA-managed allocation on Jetson. OOM denotes prefill failure. Every completed method retains a 512-MiB final physical KV cache.
Method
LongBench
4K/8K
16K/32K
64K/128K
TideKV-Snap
48.90
95.14/93.44
90.63/82.17
76.65/65.93
Matched SnapKV
48.87
95.03/93.60
89.55/80.98
74.82/OOM
KeyDiff
47.29
95.06/84.05
73.54/64.38
56.32/33.07
KIVI
48.69
94.04/93.31
91.01/84.55
OOM/OOM
HeadInfer
49.08
95.16/94.36
90.89/85.93
84.89/OOM
Table 7: LongBench averages and RULER scores of Llama-3.1-8B-Instruct on RTX 4090. Paired RULER columns indicate context lengths. The 64K matched SnapKV score uses completed examples only.
hidden
logit
Mode (selector, eviction)
rel- L2
cos
top-1
JS
Full-OneShot (ref.)
0
1.000
1.00
0
Full-Chunked (no evict)
0.012
0.9999
0.95
3×10−5
Tide-NoEvict
0.012
0.9999
0.95
3×10−5
TideKV (SnapKV, del.)
0.012
0.9999
0.95
3×10−5
Block-SnapKV (SnapKV, imm.)
0.122
0.990
0.55
0.002
Table 8: Prompt-encoding fidelity and representation drift relative to Full-OneShot (Llama-3.1-8B, S=16 K, C=512 , budget 1024 , 20 prompts). Columns: hidden-state relative- L2 and cosine similarity; logit top-1 agreement and Jensen–Shannon (JS) divergence between the prefill next-token distributions. Top: layer-major scheduling and delayed eviction stay at the bf16 chunking floor. Bottom: with the same SnapKV selector, immediate eviction drifts an order of magnitude more. Standard post-prefill SnapKV coincides with the Full-OneShot row, as its prefill forward is uncompressed.
Figure 6: Representation drift relative to Full-OneShot. Immediate Block-SnapKV drift grows with chunk index and decreases with a larger budget, whereas TideKV remains at the no-eviction chunking floor.
Platform
Context
SnapKV TTFT / peak
TideKV-Snap TTFT / peak
TTFT change
Peak change
A100
64K
12.390 s / 24.096 GiB
14.464 s / 17.283 GiB
+16.74%
−28.28 %
A100
128K
42.037 s / 33.080 GiB
43.791 s / 18.572 GiB
+4.17%
−43.86 %
RTX 4090
32K
5.500 s / 20.027 GiB
6.191 s / 16.771 GiB
+12.55%
−16.26 %
RTX 4090
64K
15.698 s / 23.447 GiB
15.865 s / 17.240 GiB
+1.06%
−26.47 %
RTX 4090
128K
OOM
47.110 s / 18.529 GiB
–
feasible
Jetson
32K
116.659 s / 19.099 GiB
126.492 s / 16.131 GiB
+8.43%
−15.54 %
Table 9: Matched 32-token latency and memory results. Values are medians; memory is the NVML process peak on A100/RTX 4090 and CUDA-managed allocation on Jetson. Each completed method starts decoding from a 512-MiB compressed KV cache.
Platform, context
Decode rate
Compression share
A100, 16K
29.77 / 36.58
21.92%
A100, 64K
30.50 / 37.06
13.38%
RTX 4090, 32K
26.86 / 32.45
17.96%
RTX 4090, 64K
27.13 / 31.59
13.73%
Jetson, 16K
2.67 / 3.46
7.16%
Jetson, 32K
2.64 / 3.45
5.89%
Table 10: Decode rate and layer-end compression share. Decode entries are SnapKV/TideKV-Snap in token/s. Compression share is the median per-run ratio of compression time to TTFT from separate profiling runs (five on A100/RTX 4090; three on Jetson).
Platform and context
C=1024
C=2048
C=4096
A100, 64K
16.783 / 17.125
14.464 / 17.283
13.899 / 17.713
RTX 4090, 64K
17.181 / 17.080
15.804 / 17.240
15.981 / 17.670
Jetson, 32K
128.27 / 16.037
125.66 / 16.131
128.62 / 16.320
Table 11: TideKV-Snap chunk-size Pareto sweep. Each entry is TTFT in seconds / peak memory in GiB (NVML on A100/RTX 4090; CUDA-managed on Jetson). RTX 4090 and Jetson timings were measured independently of Table 9 .
Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scalability bottleneck. Existing approaches largely treat KV states as equally important across time, implicitly assuming uniform precision and accessibility. However, this assumption contrasts with human memory systems, where memories vary in clarity, recall frequency, and relevance with temporal proximity.Motivated by this insight, we propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache. TTKV partitions the KV cache into temporal tiers with heterogeneous capacity and precision. The design addresses three aspects: (1) Tier Layout, decoupling fast and slow memory using HBM and DRAM; (2) Tier Content, assigning more recent KV states to faster, higher-precision tiers based on temporal proximity; and (3) Tier Interaction, employing block-wise streaming attention to overlap communication and computation when accessing slow tiers. Experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.
Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4
Harbin Institute of Technology, Shenzhen · Guangzhou University
Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.
Zhe Wang, Jiakai Li, Yujia Sun +2
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs. These methods ignore the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrading the performance of LLMs. To address this issue, we propose CompressKV, a resource-efficient KV-cache compression framework for GQA-based LLMs. Instead of aggregating attention scores from all heads, CompressKV identifies Semantic Retrieval Heads (SRHs) that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, and uses them to select tokens whose KV pairs should be retained. Furthermore, CompressKV allocates cache budgets across layers according to offline estimates of layer-wise eviction error. Experiments on LongBench and Needle-in-a-Haystack show that CompressKV consistently outperforms existing KV-cache eviction methods across memory budgets. Notably, it preserves over 97% of full-cache performance using only 3% of the KV cache on LongBench question-answering tasks and achieves 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack. These results demonstrate an improved resource--performance trade-off for long-context LLM inference. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV
Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany