TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

Yu Wang; Zhu Wang; Ning Hu; Yifan Guo; Xiu Ma; Bin Guo; Zhiwen Yu

cs.LGOct 17, 2026

TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

Authors: Yu Wang, Zhu Wang, Ning Hu, Yifan Guo, Xiu Ma, Bin Guo, Zhiwen Yu

Organizations: Northwestern Polytechnical University, No. 127 West Youyi Road, Xian, 710129, Shaanxi, China · Xi’an Aeronautics Computing Technique Research Institute, AVIC, No. 15 Jinye 2nd Road, Xi’an, 710065, Shaanxi, China · College of Computer Science and Technology, Harbin Engineering University, No. 145 Nantong Street, Harbin, 150001, Heilongjiang, China

Abstract

Long-context large language model (LLM) inference is important for modern intelligent applications but remains constrained by peak memory during prompt prefill. Existing KV cache compression reduces decoding memory but typically requires full-prompt prefill to preserve global context, retaining a high construction peak. Online chunk-level eviction lowers memory earlier but selects before each layer has observed the complete prompt, potentially degrading cache construction and decoding. We present TideKV, an on-GPU cache-construction architecture using layer-major delayed eviction. TideKV processes all prompt chunks within each Transformer layer, retains full-prompt KV only for the active layer, and evicts at layer end. This schedule preserves full-prompt-aware selection while reducing activation and temporary memory. Experiments across multiple computing platforms show that TideKV preserves the inference quality and final cache size of the evaluated compression methods. At 64K on A100, it reduces the construction-time transient footprint by 83.89%. On RTX 4090, it increases time to first token by only 1.06% at 64K and enables 128K prefill. TideKV also completes 128K on Jetson, whereas matched post-prefill compression runs out of memory on both resource-constrained platforms. These results show that delayed layer-end eviction substantially expands feasible context lengths without sacrificing cache utility.

Figures & tables

Figure 1: Four schedules for constructing a compressed KV cache during prefill. (a) Full prefill without intermediate cache release retains uncompressed KV across all layers until the prompt has been processed, preserving full-prompt visibility at a high construction peak. (b) Chunked prefill without eviction processes the prompt chunk by chunk, yet every chunk’s KV stays resident, so memory still grows with prompt length. (c) Chunked prefill with online eviction evicts after each chunk to cap memory, but it decides what to keep before the full prompt is seen, which can degrade quality through myopic eviction. (d) TideKV (ours) streams the prompt chunk by chunk in a layer-major order and defers eviction to each layer’s boundary, so every layer scores against the full prompt while full-prompt KV is confined to one active layer and temporary activations to one chunk, alongside the hidden-state bridge and retained caches.

Symbol	Meaning
$N$	number of Transformer layers
$B$	batch size
$S$	prompt (sequence) length
$C$	chunk size used during prefill
$H_{q},H_{kv}$	number of query heads, KV heads
$D$	head dimension

Table 1: Notation.

Figure 2: Layer-major delayed eviction and the corresponding tensor lifetimes. Left: full prompt KV is resident only for the active layer $l$ ; completed layers hold compressed caches and future layers hold none. Right: chunk $j$ attends to the accumulated uncompressed KV of chunks $1,\ldots,j$ , while output hidden chunks form the bridge to layer $l{+}1$ . After all $m$ chunks complete, TideKV gathers the retained KV and absolute-position metadata into $\hat{\mathcal{C}}[l]$ and releases the full KV. Hidden-state write/read and retained-KV gather are the principal added data movements; the layer-end commit is the synchronization boundary before layer $l{+}1$ .

Figure 3: Memory-controlled two-pass blockwise scoring. Instead of materializing the dense $W\times S$ observation-by-prefix score tensor, the prefix keys are split into blocks. Pass 1 scans the blocks to build the numerically stable log-sum-exp normalizers $(m_{q},Z_{q})$ , keeping only $W$ -length state. Pass 2 rescans to form normalized weights, pools them into per-key scores, and keeps the top $K_{l}-|\mathcal{R}_{\mathrm{mand}}|$ nonmandatory entries together with the mandatory recent/sink set before gathering the retained $(\hat{K},\hat{V})$ . The temporary footprint $\mathrm{Temp}=O(Wb_{s}+K_{l})$ is independent of $S$ .

Figure 4: Logical architecture of the TideKV reference implementation. The layer-major scheduler coordinates the layer-end compressor and cross-layer cache manager to transform each layer’s full KV into a compact retained cache at the layer boundary. After all layers complete, the decode adapter exposes the assembled initial cache to generation. The memory accountant records component residency and allocator/process-level peaks.

Item	Configuration
Models	Llama-3.1-8B-Instruct; Mistral-7B-Instruct-v0.2; BF16
Accelerators	A100 PCIe 40 GB; RTX 4090 24 GB; Jetson AGX Orin 64 GB (MODE_50W)
A100 stack	CUDA 11.8; PyTorch 2.3.1; Transformers 4.44.2; FlashAttention 2.7.4.post1
Inference	Batch 1; chunk 2,048; physical KV budget 4,096 unless stated otherwise

Table 2: Principal experimental configuration. Platform-specific software is held fixed within every matched comparison.

	Single-Doc. QA			Multi-Doc. QA			Summarization			Few-shot Learning			Synthetic		Code
	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PRe	Lcc	RB-P	Ave. Score
Full Cache	29.69	44.92	55.78	54.68	47.26	30.89	35.09	23.55	27.23	73.0	90.23	44.22	10.38	99.5	64.58	58.97	49.37
Llama-3.1-8B-Instruct, $Budget=256$
snap	26.9	31.93	52.54	54.55	43.2	28.41	22.19	23.81	22.22	55.6	85.27	40.94	6.75	99	61.69	54.93	44.37
tidekv-snap	26.83	32.99	53.25	54.99	43.6	28.72	21.91	23.75	22.55	55.5	86.72	41.3	5.28	99.5	62.32	55.67	44.68
pyramidkv	29.17	36.12	52.84	54.72	43.8	29.87	23.13	22.87	22.6	54.5	76.42	41.64	6.78	99.5	60.54	51.97	44.15

Table 3: Detailed results of Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2 on LongBench. Bold scores indicate the higher average within each selector pair.

Method	Budget = 256						Budget = 1024						Budget = 4096
Method	4K	16K	32K	64K	128K	Avg	4K	16K	32K	64K	128K	Avg	4K	16K	32K	64K	128K	Avg
SnapKV	68.47	65.68	62.97	60.20	43.31	60.13	88.56	78.78	74.46	68.96	58.92	73.94	95.18	89.50	81.12	76.72	65.90	81.68
TideKV-Snap	69.70	67.88	64.62	59.95	42.94	61.02	88.25	78.57	74.28	69.14	59.07	73.86	95.00	90.47	82.07	78.07	68.51	82.82
PyramidKV	61.64	61.35	58.61	53.93	39.24	54.95	80.58	73.66	70.97	65.72	56.09	69.40	95.18	87.71	77.34	71.73	63.29	79.05
TideKV-PyramidKV	61.70	61.11	59.48	54.29	40.68	55.45	80.21	73.50	70.09	65.60	56.17	69.11	95.00	85.30	77.66	71.66	63.35	78.59
AdaKV	70.05	68.21	63.98	59.38	41.43	60.61	88.50	79.90	75.42	68.85	58.46	74.23	95.09	90.05	81.93	77.25	65.70	82.00

Table 4: RULER performance (%) of Llama-3.1-8B-Instruct on A100 under different physical KV budgets and context lengths. Avg equally weights the five displayed lengths. SnapKV and PyramidKV denote selector-matched controls. Higher is better.

Figure 5: Matched construction-memory accounting on A100. (a) At 64K, both methods reach the same 15.466-GiB post-prefill allocation and retain the same 512-MiB physical cache; TideKV-Snap reduces the transient construction footprint from 7.281 to 1.173 GiB. Diamonds mark the NVML process peaks. (b,c) At 128K, the allocated-peak reduction is identical under Native and Async, while allocator retention changes the reserved and process-level realization. Percentages denote reductions from SnapKV to TideKV-Snap.

Backend	Context	Alloc.	Reserved	NVML
FA2	8K	3.74%	5.41%	5.26%
FA2	16K	8.08%	11.08%	10.79%
FA2	32K	15.53%	21.38%	20.89%
FA2	64K	26.85%	24.59%	24.13%
SDPA	8K	3.55%	4.81%	4.68%
SDPA	16K	7.04%	9.97%	9.70%

Table 5: Backend and persistent-workload robustness on A100. Positive values in panel (a) denote reductions from SnapKV to TideKV-Snap; negative values denote increases. Panel (b) reports 150 requests per configuration (600 total).

Platform	Context	SnapKV peak	TideKV-Snap peak
RTX 4090	32K	20.027	16.771
RTX 4090	64K	23.447	17.240
RTX 4090	128K	OOM	18.529
Jetson AGX Orin	32K	19.099	16.131
Jetson AGX Orin	64K	OOM	16.631
Jetson AGX Orin	128K	OOM	17.632

Table 6: First-token capacity on resource-constrained platforms. Entries are peak memory in GiB: NVML process memory on the RTX 4090 and CUDA-managed allocation on Jetson. OOM denotes prefill failure. Every completed method retains a 512-MiB final physical KV cache.

Method	LongBench	4K/8K	16K/32K	64K/128K
TideKV-Snap	48.90	95.14/93.44	90.63/82.17	76.65/65.93
Matched SnapKV	48.87	95.03/93.60	89.55/80.98	74.82/OOM
KeyDiff	47.29	95.06/84.05	73.54/64.38	56.32/33.07
KIVI	48.69	94.04/93.31	91.01/84.55	OOM/OOM
HeadInfer	49.08	95.16/94.36	90.89/85.93	84.89/OOM

Table 7: LongBench averages and RULER scores of Llama-3.1-8B-Instruct on RTX 4090. Paired RULER columns indicate context lengths. The 64K matched SnapKV score uses completed examples only.

	hidden		logit
Mode (selector, eviction)	rel- $L_{2}$	cos	top-1	JS
Full-OneShot (ref.)	$0$	$1.000$	$1.00$	$0$
Full-Chunked (no evict)	$0.012$	$0.9999$	$0.95$	$3\times 10^{-5}$
Tide-NoEvict	$0.012$	$0.9999$	$0.95$	$3\times 10^{-5}$
TideKV (SnapKV, del.)	$0.012$	$0.9999$	$0.95$	$3\times 10^{-5}$
Block-SnapKV (SnapKV, imm.)	$0.122$	$0.990$	$0.55$	$0.002$

Table 8: Prompt-encoding fidelity and representation drift relative to Full-OneShot (Llama-3.1-8B,

S{=}16

C{=}512

, budget

1024

, 20 prompts). Columns: hidden-state relative-

L_{2}

and cosine similarity; logit top-1 agreement and Jensen–Shannon (JS) divergence between the prefill next-token distributions. Top: layer-major scheduling and delayed eviction stay at the bf16 chunking floor. Bottom: with the same SnapKV selector, immediate eviction drifts an order of magnitude more. Standard post-prefill SnapKV coincides with the Full-OneShot row, as its prefill forward is uncompressed.

Panel of Figure 6 — Figure 6: Representation drift relative to Full-OneShot. Immediate Block-SnapKV drift grows with chunk index and decreases with a larger budget, whereas TideKV remains at the no-eviction chunking floor.

Platform	Context	SnapKV TTFT / peak	TideKV-Snap TTFT / peak	TTFT change	Peak change
A100	64K	12.390 s / 24.096 GiB	14.464 s / 17.283 GiB	+16.74%	$-28.28$ %
A100	128K	42.037 s / 33.080 GiB	43.791 s / 18.572 GiB	+4.17%	$-43.86$ %
RTX 4090	32K	5.500 s / 20.027 GiB	6.191 s / 16.771 GiB	+12.55%	$-16.26$ %
RTX 4090	64K	15.698 s / 23.447 GiB	15.865 s / 17.240 GiB	+1.06%	$-26.47$ %
RTX 4090	128K	OOM	47.110 s / 18.529 GiB	–	feasible
Jetson	32K	116.659 s / 19.099 GiB	126.492 s / 16.131 GiB	+8.43%	$-15.54$ %

Table 9: Matched 32-token latency and memory results. Values are medians; memory is the NVML process peak on A100/RTX 4090 and CUDA-managed allocation on Jetson. Each completed method starts decoding from a 512-MiB compressed KV cache.

Platform, context	Decode rate	Compression share
A100, 16K	29.77 / 36.58	21.92%
A100, 64K	30.50 / 37.06	13.38%
RTX 4090, 32K	26.86 / 32.45	17.96%
RTX 4090, 64K	27.13 / 31.59	13.73%
Jetson, 16K	2.67 / 3.46	7.16%
Jetson, 32K	2.64 / 3.45	5.89%

Table 10: Decode rate and layer-end compression share. Decode entries are SnapKV/TideKV-Snap in token/s. Compression share is the median per-run ratio of compression time to TTFT from separate profiling runs (five on A100/RTX 4090; three on Jetson).

Platform and context	$C=1024$	$C=2048$	$C=4096$
A100, 64K	16.783 / 17.125	14.464 / 17.283	13.899 / 17.713
RTX 4090, 64K	17.181 / 17.080	15.804 / 17.240	15.981 / 17.670
Jetson, 32K	128.27 / 16.037	125.66 / 16.131	128.62 / 16.320

Table 11: TideKV-Snap chunk-size Pareto sweep. Each entry is TTFT in seconds / peak memory in GiB (NVML on A100/RTX 4090; CUDA-managed on Jetson). RTX 4090 and Jetson timings were measured independently of Table 9 .

Memory-Efficient Inference

Memory-Efficient Inference reduces the memory needed to run neural networks while balancing accuracy, latency, throughput, and energy use.

See all 168 papers

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
Mar 27, 2026Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology, Shenzhen · Guangzhou University
DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
Oct 5, 2026Zhe Wang, Jiakai Li, Yujia Sun +2Long-Context Language Model Inference KV-Cache Eviction
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference Efficiency KV Caching
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany

Mar 27, 2026J/K move · Enter open · S save

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

Gradwell Dzikanyanga, Weihao Yang, Hao Huang, Donglei Wu+3

Harbin Institute of Technology, Shenzhen · Guangzhou University

KV-Cache Management Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Offloading

Open paper AskPDF arXiv

Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scalability bottleneck. Existing approaches largely treat KV states as equally important across time, implicitly assuming uniform precision and accessibility. However, this assumption contrasts with human memory systems, where memories vary in clarity, recall frequency, and relevance with temporal proximity.Motivated by this insight, we propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache. TTKV partitions the KV cache into temporal tiers with heterogeneous capacity and precision. The design addresses three aspects: (1) Tier Layout, decoupling fast and slow memory using HBM and DRAM; (2) Tier Content, assigning more recent KV states to faster, higher-precision tiers based on temporal proximity; and (3) Tier Interaction, employing block-wise streaming attention to overlap communication and computation when accessing slow tiers. Experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.

Abstract

Memory-Efficient Inference

Explore similar work

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference