KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

Chuangtao Chen; Grace Li Zhang; Xunzhao Yin; Cheng Zhuo; Bing Li; Ulf Schlichtmann

cs.LGApr 14, 2026

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

Authors: Chuangtao Chen, Grace Li Zhang, Xunzhao Yin, Cheng Zhuo, Bing Li, Ulf Schlichtmann

Organizations: Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China · Technische Universität Ilmenau Ilmenau, Germany

Abstract

Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.

Figures & tables

Figure 1: Comparison of KV cache reuse architectures. (a) Recomputation-based approaches require inference-time algorithms to select and recompute important tokens to repair contextual staleness; (b) the proposed KV Packet approach wraps documents with global adapters.

Figure 2: Comparison of mathematically equivalent attention maps. (a) Full Recomputation. (b) Naive Concatenation. (c) Selective Recomputation (dashed blocks for recomputed tokens). (d) KV Packet: Attention with lightweight adapters ( H eader and T railer).

Figure 3: Evaluation results (F1 score, FLOPs, Time-to-First-Token) of Llama-3.1-8B / Qwen-3-4B on datasets: Needle-in-a-Haystack, Biography, HotpotQA, and MusiQue.

Figure 4: F1 Score v.s. Compression rate of Llama-3.1-8B-Instruct model on four datasets with different compression methods

	Evaluation Dataset
Training Dataset	Biography	HotpotQA	MusiQue	NIAH
No Recompute	0.06	0.24	0.01	0.39
Biography	0.96	0.18	0.16	0.44
HotpotQA	0.88	0.44	0.39	0.36
MusiQue	0.87	0.36	0.37	0.34
Needle-in-a-Haystack (NIAH)	0.54	0.28	0.07	0.80

Table 1: Cross-domain generalization analysis. Rows represent the dataset used to train the KV Packet adapters for Llama-3.1-8B-Instruct, while columns indicate the evaluation performance (F1 score) across different domains.

Figure 5: Query-to-context attention scores of the No Recompute and KV Packet methods. The attention scores are averaged by all layers, all query tokens, and 50 randomly selected samples from each dataset. The token positions for the No Recompute method in the figure are adjusted to align with those of the KV Packet method with filler tokens.

KV Caching

Key-value (KV) caching reuses Transformer states to speed generation while managing memory and data movement.

See all 281 papers

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
Mar 27, 2026Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology, Shenzhen · Guangzhou University
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache Offloading KV Caching
University of British Columbia · Microsoft Research
PatchKV: Weight-Space Compensation of KV Cache
Sep 30, 2026Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi +2Long-Context Language Model Inference LLM Inference Acceleration
KAIST

Mar 27, 2026J/K move · Enter open · S save

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

Gradwell Dzikanyanga, Weihao Yang, Hao Huang, Donglei Wu+3

Harbin Institute of Technology, Shenzhen · Guangzhou University

KV-Cache Management Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Offloading

Open paper AskPDF arXiv

Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scalability bottleneck. Existing approaches largely treat KV states as equally important across time, implicitly assuming uniform precision and accessibility. However, this assumption contrasts with human memory systems, where memories vary in clarity, recall frequency, and relevance with temporal proximity.Motivated by this insight, we propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache. TTKV partitions the KV cache into temporal tiers with heterogeneous capacity and precision. The design addresses three aspects: (1) Tier Layout, decoupling fast and slow memory using HBM and DRAM; (2) Tier Content, assigning more recent KV states to faster, higher-precision tiers based on temporal proximity; and (3) Tier Interaction, employing block-wise streaming attention to overlap communication and computation when accessing slow tiers. Experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.

Abstract

KV Caching

Explore similar work

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

PatchKV: Weight-Space Compensation of KV Cache