KV Caching

KV: Key-Value

Key-value caching stores the keys and values computed for earlier tokens so a Transformer can generate new tokens without recomputing the full history. Because these caches can consume substantial memory and bandwidth in long-context and high-concurrency inference, methods include token eviction, low-bit quantization, low-rank compression, sparse retrieval, and offloading or reusing cached prefixes. Researchers evaluate these approaches on long-context retrieval, question answering, and reasoning benchmarks, as well as serving metrics such as time to first token, decode speed, and throughput. The central tension is reducing cache cost without losing useful evidence or changing model behavior, and the best trade-off often depends on the workload, model, and hardware.

Latest papers

281 papers0 papers in the last four weeks, down 100% on the four weeks before. 0.0% of all new papers.

Memory as a cache: Exact context reuse and deletion by construction
Sep 26, 2026Shengyao Wang, Jiang LiuKV Caching Transformer Inference
The KV Cache Is the New Memory Wall
Sep 25, 2026Tejinder SinghKV Caching Long-Context Language Model Inference
Dell Technologies, Santa Clara, CA 95054, USA
When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
Sep 24, 2026Yiyu Liu, Minlan Yu, Juncheng YangLLM Inference Efficiency KV Caching
Harvard University
Shared Global KV with Layer-Specific Local History
Sep 23, 2026Xinglang XianKV Caching Long-Context Language Model Inference
Written as a Record, Read as an Address: What a Forward Pass Leaves in an Operation's KV Cache
Sep 21, 2026Lingfeng Wu, Behzad ShomaliKV Caching LLM Interpretability
University of Bonn · Lamarr Institute
Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
Sep 20, 2026Hyunsun Chung, Taewan Noh, Minji Kim +3LLM Inference Acceleration KV-Cache Offloading
Sogang University · Samsung Electronics Co. · ETRI
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Sep 17, 2026DeepSeek-AI, :, Anyi Xu +590KV Caching Long-Context Language Model Inference
D-Quant: Driftable Entropy Coding for KV Cache Quantization
Sep 17, 2026Yi Su, Hong Liu, Guanghua Yu +1KV Caching KV-Cache Compression
Tencent
PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving
Sep 17, 2026Omkar Shewale, Deepak Kumar, Divakar Kumar YadavLLM Inference Efficiency LLM Inference
Illinois Institute of Technology · University of Wisconsin–Milwaukee
Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference Efficiency KV Caching
Tsinghua University, China · Alibaba Cloud Computing, China · Zhejiang University, China
MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators
Sep 16, 2026Dong Liu, Yanxuan YuTransformer Inference KV Caching
Columbia University New York, NY, USA
Contiguity, Not Importance: Budgeted Repair of Stale KV Caches After Document Edits
Sep 16, 2026Mingyang Mao, Wyatt Mackey, Xiaomin LinRetrieval-Augmented Generation KV Caching
University of South Florida, Electrical and Computer Engineering · DEVCOM Army Research Laboratory
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Sep 15, 2026Vivek KalyanaranganKV-Cache Offloading KV Caching
Where Should a Document Live: Context, Representations, or Parameters?
Sep 15, 2026Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias +1Memory-Augmented Language Models Knowledge Augmentation for Language Models
Amazon AGI
Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs
Sep 15, 2026Dushyant RajputLLM Inference Efficiency LLM Serving
AltSlate Labs LLP
Divergence Timing and Cumulative Disagreement under KV-Cache Eviction
Sep 15, 2026Xinyue Luo, Fei YuKV Caching KV-Cache Eviction
Wealth AI Lab, Ant Group
T-LoopFormer: Token-Level Elastic-Depth Looped Transformers with Dynamic Routing
Sep 14, 2026Mingqian Yu, Wenpeng Zhang, Shaobo Cui +1KV Caching Recurrent Transformers
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Independent Reasearcher · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China
Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries
Sep 14, 2026Frank LiKV Caching KV-Cache Management
UNSW Sydney
Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction
Sep 14, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1KV Caching KV-Cache Eviction
School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Department of Electrical and Computer Engineering, National University of Singapore
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV Caching KV-Cache Eviction
Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA
Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
Sep 10, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache Offloading KV Caching
Vrije Universiteit Amsterdam Amsterdam, the Netherlands · IBM Research Zurich Zurich, Switzerland
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Sep 10, 2026Suho Yoo, Hyunjong Ok, Jongmin Choi +2Efficient Multimodal Inference KV Caching
KAIST · POSTECH
K/V-Cache Interventions Dissociate Representation Alignment from Persona Expression in Decoder-Only Language Models
Sep 10, 2026Yu Sun, Mengyin Lu, Cong Feng +2KV Caching Decoder-Only Language Models
LinkedIn
Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving
Sep 9, 2026Hongjian Fan, Kevin Zhang, David Habinsky +1LLM Inference Efficiency KV Caching
Research Group, Seagate Technology, LLC
KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
Sep 9, 2026Xi Shi, Mengxin Zheng, Qian LouKV Caching LLM Inference Acceleration
University of Central Florida
Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
Sep 9, 2026Fumihiko Tachibana, Daisuke Miyashita, Jun DeguchiRetrieval-Augmented Generation KV Caching
Kioxia Corporation
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Sep 8, 2026Vishesh Tripathi, Abhay Kumar, Ramsha KhanEfficient Transformer Inference Grouped-Query Attention
FrontiersMind
BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agents
Sep 8, 2026Yanhong Qian, Xuanying He, Qingguo Meng +3Agent Memory Privacy-Preserving Language Models
Anhui Provincial Key Laboratory of Secure Artificial Intelligence, School of Artificial Intelligence, Anhui University, Hefei 230093, China
Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
Sep 8, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +2KV Caching KV-Cache Eviction
College of Computer Science, Sichuan University, Chengdu, China · 2Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore

Sep 26, 2026J/K move · Enter open · S save

Memory as a cache: Exact context reuse and deletion by construction

Shengyao Wang, Jiang Liu

KV Caching Transformer Inference Memory-Augmented Language Models Long-Context Language Modeling Memory-Efficient Inference

Open paper AskPDF arXiv

The KV cache of a transformer entangles every token's representation with its entire prefix: a passage encoded once cannot be reused under a different prefix or removed without recomputing everything after it, so exact cache reuse is limited to shared prefixes. We present SMem, an architecture whose context representation is a cache by construction. A block-local encoder maps each block to memory rows independently of other blocks, and a reader conditions generation on their union through cross-attention. For every parameter setting, memory composes exactly at fixed block indices, deleting a block is an exact

O(b)

update for

b

-token blocks, and the memory state is independent of the edit path. At

4\times

the training context, under the shared recipe, SMem retrieves planted needles beyond any trained-length window (exact match 0.14-0.28 at distances of 31 and 63 blocks), where learned-position, RoPE, and Block-Attention-style transformers all score at most 0.02. A fully cached context is served by computing one block alone at a near-constant 3.1-6.2 ms, whereas cold prefill grows with context; batched decode stores 34-38% fewer KV rows and runs 1.4-1.7

\times

faster when bandwidth-bound; and deletion beats suffix recomputation by 8.5

\times

at 512 blocks and 452

\times

at 4096 blocks (32-256

\times

the trained length, probing the cost model rather than a served regime). The cost is a perplexity gap of -4.7% to +2.8% (negative favors SMem) against a parameter-matched transformer with the same positional scheme, at 160M-1.5B on FineWeb-Edu across two recipes and a learning-rate search. SMem also composes with RoPE: at 160M and 410M the composite matches or leads the matched transformer and closes 29-59% of SMem's gap to a RoPE transformer. Dropping prefix entanglement thus keeps perplexity comparable while making the cache exactly composable and editable.