KV Cache Selection

KV: Key-Value

Follow this topic Ask about this topic

Latest papers

9 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

BasisKV: Value Latents for Communication and Key Routing
Nov 2, 2026Lixun Zhang, Zishan Shao, Kangning Cui +3KV Cache Selection KV-Cache Management
Duke University · Wake Forest University
EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies
Nov 1, 2026Yuhao Liu, Yuehao Li, Linghe Zhang +4KV Cache Selection Long-Context Language Model Inference
Laboratory of Solid State Optoelectronics Information Technology, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
One Distribution, Two Channels: Coverage-Calibrated KV Retention and Compensation
Sep 28, 2026Shuxin Liu, Qing Liu, Yi Du +1KV Cache Selection KV-Cache Eviction
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
Sep 3, 2026WenJie FanKV Cache Selection KV Caching
Yotta Labs
Small Frequency Corrections Can Change What Survives KV Cache Compression
Aug 27, 2026Hong Chen, Yudong Zeng, Yongwei Huang +5KV Cache Selection KV Caching
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Bosum Institute of Management Science, Shenzhen, China
KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems
Jul 27, 2026Shuo Wang, Fang Xi, Wenyuan Huang +2KV Cache Selection LLM Serving
QiYuanLab Beijing, China
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1KV Cache Selection Self-Attention
Seoul National University
PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection
Mar 23, 2026Hyoseok Park, Yeonsang ParkKV Cache Selection LLM Inference Acceleration
Department of Physics, Chungnam National University, Daejeon 34134, Republic of Korea
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3KV Cache Selection Self-Attention
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1

Nov 2, 2026J/K move · Enter open · S save

BasisKV: Value Latents for Communication and Key Routing

Lixun Zhang, Zishan Shao, Kangning Cui, Yu Liang+2

Duke University · Wake Forest University

KV Cache Selection KV-Cache Management LLM Inference Acceleration KV-Cache Compression Low-Rank Compression Parallel LLM Inference

Open paper AskPDF arXiv

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV-cache. Low-rank K/V projections provide a promising way to compress the KV-cache with minimal quality loss, but prior work has largely optimized KV-cache compression in isolation, leaving distributed inference, especially with tensor parallelism where attention computation is partitioned across multiple GPUs, largely underexplored. We make two observations: (i) tensor-parallel communication is an important cost when deploying low-rank KV-cache compression in distributed inference; and (ii) Value compression can be associated with the output projection, allowing the same low-rank Value representation to reduce both Value-cache width and tensor-parallel communication. Based on these observations, we propose BasisKV, a low-rank KV-cache compression method for communication-intensive, long-context LLM inference. BasisKV jointly optimizes the low-rank factors across per-GPU attention partitions against the complete attention output, rather than fitting each partition in isolation. The compressed Value cache is further reused to help identify which Key pages should be accessed: it predicts part of the Key information needed for routing, while a Key-specific residual stores what cannot be recovered from Value. The original Keys are then retrieved only for the selected pages to compute exact QK scores. Across multiple LLM families and system settings, BasisKV achieves a better quality--efficiency tradeoff than existing low-rank baselines, reaching up to 1.42x end-to-end speedup under 8-GPU tensor-parallel inference. Our code is available at https://github.com/alexz949/BasisKV