DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

Jinyu Guo; Zhihan Zhang; Jiehui Xie; Md. Tamim Iqbal; Dongshen Han; Lik-Hang Lee; Sung-Ho Bae; Jie Zou

cs.CLApr 21, 2026

DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

Authors: Jinyu Guo, Zhihan Zhang, Jiehui Xie, Md. Tamim Iqbal, Dongshen Han, Lik-Hang Lee, Sung-Ho Bae, Jie Zou, +2 more

Organizations: School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Bangladesh University of Engineering and Technology · The Hong Kong Polytechnic University · Kyung Hee University, School of Computing

Abstract

The quadratic computational complexity of the standard attention mechanism constitutes a fundamental bottleneck for large language models in long-context inference. While existing KV cache compression methods alleviate memory pressure, they often sacrifice generation quality and fail to address the high overhead of floating-point arithmetic. This paper introduces DASH-KV, an innovative acceleration framework that reformulates attention as approximate nearest-neighbor search via asymmetric deep hashing. Under this paradigm, we design an asymmetric encoding architecture that differentially maps queries and keys to account for their distinctions in precision and reuse characteristics. To balance efficiency and accuracy, we further introduce a dynamic mixed-precision mechanism that adaptively retains full-precision computation for critical tokens. Extensive experiments on LongBench demonstrate that DASH-KV significantly outperforms state-of-the-art baseline methods while matching the performance of full attention, all while reducing inference complexity from O(N^2) to linear O(N).

Explore similar work

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3LLM Inference Efficiency Dynamic Sparse Attention
C $^2$ KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Jul 20, 2026Chuheng Du, Junyi Chen, Hanlin Tang +7Key-Value Cache Efficient Long-Context Inference
SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5Key-Value Cache Compression Efficient Long-Context Inference

Jun 30, 2026 · cs.LGJ/K move · Enter open · S save

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi+2

Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.

LLM Inference Efficiency Dynamic Sparse Attention Key-Value Binary

Open paper Ask about it PDF

Abstract

Explore similar work

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

C2^22KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

SGD-KV: Summarization Guided KV Cache Compression

C $^2$ KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference