CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference

Chao Fei; Guozhong Li; Chenxi Liu; Panos Kalnis

cs.AIFeb 24, 2026

CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference

Authors: Chao Fei, Guozhong Li, Chenxi Liu, Panos Kalnis

Organizations: King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China

Abstract

Long-context LLMs demand accurate inference at low latency, yet decoding becomes primarily constrained by KV cache as context grows. Prior pruning methods are largely context-agnostic: their token selection ignores step-wise relevance and local semantics, which undermines quality. Moreover, their irregular accesses and selection overheads yield only limited wall-clock speedups. To address this, we propose CHESS, an algorithm-system co-design KV-cache management system. Algorithmically, CHESS introduces a context-aware, hierarchical selection policy that dynamically reconstructs a coherent context for the current decoding. System-wise, coarse granularity selection eliminates expensive data movement, fully realizing practical acceleration from theoretical sparsity. Extensive evaluations demonstrate that CHESS surpasses Full-KV quality using only 1% of the KV cache, delivers low-latency stable inference with up to \textbf{4.56 $\times$ } higher throughput, and consistently outperforms other strong baselines. Code is available at https://anonymous.4open.science/r/CHESS/.

Figures & tables

Figure 1: Context-agnostic vs. context-aware KV selection. Red indicates critical tokens, while grey denotes ignored tokens. (a) Context-agnostic ( e.g. , SnapKV) : Preserve the most important tokens based on attention scores. (b) Context-aware (Ours) : Adaptively select segments that are semantically relevant to the current generation and retain local context.

Figure 2: Escalating latency and dominant attention computation in long-context decoding.

Figure 3: Overview of the CHESS System Architecture. The system maintains a hierarchical view (Grid, Chunk, Page) over the physical KV cache to enable context-aware selection. The final context is reconstructed by combining semantically selected pages with attention sinks and the local query window.

Figure 4: Distribution of Average Entropy vs. Varentropy per KV Cache Page (Calibration Phase). This plot illustrates the density of pages from the calibration dataset, where warmer colors indicate higher concentration. Dashed lines represent the selected 99th percentile thresholds. The shaded upper-right region highlights the pruned area , corresponding to high-uncertainty outliers excluded by our method.

Method	KV Cache	Overall	Difficulty		Length( $<$ 32K;32K $\sim$ 128K; $>$ 128K)
Method	Budget	Overall	Easy	Hard	Short	Medium	Long
FullKV Inference	100%	30.2	33.9	28.0	38.3	24.2	28.7
H2O (Best)	20%	34.0	40.1	30.2	41.7	28.8	31.5
KeyDiff (Best)	8192 toks	29.2	33.3	26.7	30.6	26.5	32.4
SnapKV (Best)	4096 toks	30.2	34.9	27.3	34.4	24.2	35.2
Quest (Best)	2048 toks	32.0	34.9	30.2	40.0	25.6	31.5

Table 1: Main results on LongBench-v2. We report overall performance as well as breakdowns by difficulty (Easy, Hard) and context length (Short, Medium, Long). We compare CHESS against the best configurations of baselines: H2O (10% heavy + 10% local), KeyDiff (overall budget 8192 tokens), SnapKV (recent 512 tokens, total 4096 tokens), and Quest (budget 2048). For CHESS , we report three settings: Conservative (0.9, 0.9, 0.9), Moderate (0.8, 0.7, 0.7), and Aggressive (0.5, 0.2, 0.1).

Figure 5: Normalized accuracy on long-context tasks. Axes represent relative performance scaled to the maximum score per domain. Abbreviations: L-ICL : Long In-context Learning; Struct : Structured Data; Dialog : Dialogue History; Multi/Single-QA : Multi/Single-Doc QA.

Figure 6: End-to-end Throughput Speedup Comparison. We report the throughput ratios of CHESS and alternative sparse attention methods relative to the Full-KV baseline (normalized to

1.0\times

, indicated by the dashed line). The evaluation spans varying input context lengths (

4\text{k}

–

32\text{k}

) and batch sizes (

1

–

192

). CHESS consistently achieves the highest throughput, peaking at a

\mathbf{4.56\times}

speedup in the

32\text{k}

context setting. Gray shaded regions indicate Out-of-Memory (OOM) scenarios. Note that Quest is evaluated only at batch size 1 due to implementation constraints limiting it to single-batch inference.

Figure 7: (a) TPOT Speedup Scalability across Context Lengths (

4\text{k}

–

32\text{k}

). CHESS demonstrates superior scalability, with the performance gap widening significantly as sequence length increases. At the

32\text{k}

context, CHESS achieves a peak speedup of

\mathbf{4.56\times}

over Full KV, outperforming the strongest baseline (SnapKV) by

\mathbf{1.3\times}

. (b) Latency Stability during Long-Sequence Generation. The plot tracks the latency increase over

6\text{k}

generated tokens (following a

32\text{k}

input). While baselines exhibit linear growth (Full KV) or severe instability (SnapKV/KeyDiff), CHESS maintains a consistent, flat latency profile with negligible overhead.

Figure 8: Latency breakdown per decoding step. Comparison between Full KV and CHESS (Aggressive) with batch size 1. The ”Other” category includes scheduling, sampling, and data preparation. The overhead from the CHESS’s selection mechanism is negligible relative to the total decoding latency.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Selection Metric	Granularity	Ctx. Aware	Dyn. Recon.	Self-Correct
StreamingLLM ( Xiao et al., 2024b )	Position	Token	–	–	–
H2O ( Zhang et al., 2023 )	Attn Score	Token	–	✓	–
RocketKV ( Behnam et al., 2025 )	Attn Score	Token	–	✓	–
KeyDiff ( Park et al., 2025 )	Key Matrix	Token	–	–	–
SnapKV ( Li et al., 2024 )	Attn Score	Token	–	–	–
ChunkKV ( Liu et al., 2025 )	Attn Score	Block	–	–	–

Appendix

Table 2: Comparison with related works. Metric : The criterion used to identify important tokens; Granularity : The processing unit (Token vs. Block); Ctx. Aware : Whether selected tokens are semantically relevant to the current generation context; Dyn. Recon. : Whether the method dynamically reconstructs the KV cache context during generation; Self-Correct : Whether the method employs a mechanism to recover from retrieval errors.

Figure 9: Comparison between Fixed Window and Dynamic Construction on LongBenchV2. The trend highlights the phases of Distraction, Recovery, and Dilution.

Figure 10: End-to-end throughput speedup on a multi-GPU system (4

\times

NVIDIA A800). We report the normalized throughput of CHESS and other representative sparse attention baselines. All results are relative to the Full-KV baseline (normalized to

1.0\times

, represented by the horizontal dashed line). Higher values indicate better efficiency in processing long sequences.

Method	KV Cache	Overall	Difficulty		Length( $<$ 32K;32K $\sim$ 128K; $>$ 128K)
Method	Budget	Overall	Easy	Hard	Short	Medium	Long
FullKV Inference	100%	30.2	33.9	28.0	38.3	24.2	28.7
H2O	20%	34.0	40.1	30.2	41.7	28.8	31.5
H2O	40%	33.0	38.5	29.6	42.2	27.4	28.7
H2O	60%	31.4	34.9	29.3	41.7	23.7	29.6
KeyDiff	1024 toks	22.9	25.5	21.2	28.9	16.7	25.0

Appendix

Table 3: Evaluation of CHESS and baselines under diverse dynamic ratio configurations.

Long-Context Language Model Inference

Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.

See all 227 papers

KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache Offloading KV Caching
Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference Efficiency KV Caching
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
Mar 9, 2026Dongfang Li, Zixuan Liu, Gang Lin +2KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology · University of Electronic Science and Technology of China

May 18, 2026J/K move · Enter open · S save

KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

Jian Lin, Jiazhi Mi, Zicong Hong, Haodong Wang+4

Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China

KV-Cache Offloading KV Caching KV-Cache Management LLM Inference Acceleration Long-Context LLM Inference

Open paper AskPDF arXiv

Supporting long-context LLMs is challenging due to the substantial memory demands of the key-value (KV) cache. Existing offloading systems store the full cache in host memory and selectively fetch critical entries during decoding, but this strategy quickly hits a ceiling: sparsity cannot be pushed further without degrading accuracy. As a result, when context length and batch size grow, the volume of KV transfers rises sharply and becomes the dominant source of decoding latency. We present KVDrive, a holistic multi-tier KV cache management system spanning GPU memory, host DRAM, and SSD. Unlike prior work that pursues greater sparsity through algorithmic refinements, KVDrive tackles the problem from a systems perspective - jointly orchestrating cache placement, pipeline scheduling, and cross-tier coordination to sustain high-throughput inference under tight GPU budgets. KVDrive advances three fundamental capabilities: it adapts cache management to attention behavior to maximize reuse and minimize redundant data movement; it restructures the decoding pipeline to overlap I/O- and CPU/GPU compute-bound stages, eliminating stalls across heterogeneous resources; and it harmonizes data movement across memory tiers to unlock scalable long-context inference far beyond GPU and DRAM limits. We have implemented a fully functional prototype of KVDrive and evaluated it on long-context benchmarks with popular LLMs. The system achieves up to 1.74x higher throughput compared to state-of-the-art works while preserving accuracy.

Abstract

Long-Context Language Model Inference

Explore similar work

KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing