SparseEngine: Sparse-First Inference Engine

Jitai Hao; Quansheng Gu; Qiang Huang; Jun Yu

cs.LGSep 30, 2026

SparseEngine: Sparse-First Inference Engine

Authors: Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu

Abstract

Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.

Figures & tables

Figure 1 : Method coverage across sparse inference engines. Contours enclose representative implementations.

Figure 2 : Overview of SparseEngine. From top to bottom, the figure shows the LLM generation loop; the Transformer execution path and lifecycle hooks invoked during prefill and decoding; the interactions among SparseController , the method-specific CacheManager , AttentionView , and attention execution; and representative implementations of SnapKV and Quest.

Figure 3 : Radix prefix caching with Quest and Chain Cache with SnapKV. Conventional radix prefix caching retains complete KV blocks and attaches the summaries required by Quest. After SnapKV evicts physical KV entries, Chain Cache preserves the logical token prefix and reuses the retained KV and method state, so a continuation prefills only the new suffix.

Figure 4 : Controllable prefix-cache pruning. Within an application-specified history interval, a sparse policy selects retained KV entries; SparseEngine releases the rest while preserving the logical prefix for matching.

Figure 5 : Decode throughput on Qwen3-30B-A3B (left) and GLM-4.7-Flash (right). The upper plot shows throughput improvement over vanilla vLLM at matched batch sizes with 128K-token inputs, with vanilla vLLM defining the 0% baseline. The middle and lower plots show absolute throughput at each system–method pair’s largest successfully tested batch size with 128K- and 32K-token inputs, respectively. Bar annotations report throughput and batch size (

B

); the largest tested batch size need not be the system’s maximum capacity. Vortex encounters OOM on GLM with 128K-token inputs even at batch size 1. HiSparse and Tangram do not support the evaluated MLA.

	LongBenchV1							LongBenchV2
Method	S-Doc	M-Doc	Summ.	F-Shot	Synth.	Code	Avg.	S-Doc	M-Doc	Hist.	Learn.	Struct.	Code	Avg.
GLM-4.7-Flash $\dagger$	41.6	47.1	27.8	71.0	52.5	65.7	51.0	34.3	28.8	20.5	29.6	21.2	46.0	31.4
SnapKV	41.0	47.2	26.9	69.8	52.5	65.2	50.4	36.0	31.2	20.5	32.1	30.3	42.0	33.2
OmniKV	41.7	47.1	27.8	70.6	52.5	65.1	50.8	35.4	28.0	23.1	33.3	42.4	42.0	33.4
Quest	41.4	47.2	27.7	71.2	52.5	65.4	50.9	34.9	29.6	23.1	40.7	27.3	42.0	33.8
StreamingLLM	34.7	41.7	26.4	69.2	53.0	64.1	48.2	34.3	32.0	23.1	34.6	30.3	32.0	32.4

Table 1: Sparse-method quality on LongBench V1 and V2. Deltas below scores report SparseEngine minus the reference implementation in percentage points. Following official evaluation metrics, average score is the six-category average for V1 and accuracy over all examples for V2.

Method	Acc.	E2E Speedup
Qwen3-4B	81.7	$1.00\times$
OmniKV	80.0	$1.51\times$
Quest	76.7	$1.28\times$
Quest ( Vortex )	75.0	$0.91\times$
SnapKV ( Low )	58.3	$1.66\times$
SnapKV ( Mid )	80.0	$1.33\times$

Table 2: AIME 2024 results. Acc. (%) averages 60 responses (two per problem). Speedup is relative to Qwen3-4B-Thinking.

Table 8

Method	Resolved (%)
Vanilla	23.7
Quest	21.3
OmniKV	22.3
OmniKV ( Lag=4 )	25.0

Table 5: SWE-bench Lite resolved tasks (%) with mini-SWE-agent and tool-result prefix pruning. All runs use 300 tasks; settings are in Appendix C.4 .

SWE-lite: GLM-4.7-Flash
Method	Approx.	Time	E2E	E2E Out.
	Max BS	(min)	Speedup	tok/s
Vanilla	40	98.1	$1.00\times$	302.0
Quest	32	92.2	$1.06\times$	321.3
OmniKV	40	62.4	$1.57\times$	474.5
SnapKV ( High )	52	82.0	$1.20\times$	361.2

Table 6: End-to-end agent trace replay performance. Speedup is relative to each workload’s Vanilla at the listed batch sizes. Gasai measures system efficiency only.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\ell,a$	Transformer layer and query-head indices.
$n,i,j$	Prompt length, current query position, and a key position.
$[i]$	The causal history $\{1,\cdots,i\}$ , including the current token.
$q_{\ell,i}^{(a)}$	Query vector for head $a$ at position $i$ .
$k_{\ell,j}^{(a)},v_{\ell,j}^{(a)}$	Logical key and value used by query head $a$ at position $j$ .
$z_{i,j},\alpha_{i,j}$	Query–key logit and normalized attention weight.

Appendix

Table 7: Notation for the background equations. Layer and head indices are omitted when discussing a single layer and head.

Method	Sink	Recent	Method budget	Other settings
SnapKV	0	32	2,016 selected	32 scoring window; pooling kernel 7
Quest	–	–	2,048 selected	16-token pages; first two layers dense
OmniKV	0	32	2,048 selected	Full layers: GLM $\{0,3,8,16,19,25,31,37,44\}$ ; Llama $\{0,2,7,13,16,26\}$
H 2 O	–	–	2,048 prefill, 2,048 decode	Prefill chunk eviction; recent ratio 0.5; requested prefill scoring window 128
StreamingLLM (GLM)	8	4,096	–	–
PyramidKV (GLM)	64	512	4,096 selected	Layer ratios $0.6$ to $0.01$ (linear); 32 scoring window

Appendix

Table 8: Method-specific settings for LongBench results in Table 1 . Budgets and windows are in tokens. The method-budget column retains each method’s budget definition.

Method	Sink	Recent	Method budget	Other settings
SnapKV	0/64	256	4,096 selected	32 scoring window; pooling kernel 7
Quest	64	256	4,096 selected (4,416 total)	16-token pages; first two layers dense
OmniKV	64	256	4,096 selected	Full-attention layers: GLM $\{0,3,8,16,19,25,31,38,44\}$ ; Llama $\{0,2,7,13,16,26\}$
H 2 O	–	–	8,192 prefill; 4,096 configured decode	128 prefill scoring window; recent ratio 0.5; decode eviction disabled
StreamingLLM (GLM)	8	4,096	–	–
PyramidKV (GLM)	64	512	4,096 selected	Layer ratios $0.6$ to $0.01$ (linear); 32 scoring window

Appendix

Table 9: Method-specific settings for LongBenchV2 results in Table 1 . Budgets and windows are in tokens. The method-budget column retains each method’s budget definition. SnapKV sink values are GLM/Llama.

Method	Sink	Recent	Method budget	Other settings
OmniKV	16	64	2,048 selected	Model-profile full-attention layers
Quest (SparseEngine)	16	64	2,992 selected (3,072 total)	16-token pages; first two layers dense
Quest (Vortex)	16	64	2,048 selected (128 pages; 2,128 total)	First two layers dense
SnapKV	16	64	4,096 selected (4,176 total)	16 observation window; probability-based prefill scores; no full-attention layers; eviction every 1,024 decode steps
H 2 O	–	–	4,096 prefill, 4,096 decode	128 prefill scoring window; recent ratio 0.2; probability-based prefill scores; eviction every 1,024 decode steps
PyramidKV	16	64	7,987 configured selected	32 observation window; layer ratios $0.6$ to $0.0154712$ from layer 0; eviction every 1,024 decode steps

Appendix

Table 10: Method budgets for the AIME 2024 results in Table 2 . Budgets and windows are in tokens. Selected budgets exclude separately listed sink and recent tokens.

Method	Sink	Recent	Method budget	Prefix cache	Other settings
GLM-4.7-Flash
SnapKV	64	512	15,808 selected	Chain	32 scoring window; probability-based prefill scores; no full-attention layers
H 2 O	–	–	16,384 prefill; 8,192 decode	Chain	128 prefill scoring window; recent ratio 0.5; FP32 logit scoring; decode eviction disabled
Quest	64	512	1,472 selected (2,048 total)	Radix	16-token pages; first two layers dense
OmniKV	64	512	1,472 selected (2,048 total)	Radix	Model-profile full-attention layers
Qwen3-30B-A3B

Appendix

Table 11: Cache settings for the agent-quality results in Table 4 . Budgets are in tokens and retain each method’s configured meaning.

Figure 6 : Absolute decode throughput. Under 128K-token inputs and 2K-token outputs on Qwen3-30B-A3B (left) and GLM-4.7-Flash (right). Ours denotes SparseEngine. Available external baselines are retained; Vortex encounters OOM on GLM at batch size 1, and HiSparse and Tangram do not support the evaluated GLM MLA configuration.

Figure 7 : Absolute decode throughput. Under 32K-token inputs and 2K-token outputs on Qwen3-30B-A3B (left) and GLM-4.7-Flash (right). Ours denotes SparseEngine. HiSparse and Tangram are omitted for GLM because they do not support the evaluated MLA configuration.

Figure 8 : Decode throughput improvement over vanilla vLLM at matched batch sizes with 32K-token inputs and 2K-token outputs. Ours denotes SparseEngine; vanilla vLLM defines the 0% baseline. HiSparse and Tangram do not support the evaluated GLM MLA configuration and are omitted for that model. Hardware, hyperparameters, and the measurement protocol are specified in Appendix C.7 .

Parameter	SnapKV	Quest	OmniKV
sink_keep_tokens	64	64	64
recent_keep_tokens	512	512	512
decode_keep_tokens	7,616	1,472	1,472
full_attention_layers	–	–	auto

Appendix

Table 12: SparseEngine hyperparameters for Figure 5 . Token-budget entries are in tokens.

No.	Model	Attention Architecture	FFN Architecture
1	Qwen2.5 [ 40 ]	GQA	Dense
2	Qwen3 Dense [ 39 ]	GQA	Dense
3	Qwen3 MoE [ 39 ]	GQA	MoE
4	Qwen3.5 Dense	Hybrid linear attention	Dense
5	Qwen3.5 MoE	Hybrid linear attention	MoE
6	Qwen3.6 Dense	Hybrid linear attention	Dense

Appendix

Table 13: Supported models and their attention and FFN architectures. GQA denotes grouped-query attention, MLA denotes multi-head latent attention, and MoE denotes mixture of experts. Hybrid linear attention combines GQA with Gated DeltaNet layers.

No.	Method	Sparsity Category
1	Quest [ 32 ]	Dynamic Sparsity
2	OmniKV [ 18 ]	Dynamic Sparsity
3	RetroInfer [ 10 ]	Dynamic Sparsity
4	FlashPrefill-v2 [ 14 ]	Dynamic Sparsity
5	StreamingLLM [ 37 ]	KV Eviction
6	SnapKV [ 24 ]	KV Eviction

Appendix

Table 14: Supported sparse methods grouped into four categories. The inventory includes 15 cache methods and FlashPrefill-v2 for sparse prefill. Each method is assigned to one category according to its primary mechanism.

	Query-dependent retrieval		Physical KV eviction			Compressed KV representations
System	Quest	RetroInfer	SnapKV	Ada-SnapKV	H 2 O	Palu	LoRC	KIVI
SparseEngine (Ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
HiSparse	$\checkmark$	$\triangle$	$\triangle$	$\triangle$	$\triangle$	–	–	–
Tangram	–	–	$\checkmark$	$\checkmark$	$\triangle$	–	–	–
Vortex	$\checkmark$	$\triangle$	$\triangle$	$\triangle$	$\triangle$	–	–	–
SPIN	$\checkmark$	$\checkmark$	$\triangle$	$\triangle$	$\triangle$	–	–	–

Appendix

Table 15: Abstraction-level compatibility with representative sparse methods.

\checkmark

: the method’s defining computation and KV-state semantics fit the documented extension boundary;

\triangle

: only part of the method fits; –: a mapping is not established by that contract. These are design assessments, not implementation or performance results. New method modules are allowed; changes to system-owned storage/transport contracts are not. Physical eviction must release KV storage, rather than merely mask reads or offload entries. Appendix D gives the mapping criteria and evidence.

Explore similar work

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8Dynamic Sparse Attention Key-Value Cache
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents
Jun 4, 2026Zhuoming Chen, Xinrui Zhong, Qilong Feng +5Dynamic Sparse Attention Large Language Model Serving
1Carnegie Mellon University · 2Rice University · 3National University of Singapore
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3LLM Inference Optimization Dynamic Sparse Attention
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China

Apr 29, 2026, cs.LGJ/K move · Enter open · S save

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

Zihan Zhao, Baotong Lu, Shengjie Lin, Yizou Chen+7

University of Virginia · Microsoft Research · Georgia Institute of Technology+1

Dynamic Sparse Attention Key-Value Cache

Open paper AskPDF arXiv

Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step and extending the KV storage to CPU memory. In practice, however, these algorithmic savings rarely translate into end-to-end system-level gains because sparse methods typically operate at different granularities and thus rely on ad hoc, per-algorithm implementations. At the same time, hierarchical KV storage introduces a new systems bottleneck: retrieving fine-grained, irregular KV subsets across the GPU-CPU boundary can easily erase the benefits of sparsity. We present SPIN, a sparse-attention-aware inference framework that co-designs the execution pipeline with hierarchical KV storage through three techniques: (1) a unified partition abstraction that maps different sparsity granularities onto a shared page-based KV substrate; (2) a locality-aware KV cache manager that dynamically sizes per-request HBM budgets and uses a GPU-friendly bucketed LRU policy to cut PCIe round-trips; and (3) a two-level hierarchical metadata layout sized to the active working set rather than the worst-case address space. Built on vLLM with three representative sparse attention algorithms, SPIN delivers 1.66-5.66x higher end-to-end throughput and 7-9x lower TTFT than vLLM, and reduces TPOT by up to 58% over the original sparse-attention implementations.

Abstract

Explore similar work

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference