Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

Naji Alazhar; Wassel Alazhar

cs.AIOct 9, 2026

Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

Authors: Naji Alazhar, Wassel Alazhar

Organizations: Umans AI

Abstract

Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45 $\times$ HiSparse and 1.27 $\times$ non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4 $\times$ the concurrency and 2.3 to 3.7 $\times$ the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85 $\times$ the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.

Figures & tables

Figure 1: GLM-5.3 at 32K input and 8K output cap, using the input and output lengths of the HiSparse evaluation. (a) Generation throughput against offered concurrency, (b) mean TTFT on a logarithmic axis and (c) mean TPOT against generation throughput.

Figure 2: One decode step and where its KV lives. The indexer reads index state and emits the top-

k

positions, colored here by where each row is found. The resolver first takes rows that are in the tail or in optional history (1), then scans the hot cache (2). A row found in none of them is copied from host DRAM into the hot cache (3) before attention reads the selected rows. Optional history holds imported history in reclaimable capacity of the KV pools, drawn schematically as one divider that moves with load. The hot cache and the tail are kept per request and KV source and are never lent. Host backup can be deferred while HBM is spare: a row without host backing stays protected on the GPU, so every missing row has host backing. A lookup skips steps 2 and 3 only when every selected row resolves in step 1. Row positions are illustrative.

Figure 3: One speculative round over the KV hierarchy. The drafter, conditioned on target features, extends the anchor D into a block, and the target model verifies its

K

queries in one pass. Here E and F are accepted, G is rejected, and the correction G ∗ opens the next round. (1) Draft KV is held in a ring bounded by the drafter’s window. (2) For each request and KV source, the resolver handles the top-

k

selections of the

K

queries together, so each distinct missing row is copied from host once (a is selected by three queries, b by two), and each query still attends over its own selection. (3) Rows that do not fit the hot cache go to a staging arena, reused across layer groups after its readers finish and promoted rows are copied back. (4) The target writes the rows of all

K

queries to the tail. Acceptance commits the computed rows of the anchor and the accepted drafts; the rejected row is dropped and G ∗ has no row until the next forward. Host backup of the committed rows follows verification. Sizes are illustrative (

K=4

k=3

Method	32K	100K	100K
	GPU, required		Host backup
Resident	1.456	4.444	–
HiSparse	0.259	0.432	4.186
Adaptive HiSparse	0.259 †	0.432 †	4.186
Resident + DFlash2	1.481	4.469	–
Adaptive DHiSparse	0.319 †	0.492 †	4.186

Table 1: Memory per session for GLM-5.3, in GiB. Required GPU memory at 32K and 100K context, and the full host backup at 100K. Model weights and shared workspaces are excluded.

\dagger

excludes optional residency, which only uses HBM that is spare.

Figure 4: GLM-5.3 serving at 100K context. (a) Output throughput against offered concurrency, (b) mean TTFT and (c) mean TPOT against output throughput. Color gives the KV policy and filled markers add DFlash2. TTFT in (b) is on a logarithmic axis. Adaptive DHiSparse uses static blocks of 8 at C4 to C32, 6 at C64 and 4 at C128 and C256.

			Against	TTFT (s)			TPOT	Active
$C$	Method	Output	HiSparse	mean	post-wave	post-wave p99	(ms)	/ slots
128	HiSparse	548	–	10.68	5.11	26.97	50.3	121.1 / 208
	Adaptive HiSparse	622	1.14 $\times$	6.13	2.48	11.85	46.6	125.6 / 208
	Adaptive DHiSparse	793	1.45 $\times$	5.13	2.25	9.67	35.9	124.7 / 140
256	HiSparse	730	–	47.61	42.54	102.89	58.0	190.5 / 208
	Adaptive HiSparse	823	1.13 $\times$	34.36	31.01	81.99	54.4	201.5 / 208

Table 2: GLM-5.3 at 100K context, offered concurrency 128 and 256. Output is in tokens/s per GPU. Post-wave latency excludes the first

C

submitted requests. Active is the mean number of requests decoding at once over the central half of a run.

Figure 5: DeepSeek-V4.1-Flash serving at 500K context. Panels and encoding as in Figure 4 ; filled markers add DSpark. HiSparse is our optimized port to DeepSeek-V4.1-Flash.

Figure 6: Residency under a load step. GLM-5.3 at 100K context, without speculation: the same build with optional residency on and off, offered C16, then C256, then C16 for five minutes each; in-flight requests drain when the load falls. (a) Allocated request slots. (b) Coverage: ready optional history over eligible history. (c) Share of selected rows fetched from host. One-second counters shown in five-second bins. Per-phase request statistics are in Table A1 .

Figure 7: Block size for DFlash2 at 100K context. (a) Emitted tokens per speculative round and (b) output throughput for static blocks of 4, 6 and 8 target queries at three offered loads.

Figure 8: What resolution costs. (a) GLM-5.3, verification width 8, batch 32 per GPU,

B=4{,}096

: resolver time against the unique host-miss payload of a call, for synthetic shared, partial and disjoint selections across the eight queries, with a cold or a retained cache. The dashed line is the least-squares fit, 49.4 GB/s (

R^{2}=0.9992

). One query’s top-

k

over the 32 requests of a call is 37.75 MB, so disjoint selections fetch eight such payloads and shared selections one. (b) DeepSeek-V4.1-Flash at 500K, single-query resolver with no host miss: time against hot-cache size for two batch sizes. The dotted line is the direct path at batch 8. (c) The same resolver at

B=4{,}096

with and without 77 host misses per request. Error bars span the minimum and maximum of nine timings in (a) and the 10th to 90th percentiles in (b) and (c). These are isolated kernel measurements with synthetic selections and real payload copies.

Figure 9: Output and host fetches against the hot-cache budget. (a–d) Output per decode GPU and (e–h) share of selected rows fetched from host, for HiSparse and Adaptive HiSparse as

B/k

varies, without speculation. GLM-5.3 runs at 100K context on one prepared prefix, and DeepSeek-V4.1-Flash at 500K on 16 prepared prefixes.

	GLM-5.3	DeepSeek
Selecting layers $\times$ $k$	21 $\times$ 2,048	8 $\times$ 512
Selections per token	43,008	4,096
KV sources	78, one per layer	4, shared
Bytes per missed selection	2.4 to 2.5 KB	288 B
Host bytes per token	34 to 4.2 MB	0.42 to 0.11 MB
Hot cache per session	184 MB	6 MB

Table 3: Retrieval volume of the two models. Per generated token and request. Ranges run from the smallest to the largest budget of the sweep, for HiSparse at C128. The hot cache is that of Adaptive HiSparse at the budgets of Section 4.1 .

Figure 10: What the cache costs inside a step, Adaptive HiSparse at C128. (a) Summed duration of the cache kernels per GPU step as

B/k

varies, on one scale for both models: the fused resolver and, on GLM-5.3, the copies that load selected rows into the follower payload layers. A GLM-5.3 step holds 24 to 28 sessions per GPU up to

B/k=4

and 13 at

B/k=8

; a DeepSeek-V4.1-Flash step holds 62 to 63. (b) Share of step time by kernel family at the budget used in the paper (

B/k=2

for GLM-5.3, 16 for DeepSeek-V4.1-Flash), each family counted where it runs alone. “Several families at once” is time in which kernels of more than one family run; “no kernel running” is time between kernels, 1.5% and 1.9% of the step.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,090 papers

MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding
Oct 3, 2026Dich Nhat Minh Nguyen, Tran Dang Duong NguyenLanguage Model Decoding LLM Inference Acceleration
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM Serving KV-Cache Offloading
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache Offloading Long-Context Language Model Inference
Huazhong University of Science and Technology Wuhan, China · UCloud Technology Co., Ltd. Shanghai, China

Oct 3, 2026J/K move · Enter open · S save

MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding

Dich Nhat Minh Nguyen, Tran Dang Duong Nguyen

Language Model Decoding LLM Inference Acceleration Sparse Attention

Open paper AskPDF arXiv

Long-context decoding is limited by memory bandwidth, because every output token reads the KV cache of every layer. Sparse decoding reduces this cost by reading only part of the KV cache. We observe that the number of pages a query needs varies widely across KV heads, layers and steps. Fixed budgets are simple, but they are sized for demanding cases and tuned per workload; adaptive budgets follow this variation more flexibly, but existing designs pay for it with extra selection cost or training. At the kernel level, FlashAttention-3 (FA3) and FlashInfer are designed for rows of similar length: with page lists whose length differs per KV head, they either pad the lists (forfeiting much of the sparse saving), leave thread blocks unbalanced, or rely on a host-side plan that runs outside the CUDA graph. We propose MOIRA, a training-free sparse decode path in vLLM whose budget adapts per KV head and per layer. For every request, layer, KV head and step, a coverage rule keeps the smallest set of pages whose estimated attention mass reaches a fraction

γ

. A new kernel, self-planning attention, lets each thread block derive its own share of the work from the list lengths, so the whole decode step stays inside the CUDA graph. On an H200, at RULER's 128k context, MOIRA with

γ=0.99

matches dense accuracy while reading about 30% of the pages and reduces the time per output token (TPOT) by 2.2-2.5

\times

relative to dense FA3; with

γ=0.98

it reduces TPOT by 2.7

\times

and stays within the noise of dense. Under high serving load it raises throughput by up to 51%. These results suggest that a budget adapted per head and layer, paired with a kernel that keeps such budgets inside the CUDA graph, makes sparse decoding both flexible and fast.

Abstract

LLM Inference Acceleration

Explore similar work

MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding