AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill

Liquan Liu; Yifan Zhang; Bowei Xu

cs.LGOct 5, 2026

AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill

Authors: Liquan Liu, Yifan Zhang, Bowei Xu

Organizations: Zeraix

Abstract

Block drafters such as DSpark propose ranked candidates for several positions in one forward pass, and a tree verifier checks them in one pass of the target. The number of rows to verify trades the tokens a wider tree is expected to accept against the time a wider verify takes. Most schedulers that choose this number take the verify time from a table or model measured before serving, corrected online by at most one scale factor, and take acceptance from the drafter's confidence estimates or from a map fitted offline. AdaSpark learns both quantities while it serves, with no profile, calibration or sweep in advance. It learns which verify widths are worth offering and fits each one's verify time as a function of context. It fits each candidate's acceptance probability to the target's verify outcomes, with the drafter's confidence head as one input, and orders and sizes the tree by that fit instead of by the head. The same model prices n-gram continuations of the request's own text, so drafted and text-derived candidates compete for rows in one best-first order. The width is chosen by pricing time at the long-run decode rate. On single- and multi-turn conversations from six public datasets, on three dense targets and one mixture-of-experts target, AdaSpark decodes 1.5-3.1x faster than llama.cpp's DSpark with the same drafters. Our imparo engine with AdaSpark is 1.17-1.52x faster than imparo running with a three-token chain (the default llama.cpp setting); this gain comes from the scheduler alone. Without a width sweep, AdaSpark is never more than 0.3% slower than the best pinned tree width on any dense target or context band. On the mixture-of-experts target it ties the best pinned width, and the other pinned widths from 4 to 16 rows are 5-14% slower.

Figures & tables

Figure 1: Shape of the candidate tree, drawn for a block of

B=4

positions and

K=3

candidates per position. Dark blue nodes are the drafter’s picks, light blue nodes its siblings, orange squares an n-gram chain. The targets here use

B=7

or 9 and

K=8

; a verify takes the best-first prefix of these nodes (§ 3.1 ).

Figure 2: One AdaSpark round. Solid arrows are the forward path; dashed arrows are the feedback that trains the two online models and extends the n-gram index. Both models persist across requests. The acceptance model’s per-request intercept, and the per-request offset of the head-based estimate it competes with, restart with each request.

Figure 3: Verify time against the number of rows verified, on each target, at two context lengths (median over two passes of the per-run median over rounds that verified exactly

n

rows). Each panel has its own vertical scale. T(16)/T(2) at 1,596 tokens: LFM2.5-2.6B 1.11, LFM2.5-8B-A1B 2.16, Qwen3-4B 1.57, Qwen3-8B 1.53.

target	rows	$T$ (443)	$T$ (1,596)	$T$ (8,444)	$\alpha$ (ms)	$\beta$ (ms per 1k tokens)	max residual
LFM2.5-2.6B	8	24.0	24.1	25.0	23.9	0.13	0.1%
LFM2.5-2.6B	16	26.0	25.9	27.5	25.8	0.20	0.7%
LFM2.5-2.6B	32	43.9	44.4	47.3	43.7	0.43	0.0%
LFM2.5-8B-A1B	8	24.2	23.0	23.7	23.6	$-$ 0.01	2.7%
LFM2.5-8B-A1B	16	32.3	30.3	32.4	31.3	0.11	3.9%
LFM2.5-8B-A1B	32	54.8	54.4	57.3	54.3	0.35	0.8%

Table 1: Verify time (ms) at three contexts for three row counts on each target, and the fit

T(c)=\alpha+\beta\cdot c

of Eq. ( 3 ).

target	requests	nodes	head-based	fitted	change	requests won	priced	first request
LFM2.5-2.6B	91	580,495	0.1410	0.1238	$-$ 12.2%	91/91	100%	157/436
LFM2.5-8B-A1B	94	592,660	0.1520	0.1360	$-$ 10.5%	94/94	100%	322/644
Qwen3-4B	87	567,698	0.1644	0.1400	$-$ 14.8%	87/87	100%	779/838
Qwen3-8B	93	634,175	0.1572	0.1336	$-$ 15.0%	93/93	100%	749/864

Table 2: Acceptance prediction on the evaluation subset: log loss per labelled node of the head-based estimate (the drafter’s confidence head with its per-request offset) and of the fitted model, both scored on each round’s labels before either learned from them, over every request after the first that did not reach the token cap. priced is the share of rounds whose tree the fitted model priced; the last column gives rounds priced over tree rounds for the first request, which starts from an empty store.

Figure 4: The acceptance model while serving the evaluation subset from an empty store (looped replies left out): for each request, how much lower the fitted model’s log loss per labelled node is than that of the head-based estimate it competes with (1

-

fitted / head-based), both scored before either learns from the request’s labels. A bar above the dashed zero line (blue) is a request the fitted model predicted better. Shaded bands mark the six sources in serving order (SB Spec-Bench, SP SPEED-Bench, WC WildChat, CF ConvFinQA, TA ToolACE, LB LongBench). The fitted model is lower on 91 of 91, 94 of 94, 87 of 87, 93 of 93 requests after the first, in the order of the panels. At the 20 changes of source (five on each target), the first request of the new source ranks 0.47 on average among that source’s other requests by this reduction (0 lowest, 1 highest) and below their median at 8 of them.

target	replies	n	4	5	6	8	12	16	AdaSpark / best	again
LFM2.5-2.6B	all replies	92	–	–	–	0.907	0.968	0.998	1.002 $\times$ [0.996, 1.008]	0.995
	under 2,048	75	–	–	–	0.909	0.970	1.000	1.000 $\times$ [0.995, 1.007]	0.995
	2,048 and over	17	–	–	–	0.899	0.962	0.990	1.010 $\times$ [0.991, 1.025]	0.995
LFM2.5-8B-A1B	all replies	91	0.857	0.920	0.953	0.997	0.926	0.900	1.003 $\times$ [0.989, 1.019]	0.995
	under 2,048	74	0.847	0.910	0.947	0.996	0.930	0.902	1.004 $\times$ [0.990, 1.020]	1.002
	2,048 and over	17	0.899	0.969	0.980	0.998	0.911	0.891	1.002 $\times$ [0.950, 1.042]	0.964

Table 3: The width on the evaluation subset: AdaSpark against the same scheduler with its width pinned, every other part unchanged (both online models, the n-gram fill, a store learning from empty). The width columns are each pinned arm’s decode speed over AdaSpark’s, geometric mean of per-reply ratios (above 1 is faster than AdaSpark), with the best pinned width in bold, for all replies and by the context a reply was decoded at (prompt plus half the reply). AdaSpark runs before the pinned arms and again after them, and its speed on a reply is the mean of the two runs. The routed target is also pinned at 4, 5 and 6 rows, in a second stage run the same way and compared with its own two AdaSpark runs; a dash marks a width not run (§ 3.4 ). AdaSpark / best is AdaSpark over the best pinned width with its 95% bootstrap interval over conversations; again is AdaSpark’s second run over its first. Looped and single-delta replies are left out as in Table 4 .

Figure 5: What the second eight rows of a tree cost and what they buy. Each point is one target at one context (443, 1,596 or 8,444 tokens) and one of three prompts, from runs of 256 tokens at a fixed width of 8 and of 16 rows (acceptance model on, n-gram candidates off; setup notes): the extra time per round of a 16-row tree over an 8-row tree (the drafter included), against the extra tokens it accepts per round, both relative to the 8-row tree. Above the diagonal the wider tree decodes faster. The diagonal is the break-even line for every target.

	evaluation subset: tok/s and ratio					over own AR
target	llama.cpp DSpark	imparo DSpark	AdaSpark	AdaSpark / llama.cpp	looped	llama.cpp DSpark	AdaSpark
LFM2.5-2.6B	66.9	83.8	126.3	1.87 $\times$ [1.80, 1.95]	3	1.55 $\times$	2.74 $\times$
LFM2.5-8B-A1B	79.1	105.5	122.1	1.54 $\times$ [1.52, 1.57]	2	0.80 $\times$	1.16 $\times$
Qwen3-4B	35.7	79.1	100.4	2.83 $\times$ [2.73, 2.92]	7	0.83 $\times$	2.33 $\times$
Qwen3-8B	21.7	50.9	68.2	3.11 $\times$ [3.05, 3.18]	1	0.85 $\times$	2.64 $\times$

Table 4: End to end. Left: decode tok/s on the evaluation subset (Appendix A ; generated tokens over decode time, summed over replies) and AdaSpark over llama.cpp DSpark (geometric mean of per-reply ratios, 95% bootstrap interval over conversations). A reply that reached the 8,192-token cap is a repetition loop under greedy decoding: each tok/s column leaves out its own arm’s looped replies and each ratio the replies that looped in either of its two arms; looped counts the replies that looped in any arm. AdaSpark’s speed on a reply is the mean of its two runs (§ 3.4 ). A reply streamed as a single delta (a bare tool call) has no client-clock decode interval and is left out as well (none on LFM2.5-2.6B, 2 on LFM2.5-8B-A1B, none on Qwen3-4B, none on Qwen3-8B). Right: each engine’s speculation speedup over its own autoregressive (AR) decode: its tok/s on the evaluation subset over its AR tok/s, which is measured on the short subset (setup notes) only (llama.cpp and imparo: LFM2.5-2.6B 43.2 and 46.1; LFM2.5-8B-A1B 99.0 and 105.0; Qwen3-4B 42.9 and 43.2; Qwen3-8B 25.6 and 25.8 tok/s). AR speed barely depends on the text; the short subset’s contexts are shorter, where AR decode is faster, so these speedups are if anything understated.

Figure 6: Where the gain over llama.cpp comes from. The engine factor is imparo DSpark over llama.cpp DSpark, the two engines running the same algorithm (a chain of at most three draft tokens); the scheduler factor is AdaSpark over imparo DSpark, on the same engine; the total is AdaSpark over llama.cpp DSpark, over the replies both engines served; it is close to the product of the two, whose reply sets differ only by replies that looped in one arm. Whiskers are 95% bootstrap intervals over conversations.

Figure 7: Anatomy of an AdaSpark round on each target, at the width its rounds most often ran: the drafter’s forward, the tree build, the verify and the commit (medians), and the tokens committed per round at that width. The text beside each bar also gives the tokens per round and milliseconds per token over all rounds, other widths included, beside the same engine’s autoregressive (AR) milliseconds per token on the short subset. LFM2.5-2.6B: draft 9.6, verify 26.1 ms at 16 rows (90% of rounds); LFM2.5-8B-A1B: draft 7.1, verify 22.8 ms at 8 rows (34% of rounds); Qwen3-4B: draft 8.0, verify 27.1 ms at 8 rows (97% of rounds); Qwen3-8B: draft 11.2, verify 43.2 ms at 8 rows (97% of rounds).

	evaluation subset		held out		whole test set
target	ratio	replies	ratio	replies	ratio	replies	looped, held out
LFM2.5-2.6B	1.87 $\times$ [1.80, 1.95]	92	1.80 $\times$ [1.73, 1.89]	64	1.84 $\times$ [1.79, 1.90]	156	3
LFM2.5-8B-A1B	1.54 $\times$ [1.52, 1.57]	93	1.50 $\times$ [1.46, 1.54]	65	1.52 $\times$ [1.50, 1.55]	158	1
Qwen3-4B	2.83 $\times$ [2.73, 2.92]	88	2.88 $\times$ [2.83, 2.94]	65	2.85 $\times$ [2.79, 2.91]	153	2
Qwen3-8B	3.11 $\times$ [3.05, 3.18]	94	3.07 $\times$ [3.00, 3.14]	67	3.10 $\times$ [3.05, 3.14]	161	0

Table 5: AdaSpark over llama.cpp DSpark on the evaluation subset, on the held-out conversations (the 31 conversations, 61 turns, of the test set outside the subset, which no arm had run when the subset was measured) and on the whole test set (73 conversations, 148 turns): geometric mean of per-reply decode ratios with the 95% bootstrap interval over conversations, and the replies it rests on. Looped and single-delta replies are left out as in Table 4 ; the last column counts the held-out replies that reached the cap.

arm	SB	SP	WC	CF	TA	LB	all replies
LFM2.5-2.6B
llama.cpp DSpark (tok/s)	67.2	66.6	60.5	72.9	64.9	71.8	66.9
imparo DSpark (3-token chain)	1.24	1.23	1.27	1.21	1.20	1.26	1.23 $\times$ [1.21, 1.25]
full-block chain	1.62	1.59	1.60	1.76	1.61	1.71	1.64 $\times$ [1.59, 1.68]
fixed 16-row tree	1.85	1.78	1.79	1.91	1.78	1.91	1.82 $\times$ [1.78, 1.87]
cost-model width	1.83	1.77	1.79	1.91	1.72	1.92	1.81 $\times$ [1.76, 1.86]

Table 6: Ablation on the dense and the routed LFM2.5 targets. The first row of each block is llama.cpp DSpark’s tok/s per source; every other cell is the arm over it, the geometric mean of per-reply decode ratios (looped replies excluded), with the 95% bootstrap interval over conversations in the last column. From the fixed tree down, each row adds to the row above: the cost model’s width, then the fitted acceptance model and n-gram candidates together. Sources: SB Spec-Bench, SP SPEED-Bench, WC WildChat, CF ConvFinQA, TA ToolACE, LB LongBench.

	LFM2.5-2.6B	LFM2.5-8B-A1B	Qwen3-4B	Qwen3-8B
verified rows held by n-gram-only nodes	15.2%	9.4%	7.2%	8.1%
accepted draft tokens from n-gram-only nodes	15.8%	7.8%	5.1%	7.8%
accepted draft tokens from nodes both sources proposed	18.1%	18.5%	22.1%	22.4%
acceptance rate, drafter-only nodes	24%	46%	33%	34%
acceptance rate, nodes both proposed	70%	72%	64%	70%
acceptance rate, n-gram-only nodes	30%	40%	26%	37%

Table 7: What n-gram fill contributes, from the instrumented AdaSpark run on the short subset. Rows are the verified nodes past each round’s anchor; accepted tokens are the draft tokens committed. A round is widened when its n-gram candidates move the chooser to a wider class than it takes with them priced at zero.

source	LFM2.5- 2.6B	LFM2.5- 8B-A1B	Qwen3-4B	Qwen3-8B
Spec-Bench	1.51	1.14	1.28	1.33
SPEED-Bench	1.51	1.19	1.24	1.31
WildChat	1.43	1.12	1.24	1.32
ConvFinQA	1.71	1.24	1.36	1.41
ToolACE	1.41	1.19	1.23	1.24
LongBench	1.73	1.13	1.26	1.28

Table 8: The scheduler’s gain by workload: AdaSpark over imparo DSpark (same engine, a chain of at most three draft tokens), geometric mean of per-reply decode ratios per source, looped replies excluded. Darker cells are larger gains.

Figure 8: Share of rounds at each verified width, per target, from AdaSpark’s rounds on the instrumented run over the short subset (setup notes), cold-start rounds excluded. LFM2.5-2.6B: 7 rows 1%, 8 rows 3%, 16 rows 90%, 32 rows 4%, other widths under 1% (1,610 rounds); LFM2.5-8B-A1B: 2 rows 13%, 3 rows 4%, 4 rows 21%, 5 rows 3%, 6 rows 2%, 7 rows 11%, 8 rows 34%, 10 rows 4%, 12 rows 2%, 16 rows 2%, 32 rows 1%, other widths under 1% (3,066 rounds); Qwen3-4B: 8 rows 97%, 16 rows 2%, other widths under 1% (3,296 rounds); Qwen3-8B: 8 rows 97%, 16 rows 2%, other widths under 1% (4,094 rounds).

target	all replies	under 2,048	2,048 and over	learned widths	kernel-class widths
LFM2.5-2.6B	0.997 $\times$ [0.992, 1.001]	0.995 $\times$ [0.990, 1.001]	1.005 $\times$ [0.995, 1.012]	2–4, 6, 8–10, 12–17, 20, 22, 24, 26, 28, 32, 34, 36, 40, 48, 64	8, 16, 24, 32, 40, 47, 48, 56, 64
LFM2.5-8B-A1B	0.996 $\times$ [0.990, 1.003]	0.994 $\times$ [0.987, 1.002]	1.005 $\times$ [0.992, 1.019]	2–17, 20, 24, 28, 32, 64	8, 16, 24, 32, 40, 48, 56, 64
Qwen3-4B	0.998 $\times$ [0.986, 1.008]	1.003 $\times$ [0.997, 1.010]	0.985 $\times$ [0.947, 1.012]	2–18, 20, 24, 28, 30, 32, 44, 57	8, 16, 24, 32, 40, 48, 56, 64
Qwen3-8B	1.002 $\times$ [0.999, 1.006]	1.003 $\times$ [1.001, 1.006]	0.999 $\times$ [0.989, 1.010]	2–17, 20, 22, 24, 26, 28, 30, 32, 57	8, 16, 24, 32, 40, 48, 56, 64

Table 9: Learned against kernel width classes on the evaluation subset: AdaSpark’s decode speed with learned widths (§ 3.2 ) over its speed with the widths taken from the 8-bit matrix kernel’s row classes, geometric mean of per-reply ratios with its 95% bootstrap interval over conversations, for all replies and by the context a reply was decoded at; above 1 the learned widths are faster. Both arms start from an empty store and ran on one build; the learned arm runs before the kernel-class arm and again after it, and its speed on a reply is the mean of the two runs. The last columns give the widths each arm’s table offered at the end of its run (setup notes).

Figure 9: LFM2.5-2.6B: measured verify time of each round (dots; hollow dots are cold-start rounds at a width chosen for measurement) and the cost model’s estimate for the width it chose (line), from an empty store, over the first 160 rounds. Over all 1,677 rounds, 67 of them cold-start rounds, the median absolute error of the estimate over the second half of the run is 0.9%.

Figure 10: Accepted nodes per round against the expected accepted nodes

S

at the chosen width, in ten equal-count bins of

S

per series, on the instrumented run over the short subset, over the rounds whose width the chooser set from

S

(cold-start rounds have none). The dashed diagonal is perfect calibration. Realised over expected accepted nodes: LFM2.5-2.6B, fitted 0.997 (1,460 rounds); LFM2.5-8B-A1B, fitted 0.981 (1,998 rounds); Qwen3-4B, fitted 0.973 (3,208 rounds); Qwen3-8B, fitted 0.986 (3,972 rounds); LFM2.5-2.6B, drafter head 1.004 (1,642 rounds). The drafter-head series is the cost-model arm, which prices the tree with the head-based estimate.

target	all replies	first turns
LFM2.5-2.6B	81/95	39/42
LFM2.5-8B-A1B	73/95	34/42
Qwen3-4B	78/95	37/42
Qwen3-8B	85/95	38/42

Table 10: Replies of the evaluation subset whose text, reasoning included, is byte-identical between AdaSpark’s first run and imparo’s three-pick DSpark. Later turns carry each arm’s own earlier replies, so one early difference makes every later turn of that conversation differ; first turns share their history and isolate the verify.

method	size chosen per round	verify cost from	acceptance from
AdaSpark	yes: tree width, from widths it learns, 2–64 rows	fitted online from served rounds, per learned width, as a function of context	fitted online per node, with terms for n-gram candidates
CAST [ 31 ]	yes: per step, by a cost-benefit test with thresholds set by hand per model	latency table over batch size, context and tokens fed, profiled per device before serving	drafter probabilities
EVICT [ 18 ]	yes: tree prefix, 1–32 tokens	table profiled at startup per verify length, no context term	drafter probabilities
Bastion [ 20 ]	yes: grows the tree until the estimated speedup falls	roofline model in width and context, calibrated offline, optional online scale factor	drafter probabilities
TreeSpark [ 9 ]	yes: stops at a price per verified token	latency ladder measured once per deployment	two-parameter map fitted offline to target outcomes
DSpark [ 8 ] , vLLM [ 17 ] , SGLang [ 19 ]	yes: a chain length per request under a batch budget	table profiled at engine start [ 8 ] ; at startup at 8,192 tokens [ 17 ] ; offline, at contexts under about 600 tokens [ 19 ]	confidence head, calibrated offline [ 8 ]

Table 11: How schedulers size a verify. “Per round” means the size is chosen anew for each verify from the listed estimates. Entries are taken from each paper or, for the engines, from their source code (§ 6 ).

	vLLM adaptive verification	SGLang DSpark scheduler	AdaSpark
objective	expected tokens / cost	expected tokens $\times$ steps per second	expected tokens $-$ $\rho$ $\cdot$ time (Eq. 7)
source of cost	profiled at startup	offline profiler, loaded from a file	fitted from served rounds (§ 3.2 )
context	one length, 8,192 tokens by default	16-token prompts decoded for about 570 steps (contexts up to about 600 tokens)	per-class law in context, refitted while serving
where the table is not monotone	running maximum over narrower widths: a cheaper wide width is priced up and never used	lookup at the nearest profiled size below	minimum over wider classes: a cheaper wide class is used for the narrower widths
source of acceptance	drafter confidence head	drafter confidence head, optional offline calibration	fitted multinomial (§ 3.3 )
candidate sources per verify	one	one	two, priced separately (§ 3.3 )

Table 12: DSpark schedulers in vLLM (read from vLLM commit 41c4a3ed4e) and SGLang (read from SGLang commit 8b2ca8ecc2), and AdaSpark.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

source and content	licence	categories	conv.	turns	subset turns	change to the text
Spec-Bench [ 65 ] MT-Bench two-turn items, translation, summarisation, QA, math reasoning, RAG	Apache-2.0	13	18	26	21	none
SPEED-Bench [ 66 ] qualitative split	source’s own	11	19	23	15	none
WildChat [ 67 ] real user conversations; user turns only	ODC-BY	1	8	32	16	cut to the first four user turns
ConvFinQA [ 68 , 69 ] several questions about one earnings report	not stated	1	8	29	16	later turns reduced to their own question
ToolACE [ 70 ] tool use with the source’s schemas and recorded tool results	Apache-2.0	1	8	26	13	Python-style parameter types ( dict , float , int ) renamed to the JSON Schema types they name
LongBench [ 71 ] long documents with the benchmark’s own task prompt	MIT	6	12	12	6	a document over the cap cut from the middle

Appendix

Table 13: The test set: six public sources, 73 conversations, 148 turns, and the evaluation subset drawn from it. Licences as each source states them. The last column lists every change made to source text.

part	constant	value	set by
cost model (§ 3.2 )	quantile level $\tau$	0.15	hand (Proposition 3 )
	context basis unit	8,192 tokens	hand
	L1 and L2 weights $\lambda_{1}$ , $\lambda_{2}$ (on time normalised by the class’s scale)	0.002, 0.01	hand
	FTRL rate and stabiliser	0.05, 1	hand
	squared-gradient cap (rate floor)	400	hand
	sample bank per context band; samples a refit is judged on	16 samples; the newest quarter of a band, all of a band with fewer than four	hand

Appendix

Table 14: The scheduler’s constants.

alternative	what went wrong	measured
Per-round ratio without the fixed cost $D$	narrow rounds looked cheap	+2.2% against a fixed width of 16 (1,596-token context); $-$ 0.8% with $D$
Argmax of the value with no margin	the choice flipped on noise near the boundary	+1.70% / $-$ 0.95% against the best fixed width at two contexts; $-$ 0.40% / $-$ 0.42% with the margin
One acceptance curve per width, learned from rounds at that width	a round at $n$ rows says nothing about wider trees, so the learner never left its start	stayed at its initial width in 56 of 56 rounds from 8 rows and 47 of 47 from 16
Drafter candidates priced without the contradicted term	picks next to a strong n-gram proposal were over-priced; the drafter’s gain fell to 0.5 and trees stayed at 8 rows	contradicted picks: 0.598 predicted, 0.539 accepted; their siblings 0.041 and 0.012. With the term, picks contradicted by a copy of $\geq 16$ tokens: 0.129 predicted, 0.147 accepted
Agreed candidates starting from the drafter’s own estimate	agreement was not counted	agreed picks: 0.865 predicted, 0.970 accepted
Monotone cost table (running maximum over narrower widths)	a cheaper wide class can never be chosen	by construction; 48 rows are 4% cheaper than 40 (§ 3.2 )

Appendix

Table 15: Design alternatives that were built and measured before the form of § 3 , all measured on LFM2.5-2.6B during development. Changes are in milliseconds per token; positive is slower.

Explore similar work

Jul 6, 2026cs.AI

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification. While recent parallel drafters efficiently propose long token sequences in a single forward pass, they suffer from rapid acceptance decay due to a lack of inter-token dependencies. Furthermore, indiscriminately verifying these extended blocks wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput in high-concurrency serving systems. We introduce DSpark, a speculative decoding framework that unifies high-throughput parallel generation with adaptive, load-aware verification. To maintain draft quality, DSpark utilizes a semi-autoregressive architecture, coupling a parallel backbone with a lightweight sequential module, to introduce intra-block dependency modeling and mitigate suffix decay. To optimize system efficiency, DSpark employs confidence-scheduled verification, dynamically tailoring the verification length for each request based on estimated prefix survival probabilities and engine-specific throughput profiles. On offline benchmarks across diverse domains, DSpark substantially improves the accepted length over state-of-the-art autoregressive and parallel drafters. When deployed within the DeepSeek-V4 serving system under live user traffic, DSpark successfully mitigates verification waste. Compared to the established production baseline (MTP-1), DSpark accelerates per-user generation speeds by 60 to 85 percent at matched throughput levels. More importantly, by preventing severe throughput degradation under strict interactivity constraints, it enables performance tiers that were previously unattainable, shifting the Pareto frontier of our serving system.

Xin Cheng, Xingkai Yu, Chenze Shao +30

1Peking University · 2DeepSeek-AI

Speculative Decoding Self-Speculative Autoregressive Generation Deepseek

Ask PDF

Sep 29, 2026cs.LG

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.

Hao-Yuan He, Peng-Fei Liu, Si Shen +1

Speculative Decoding Block-Diffusion Drafter Drafter Autoregressive Model Long-Context Reasoning

Ask PDF

Aug 3, 2026cs.CL

From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding

Speculative decoding accelerates LLM inference only when drafted continuations survive target-model verification. Semi-autoregressive drafters such as DSpark predict an entire token block with one backbone forward and refine it with a lightweight Markov head. However, DSpark decodes this block as a single chain, so an early mismatch invalidates the remaining suffix and limits the benefit of large draft blocks. We show that the conditional structure already learned by DSpark can support multiple parent-consistent continuations without retraining or additional backbone passes. We introduce Parent-Conditioned Drafting Tree (PCTree), which uses the pretrained Markov head to score alternative children separately for each concrete parent and allocates a fixed verification budget to the most probable paths. This converts DSpark's linear draft into a tree while preserving its one-pass parallel backbone. Across Qwen3-{4B,8B,14B} and nine benchmarks, at

B{=}7

, measured speedup gains over autoregressive (AR) decoding, relative to matched DSpark, range from

3.1\%

29.5\%

. On Qwen3-4B GSM8K at

B{=}16

, PCTree increases mean acceptance length from

9.41

11.16

and three-run mean AR speedup from

6.14{\times}

6.60{\times}

. These show that parent-conditioned branching can turn conditional capacity already present in a semi-autoregressive drafter into end-to-end inference gains through an inference-only change.

Zixian Li, Tong Li, Chi Xie +2

OPPO AI Center

Speculative Decoding Drafts Autoregressive Decoding LLM Inference Optimization Continuation Trees

Ask PDF