LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models

Zhiyuan Ma

cs.LGSep 20, 2026

LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models

Authors: Zhiyuan Ma

Abstract

Tree speculative decoding for hybrid language models must preserve one coherent continuation across recurrent, convolution, and attention state. We present LumoTree, a verifier that executes recurrent paths in parallel, reuses state tiles within each path, and coordinates native recurrent replay, convolution-history gathering, and attention-cache remapping through a shared logical tree. Fused candidate selection, GPU-resident acceptance, and grouped split-K attention support the verification cycle. Component experiments show exact candidate-selection parity and recurrent agreement within paired error bounds. An exploratory Qwen3.8-27B NVFP4 deployment on a single NVIDIA DGX Spark (GB10) records 25.63 pooled tokens/s on ten SWE-bench Verified Astropy tasks. The results characterize component-level numerical agreement and coding-agent deployment; complete-model continuation and controlled application speedups remain open.

Figures & tables

Panel of Figure 1 — Fig. 1: Tree verification and accepted-path commitment. (a) Drafting provides a candidate tree and proposal probabilities $q$ . (b) Branch-local computation produces target probabilities $p$ and retains temporary state. (c) Sampling selects a materialized path (here $a,d$ ); recurrent replay, convolution gathering, and KV remapping publish $\mathcal{S}(mad)$ . The newly sampled token $b^{*}$ remains pending. One descriptor keeps logical ancestry and physical storage aligned.

Method	Verification work	Continuation state
Weaver [ 7 ]	Ancestor-masked triangular solve	Short selected-path recurrence
SpecLA [ 6 ]	Resident state tiles; parallel chains	Buffered factors; later fused update
Bole [ 28 ]	Finite-Neumann correction solve	Compact factors; state reconstruction
TreeWY [ 30 ]	Triangular solve in fused kernel	Prior-path reconstruction with next verify
ReplaySSM [ 31 ]	Corrected inputs; cached transitions	Deferred state update
OneLA [ 32 ]	Shared-base projections across beams	Compact per-beam transition records

TABLE I: Recurrent verification and continuation policies. These mechanism comparisons do not rank the systems by performance.

Panel of Figure 3 — Fig. 2: The accepted-prefix interface. Recurrent, convolution, and attention-KV state must all describe the same materialized prefix. A newly sampled correction or bonus token can remain pending until the next forward.

Panel of Figure 4 — Fig. 3: State lifetime in the path-scan verifier. Temporary cut states connect path groups; recorded operands support selected-path replay. Only the accepted frontier becomes native persistent state.

Starting work or constraint	LumoTree change	What must remain consistent
Host path-list construction between acceptance and publication	Device-selected path feeds captured native replay, convolution, KV and drafter selection	All consumers select the same materialized prefix; pending and padded tokens do not advance state
Logical ancestry does not determine physical KV placement	Spine-contiguous slots with coordinated writes, mask columns and accepted-path remap	Logical query order, physical source slots and native destination positions agree
Per-node convolution histories and repeated layer preparation	Static ancestry gathers, fused ordered taps and per-layer operand buffers	Tap order, cast boundaries, history padding and layer identity
Separate score preparation and argmax/top-k operations	One shared logits tensor and fused selection into graph buffers	Both reference tie conventions, candidate order and duplicate behavior
Repeated KV staging and limited decode parallelism	Grouped query heads and split-K in the patched tree-attention kernel	Tree visibility and slot mapping across partitions; characterized reduction error

TABLE II: Concrete implementation changes around the path verifier. The third column identifies the coordination or numerical constraint that makes each change nontrivial. Workload rates measure their combined execution.

Metric	Value
Accepted drafts per event	4.05
Target span (ms)	114.89
Drafter span (ms)	54.76
Dispatch span (ms)	20.15

TABLE III: Acceptance and mean GPU span durations for the ten-task deployment. Dispatch includes acceptance, path handling, and state commitment.

Operand sets	BF16 output bitwise	FP32 state bitwise	Paired rule passed
Calibration	3,004/3,072	2,688/2,688	5,760/5,760
Held-out	3,018/3,072	2,688/2,688	5,760/5,760

TABLE IV: GDN component checks on GB10. Bitwise columns compare with the native chain; the paired rule compares both implementations with a float64 reference. Counts are per block and process.

Configuration	Output cells	State cells
Native GPU recurrence reference
LumoTree	0	0
Sequential software references
Weaver author-default	53,745	1,154
Weaver (FP32 preparation)	57,707	388
TreeWY author-default	37,528	6,912

TABLE V: Recurrent agreement on GB10 under distinct sequential references. Counts give cells outside the paired error bounds among 64,512 output cells and 6,912 state cells per configuration, with identical counts in both repeats. The two reference groups are not a common accuracy ranking.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	LumoTree / vLLM
Target	Qwen3.8-27B; ModelOpt mixed NVFP4, BF16 execution
Proposals	27 drafts plus root, padded to 32 rows; maximum depth 11
Attention	Patched FlashAttention-2; head grouping 2, split-K 4
Context	131,072 tokens
Scheduling	4,096 scheduled tokens; one sequence
Response cap	24,000 tokens

Appendix

TABLE VI: Serving configuration for the ten-task LumoTree deployment. Exact software versions and launch settings accompany the artifact.

Task suffix	Agent time (min)	Evaluation
13977	25	Failed tests
14096	23	Resolved
14182	7	Failed tests
14309	2	Resolved
14365	7	Failed tests
14369	28	Resolved

Appendix

TABLE VII: Task-level results for the ten-task LumoTree deployment. Task IDs have prefix astropy__astropy- . Durations are rounded to minutes.

Starting-state control	Greedy	State/ logits	Joint choices
Natural prefill	70/84	0/84	–
Common target state	84/84	84/84	–
Common target and MTP state	84/84	65/84	68/84
With fixed kernel selection	84/84	65/84	68/84

Appendix

TABLE VIII: Cross-process native-reference agreement, in paths out of 84. State/logits requires identical published target state and full target logits. Joint choices requires the next target decision and the actual MTP spine and ordered top-three selections at every tested phase. A dash denotes an untested joint MTP condition.

Explore similar work

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models
Aug 3, 2026Li Wang, Yi Su, Xiabao Wu +9Speculative Decoding Self-Speculative
State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · Shanghai Jiao Tong University +1
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferKimi Delta Attention Gated Deltanet
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding Decoding
Thoughtworks · Nvidia

Aug 3, 2026, cs.DCJ/K move · Enter open · S save

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

Li Wang, Yi Su, Xiabao Wu, Chiran You+8

State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · Shanghai Jiao Tong University+1

Speculative Decoding Self-Speculative Strongest Prior Asynchronous Baseline Trees

Open paper AskPDF arXiv

Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound. Tree speculative decoding offers an attractive acceleration path, but existing tree-speculation systems are designed around the key--value caches of full-attention models. On hybrid models, they traverse recurrent layers branch by branch and materialize a full state for every proposal node, causing verification latency and transient memory to scale poorly with tree and batch sizes. We present Bole, a kernel--runtime co-design that enables efficient tree speculation for hybrid-attention LLMs. Bole transforms the linear-attention recurrence into a tree-structured closed form and realizes it with a resource-efficient GPU kernel, verifying all proposal nodes in parallel and accelerating linear-attention tree verification by 3.4--7.7

\times

. It losslessly encodes speculative state updates as token-level factors and reconstructs only the state selected after sampling, reducing transient state memory by 82--99

\times

and freeing GPU capacity for KV caches. Its integration into SGLang, a widely deployed production LLM serving engine, couples efficient state management with a batch-wide verification budget calibrated to the complete hybrid forward. Across four models, two GPU platforms, and diverse datasets, Bole delivers up to

4.72\times

the offline decode throughput of autoregressive decoding and up to

2.03\times

that of the strongest tree-speculative baseline. Under online agent workloads, it reduces TTFT and TPOT by up to

67.6%

and

49.9%

, respectively, over the strongest tree-speculative baseline.

Abstract

Explore similar work

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding