HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding

Jongwoo Ko; Pashmina Cameron

cs.CLNov 5, 2026

HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding

Authors: Jongwoo Ko, Pashmina Cameron

Organizations: Microsoft Applied Sciences Group

Abstract

Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, drafters must model long-range dependencies and fine-grained draft-position interactions without the growing KV-cache overhead of full-context attention. We propose HyFlash, a compact parallel drafter that introduces a prefix-memory prior to separate long-range context modeling from fine-grained draft-position interactions with limited capacity. Local attention captures recent context and draft-position interactions, while recurrent memory compresses historical target features into fixed-size states. This inductive bias enables effective local and long-range modeling with bounded draft-side memory. Experiments show that HyFlash achieves higher acceptance than full-context attention with substantially less KV storage and with larger gains for shallow drafters and long contexts.

Figures & tables

Figure 1: Motivation for memory-constrained drafting. (a) Analytical target memory usage under an illustrative 12 GiB inference budget for 16 GB-class devices ( Appendix B ). (b) Acceptance across evidence distances for independently trained two-layer drafters with equal 5 MiB logical context budgets.

Figure 2: HyFlash framework. Target-conditioned hybrid layers produce representations for a separate proposal readout. The same target verifies proposals and returns the next anchor; only newly committed target features update memory.

Target	Drafter	Math						Code						Chat
		GSM8K		MATH-500		AIME25		HumanEval		MBPP		LCB		MT-Bench		Avg.
		Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Q3-4B	DFlash	1.89	3.19	1.80	3.28	1.64	3.12	1.70	3.02	1.63	2.89	1.36	2.75	1.53	2.63	1.64	2.98
	DSpark	2.28	3.92	2.11	4.02	1.93	3.82	2.04	3.73	1.91	3.54	1.66	3.46	1.79	3.19	1.95	3.67
	HyFlash	2.53	4.38	2.30	4.48	2.16	4.38	2.23	4.13	2.08	3.94	1.82	3.83	1.98	3.59	2.15	4.10
P4-14B	DFlash	2.09	3.13	2.38	3.58	2.04	3.10	2.21	3.26	2.20	3.17	1.75	2.78	1.62	2.28	2.03	3.04

Table 1: Decoding speedup over autoregressive decoding and average acceptance length (

\tau

) across math, code, and chat benchmarks on Qwen3-4B and Phi-4-reasoning (14B). Both targets use thinking mode, with a maximum generation length of 4,096 tokens.

Context	HotpotQA			QASPER			GovReport
Context	Full	SWA	Hy.	Full	SWA	Hy.	Full	SWA	Hy.
2K–4K	2.78	2.72	3.02	2.61	2.45	2.70	2.43	1.88	2.50
4K–8K	3.23	2.57	3.25	2.52	2.31	2.64	2.50	1.66	2.61
8K–16K	1.24	2.25	3.11	1.79	2.12	2.77	1.19	1.57	2.60
16K–32K	1.02	2.13	2.72	0.97	1.81	2.83	0.86	1.58	2.44

Table 2: Average acceptance length (

\tau

) of Phi-4-reasoning on LongBench across context lengths.

Intervention	Nearby ( $\uparrow$ )	Distance ( $\uparrow$ )	Late-slot Acc. ( $\uparrow$ )
HyFlash	6.32	5.62	0.77
Recurrent history
SSM: Last 512	6.32	3.09	0.77
SSM: Last feature	4.66	2.19	0.68
Intra-block interactions
Cross-slot off	5.68	4.95	0.71

Table 3: Ablations of recurrent memory and intra-block attention. Nearby/Distance: average acceptance length on synthetic retrieval probes; Late-slot Acc.: conditional acceptance at draft positions 4–7 on natural prefixes.

Figure 3: Average acceptance length (

\tau

) of HyFlash and full-context attention drafters across different depth.

Method	MATH-500		AIME25		MBPP		LCB
Method	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
BF16	2.87	4.21	2.52	3.78	2.95	4.12	2.18	3.36
INT4	2.98	4.18	2.61	3.72	3.08	4.09	2.30	3.35
$\Delta$ Rel. (%)	3.8	$-$ 0.7	3.6	$-$ 1.6	4.4	$-$ 0.7	5.5	$-$ 0.7

Table 4: Effect of 4-bit weight quantization on decoding speedup (Spd.) and average acceptance length (

\tau

) of HyFlash across math and code benchmarks. Rel. denotes performance relative to BF16.

Method	MATH-500		AIME25		MBPP		LCB
Method	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
Mamba-2	2.87	4.21	2.52	3.78	2.95	4.12	2.18	3.36
GDN-2	2.85	4.14	2.47	3.59	2.76	4.01	2.18	3.38

Table 5: Effect of recurrent operators on decoding speedup (Spd.) and average acceptance length (

\tau

Method	MATH-500		AIME25		MBPP		LCB
Method	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
DSpark	2.87	4.21	2.52	3.78	2.95	4.12	2.18	3.36
DFlash2	2.92	4.48	2.61	3.94	3.05	4.30	2.26	3.58

Table 6: Effect of proposal readout mechanisms on decoding speedup (Spd.) and average acceptance length (

\tau

). Both methods use the same HyFlash backbone, differing only in their proposal readout mechanisms.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Platform (Processor)	RAM	Peak AI Compute
Microsoft Surface Laptop 7 (X Elite)	16 GB	45 TOPS
Apple Mac mini (M4)	16 GB	38 TOPS
ASUS Vivobook S 15 (X Elite)	16 GB	45 TOPS
NVIDIA Jetson Orin NX	16 GB	157 TOPS
NVIDIA Jetson Orin Nano Super	8 GB	67 TOPS
Qualcomm Snapdragon Dev Kit (X Elite)	32 GB	45 TOPS

Appendix

Table 7: Representative on-device AI hardware configurations. RAM denotes total physical memory, while peak AI compute is reported in tera operations per second (TOPS). Vendor-reported TOPS may reflect different compute units, numerical precisions, and sparsity settings and are not directly comparable. All specifications are taken from publicly available manufacturer documentation ( Microsoft, 2024 ; Apple, 2024 ; ASUS, 2024 ; Maheshwari and Su, 2025 ; Qualcomm, 2024 ) .

Method	AIME25				LCB
Method	Fixed		Adaptive		Fixed		Adaptive
	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
DSpark	2.15	3.42	2.40	3.94	1.82	2.97	2.03	3.42
HyFlash	2.52	3.81	2.74	4.29	2.17	3.44	2.41	3.94

Appendix

Table 8: Effect of tree drafting strategies on decoding speedup (Spd.) and average acceptance length (

\tau

). Fixed uses width 4 and depth 7, while Adaptive follows BASTION ( Oh et al., 2026 ) .

Method	$k=3$		$k=5$		$k=7$		$k=9$
Method	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
DSpark	1.88	2.73	2.12	3.10	2.22	3.28	2.18	3.27
DSpark + SWA	1.55	2.28	1.93	2.89	2.09	3.15	2.03	3.04
HyFlash	1.95	2.89	2.45	3.68	2.58	3.79	2.62	3.98

Appendix

Table 9: Sensitivity to inference-time draft length. Average acceptance length (

\tau

) and decoding speedup for

k\in\{3,5,7,9\}

, using drafters trained with

K=7

Method	MATH-500		AIME25		MBPP		LCB
Method	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
Greedy	3.35	4.86	2.90	4.29	4.09	5.77	3.13	4.69
Moderate	3.22	4.75	2.79	4.25	3.00	4.39	2.54	3.97
Untruncated	2.87	4.21	2.52	3.78	2.95	4.12	2.18	3.36

Appendix

Table 10: Sensitivity to sampling parameters on decoding speedup (Spd.) and average acceptance length (

\tau

Method	MATH-500		AIME25		MBPP		LCB
Method	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$
4K	2.87	4.21	2.52	3.78	2.95	4.12	2.18	3.36
16K	2.00	5.46	1.41	2.95	1.64	3.69	1.78	4.15

Appendix

Table 11: Sensitivity to context length on decoding speedup (Spd.) and average acceptance length (

\tau

Figure 4: Latency analysis at 4K and 16K context lengths, excluding prefill. (a) Per-round latency breakdown on AIME25. (b) Draft-backbone latency across four benchmarks.

Parallel Speculative Decoding

See all 37 papers

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
Oct 14, 2026Shuxuan Li, Jialiang Kang, Yanghao Zhou +2Speculative Decoding Long-Context Language Model Inference
Sun Yat-sen University · Canva Research · Peking University
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Sep 21, 2026Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn +6KV-Cache Management Speculative Decoding
Harvard University · Capital One · Red Hat +2
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding LLM Inference Acceleration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China

Oct 14, 2026J/K move · Enter open · S save

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu+1

Sun Yat-sen University · Canva Research · Peking University

Speculative Decoding Long-Context Language Model Inference Memory-Augmented Language Models

Open paper AskPDF arXiv

Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet most drafters are trained on short sequences and degrade on long inputs, while long-context training is costly. Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries. Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history attention affordable. We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position. To counter attention dilution as the memory grows, we sharpen attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window. Across six target-drafter pairs and diverse benchmarks of up to 256K tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08

\times

to 4.22

\times

Abstract

Parallel Speculative Decoding

Explore similar work

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting