FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

Shuxuan Li; Jialiang Kang; Yanghao Zhou; Ziye Zhu; Xiao Yang

cs.CLOct 14, 2026

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

Authors: Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu, Xiao Yang

Organizations: Sun Yat-sen University · Canva Research · Peking University

Abstract

Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet most drafters are trained on short sequences and degrade on long inputs, while long-context training is costly. Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries. Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history attention affordable. We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position. To counter attention dilution as the memory grows, we sharpen attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window. Across six target-drafter pairs and diverse benchmarks of up to 256K tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08 $\times$ to 4.22 $\times$ .

Figures & tables

Figure 1: Latency breakdown across context lengths for Llama-3.1-8B (vLLM, one H100 GPU, batch size 1), for an AR step and a DFlash SD round with block size 10. From 1K to 128K tokens, target KV-cache attention grows from 5% to 46% of the AR step, while drafter KV-cache attention stays below 7% of the SD round.

Figure 2: Long-context degradation of DFlash drafters. Speedup over AR decoding for Llama-3.1-8B and Gemma-4-12B on SPEED-Bench (1K–32K tokens) Abramovich et al. (2026) , LongBench v2 Bai et al. (2025) (grouped by input length), and Retr.KV Zhang et al. (2024) .

Figure 3: Diagnosing long-context drafting on Llama-3.1-8B. Accuracy of a linear probe predicting the order of two entries from the difference of their context features or token embeddings, by separation, on SPEED-Bench 32K (solid) and LBv2 (dashed), with bands of one standard deviation over five document-level folds.

Figure 4: Overview of FOLD. (a) Target hidden states of the ordered prefix are projected into context features $c_{j}$ . (b) The draft block and recent entries keep their true relative distances, and all earlier entries share the virtual position $t-D$ . (c) The window and the far memory are attended separately, and $\alpha$ sharpens the memory read while its mass grows as $N^{1/\alpha}$ rather than $N$ for $N$ equally scored entries. The outputs are merged by their masses.

Target	Drafter	$B$	Attention layers	Train	$W$
Llama-3.1-8B	DFlash	10	5 full	3K	2,048
Qwen3.5-9B	DFlash	16	5 SW (4K) + 1 full	40K	4,096
Gemma-4-12B	DFlash, DSpark	7	5 full, partial RoPE	4K	4,096
Qwen3-8B	DFlash, DSpark	7	5 full	4K	2,048

Table 1: Drafters. Block size

B

, attention layers (SW: sliding window), training length, and the window

W

of FOLD. Checkpoints are listed in Appendix B.1 .

Llama-3.1-8B
Method	Gov		LBv2		Retr.KV		En.		LCA		RepoBench		SWE-bench		Avg.
Method	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.
DFlash	2.49	2.35 $\times$	2.88	2.62 $\times$	1.35	1.24 $\times$	1.77	1.64 $\times$	2.75	2.53 $\times$	2.87	2.68 $\times$	1.86	1.74 $\times$	2.28	2.11 $\times$
StreamingLLM	2.88	2.71 $\times$	3.88	3.57 $\times$	1.79	1.69 $\times$	2.70	2.57 $\times$	4.06	3.79 $\times$	4.07	3.79 $\times$	2.36	2.22 $\times$	3.11	2.91 $\times$
YaRN	3.49	3.18 $\times$	3.87	2.91 $\times$	1.66	1.12 $\times$	2.65	1.87 $\times$	4.01	3.03 $\times$	3.94	3.22 $\times$	3.02	2.42 $\times$	3.23	2.54 $\times$
CMR	3.39	2.58 $\times$	4.05	2.98 $\times$	1.76	1.44 $\times$	2.87	2.36 $\times$	4.34	3.41 $\times$	4.19	3.22 $\times$	2.82	2.21 $\times$	3.35	2.60 $\times$

Table 2: Main results with DFlash drafters. Acceptance length

\tau

and speedup (Spd.) over AR decoding. Layer coverage for Qwen3.5-9B is specified in Appendix B.3 . Bold and italics mark the best and second-best values per column and model.

Method	Qwen3-8B		Gemma-4-12B
	General		General		Code
	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.
DSpark	4.02	3.41 $\times$	3.03	2.62 $\times$	3.68	3.22 $\times$
StreamingLLM	4.19	3.54 $\times$	2.68	2.33 $\times$	3.54	3.13 $\times$
YaRN	4.27	3.32 $\times$	3.21	2.45 $\times$	3.72	3.03 $\times$
CMR	4.25	2.91 $\times$	2.94	2.25 $\times$	3.79	3.00 $\times$

Table 3: Results with DSpark drafters. Acceptance length

\tau

and speedup over AR decoding on the general and code tasks. General includes only inputs of at least 16K tokens and, for Qwen3-8B, only Gov and LBv2.

Figure 5: Speedup over AR by input length. Top: LCA inputs of fixed length, where the 180K and 240K inputs come from its huge-context split (dotted line). Bottom: LBv2 requests grouped by length. Llama-3.1-8B does not admit inputs beyond 128K. Appendix C.1 shows the corresponding

\tau

	DFlash $\rightarrow$ Window			Window $\rightarrow$ FOLD
	32K	64K	122K	32K	64K	122K
Llama	$+1.07$	$+1.31$	$+1.53$	$+1.02$	$+1.07$	$+0.93$
Qwen3.5	$-0.22$	$-0.08$	$+0.15$	$+1.11$	$+1.28$	$+1.29$
Gemma	$+0.09$	$+0.13$	$+0.36$	$+0.15$	$+0.29$	$+0.21$

Table 4: Decomposition of the difference in

\tau

between FOLD and DFlash on LCA into Window minus DFlash and FOLD minus Window. For Qwen3.5-9B, Window restricts only the full-attention layer, while FOLD lets all six layers read distant entries.

Model	Tasks	$\tau_{\alpha=2}$	$\alpha=2$ vs. 1	$\alpha=4$ vs. 2
Llama	General	3.17	1.150 [1.137, 1.162]	1.018 † [1.014, 1.022]
	Code	4.52	1.067 [1.064, 1.072]	0.992 [0.991, 0.994]
Qwen3	General	3.26	1.160 [1.138, 1.183]	1.012 † [1.006, 1.019]
Qwen3.5	General	3.59	2.793 [2.743, 2.845]	1.146 [1.133, 1.159]
	Code	6.86	4.173 [4.068, 4.285]	1.001 [1.000, 1.003]
Gemma	General	2.88	1.071 [1.066, 1.077]	1.001 † [0.998, 1.004]

Table 5: Effect of the concentration factor

\alpha

on requests longer than the window. General and Code follow the task groups of § 4.1 , and for Qwen3-8B, General covers only Gov and LBv2. Both columns are means of per-request ratios over all requests of a group, with 95% bootstrap intervals, and values above 1 favor the larger

\alpha

. † The ratio of the overall

\tau

of all requests leads to a different conclusion (Appendix D.2 ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Target	Drafter checkpoint
NousResearch/Meta-Llama-3.1-8B-Instruct	z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat
Qwen/Qwen3.5-9B	z-lab/Qwen3.5-9B-DFlash
google/gemma-4-12B-it	deepseek-ai/dflash_gemma4_12b_block7
google/gemma-4-12B-it	deepseek-ai/dspark_gemma4_12b_block7
Qwen/Qwen3-8B	deepseek-ai/dflash_qwen3_8b_block7
Qwen/Qwen3-8B	deepseek-ai/dspark_qwen3_8b_block7

Appendix

Table 6: Released checkpoints on Hugging Face.

		Llama-3.1-8B		Qwen3.5-9B		Gemma-4-12B
Task	Gen. limit	$n$	Input length	$n$	Input length	$n$	Input length
Gov	512	284	1.8K–27K	275	1.1K–28K	275	1.1K–28K
LBv2	128	320	9.8K–128K	379	10K–253K	378	10K–250K
Retr.KV	128	488	121K–122K	478	165K–166K	478	170K–170K
En.Sum	1200	48	65K–125K	72	66K–235K	72	66K–240K
En.QA	512	112	64K–128K †	281	64K–256K †	281	64K–256K †

Appendix

Table 7: Evaluation data. Number of requests

n

, input length range in tokens, and generation limit per task. † Range of the length bins that contain the requests. LCA inputs have exactly the stated length, and SPEED-Bench inputs have the nominal length of their tier.

Figure 6: Acceptance length by input length , on the same requests and groups as Figure 5 . Top: LCA. Bottom: LBv2.

Llama-3.1-8B
Method	8K		16K		32K
Method	$\tau$	Spd.	$\tau$	Spd.	$\tau$	Spd.
DFlash	3.02	2.84 $\times$	2.69	2.52 $\times$	2.21	2.07 $\times$
StreamingLLM	3.50	3.27 $\times$	3.18	2.99 $\times$	2.89	2.72 $\times$
YaRN	4.07	3.70 $\times$	3.62	3.17 $\times$	3.34	2.74 $\times$
CMR	3.87	2.92 $\times$	3.70	2.82 $\times$	3.33	2.57 $\times$

Appendix

Table 8: Results on SPEED-Bench. Acceptance length

\tau

and speedup over AR decoding on the throughput tiers of 8K, 16K, and 32K tokens. Bold and italics mark the best and second-best values per column and model.

Method	Gov		LBv2
Method	$\tau$	Spd.	$\tau$	Spd.
DFlash	2.68	2.52 $\times$	4.08	3.58 $\times$
StreamingLLM	2.55	2.38 $\times$	5.28	4.59 $\times$
YaRN	2.84	2.58 $\times$	5.36	4.26 $\times$
CMR	2.77	2.12 $\times$	5.33	3.71 $\times$
ReRoPE	2.80	2.60 $\times$	4.32	3.72 $\times$

Appendix

Table 9: Results for Qwen3-8B with its DFlash drafter. Acceptance length

\tau

and speedup over AR decoding. Bold and italics mark the best and second-best values per column.

	All 1,531 requests			Common set (1,437)
Method	$\tau$	Speedup	Diff. (AR)	$\tau$	Speedup
DFlash	1.96	1.54 $\times$	83	1.97	1.55 $\times$
SGLang window	2.70	2.18 $\times$	82	2.70	2.18 $\times$
StreamingLLM	2.74	2.11 $\times$	84	2.75	2.11 $\times$
CMR	2.93	2.16 $\times$	82	2.93	2.16 $\times$
ReRoPE	3.21	2.46 $\times$	79	3.22	2.47 $\times$

Appendix

Table 10: Llama-3.1-8B in SGLang. Averages over the seven general tasks with equal weight, with the speedup measured against AR on the same GPU. The Diff. (AR) column counts requests whose outputs differ from AR. The common set excludes the 94 requests that differ for any method. Bold indicates the highest acceptance length and speedup.

Llama-3.1-8B
Bucket	$n$	$\tau_{\mathrm{DFlash}}$	$\tau_{\mathrm{FOLD}}$	DFlash $\rightarrow$ Window	Window $\rightarrow$ FOLD
8–16K	28	2.63	4.07	+0.57 [+0.36, +0.79]	+0.87 [+0.52, +1.25]
16–32K	90	2.85	3.99	+0.66 [+0.54, +0.78]	+0.48 [+0.32, +0.64]
32–64K	72	3.12	4.44	+1.09 [+0.91, +1.28]	+0.23 [+0.06, +0.43]
64–128K	130	2.83	4.18	+1.34 [+1.16, +1.52]	+0.01 [-0.08, +0.09]
Qwen3-8B

Appendix

Table 11: Decomposition of the difference in

\tau

on LBv2 by input length. The last two columns are changes in

\tau

paired by request, which sum to the difference between FOLD and DFlash, with pointwise 95% bootstrap intervals.

Model	Tasks	$n$	$\tau_{\alpha=2}$	$\alpha=2$ vs. 1	$\alpha=4$ vs. 2
Llama	Natural	1137	3.15	1.174 [1.158, 1.191]	1.027 † [1.022, 1.032]
	SPEED	1816	4.24	1.039 [1.036, 1.043]	0.991 [0.989, 0.992]
	QA/MC/Dia	391	3.22	1.078 [1.070, 1.086]	0.993 [0.991, 0.996]
Qwen3	Natural	359	3.26	1.160 [1.138, 1.183]	1.012 † [1.006, 1.019]
	SPEED	1150	3.51	1.194 [1.185, 1.203]	1.012 [1.010, 1.014]
Qwen3.5	Natural	1138	3.75	2.709 [2.632, 2.787]	1.217 [1.198, 1.236]

Appendix

Table 12: Effect of

\alpha

on the general tasks and SPEED-Bench. Ratios and † as in Table 5 , with

n

the number of requests. Natural covers Gov, LBv2, Retr.KV, and En.Sum, QA/MC/Dia covers En.QA, En.MC, and En.Dia, and SPEED covers all SPEED-Bench requests of each model that are longer than the window.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Aug 31, 2026Tong Yuan, Chengxi Liao, Zeyi WenMemory-Augmented Language Models KV Caching
Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou)
Long-Context Drafting Does Not Require Long-Context Drafters
Oct 14, 2026Aref Jafari, Vansh Bhatia, Matt Qin +5Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices Inc. (AMD) · University of Waterloo
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding LLM Inference Acceleration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China

Aug 31, 2026J/K move · Enter open · S save

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache

Tong Yuan, Chengxi Liao, Zeyi Wen

Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou)

Memory-Augmented Language Models KV Caching Speculative Decoding LLM Inference Acceleration KV-Cache Compression Long-Context LLM Inference

Open paper AskPDF arXiv

Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of tokens, making decoding latency a major bottleneck. Speculative decoding (SD) reduces latency without changing model outputs, but its speedup depends on both accepted draft tokens and draft-step latency: Lightweight drafts are fast but lack the capacity to capture long-range dependencies, whereas strong independent drafts recover acceptance but incur growing KV-access cost at long prefixes. We introduce memory-augmented drafting for long-context SD, equipping a strong independent draft with compressed draft-side KV memory: A lightweight adaptor constructs and incrementally updates this memory to retain distant information and exact recent context. The target verifier retains its full KV cache and applies the standard accept/reject rule, preserving SD's lossless guarantee. Experiments on Llama~3.1-8B and 70B targets at prefix lengths up to 32K show that our method reduces draft-side memory by over 70%. It achieves speedups of up to 2.08x and 3.33x , respectively, over autoregressive decoding.

Abstract

Speculative Decoding

Explore similar work

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache

Long-Context Drafting Does Not Require Long-Context Drafters

DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting