ConFu: Contemplate the Future for Better Speculative Sampling

Zongyue Qin; Raghavv Goel; Mukul Gagrani; Risheek Garrepalli; Mingu Lee; Yizhou Sun

cs.CLMar 9, 2026

ConFu: Contemplate the Future for Better Speculative Sampling

Authors: Zongyue Qin, Raghavv Goel, Mukul Gagrani, Risheek Garrepalli, Mingu Lee, Yizhou Sun

Organizations: University of California Los Angeles, United States · Qualcomm AI Research, United States.

Abstract

Speculative decoding has emerged as a powerful approach to accelerate large language model (LLM) inference by employing lightweight draft models to propose candidate tokens that are subsequently verified by the target model. The effectiveness of this paradigm critically depends on the quality of the draft model. While recent advances such as the EAGLE series achieve state-of-the-art speedup, existing draft models remain limited by error accumulation: they condition only on the current prefix, causing their predictions to drift from the target model over steps. In this work, we propose ConFu (Contemplate the Future), a novel speculative decoding framework that enables draft models to anticipate the future direction of generation. ConFu introduces (i) contemplate tokens and soft prompts that allow the draft model to leverage future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate token mechanism with MoE to enable context-aware future prediction, and (iii) a training framework with anchor token sampling and future prediction replication that learns robust future prediction. ConFu improves token acceptance rates and generation speed over EAGLE-3 by 8--11% on Llama-3 3B/8B and by approximately 20% on Qwen-3 4B across downstream tasks. We believe our work is the first to bridge speculative decoding with continuous reasoning tokens, offering a new direction for accelerating LLM inference.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the purpose of future generation direction prediction

Figure 2: Overview of ConFu’s inference pipeline. Given the input tokens, the target model first produces the next output token along with a future prediction vector

\bm{f}

, using both prompt tokens and contemplate tokens. The draft model then conditions on

\bm{f}

as an additional future token to autoregressively generate draft tokens. Throughout the drafting process, the future token

\bm{f}

remains fixed and is always appended to the end of the input sequence.

Figure 3: Verification with contemplate tokens in ConFu. Let

t_{1},t_{2},t_{3}

denote draft tokens in the speculative tree. We insert one contemplate token after each draft token so that the target model can simultaneously verify draft candidates and generate the corresponding future predictions. The tree attention mask is adjusted accordingly to ensure correct verification and alignment of future predictions with accepted tokens.

Figure 4: Illustration of Dynamic Contemplate Tokens with MoE. The input tokens contain both accepted tokens and the draft tokens of the current iteration. The MoE module only takes the hidden representation of the last accepted token as inputs. Then it computes the expert weights with a linear layer (router) and outputs the weighted sum of the selected learnable embeddings as the final contemplate token embedding. For simplicity a single [con] token is shown instead of 3 (1 for ’like’ and 2 for draft tokens)

Method	Nodes	T	WRIT		RP		CODE		EXT		STEM		HUM		SUMM		TRAN		QA		RAG		M/R		AVG
			$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR
Eagle3	30	0.0	4.00	1.97	3.54	1.73	5.13	2.54	4.38	2.07	4.26	2.10	4.04	1.97	3.50	1.71	3.64	1.79	3.58	1.72	4.07	1.37	4.59	2.06	4.00	1.83
ConFu	30	0.0	4.55	2.18	4.18	1.96	5.72	2.75	4.96	2.27	4.85	2.33	4.59	2.18	3.95	1.88	4.16	1.98	3.91	1.83	4.52	2.00	5.21	2.45	4.41	2.11
Eagle3	30	0.7	3.78	1.78	3.41	1.60	4.86	2.30	4.17	1.90	4.00	1.89	3.75	1.75	3.44	1.60	3.40	1.60	3.43	1.60	3.90	1.29	4.32	2.01	3.81	1.70
ConFu	30	0.7	4.24	1.97	3.99	1.78	5.43	2.54	4.77	2.09	4.61	2.15	4.15	1.93	3.82	1.8	3.95	1.8	3.75	1.7	4.42	1.86	4.86	2.24	4.29	1.95
Eagle3	30	1.0	3.23	1.51	2.90	1.37	4.47	2.12	4.08	1.85	3.24	1.54	3.19	1.52	3.22	1.51	3.03	1.43	3.09	1.43	3.44	1.52	3.97	1.85	3.58	1.59

Table 1: Llama3.2-3B-Instruct comparison on SpecBench tasks across temperature={

0.0,0.7,1.0

} and draft nodes={

30,60

}. WRIT=writing, QA=question-answer, SUMMAR=summarization, TRANS=translation, CODE=coding, M/R=math/reasoning. Both metrics are higher the better.

Method	Nodes	T	WRIT		RP		CODE		EXT		STEM		HUM		SUMM		TRAN		QA		RAG		M/R		AVG
			$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR
Eagle3	30	0.0	4.39	2.43	4.26	2.31	5.65	3.15	4.89	2.59	5.02	2.80	4.60	2.59	3.76	2.04	4.28	2.35	4.35	2.32	4.78	1.59	5.16	2.82	4.59	2.36
ConFu	30	0.0	5.03	2.74	4.64	2.50	6.00	3.32	5.19	2.75	5.47	3.03	5.18	2.89	4.07	2.20	4.87	2.66	4.63	2.46	5.19	2.60	5.65	3.08	5.01	2.69
Eagle3	30	0.7	4.09	2.23	3.91	2.14	5.36	2.95	4.72	2.48	4.45	2.45	4.27	2.34	3.64	1.96	4.03	2.19	3.99	2.12	4.49	2.27	4.88	2.63	4.31	2.31
ConFu	30	0.7	4.57	2.42	4.33	2.25	5.74	3.07	5.07	2.59	4.99	2.67	4.77	2.57	3.99	2.08	4.53	2.39	4.16	2.15	4.84	2.36	5.34	2.81	4.70	2.44
Eagle3	30	1.0	3.35	1.81	3.34	1.80	4.84	2.63	4.42	2.29	4.01	2.19	3.02	1.65	3.30	1.75	3.64	1.95	3.57	1.88	3.96	1.99	4.48	2.41	3.85	2.04

Table 2: Llama3.1-8B-Instruct comparison on SpecBench tasks across temperature={

0.0,0.7,1.0

} and draft nodes={

30,60

}. Both metrics are higher the better.

Method	Nodes	T	WRIT		RP		CODE		EXT		STEM		HUM		SUMM		TRAN		QA		RAG		M/R		AVG
			$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R	$\tau$	S/R
Eagle3	30	0.0	3.69	2.53	3.14	2.18	4.03	2.88	4.05	2.88	3.83	2.83	3.45	2.52	3.05	2.05	3.16	2.27	3.37	2.41	3.56	2.18	4.21	2.80	3.57	2.44
ConFu	30	0.0	4.57	3.10	3.69	2.53	4.98	3.53	4.92	3.39	4.49	3.29	4.08	2.95	3.59	2.34	3.86	2.74	4.01	2.83	4.37	3.05	5.15	3.59	4.32	3.00
Eagle3	30	0.7	3.61	2.46	3.22	2.22	3.96	2.78	4.24	2.89	3.65	2.68	3.41	2.46	3.05	2.03	3.12	2.22	3.35	2.36	3.56	2.50	4.19	2.91	3.56	2.48
ConFu	30	0.7	4.43	3.03	3.60	2.50	4.93	3.50	4.98	3.48	4.39	3.24	4.04	2.94	3.63	2.38	3.76	2.69	3.88	2.76	4.37	3.06	5.15	3.60	4.28	2.99
Eagle3	30	1.0	3.60	2.18	3.13	2.16	3.93	2.79	4.23	2.95	3.70	2.70	3.27	2.37	3.09	2.06	3.10	2.21	3.26	2.31	3.52	2.47	4.18	2.93	3.53	2.46

Table 3: Qwen3-4B comparison on SpecBench tasks across temperature={

0.0,0.7,1.0

} and draft nodes={

30,60

}. Both metrics are higher the better.

Method	T	WRIT		RP		CODE		EXT		STEM		HUM		SUMM		TRAN		QA		RAG		M/R		AVG
		$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR	$\tau$	SR
Eagle3	0.0	4.39	2.43	4.26	2.31	5.65	3.15	4.89	2.59	5.02	2.80	4.60	2.59	3.76	2.04	4.28	2.35	4.35	2.32	4.78	1.59	5.16	2.82	4.59	2.36
ConFu	0.0	5.03	2.74	4.64	2.50	6.00	3.32	5.19	2.75	5.47	3.03	5.18	2.89	4.07	2.20	4.87	2.66	4.63	2.46	5.19	2.60	5.65	3.08	5.01	2.69
$-$ MoE	0.0	4.97	2.73	4.50	2.44	5.94	3.30	5.23	2.76	5.46	3.04	5.06	2.84	4.04	2.19	4.83	2.65	4.60	2.46	5.12	2.57	5.59	3.05	4.97	2.67
$-$ MoE & Replicate	0.0	4.77	2.59	4.37	2.36	5.82	3.22	5.07	2.69	5.19	2.88	4.81	2.69	3.96	2.13	4.59	2.5	4.49	2.38	4.90	2.22	5.49	2.98	4.81	2.55
Eagle3	0.7	4.09	2.23	3.91	2.14	5.36	2.95	4.72	2.48	4.45	2.45	4.27	2.34	3.64	1.96	4.03	2.19	3.99	2.12	4.49	2.27	4.88	2.63	4.31	2.31

Table 4: Llama3.1-8B-Instruct ablation comparison on SpecBench tasks across temperature={

0.0,0.7

} and draft nodes={

30

}. WRIT=writing, QA=question-answer, SUMMAR=summarization, TRANS=translation, CODE=coding, M/R=math/reasoning. Both metrics are higher the better. Bold numbers indicate best performance under that temperature.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model (60 nodes)	Eagle3 (GB)	ConFu (GB)
Llama-3.2-3B-Instruct	7.50	7.52
Llama-3.1-8B-Instruct	17.08	17.12

Appendix

Table 5: Peak GPU memory usage (GB) when running with 60 draft nodes. Results are averaged over 10 SpecBench samples.

Length	Nodes	T	WRIT	RP	CODE	EXT	STEM	HUM	SUMM	TRAN	QA	RAG	M/R	AVG
4	30	0.0	4.72	4.10	5.70	4.95	4.85	4.48	4.17	3.93	3.88	4.56	5.16	4.50
7	30	0.0	4.55	4.18	5.72	4.96	4.85	4.59	3.95	4.16	3.91	4.52	5.21	4.41
10	30	0.0	4.64	4.02	5.76	4.99	4.79	4.51	4.17	3.96	3.91	4.57	5.15	4.50
4	30	0.7	4.35	3.87	5.37	4.67	4.55	4.30	3.88	3.78	3.70	4.35	4.94	4.27
7	30	0.7	4.24	3.99	5.43	4.77	4.61	4.15	3.82	3.95	3.75	4.42	4.86	4.29
10	30	0.7	4.35	3.97	5.62	4.74	4.59	4.22	3.88	3.81	3.63	4.34	4.86	4.26

Appendix

Table 6: Llama3.2-3B-Instruct ConFu average acceptance rate (

\tau

, higher is better) across temperature={

0.0,0.7,1.0

} and draft nodes={

30,60

}, for training block lengths {4,7,10}. Bold indicates the best length for each (Nodes, T) setting. WRIT=writing, QA=question-answer, SUMMAR=summarization, TRANS=translation, CODE=coding, M/R=math/reasoning

Figure 5: Survival function of accepted draft length, showing the probability that at least

l

consecutive draft tokens are accepted. ConFu consistently exhibits higher tail acceptance than EAGLE‑3, indicating more robust acceptance of long draft trajectories under strict decoding.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Apr 14, 2026Zhuofan Wen, Yang FengSpeculative Decoding LLM Inference Acceleration
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Mar 19, 2026J/K move · Enter open · S save

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang+13

Nanyang Technological University · SpecForge Team · Meituan+2

Speculative Decoding LLM Inference Acceleration Efficient Language Model Training

Open paper AskPDF arXiv

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.

Abstract

Speculative Decoding

Explore similar work

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Draft-OPD: On-Policy Distillation for Speculative Draft Models

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration