Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling

Yongchang Hao; Lili Mou

cs.LGApr 5, 2026

Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling

Authors: Yongchang Hao, Lili Mou

Organizations: Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair

Abstract

Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that of the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier's distribution, such as sampling with top- $k$ or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation, we propose Cactus (constrained acceptance speculative sampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness of our approach.

Figures & tables

Table 1: The results on three benchmarks: GSM8K, IFEval, and GPQA. We report the “strict-match” accuracy as the score with the standard regex pattern for each task. AL m indicates the number of accepted tokens when the draft length is

m

. Rej denotes the total number of rejected tokens throughout generation in relative scale, where we use the SpS runs as the reference (labeled as “Ref”).

Figure 1: Accuracy-acceptance across benchmarks and model settings. The $x$ -axis shows the average accepted length (AL), and the $y$ -axis shows accuracy normalized by the standard deviation from the verifier.

Figure 2: Task score vs. acceptance rate for the 0.6B+14B Qwen 3 combination without top- $p$ /top- $k$ sampling arguments. Solid and dashed lines are degree-2 polynomial fits.

Figure 3: Wall-time normalized throughput ( $y$ -axis) across different model sizes and draft lengths. The wall time of a single verifier model is always normalized to 1.

Figure 4: Evaluating on GSM8K with three model pairs.

		GSM8K		IFEval		GPQA
$m$	Name	Score ↑	AL ${}_{m}^{\uparrow}$	Score ↑	AL ${}_{m}^{\uparrow}$	Score ↑	AL ${}_{m}^{\uparrow}$
10	SpS	95.30	5.03	83.36	2.61	40.40	3.73
	TAS	94.10	7.02	83.73	4.16	40.40	6.12
	Ours ( $\delta=1$ )	94.40	7.13	85.21	4.47	41.92	6.36

Table 2: The results of Qwen 3 32B as verifier and Qwen 3 1.7B as drafter on three benchmarks: GSM8K, IFEval, and GPQA. We report the “strict-match” accuracy and the acceptance length (AL).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

		GSM8K			IFEval			GPQA
$m$	Name	Score ↑	AL ${}_{m}^{\uparrow}$	Wall ↓	Score ↑	AL ${}_{m}^{\uparrow}$	Wall ↓	Score ↑	AL ${}_{m}^{\uparrow}$	Wall ↓
10	SpS	91.12	4.27	1.00x	85.03	2.19	1.00x	39.39	3.37	1.00x
	Mentored	91.66	4.51	1.20x	61.37	2.88	0.96x	40.91	4.31	0.93x
	SpecCas	88.40	6.42	0.85x	69.50	5.02	0.54x	32.83	6.27	0.68x
	TAS	92.65	5.24	0.86x	86.14	3.00	0.82x	38.89	4.99	0.72x
	Cactus 1	93.10	5.44	0.87x	85.96	3.03	0.78x	43.43	5.16	0.70x

Appendix

Table 3: The results with Qwen 3 14B as verifier and Qwen 3 0.6B as drafter.

	MT Bench	Trans.	Summ.	QA	Math	RAG	AL 10	Overall
SpS	2.01 $\times$	1.40 $\times$	1.92 $\times$	1.85 $\times$	1.83 $\times$	1.86 $\times$	3.20	1.81 $\times$
Cactus ( $\delta=1$ )	2.09 $\times$	1.40 $\times$	2.04 $\times$	1.95 $\times$	1.86 $\times$	1.92 $\times$	3.29	1.88 $\times$

Appendix

Table 4: Speedup comparison on Spec-Bench using Qwen 3 14B as the verifier and Qwen 3 0.6B as the drafter. We report the speedup ratio relative to standard autoregressive decoding. “Accepted” denotes the mean number of accepted tokens per step.

Draft Size	Score	AL	Rej
Verifier (Oracle)	91.71	-	-
0.6B	93.10	5.44	-32%
1.7B	92.50	6.78	-60%
4B	92.57	7.76	-76%

Appendix

Table 5: Ablation on GSM8K using Qwen 3 14B verifier with different drafter sizes (

\delta=1

Task

Question: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market?

\delta=1

<think>\nOkay , let me try to figure out how much Janet makes each day at the farmers’ market. Hmm, so the problem says she has ducks that lay 16 eggs per day. Then she eats three for breakfast and bakes muffins with four eggs each day. Then she sells the remaining eggs for

2 each. Alright, let’s break this down step by step.\n \nFirst , total eggs per day: 16 eggs. Then she uses some for breakfast and baking. Let’s see, for breakfast, she eats 3 eggs. Then for baking, she uses 4 eggs. So total eggs used each day would be 3 + 4 = 7 eggs. Therefore, the remaining eggs would be 16 - 7 = 9 eggs. Then she sells those 9 eggs at

2 each. So, how much does she make? 9 eggs multiplied by

2, which is 18 dollars.\n \nWait , let me check again. If she uses 3 eggs for breakfast and 4 for baking, that’s 7. 16 minus 7 is indeed 9. Then 9 times 2 is 18. Yeah, that seems right. So her daily earnings would be

18. Hmm, is there any possibility I made a mistake here? Maybe I should re-read the problem to make sure I didn’t miss anything.\n \nThe problem says she lays 16 eggs per day. Then she eats 3 for breakfast and bakes muffins each day with four. So the total used is 3+4=7. Subtract that from 16, which leaves 9. Multiply by

2 for selling, 9*2=18. Yep, that’s correct. I think that’s the answer.\n </think>\n \nJanet ’s ducks lay 16 eggs per day. She uses 3 eggs for breakfast and 4 eggs for baking muffins, totaling 7 eggs used. The remaining eggs are

16 - 7 = 9$. She sells these at

2 per egg, earning

9 \times 2 = 18$. \n \n ######## 18

\delta=10^{6}

<think>\nOkay , so let\’s try to figure out how much Janet makes every day at the farmers\’ market. Let me start by breaking down the problem step by step. [lengthy CoT omitted] so total eggs used is 4 * 4 = 16. Then, after eating 3 for breakfast, leftover is 16 - 3 = 13, then she sells them at 2 each: 13 * 2 =

26. \n \nYes , that seems to make sense. Therefore, the answer should be 26. But let me check if the question is asking "how much money does she make every day at the farmers\’ market?" So yes, selling the remainder at

2 per egg. Therefore, answer is

26. \n \nTherefore , the answer is 26, which in the previous problems they had 5. This is a different case. So I think that\’s correct.\n </think>\n \nAnswer : 26\n \nThe problem states that Janet\’s ducks lay 16 eggs per day. She eats 3 eggs for breakfast and 4 eggs to make muffins daily. After these expenses, the remaining eggs are

16 - 3 - 4 = 9

, which are sold at

2 per egg. Therefore, the total money she earns at the farmers\’ market is

9 \\times 2 = 18

. \n \nHowever , the phrase "bakes muffins for her friends every day with four" might imply that she uses 4 eggs per muffin, meaning she makes 4 muffins. If she uses 4 eggs each muffin, the total eggs consumed would be

4 \\times 4 = 16

. Since she eats 3, the remaining eggs are

16 - 3 = 13

, which are sold for

13 \\times 2 = 26

. Hence, the correct answer is 26. \n \n ######## 26

Appendix

Table 6: Generations for different values of

\delta

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference
Apr 15, 2026Xuwen Zhou, Fangxin Liu, Chao Wang +5Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Alibaba Cloud Computing
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaOn-Device Language Model Inference Speculative Decoding
University of California, San Diego
Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University

Apr 15, 2026J/K move · Enter open · S save

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Xuwen Zhou, Fangxin Liu, Chao Wang, Xiao Zheng+4

Shanghai Jiao Tong University · Alibaba Cloud Computing

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of "Frequency-Guided Candidate Selection and Probability-Guarded Acceptance," CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments.

Abstract

Speculative Decoding

Explore similar work

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Entropy-Aware Token Rejection for Improving Speculative Decoding