RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding

Mingkang Liu; Weiming Meng; Peipei Jiao; Jiawei Zhang; Tianyuan Liu; Junyan Li; Lichen Pang

cs.CLOct 21, 2026

RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding

Authors: Mingkang Liu, Weiming Meng, Peipei Jiao, Jiawei Zhang, Tianyuan Liu, Junyan Li, Lichen Pang

Abstract

Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk. We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length. RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens. Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.

Figures & tables

Figure 1: Misalignment between KL divergence and acceptance rate. Due to limited parameters, the draft model cannot perfectly replicate the target distribution. However, low KL loss may still lead to token rejection. This discrepancy arises because speculative decoding acceptance depends on the relative ranking of top candidate tokens rather than exact distribution matching.

Figure 2: Output Throughput (TPS) and Mean Accept Length across model configurations. Evaluated on Llama-3.1-8B-Instruct with SGLang 5.5.0, the results demonstrate that SFT+DPO and SFT+GRPO consistently outperform the baseline SFT approach on both metrics.

Figure 3: Overview of the RCSD framework. Given requests and dataset samples, a draft model $\pi_{\theta}$ first proposes $k$ tokens via speculative decoding, and a target model $\pi_{T}$ performs parallel verification. The verification stage partitions outputs into accepted tokens (forming the chosen sequence $y^{+}$ ) and early-rejected tokens (forming the rejected sequence $y^{-}$ ), from which preference or reward signals are constructed. The resulting pairs $(x,y^{+},y^{-})$ or response groups $G$ are then used to update $\pi_{\theta}$ via DPO- or GRPO-based optimization. In DPO-based optimization, target-model preferences between $y^{+}$ and $y^{-}$ guide parameter updates. In GRPO-based optimization, grouped responses are optimized using an acceptance-rate reward.

Dataset	Model	Step 3	Step 4	Step 5	Step 6
HumanEval	HASS	2.322	2.446	2.510	2.533
	HASS + DPO	2.616	2.780	2.922	3.014
	HASS + GRPO	2.629	2.807	2.928	3.015
MT-Bench	HASS	1.986	2.047	2.072	2.092
	HASS + DPO	2.333	2.442	2.537	2.571
	HASS + GRPO	2.346	2.467	2.549	2.601

Table 1: Qwen3-1.7B: results at Steps 3–6. Baseline HASS, HASS+DPO, and HASS+GRPO are compared. Higher values indicate better acceptance length. HASS+GRPO generally achieves better performance.

Dataset	Model	Step 3	Step 4	Step 5	Step 6
HumanEval	HASS	2.731	2.950	3.103	3.202
	HASS + DPO	2.896	3.185	3.391	3.540
	HASS + GRPO	2.897	3.189	3.402	3.558
MT-Bench	HASS	2.282	2.397	2.438	2.497
	HASS + DPO	2.420	2.595	2.674	2.731
	HASS + GRPO	2.426	2.596	2.692	2.736

Table 2: Llama-3.1-8B: results at Steps 3–6. HASS+GRPO generally achieves better performance.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University
Acceptance-Aware Draft Model Training for Speculative Decoding
Sep 21, 2026Tianhua Xia, Mugilan Ganesan, Yifei Feng +3Speculative Decoding LLM Inference Acceleration
New York University · University of Waterloo · Independent Researcher
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas

Dec 29, 2025J/K move · Enter open · S save

Entropy-Aware Token Rejection for Improving Speculative Decoding

Tiancheng Su, Meicong Zhang, Guoxiu He

School of Economics and Management East China Normal University

Speculative Decoding LLM Inference Acceleration LLM Reasoning

Open paper AskPDF arXiv

Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronger target model to verify them. However, standard SD is mainly designed for acceleration, and its output quality is typically constrained by the target model. In this work, we propose Entropy-Aware Speculative Decoding (EASD), a lightweight and training-free extension of SD that improves reasoning quality through token-level entropy-guided rejection. EASD detects cases where both draft and target models exhibit high uncertainty while strongly overlapping in their top predictions. In such uncertain-agreement cases, EASD rejects the aligned token and resamples from the target distribution, preventing low-confidence errors from propagating. Experiments on challenging reasoning benchmarks show that EASD consistently improves accuracy over standard SD and reward-guided variants while maintaining comparable inference efficiency. Notably, EASD can surpass the standalone performance of the target model, suggesting that speculative decoding can serve not only as an acceleration method but also as an effective mechanism for improving reasoning quality. The code is available at https://github.com/ECNU-Text-Computing/EASD.

Abstract

Speculative Decoding

Explore similar work

Entropy-Aware Token Rejection for Improving Speculative Decoding

Acceptance-Aware Draft Model Training for Speculative Decoding

Beyond the Target: From Imitation to Collaboration in Speculative Decoding