S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

Yuhang He; Hongyu Ge; Zihan Lin; Siyi Liu; Yongqi Zhang

cs.LGOct 9, 2026

S $^3$ R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

Authors: Yuhang He, Hongyu Ge, Zihan Lin, Siyi Liu, Yongqi Zhang

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72--76% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (S $^3$ R). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under S $^3$ R, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S $^3$ R maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, S $^3$ R achieves $1.39\times~1.35\times$ the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.

Figures & tables

Panel of Figure 1 — Figure 1: ( 1(a) ) Attention accounts for 72–76% of AR-decoding kernel time as KV volume grows (Full results in Appendix F.2.1 ). ( 1(b) ) Both mild and aggressive sparsity exhibit training collapse without training-side mismatch correction. ( 1(c) ) Cumulative KV pages recomputed or transferred; our scheduler substantially reduces repeated KV reconstruction and transfer.

Figure 2: Overview of S 3 R’s rollout mechanism and runtime support. Sparse drafting reduces RL rollout cost, while dense verification preserves rollout fidelity with KV-aware runtime support.

Figure 3: Trade-off costs on Qwen3-8B (

s=0.05

k=6

Model	Method	Benchmarks		Efficiency
Model	Method	AIME26	ArXiv-Math	$\tau$	Rollout (min) $\downarrow$	Step (min) $\downarrow$	Throughput/Speedup
Qwen3-1.7B	AR	37.2	9.95	N/A	10.5 (base)	13.9 (base)	2308.6 / $1.00\times$
	N-gram Lookup	37.1	9.84	1.31	12.5 ( $+19.1\%$ )	19.0 ( $+36.3\%$ )	1671.0 / $0.72\times$
	History Retrieval	36.0	9.77	2.03	10.4 ( $-1.7\%$ )	13.3 ( $-4.2\%$ )	2412.7 / $1.05\times$
	Auxiliary Proposer	36.7	9.87	1.89	9.1 ( $-13.9\%$ )	12.4 ( $-10.5\%$ )	2562.0 / $1.11\times$
	Ours	37.1	9.89	5.59	6.5 (-38.6%)	9.9 (-28.5%)	3213.8 / 1.39 $\times$

Table 1: Main results on Qwen3-1.7B and Qwen3-8B. Benchmark scores are pass@1 (%) from the best checkpoint. Timing and throughput cover the full training run after the first warm-up step. Best and second-best results within each model size are shown in bold and underlined, respectively.

Panel of Figure 4 — Figure 4: Training stability under sparse rollout variants. ( 4(a) ) Reward and gradient norm for S 3 R and Direct sparse + mismatch correction; ( 4(b) ) reward across sparse budgets.

Panel of Figure 5 — Figure 5: Qwen3-8B rollout diagnostics during RL training. ( 5(a) ) End-to-end training throughput. ( 5(b) ) Generation time per processed token (prompt and response) versus draft-token acceptance; outlined markers denote method medians. ( 5(c) ) Policy entropy and relative change in draft-token acceptance.

Figure 6: Qwen3-8B budget sweep at a frozen step-60 checkpoint (

B=64

, initial context 8K,

k=6

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

State	Before actor update	Before the next rollout
Requests and KV	Release request state and KV resources.	Start new requests with KV capacity fixed for the rollout.
Rollout model	Retain parameter buffers on GPU.	Synchronize the updated actor snapshot into these buffers.
CUDA graphs	Discard captures tied to released KV storage; retain the planned batch sizes.	Recapture the planned sizes for graph replay during generation.

Appendix

Table 3: Resource lifetimes across the rollout and actor-update phases.

Panel of Figure 7 — Figure 7: Checkpoint portability and periodic online drafter adaptation. ( 7(a) ) Draft-token acceptance and ( 7(b) ) generation speedup relative to paired AR across target checkpoints; error bars summarize placement variability checked against additional runs. ( 7(c) ) Mean acceptance length during periodic co-training, with faint observations and bold seven-step rolling means.

Panel of Figure 8 — Figure 8: Reward and response-length trajectories of AR and S 3 R across training steps on Qwen3-1.7B and Qwen3-8B.

Panel of Figure 9 — Figure 9: S 3 R draft-token acceptance across training checkpoints for ( 9(a) ) Qwen3-1.7B and ( 9(b) ) Qwen3-8B. Cells report acceptance at each draft position.

Panel of Figure 10 — Figure 10: Qwen3-1.7B rollout diagnostics. ( 10(a) ) End-to-end training throughput; ( 10(b) ) generation time per processed token versus draft-token acceptance; outlined markers denote method medians.

Item	Setting
Model	Qwen3-8B, DAPO step-60 checkpoint
Hardware	$2\times$ NVIDIA H100 80GB HBM3
Parallelism	one rollout-engine replica, TP ${=}2$
Precision	BF16
Workload	equal-length synthetic prompts, fixed concurrency, ignore EOS
Sampling	temperature 0 (deterministic systems workload)

Appendix

Table 4: Settings for the component measurements in Tables 5 – 7 and Figure 11 .

$N_{\mathrm{KV}}$	$S$	$B$	Attention	FFN	QKVO	LM head	Other
8K	2K	4	0.96 (19.4%)	2.18 (43.8%)	0.83 (16.6%)	0.21 (4.1%)	0.80 (16.1%)
8K	4K	2	0.93 (18.8%)	2.18 (44.2%)	0.82 (16.6%)	0.20 (4.2%)	0.80 (16.2%)
8K	8K	1	0.98 (19.8%)	2.17 (43.7%)	0.82 (16.5%)	0.20 (4.1%)	0.79 (16.0%)
16K	2K	8	1.16 (22.4%)	2.19 (42.1%)	0.83 (16.0%)	0.21 (4.0%)	0.81 (15.6%)
64K	4K	16	2.34 (36.2%)	2.24 (34.6%)	0.85 (13.2%)	0.21 (3.3%)	0.82 (12.7%)
128K	8K	16	3.85 (48.4%)	2.23 (28.1%)	0.85 (10.7%)	0.21 (2.6%)	0.82 (10.3%)

Appendix

Table 5: Full-attention AR decoding costs: ms per iteration (share of GPU kernel time), excluding TP collectives. The upper rows cover small-KV rollout-tail workloads; the lower rows show the transition to attention-dominated decoding. QKVO denotes query, key, value, and output projections.

Figure 11: Attention-to-FFN kernel latency ratio for full-attention AR decoding across batch sizes and context lengths. The black contour marks equal latency; hatched cells exceed the measured KV-capacity limit.

$B$	$S$	$N_{\mathrm{KV}}$	Full attention	S 3 R attention	Reduction
16	4K	64K	2.31	1.42	38.3%
16	8K	128K	3.83	1.89	50.6%
16	16K	256K	6.82	2.79	59.2%
32	4K	128K	3.80	1.96	48.5%
32	8K	256K	6.83	2.85	58.3%
32	16K	512K	12.91	4.68	63.7%

Appendix

Table 6: Attention time in AR and S 3 R, in ms per scheduler iteration per TP rank (median over three runs). S 3 R includes sparse drafting and dense verification, averaged over the full window, including idle iterations. These component savings must still be weighed against accepted output and other execution costs.

$B$	$S$	$N_{\mathrm{KV}}$	AR ( $7T_{p}$ )	Draft ( $6T_{q}$ )	Verify ( $T_{V}$ )	Total	$\alpha_{\mathrm{BE}}$
16	4K	64K	54.42	40.06	28.06	68.12	$>100\%$
16	8K	128K	65.55	39.15	27.97	67.12	$>100\%$
16	16K	256K	88.51	42.38	28.10	70.48	76.2%
32	4K	128K	66.70	40.89	29.03	69.92	$>100\%$
32	8K	256K	88.12	43.01	26.86	69.87	75.8%
32	16K	512K	133.18	43.73	28.56	72.29	46.7%

Appendix

Table 7: Drafting and verification costs for

k=6

(ms, medians over three runs). The AR reference

7T_{p}

generates as many tokens as a fully accepted round.

\alpha_{\mathrm{BE}}

is the acceptance needed to break even under sequential stage accounting. AR and drafting use CUDA Graphs; verification runs eagerly. The summed costs exclude idle iterations and cross-request phase mixing.

Panel of Figure 12 — Figure 12: Sparse-budget sweeps for ( 12(a) ) Qwen3-1.7B and ( 12(b) ) Qwen3-8B at $B=64$ , initial $S=8$ K, and $k=6$ . Black circles show median measured speedup; squares show estimates from Eq. 24 at the same runs’ measured acceptance. Orange diamonds mark budgets excluded from fitting. Blue stars show separate RL observations, discussed in Appendix F.3 .

Panel of Figure 13 — Figure 13: Estimated speedup over sparse budget and acceptance for ( 13(a) ) Qwen3-1.7B and ( 13(b) ) Qwen3-8B at $B=64$ , $S\approx 8$ K, and $k=6$ . The background evaluates Eq. 24 ; the dark contour marks $1\times$ speedup. Circles and diamonds show calibration and held-out observations. Blue stars locate separate RL observations at $s=0.05$ ; their annotations report measured speedups.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 311 papers

Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models
Jun 7, 2026Yang Zhou, Ranajoy Sadhukhan, Zhaofeng Sun +7RL for Language Model Reasoning Sparse Attention
Carnegie Mellon University · Cornell University · Intel +1
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Jun 17, 2026Minseo Kim, Minjae Lee, Seunghyuk Oh +7RL for Language Models Reinforcement Learning
FuriosaAI · University of California, Berkeley
SPARK: Learning from Verification in Speculative GRPO Rollouts
Aug 5, 2026Nhat Minh Pham, Duy Tung Doan, Quang Huy Phung +3RL for Language Model Reasoning Speculative Decoding
VNU University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam · Viettel AI, Viettel Group, Hanoi, Vietnam

Jun 7, 2026J/K move · Enter open · S save

Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models

Yang Zhou, Ranajoy Sadhukhan, Zhaofeng Sun, Zhuoming Chen+6

Carnegie Mellon University · Cornell University · Intel+1

RL for Language Model Reasoning Sparse Attention Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Despite being powerful, reinforcement learning with verifiable rewards (RLVR) induces extremely long COT, making it computationally expensive. Since RLVR per-step cost is dominated by long-context rollout generation, sparse attention offers a promising way to accelerate dense rollout. However, sparse rollouts require a delicate stability-efficiency tradeoff: overly aggressive sparsity causes collapse, while overly lenient sparsity gives insufficient speedup. In this work, we study this tradeoff through sparse-to-dense actor-policy mismatch. We first observe that sparse rollout collapse is not driven by uniform degradation across tokens: most sparse tokens align perfectly with dense even under aggressive sparsity. Motivated by this, we hypothesize that sparse rollout training remains stable if the lower tail of per-token actor-policy mismatch stays above a critical threshold throughout the trajectory. We introduce a dynamic sparsity schedule that keeps this tail statistic constant during generation and validate our hypothesis. Across Qwen3 thinking-family models, keeping the tail mismatch statistic near a consistent threshold generally enables stable training. We then use a cost model to find the sparsity schedule for maximum speedup under this mismatch threshold, achieving 2.2x, 2.4x, and 2.0x rollout speedups when training Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Empirically, we show the thresholds generalize to a larger model (Qwen3-14B) and another RL domain (coding). Finally, our analysis naturally motivates DistillSparse: lightweight LoRA-based distillation on sparse rollout lets more aggressive sparsity reach the same sparse-to-dense mismatch threshold, yielding higher speedup.

Abstract

Speculative Decoding

Explore similar work

Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

SPARK: Learning from Verification in Speculative GRPO Rollouts