Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72--76% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (S3R). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under S3R, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S3R maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, S3R achieves 1.39×1.35× the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.
Figures & tables
Figure 1: ( 1(a) ) Attention accounts for 72–76% of AR-decoding kernel time as KV volume grows (Full results in Appendix F.2.1 ). ( 1(b) ) Both mild and aggressive sparsity exhibit training collapse without training-side mismatch correction. ( 1(c) ) Cumulative KV pages recomputed or transferred; our scheduler substantially reduces repeated KV reconstruction and transfer.
Figure 2: Overview of S 3 R’s rollout mechanism and runtime support. Sparse drafting reduces RL rollout cost, while dense verification preserves rollout fidelity with KV-aware runtime support.
Table 1: Main results on Qwen3-1.7B and Qwen3-8B. Benchmark scores are pass@1 (%) from the best checkpoint. Timing and throughput cover the full training run after the first warm-up step. Best and second-best results within each model size are shown in bold and underlined, respectively.
Figure 4: Training stability under sparse rollout variants. ( 4(a) ) Reward and gradient norm for S 3 R and Direct sparse + mismatch correction; ( 4(b) ) reward across sparse budgets.
Figure 5: Qwen3-8B rollout diagnostics during RL training. ( 5(a) ) End-to-end training throughput. ( 5(b) ) Generation time per processed token (prompt and response) versus draft-token acceptance; outlined markers denote method medians. ( 5(c) ) Policy entropy and relative change in draft-token acceptance.
Figure 6: Qwen3-8B budget sweep at a frozen step-60 checkpoint ( B=64 , initial context 8K, k=6 ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
State
Before actor update
Before the next rollout
Requests and KV
Release request state and KV resources.
Start new requests with KV capacity fixed for the rollout.
Rollout model
Retain parameter buffers on GPU.
Synchronize the updated actor snapshot into these buffers.
CUDA graphs
Discard captures tied to released KV storage; retain the planned batch sizes.
Recapture the planned sizes for graph replay during generation.
Appendix
Table 3: Resource lifetimes across the rollout and actor-update phases.
Figure 7: Checkpoint portability and periodic online drafter adaptation. ( 7(a) ) Draft-token acceptance and ( 7(b) ) generation speedup relative to paired AR across target checkpoints; error bars summarize placement variability checked against additional runs. ( 7(c) ) Mean acceptance length during periodic co-training, with faint observations and bold seven-step rolling means.
Figure 8: Reward and response-length trajectories of AR and S 3 R across training steps on Qwen3-1.7B and Qwen3-8B.
Figure 9: S 3 R draft-token acceptance across training checkpoints for ( 9(a) ) Qwen3-1.7B and ( 9(b) ) Qwen3-8B. Cells report acceptance at each draft position.
Figure 10: Qwen3-1.7B rollout diagnostics. ( 10(a) ) End-to-end training throughput; ( 10(b) ) generation time per processed token versus draft-token acceptance; outlined markers denote method medians.
Item
Setting
Model
Qwen3-8B, DAPO step-60 checkpoint
Hardware
2× NVIDIA H100 80GB HBM3
Parallelism
one rollout-engine replica, TP =2
Precision
BF16
Workload
equal-length synthetic prompts, fixed concurrency, ignore EOS
Sampling
temperature 0 (deterministic systems workload)
Appendix
Table 4: Settings for the component measurements in Tables 5 – 7 and Figure 11 .
NKV
S
B
Attention
FFN
QKVO
LM head
Other
8K
2K
4
0.96 (19.4%)
2.18 (43.8%)
0.83 (16.6%)
0.21 (4.1%)
0.80 (16.1%)
8K
4K
2
0.93 (18.8%)
2.18 (44.2%)
0.82 (16.6%)
0.20 (4.2%)
0.80 (16.2%)
8K
8K
1
0.98 (19.8%)
2.17 (43.7%)
0.82 (16.5%)
0.20 (4.1%)
0.79 (16.0%)
16K
2K
8
1.16 (22.4%)
2.19 (42.1%)
0.83 (16.0%)
0.21 (4.0%)
0.81 (15.6%)
64K
4K
16
2.34 (36.2%)
2.24 (34.6%)
0.85 (13.2%)
0.21 (3.3%)
0.82 (12.7%)
128K
8K
16
3.85 (48.4%)
2.23 (28.1%)
0.85 (10.7%)
0.21 (2.6%)
0.82 (10.3%)
Appendix
Table 5: Full-attention AR decoding costs: ms per iteration (share of GPU kernel time), excluding TP collectives. The upper rows cover small-KV rollout-tail workloads; the lower rows show the transition to attention-dominated decoding. QKVO denotes query, key, value, and output projections.
Figure 11: Attention-to-FFN kernel latency ratio for full-attention AR decoding across batch sizes and context lengths. The black contour marks equal latency; hatched cells exceed the measured KV-capacity limit.
B
S
NKV
Full attention
S 3 R attention
Reduction
16
4K
64K
2.31
1.42
38.3%
16
8K
128K
3.83
1.89
50.6%
16
16K
256K
6.82
2.79
59.2%
32
4K
128K
3.80
1.96
48.5%
32
8K
256K
6.83
2.85
58.3%
32
16K
512K
12.91
4.68
63.7%
Appendix
Table 6: Attention time in AR and S 3 R, in ms per scheduler iteration per TP rank (median over three runs). S 3 R includes sparse drafting and dense verification, averaged over the full window, including idle iterations. These component savings must still be weighed against accepted output and other execution costs.
B
S
NKV
AR ( 7Tp )
Draft ( 6Tq )
Verify ( TV )
Total
αBE
16
4K
64K
54.42
40.06
28.06
68.12
>100%
16
8K
128K
65.55
39.15
27.97
67.12
>100%
16
16K
256K
88.51
42.38
28.10
70.48
76.2%
32
4K
128K
66.70
40.89
29.03
69.92
>100%
32
8K
256K
88.12
43.01
26.86
69.87
75.8%
32
16K
512K
133.18
43.73
28.56
72.29
46.7%
Appendix
Table 7: Drafting and verification costs for k=6 (ms, medians over three runs). The AR reference 7Tp generates as many tokens as a fully accepted round. αBE is the acceptance needed to break even under sequential stage accounting. AR and drafting use CUDA Graphs; verification runs eagerly. The summed costs exclude idle iterations and cross-request phase mixing.
Figure 12: Sparse-budget sweeps for ( 12(a) ) Qwen3-1.7B and ( 12(b) ) Qwen3-8B at B=64 , initial S=8 K, and k=6 . Black circles show median measured speedup; squares show estimates from Eq. 24 at the same runs’ measured acceptance. Orange diamonds mark budgets excluded from fitting. Blue stars show separate RL observations, discussed in Appendix F.3 .
Figure 13: Estimated speedup over sparse budget and acceptance for ( 13(a) ) Qwen3-1.7B and ( 13(b) ) Qwen3-8B at B=64 , S≈8 K, and k=6 . The background evaluates Eq. 24 ; the dark contour marks 1× speedup. Circles and diamonds show calibration and held-out observations. Blue stars locate separate RL observations at s=0.05 ; their annotations report measured speedups.
Despite being powerful, reinforcement learning with verifiable rewards (RLVR) induces extremely long COT, making it computationally expensive. Since RLVR per-step cost is dominated by long-context rollout generation, sparse attention offers a promising way to accelerate dense rollout. However, sparse rollouts require a delicate stability-efficiency tradeoff: overly aggressive sparsity causes collapse, while overly lenient sparsity gives insufficient speedup. In this work, we study this tradeoff through sparse-to-dense actor-policy mismatch. We first observe that sparse rollout collapse is not driven by uniform degradation across tokens: most sparse tokens align perfectly with dense even under aggressive sparsity. Motivated by this, we hypothesize that sparse rollout training remains stable if the lower tail of per-token actor-policy mismatch stays above a critical threshold throughout the trajectory. We introduce a dynamic sparsity schedule that keeps this tail statistic constant during generation and validate our hypothesis. Across Qwen3 thinking-family models, keeping the tail mismatch statistic near a consistent threshold generally enables stable training. We then use a cost model to find the sparsity schedule for maximum speedup under this mismatch threshold, achieving 2.2x, 2.4x, and 2.0x rollout speedups when training Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Empirically, we show the thresholds generalize to a larger model (Qwen3-14B) and another RL domain (coding). Finally, our analysis naturally motivates DistillSparse: lightweight LoRA-based distillation on sparse rollout lets more aggressive sparsity reach the same sparse-to-dense mismatch threshold, yielding higher speedup.
Yang Zhou, Ranajoy Sadhukhan, Zhaofeng Sun +7
Carnegie Mellon University · Cornell University · Intel +1
Reinforcement learning (RL) has become a representative post-training paradigm for large language models (LLMs), enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive (AR) sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 24.2% and 15.6%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.
Group Relative Policy Optimization (GRPO) has become a standard approach for improving the reasoning abilities of large language models, but its training is costly because each step requires generating many long reasoning responses. Recent methods use speculative decoding to accelerate the rollouts, where a draft model proposes tokens that the target model verifies. However, we find that the draft model repeatedly misses the same tokens within a response and across responses in the same GRPO group, and the verification feedback on these misses is not used to update the draft during the rollout. Motivated by this analysis, we introduce SPARK, a lightweight self-adaptive mechanism designed to enhance draft models used in speculative GRPO training. SPARK leverages target verification feedback to refine draft predictions during generation while preserving their existing training procedures. Averaged across 21 model-dataset settings, SPARK improves generation speedup by 32.7% and 11.4%, and end-to-end GRPO training speedup by 32.2% and 8.5%, over EAGLE3-style and DFlash-style baselines, respectively.
Nhat Minh Pham, Duy Tung Doan, Quang Huy Phung +3
VNU University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam · Viettel AI, Viettel Group, Hanoi, Vietnam