Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks. During rollout, agentic RL exhibits a workload pattern distinct from that of traditional RL: multi-turn interactions repeatedly introduce prefill requests, causing prefill and decode workloads to coexist throughout rollout. This workload makes prefill-decode (PD) disaggregation, a design widely used for online LLM serving, promising for speeding up agentic RL rollout. This paper shows that PD disaggregation can improve rollout efficiency by (a) mitigating interference among data-parallel attention modules, (b) enabling stage-specific parallelism, and (c) reducing end-to-end trajectory completion time. However, a fixed prefill-decode allocation may fail to sustain these benefits as workload composition varies across tasks and over time. We therefore present Prorsa, an agentic RL framework for prediction-guided prefill-decode resource allocation. Prorsa observes the lifecycle of every trajectory to forecast short-horizon prefill and decode demand. These forecasts then guide worker allocation decisions and trigger role switching of rollout engines, aligning prefill and decode capacity with anticipated workload demand. Across synchronous and asynchronous agentic RL settings, Prorsa achieves end-to-end rollout speedups of 1.11x-4.05x over prefill-decode colocation.
Figures & tables
Figure 1: Normalized cumulative prefill and decode tokens for different RL tasks.
Figure 2
Figure 3: Prefill-to-decode token ratios across and within steps with a maximum staleness of 8.
Figure 4: Prorsa architecture overview.
Figure 5: Overall performance of Prorsa and PD colocation.
Model
Qwen3-235B
DeepSeek-V3.2
MiMo-V2.5-Pro
Staleness
On-policy
2
8
On-policy
2
8
On-policy
2
8
PD colocation
133.88
169.14
176.55
713.05
723.96
939.54
253.17
198.08
252.36
Prorsa
78.98
86.96
80.40
119.47
123.21
128.12
195.35
158.90
216.42
(1.7 ×↓ )
(1.9 ×↓ )
(2.2 ×↓ )
(6.0 ×↓ )
(5.9 ×↓ )
(7.3 ×↓ )
(1.3 ×↓ )
(1.2 ×↓ )
(1.2 ×↓ )
Table 2: Average end-to-end trajectory completion time (seconds) on the general agent task.
Figure 7Figure 8
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Inter-step prefill-to-decode token ratios for the general agent and coding agent tasks over seven training steps.
Figure 11: Intra-step prefill-to-decode token ratios within the first training step, measured over 10-second windows.