PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning

Jiaan Zhu; Wei Gao; Youhui Bai; Zewen Jin; Ju Huang; Siran Yang; Jiamang Wang; Lin Qu

cs.AISep 28, 2026

PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning

Authors: Jiaan Zhu, Wei Gao, Youhui Bai, Zewen Jin, Ju Huang, Siran Yang, Jiamang Wang, Lin Qu, +1 more

Organizations: USTC China

Abstract

Multi-turn rollout dominates the cost of agentic reinforcement learning (RL). Asynchronous execution and elastic GPU resources can accelerate this stage, but adding rollout replicas yields diminishing returns while training GPUs remain idle between updates. We observe that effective resource use also depends on the prefill--decode (PD) configuration. Both the choice between colocation and disaggregation and the optimal PD ratio vary with the workload, making resource scaling and PD configuration interdependent. Exploiting this opportunity requires selecting effective configurations and realizing their benefits within transient resource-availability windows despite reconfiguration costs. We present PEARL, an asynchronous agentic RL system that coordinates external resource elasticity, temporary reuse of idle training GPUs, and adaptive PD execution. PEARL maintains a unified GPU--worker--role state and uses runtime profiles to predict rollout batch completion time, accounting for environment-induced reductions in decode concurrency. It selects the PD mode and ratio under the current GPU budget and translates each decision into an incremental transition plan that minimizes worker and role changes. Cost-aware switching and borrowing policies suppress transitions with insufficient expected benefit while ensuring timely return of training GPUs. Our evaluation show that PEARL achieves $2.17$ -- $2.79\times$ the throughput of fixed-resource ROLL across different LLMs. Compared with RLBoost+, throughput improves by up to approximately 26.9% for Qwen3-8B and 36.3% for Qwen3-30B-A3B.

Figures & tables

Figure 1 . Agentic RL workflow and multi-turn rollout.

Panel of Figure 2 — Figure 2 . Computational characteristics of agentic RL workflows.(a) Rollout and training durations for Qwen3-8B as rollout resources increase from 16 to 32 GPUs, with training fixed at 8 GPUs.(b) Generation throughput of Qwen3-30B-A3B during multi-turn rollout. (c) Rollout throughput across batch sizes and PD ratios for Qwen3-30B-A3B.

Category	System	Adaptation		PD Configuration
Category	System	Resource	Workload	Mode	P/D Ratio
Fixed & Async RL	AReaL ( Fu et al., 2025 )	\ding 55	\ding 55	Colocated	–
	ROLL ( Wang et al., 2025b )	\ding 55	\ding 55	Colocated	–
	Laminar ( Sheng et al., 2026 )	\ding 55	\ding 55	Colocated	–
Elastic RL	RLBoost ( Wu et al., 2026 )	Inter-scale	\ding 51	Colocated	–
	ROSE ( Gao et al., 2026a )	Inter-scale	\ding 51	Colocated	–

Table 1 . Comparison of rollout deployment adaptation and PD configurations in representative RL systems. Resource adaptation denotes inter-scale elasticity and intra-scale training-GPU reuse; workload adaptation denotes deployment changes in response to workload.

Figure 4 . Dedicated rollout GPU availability and the three two-hour segments used in our experiments. The count includes 4 reserved GPUs and up to 12 preemptible GPUs; the 16 reserved training GPUs are excluded.

Figure 5 . Average system throughput for each model and trace segment. Bar annotations are normalized to ROLL. RLBoost+ shares PEARL ’s external rollout resource availability and training GPU allocation; ROLL-Large uses a fixed, larger reserved rollout pool.

Figure 6 . Per-step system throughput over three segments. The stacked background shows the shared physical GPU budget: 16 reserved training GPUs, 4 reserved rollout GPUs, and up to 12 preemptible rollout GPUs. PEARL and RLBoost+ can utilize all the GPUs, while ROLL uses the reserved allocation and ROLL-Large uses a fixed rollout pool of 14, 10, or 12 GPUs.

Figure 7 . Ablation study with 16 training GPUs and fixed allocations of 4–16 rollout GPUs. Throughput is normalized to ROLL. A: optimal PD-disaggregation ratio; B: ungated intra-scale toggle; C: benefit-aware intra-scale gate. PEARL combines all three components.

Figure 8 . Workload sensitivity. (a) Rollout batch size varies with the maximum trajectory length fixed at 32K tokens. (b) Maximum trajectory length varies with the rollout batch size fixed at 64.

Dedicated GPUs	Overlap	Selected	Empirical best
10	No	$2P3D$	$2P3D$
10	Yes	$\boldsymbol{4P9D}$	$\boldsymbol{5P8D}$
12	No	$3P3D$	$3P3D$
12	Yes	$4P10D$	$4P10D$
16	No	$3P5D$	$3P5D$
16	Yes	$4P12D$	$4P12D$

Table 2. PD-ratio selection for Qwen3-8B.

Operation	Qwen3-8B	Qwen3-30B-A3B
Inter-scale preparation	56.29	67.06
Inter-scale finalization	11.05	17.89
Intra-scale activation	6.26	6.09
Intra-scale deactivation	3.18	4.38

Table 3. Mean reconfiguration latency in seconds.

Explore similar work

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
Aug 11, 2026Yanyu Ren, Xizheng Wang, Xiao Liu +8
Tsinghua University · Zhongguancun Laboratory
Libra: Efficient Resource Management for Agentic RL Post-Training
Jun 2, 2026Kaiwen Chen, Xin Tan, Jingzong Li +6Agentic Reinforcement Learning Resource-Efficient
The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher
WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
Jul 19, 2026Ryan Xu, Atlas Zhao, David Bao +1Agentic Reinforcement Learning Rollout

Aug 11, 2026, cs.LGJ/K move · Enter open · S save

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

Yanyu Ren, Xizheng Wang, Xiao Liu, Bowen Lv+7

Tsinghua University · Zhongguancun Laboratory

Open paper AskPDF arXiv

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state,

\textrm{RA}^2\textrm{P}

selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6

\times

over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58

\times

, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.

Abstract

Explore similar work

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

Libra: Efficient Resource Management for Agentic RL Post-Training

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning