cs.AIMar 19, 2026
SaveProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
Abstract
Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.
Figures & tables
Figure 1: Coupled vs. decoupled designs. Left: Existing frameworks often embed the full agentic rollout lifecycle inside the RL training stack. Right: ProRL Agent treats rollout as an independent HTTP service. The trainer submits rollout requests and receives completed trajectories and rewards, while the rollout server handles environment execution, tool use, evaluation, and inference coordination. This decoupled design improves resource isolation, portability, and extensibility.
| Frameworks | Training-Rollout Decoupled? | Rootless Sandbox? | Scaffold-Independent? |
| SkyRL-Agent ( Cao et al., 2025b ) | ✗ | ✗ | ✓ |
| VeRL-Tool ( Jiang et al., 2025 ) | ✗ | ✗ | ✓ |
| Agent Lightning ( Luo et al., 2025c ) | ✗ | ✗ | ✗ |
| rLLM ( Tan et al., 2025 ) | ✗ | ✗ | ✓ |
| GEM ( Liu et al., 2025b ) | ✗ | ✗ | ✓ |
| ProRL Agent (Ours) | ✓ | ✓ | ✓ |
Table 1: Comparison of ProRL Agent with existing frameworks for multi-turn agent RL. ProRL Agent decouples rollout from training, supports rootless sandboxing for shared HPC environments, and is independent of any specific training framework.
Figure 2: Overview of the ProRL Agent architecture. The system consists of three components. (1) Sandbox Environment : each rollout is executed inside a SingularityRuntime container and orchestrated via AgentHandler , which exposes three lifecycle methods including init() , run() , and eval() for environment setup, multi-turn agent execution, and reward scoring, respectively. (2) ProRL Agent Server : an HTTP service that manages rollouts through a three-stage asynchronous pipeline (INIT RUN EVAL) with independent worker pools, and maintains a min-heap LLM backend pool supporting dynamic registration and checkpoint swapping. (3) RL Trainer : any training framework (e.g., veRL, NeMo RL) interacts with the server solely via HTTP, submitting jobs via POST process and managing backends via add_llm_server and /cancel ; completed trajectories and rewards are returned to the trainer to update the policy.
Figure 3: Comparison of DAPO implementations ( ). Our efficient implementation optimizes worker synchronization, significantly reducing the idle time (waiting period) between rollout generations compared to the baseline batch-by-batch approach.
| Size | Model | Reproduced | Reported |
| 4B | Qwen3-4B-Instruct-2507 | 14.8 | – |
| ProRL Agent-4B (RL) | 21.2 | – | |
| 8B | Qwen3-8B | 9.6 | – |
| SkyRL-Agent-8B-v0 | – | 9.4 | |
| ProRL Agent-8B (RL) | 18.0 | – | |
| 14B | Qwen3-14B | 15.4 | – |
Table 2: Comparison of performance on SWE-Bench Verified across models of different scales. We report the reproduced performance and, where available, the reported results from prior work. Across all model sizes
Figure 4: Training curves for ProRL Agent across three agent domains. From left to right: mean reward during RL training of the STEM agent, Pass@1 on AMC during RL training of the math agent, and Pass@1 on Codeforces during RL training of the code agent. All three curves show steady improvement during training, demonstrating the generality of ProRL Agent beyond software engineering tasks.
| Load Balancing | Efficient Bash | Stale Job Cleanup | Action Time (s) | GPU Util (%) | Throughput (instance/sec) |
| ✓ | ✓ | ✓ | 0.42 | 78 | 0.37 |
| ✓ | ✓ | 0.42 | 42 | 0.25 | |
| ✓ | ✓ | 0.78 | 68 | 0.29 | |
| ✓ | ✓ | 0.42 | 65 | 0.30 |
Table 3: Ablation study of the proposed system components. Action Time denotes the average time required to execute shell-command actions. Each component improves rollout throughput, either by increasing GPU utilization or by reducing action execution time.
Figure 5: Rollout throughput (instances/sec) on software engineering tasks versus the number of compute nodes. The near-linear increase in throughput demonstrates that ProRL Agent scales efficiently with additional compute resources.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: ProRL Agent separates the full agentic rollout lifecycle, spanning environment management to reward computation, from GPU-intensive training, thereby decoupling I/O-intensive rollout from training.
Figure 7: SkyRL-Agent. The training driver runs concurrent trajectory-generation coroutines on a single CPU process. It controls the multi-turn agent loop, queries a remote vLLM server for inference, and interacts with remote environment containers for execution. Although inference and environment execution are offloaded, rollout control remains inside the training driver.
Figure 8: Agent Lightning. Agent Lightning places the training loop, the LightningStoreServer , and all rollout workers within a single process tree. The store runs as a background thread, while rollout workers are spawned as child processes from the trainer. As a result, rollout does not have an independent service lifecycle: if the training process terminates, the store also stops and the rollout workers are disrupted. Thus, rollout remains managed within the training stack rather than being cleanly decoupled.
Figure 9: VeRL-Tool. VeRL-Tool extends the standard veRL trainer to support multi-turn agent rollouts. The training system manages the agent loop and trajectory collection, while tool execution is offloaded to a separate CPU-based environment service. In this design, rollout control remains inside the trainer.
Figure 10: rLLM: rollout embedded in a monolithic training driver. rLLM is built on a heavily modified fork of veRL. The agent loop, environment management, and trajectory orchestration all reside within a single driver process. There is no independent rollout service, no persistent trajectory buffer, and no possibility of the rollout surviving independently of the training driver. The full rollout lifecycle remains tightly coupled with the training stack.
Figure 11: GEM. GEM keeps environment execution inside the training process. Environments are instantiated as in-memory Python objects, and environment stepping is performed through direct env.step() calls, with parallelism provided only by ThreadPoolExecutor threads in the same address space. A single driver process orchestrates both rollout and training, while GPU workers are accessed remotely via Ray RPC. As a result, the environment and rollout lifecycle remain fully embedded in the training stack.
More in this topic
LLM Agent Training
Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.
See all 564 papers