SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Dayang Liang; Lang Feng; Bo An; Yunlong Liu

cs.AIAug 20, 2026

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Authors: Dayang Liang, Lang Feng, Bo An, Yunlong Liu

Organizations: Xiamen University, China

Abstract

Agentic reinforcement learning (RL) has emerged as an important post-training approach for enhancing the capabilities of Large Language Models (LLMs). However, existing methods face a trade-off between policy performance and resource efficiency. Conventional Proximal Policy Optimization (PPO) implementations incur substantial memory overhead from a separate critic, whereas critic-free group-relative methods require multiple rollouts and face potential learning bottlenecks on long-horizon tasks. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), an efficient PPO-style framework that unifies policy optimization and value learning within a single causal language model. SAPO exploits the autoregressive structure of LLMs to sequentially generate action and value estimation at distinct causal boundaries with shared parameters, and then jointly optimizes the PPO objectives and an auxiliary on-policy SARSA objective with turn-level generalized advantage estimation, where the latter is designed to facilitate value learning. Extensive experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B and Qwen3-14B demonstrate that SAPO reduces peak GPU memory usage by 23.1% and per-iteration runtime by 24.8% over strong PPO baseline, while matching or slightly improving task success rate. Our experiments also show that SAPO outperforms Group Relative Policy Optimization (GRPO) and recent cutting-edge variants in both task success and training stability.

Figures & tables

Figure 1: Overview of SAPO. Left : Value and policy autoregressive processes. A shared autoregressive language model generates actions and reads $V$ and $Q$ from logits at the pre- and post-action boundaries. Right : Loss structure and training pipeline. One rollout per task supplies turn-level value $V$ targets, on-policy SARSA $Q$ target and advantages, which are broadcast to response tokens for policy optimization.

Type	Method	ALFWorld							WebShop
Type	Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-Source Model
Prompting	GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Prompting	Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Qwen2.5-1.5B-Instruct
Prompting	ReAct	17.4	20.5	15.7	6.2	7.7	2.0	12.8	40.1	11.3

Table 1: Performance results on ALFWorld and WebShop. For each RL training method, we report the mean and standard deviation over three random seeds. The ALFWorld contains six categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). The baseline GiGPO w/o std replaces the group std with one ( Feng et al., 2026 ) .

Figure 2: Computational resource comparison between SAPO and PPO using Qwen2.5-1.5B and Qwen2.5-7B on ALFWorld and WebShop under matched experimental settings. The left panel reports the per-iteration runtime breakdown across training stages in seconds on a logarithmic scale, where N/A denotes stages not used by SAPO. The right panel reports peak GPU memory usage in GB during training on NVIDIA H200 GPUs; the 1.5B experiments use one GPU and the 7B experiments use two GPUs.

Method	ALFWorld
Method	Pick	Clean	Cool	Look	Heat	Pick2	All
SAPO (1.5B)	96.4 ±1.4	81.2 ±5.1	100.0 ±0.0	100.0 ±0.0	97.2 ±3.2	87.7 ±5.1	94.6 ±1.4
$-\cal{L}_{Q}$ (1.5B)	100.0 ±0.0	81.6 ±13.3	98.6 ±2.4	100.0 ±0.0	85.3 ±9.4	84.3 ±5.6	92.2 ±0.8
$-Norm$ (1.5B)	97.1 ±2.7	84.1 ±14.1	91.7 ±10.3	96.7 ±4.5	84.8 ±7.4	83.6 ±8.6	89.1 ±3.4
SAPO (7B)	98.8 ±1.6	85.5 ±11.2	99.1 ±2.0	100.0 ±0.0	95.6 ±4.7	92.3 ±8.2	96.1 ±1.1
$-\cal{L}_{Q}$ (7B)	95.9 ±4.4	82.7 ±4.3	96.2 ±6.6	97.4 ±4.4	82.3 ±5.4	87.4 ±11.0	91.2 ±2.0

Table 2: Ablation study comparing variants without action value learning loss (

-\cal{L}_{Q}

) and advantage normalization (

-Norm

). The best performance under the same setting is highlighted in bold. We report the average success rate (%) from three seeds.

Figure 3: Pearson correlations of predicted $Q$ (left) and $V$ (right) values with their corresponding Monte Carlo return estimates.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The prompt template for ALFWorld environment.

Panel of Figure 5 — Figure 5: The prompt template for WebShop environment.

Figure 6: Case study of SAPO on ALFWorld using Qwen2.5-1.5B-Instruct.

Figure 7: Case study of SAPO on ALFWorld using Qwen2.5-1.5B-Instruct.

Figure 8: Case study of SAPO on ALFWorld using Qwen2.5-1.5B-Instruct.

Figure 9: Case study of SAPO on ALFWorld using Qwen2.5-1.5B-Instruct.

Explore similar work

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic Learning Autoregressive Rollout
Tsinghua University · Work done while ZH and YL interned at Z.AI.
Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning
Jul 15, 2026Bowei He, Yankai Chen, Xiaokun Zhang +1Frictive Policy Optimization Offline Reinforcement Learning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · McGill Univeristy, Montreal, Canada · City University of Hong Kong, Hong Kong SAR.
3SPO: State-Score-Supervised Policy Optimization for LLM Agents
Jun 8, 2026Yu Han, Kailing Li, Yang Jiao +4Large Language Model Reinforcement Learning Offline Reinforcement Learning
School of Computer Science and Technology, East China Normal University · Fudan University · KAUST +1

Jul 8, 2026, cs.LGJ/K move · Enter open · S save

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

Tsinghua University · Work done while ZH and YL interned at Z.AI.

Agentic Learning Autoregressive Rollout Asynchronous Execution

Open paper AskPDF arXiv

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Abstract

Explore similar work

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

3SPO: State-Score-Supervised Policy Optimization for LLM Agents