SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

Kai Ruan; Jinghao Lin; Qianshan Wei; Ziqi Zhou; Zihe Huang

cs.AIAug 25, 2026

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

Authors: Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Independent Researcher · Institute of Automation, Chinese Academy of Sciences · Duke University ·

Abstract

Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.

Figures & tables

Figure 1: SPO++ acquires reward faster than SPO. Thin lines show matched replicates, thick lines their mean, and shaded regions one standard error. Curves use a five-step trailing mean; Table 1 reports unsmoothed aggregates.

	ALFWorld		Math-TIR
	Qwen3.5-0.8B	Qwen3.5-2B	Qwen3.5-0.8B
SPO area	$0.532$	$0.522$	$0.217$
SPO++ area	$\mathbf{0.722}$	$\mathbf{0.681}$	$\mathbf{0.242}$
$\Delta$ area (points)	$+19.00\pm 8.95$	$+15.92\pm 10.25$	$+2.50\pm 1.64$
$\Delta$ end reward (points)	$+7.86\pm 7.18$	$+4.88\pm 6.83$	$+5.03\pm 4.74$

Table 1: SPO++ improves online learning efficiency in every evaluated setting. Paired differences are reported in percentage points on the normalized

[0,1]

scale, as mean

\pm

sample s.d.; end reward averages the final five plotted steps.

Figure 2: Recipe-level optimization diagnostics for SPO and SPO++ . Curves show one matched Math-TIR run with complete per-update logging; pale traces are raw updates and dark curves are five-step trailing means. Other runs record a reduced diagnostic subset.

Variant	Normalization measure	$\Delta$ learning composite
SPO++ (trajectory norm.)	Trajectory	$+2.29\pm 9.66$
SPO++	Action token	$\mathbf{+12.99\pm 8.21}$
Action-token vs. trajectory	Paired difference	$\mathbf{+10.70\pm 3.98}$

Table 2: Matched normalization ablation on 0.8B ALFWorld over ten training steps and three seeds. Values are changes in the learning composite

C_{10}

, in percentage points; the first two rows are relative to SPO and the last row is the paired difference. We report mean

\pm

sample s.d.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Item	ALFWorld	Math-TIR
Model	Qwen3.5-0.8B / 2B	Qwen3.5-0.8B (shared SFT)
Training prompts	128 canonical tasks	1,500 prompts
Offline value rollouts	$1{,}024$ (8 / prompt)	$12{,}000$ (8 / prompt)
Online trajectories / method	$6{,}400$	$25{,}600$
Update / publication	128 / every update	128 / every two updates
Train / rollout GPUs	2 / 6	4 / 4

Appendix

Table 3: Training configuration for the reported experiments.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 612 papers

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Aug 20, 2026Dayang Liang, Lang Feng, Bo An +1RL for Language Models Agentic RL
Xiamen University, China
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Sep 30, 2026Bo-Wen Zhang, Junwei He, Maoqi Liu +6Agentic RL Reward Shaping
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance
Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
Jul 5, 2026Mingxuan Fan, Peiyang LiuAgentic RL Group Relative Policy Optimization
Baidu Inc., China · Peking University, China

Aug 20, 2026J/K move · Enter open · S save

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Dayang Liang, Lang Feng, Bo An, Yunlong Liu

Xiamen University, China

RL for Language Models Agentic RL Proximal Policy Optimization RL for LLM Agents

Open paper AskPDF arXiv

Agentic reinforcement learning (RL) has emerged as an important post-training approach for enhancing the capabilities of Large Language Models (LLMs). However, existing methods face a trade-off between policy performance and resource efficiency. Conventional Proximal Policy Optimization (PPO) implementations incur substantial memory overhead from a separate critic, whereas critic-free group-relative methods require multiple rollouts and face potential learning bottlenecks on long-horizon tasks. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), an efficient PPO-style framework that unifies policy optimization and value learning within a single causal language model. SAPO exploits the autoregressive structure of LLMs to sequentially generate action and value estimation at distinct causal boundaries with shared parameters, and then jointly optimizes the PPO objectives and an auxiliary on-policy SARSA objective with turn-level generalized advantage estimation, where the latter is designed to facilitate value learning. Extensive experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B and Qwen3-14B demonstrate that SAPO reduces peak GPU memory usage by 24.8% and per-iteration runtime by 23.1% over strong PPO baseline, while matching or slightly improving task success rate. Our experiments also show that SAPO outperforms Group Relative Policy Optimization (GRPO) and recent cutting-edge variants in both task success and training stability.

Abstract

RL for LLM Agents

Explore similar work

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning