SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

Tianyi Wang; Yixia Li; Long Li; Yibiao Chen; Shaohan Huang; Yun Chen; Peng Li; Yang Liu

cs.AIApr 10, 2026

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

Authors: Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, +1 more

Organizations: Southern University of Science and Technology · Beijing University of Posts and Telecommunications · INFLY TECH · Microsoft Research Asia · Shanghai University of Finance and Economics · Tsinghua University

Abstract

Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.

Figures & tables

Figure 1: Analysis of the “Tail Effect”. We visualize Critic value dynamics $V(s_{t})$ to diagnose inefficiencies. Blue and red lines denote correct and incorrect trajectories, respectively. The Critic discriminates only near the sequence tail. For correct paths, $V(s_{t})$ rises late, causing $\hat{A}_{t}$ to vanish; for incorrect ones, it fails to penalize intermediate steps. This indicates credit assignment based on token position rather than semantic contribution. The Critic was trained under 8192 context window. Additional randomly sampled visualizations are provided in Appendix B .

Figure 2: Visualization of the GRPO Advantage Function. derived under the Bernoulli assumption (see Appendix A ). The plot illustrates how GRPO implicitly models the reasoning task as a Contextual Bandit : instead of a static reward, the advantage is dynamically scaled based on the prompt’s estimated difficulty

\hat{p}(s_{p})

, contrasting success (Blue) against failure (Red).

Figure 3: Overview of SPPO. Motivated by the implicit bandit behavior of GRPO, SPPO explicitly reformulates reasoning as a Sequence-Level Contextual Bandit, utilizing a scalar value function

V(s_{p})

DeepSeek-R1-Distill-Qwen-1.5B
Method	A24	A25	AMC	MATH	Minerva	Avg
Base Model	27.50	21.67	71.56	83.73	20.35	44.96
PPO	27.50	20.83	70.63	81.38	19.89	44.06
ReMax	31.67	25.42	71.88	84.38	20.40	46.74
RLOO	30.42	21.67	72.81	84.10	21.73	46.15
GRPO ( $N=8$ )	30.00	26.25	73.13	83.88	22.15	47.08

Table 1: Performance comparison on 1.5B and 7B scales. SPPO consistently outperforms baselines. The Small Critic variant (1.5B Critic aligning 7B Policy) achieves the top average score.

Figure 4: Ablation Analysis of the Optimization Objective. We compare SPPO against Standard PPO and a control baseline (“PPO + BCE”) that integrates the BCE loss into the token-level framework. The failure of the control baseline demonstrates that the performance gains do not stem from the loss function itself, but from the Sequence-Level Contextual Bandit formulation , which propagates a unified advantage signal to resolve credit assignment ambiguity.

Figure 5: Training Efficiency on Deepseek-R1-Distill-Qwen-7B (Performance vs. Wall-clock Time). The plot compares the trajectory of SPPO against strong baselines (GRPO, PPO, RLOO, ReMax) on the DAPO-17k dataset ( Yu et al., 2025 ) . Solid Red: SPPO with a matched 7B Critic. Dashed Pink: SPPO with a decoupled, smaller 1.5B Critic (Deepseek-R1-Distill-Qwen-1.5B). The y-axis noted as the Avg@8 score evaluated on AIME24 , AIME25 , AMC23 , MATH500 , and Minerva Math . SPPO achieves optimal performance significantly faster than group-based methods, and the decoupled critic maintains performance while reducing memory overhead.

Figure 6: GPU Memory Allocation Analysis. Comparison of normalized peak VRAM usage during the training of a 7B policy. The “Decoupled Critic” (7B+1.5B) approach, combined with the system-level optimizations in verl , significantly reduces memory bottlenecks compared to symmetric actor-critic setups (7B+7B), making efficient RL alignment accessible on consumer-grade hardware.

Figure 7: Correlation analysis between the Critic’s predicted difficulty ( $y$ -axis) and the empirical AVG@ $k$ rate ( $x$ -axis) (Where k = 64). The plot reveals a clear positive correlation (Pearson $r=0.642$ ), indicating the Critic successfully distinguishes between hard and easy tasks. The marginal histograms (top and right) contrast the bimodal distribution of real task difficulty with the more conservative, quasi-normal distribution of the Critic’s predictions.

Figure 8: RLVR Benchmark Results. Comparison of SPPO (Blue Solid) and Standard PPO (Red Dashed) across five control tasks with sparse outcome rewards (

\gamma=1.0

). SPPO demonstrates robust convergence in complex control tasks where Standard PPO exhibits instability or failure (e.g., Hopper , MountainCar ), while achieving superior sample efficiency in precision tasks like CartPole .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Extended Analysis of Critic Value Dynamics (10 Random Samples). Each subplot represents a distinct mathematical problem sampled from the validation set. Blue Lines: Value estimates for correct trajectories ( $R=1$ ). Red Lines: Value estimates for incorrect trajectories ( $R=0$ ). The consistent overlap of value curves until the sequence tail demonstrates the systematic failure of token-level value estimation in distinguishing intermediate reasoning quality.

Category	Resource / Link	License
Models	DeepSeek-R1-Distill-Qwen-1.5B	MIT
	DeepSeek-R1-Distill-Qwen-7B	MIT
Training	DAPO-Math-17k	Apache 2.0
Datasets	DeepScaleR-Preview	MIT
Benchmarks	AIME 2024 , AIME 2025	–
	AMC 23 , MATH-500	–

Appendix

Table 2: Summary of resources and licenses used in this work. Click on the resource names to visit their HuggingFace pages.

Explore similar work

Soft Sequence Policy Optimization
Feb 22, 2026Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman IschenkoLLM Alignment RL Post-Training
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
Apr 14, 2026Xingyu Lin, Yilin Wen, Du Su +5RL for Language Model Reasoning Policy Optimization
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Oct 9, 2026Zhicheng Cai, Weinan Dai, Yuyan Zhou +3Reinforcement Learning RL for Language Model Reasoning
Institute for AI Industry Research (AIR), Tsinghua University SIA-Lab of Tsinghua AIR and ByteDance Seed

Feb 22, 2026J/K move · Enter open · S save

Soft Sequence Policy Optimization

Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko

Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia

LLM Alignment RL Post-Training RL for Language Model Reasoning Off-Policy RL Importance Sampling

Open paper AskPDF arXiv

A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.

Abstract

Explore similar work

Soft Sequence Policy Optimization

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood

Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL