SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Authors: Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, +1 more
Organizations: Southern University of Science and Technology · Beijing University of Posts and Telecommunications · INFLY TECH · Microsoft Research Asia · Shanghai University of Finance and Economics · Tsinghua University
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.
Figures & tables
Figure 1: Analysis of the “Tail Effect”. We visualize Critic value dynamics V(st) to diagnose inefficiencies. Blue and red lines denote correct and incorrect trajectories, respectively. The Critic discriminates only near the sequence tail. For correct paths, V(st) rises late, causing A^t to vanish; for incorrect ones, it fails to penalize intermediate steps. This indicates credit assignment based on token position rather than semantic contribution. The Critic was trained under 8192 context window. Additional randomly sampled visualizations are provided in Appendix B .
Figure 2: Visualization of the GRPO Advantage Function. derived under the Bernoulli assumption (see Appendix A ). The plot illustrates how GRPO implicitly models the reasoning task as a Contextual Bandit : instead of a static reward, the advantage is dynamically scaled based on the prompt’s estimated difficulty p^(sp) , contrasting success (Blue) against failure (Red).
Figure 3: Overview of SPPO. Motivated by the implicit bandit behavior of GRPO, SPPO explicitly reformulates reasoning as a Sequence-Level Contextual Bandit, utilizing a scalar value function V(sp) .
Method
A24
A25
AMC
MATH
Minerva
Avg
DeepSeek-R1-Distill-Qwen-1.5B
Base Model
27.50
21.67
71.56
83.73
20.35
44.96
PPO
27.50
20.83
70.63
81.38
19.89
44.06
ReMax
31.67
25.42
71.88
84.38
20.40
46.74
RLOO
30.42
21.67
72.81
84.10
21.73
46.15
GRPO ( N=8 )
30.00
26.25
73.13
83.88
22.15
47.08
Table 1: Performance comparison on 1.5B and 7B scales. SPPO consistently outperforms baselines. The Small Critic variant (1.5B Critic aligning 7B Policy) achieves the top average score.
Figure 4: Ablation Analysis of the Optimization Objective. We compare SPPO against Standard PPO and a control baseline (“PPO + BCE”) that integrates the BCE loss into the token-level framework. The failure of the control baseline demonstrates that the performance gains do not stem from the loss function itself, but from the Sequence-Level Contextual Bandit formulation , which propagates a unified advantage signal to resolve credit assignment ambiguity.
Figure 5: Training Efficiency on Deepseek-R1-Distill-Qwen-7B (Performance vs. Wall-clock Time). The plot compares the trajectory of SPPO against strong baselines (GRPO, PPO, RLOO, ReMax) on the DAPO-17k dataset ( Yu et al., 2025 ) . Solid Red: SPPO with a matched 7B Critic. Dashed Pink: SPPO with a decoupled, smaller 1.5B Critic (Deepseek-R1-Distill-Qwen-1.5B). The y-axis noted as the Avg@8 score evaluated on AIME24 , AIME25 , AMC23 , MATH500 , and Minerva Math . SPPO achieves optimal performance significantly faster than group-based methods, and the decoupled critic maintains performance while reducing memory overhead.
Figure 6: GPU Memory Allocation Analysis. Comparison of normalized peak VRAM usage during the training of a 7B policy. The “Decoupled Critic” (7B+1.5B) approach, combined with the system-level optimizations in verl , significantly reduces memory bottlenecks compared to symmetric actor-critic setups (7B+7B), making efficient RL alignment accessible on consumer-grade hardware.
Figure 7: Correlation analysis between the Critic’s predicted difficulty ( y -axis) and the empirical AVG@ k rate ( x -axis) (Where k = 64). The plot reveals a clear positive correlation (Pearson r=0.642 ), indicating the Critic successfully distinguishes between hard and easy tasks. The marginal histograms (top and right) contrast the bimodal distribution of real task difficulty with the more conservative, quasi-normal distribution of the Critic’s predictions.
Figure 8: RLVR Benchmark Results. Comparison of SPPO (Blue Solid) and Standard PPO (Red Dashed) across five control tasks with sparse outcome rewards ( γ=1.0 ). SPPO demonstrates robust convergence in complex control tasks where Standard PPO exhibits instability or failure (e.g., Hopper , MountainCar ), while achieving superior sample efficiency in precision tasks like CartPole .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Extended Analysis of Critic Value Dynamics (10 Random Samples). Each subplot represents a distinct mathematical problem sampled from the validation set. Blue Lines: Value estimates for correct trajectories ( R=1 ). Red Lines: Value estimates for incorrect trajectories ( R=0 ). The consistent overlap of value curves until the sequence tail demonstrates the systematic failure of token-level value estimation in distinguishing intermediate reasoning quality.
Category
Resource / Link
License
Models
DeepSeek-R1-Distill-Qwen-1.5B
MIT
DeepSeek-R1-Distill-Qwen-7B
MIT
Training
DAPO-Math-17k
Apache 2.0
Datasets
DeepScaleR-Preview
MIT
Benchmarks
AIME 2024 , AIME 2025
–
AMC 23 , MATH-500
–
Appendix
Table 2: Summary of resources and licenses used in this work. Click on the resource names to visit their HuggingFace pages.
A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.
Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly in their mathemat ical reasoning performance. However, GRPO and related entropy regularization methods still struggle with token-level sparse-rewards, which is an inherent chal lenge in chain-of-thought (CoT) reasoning. These approaches often rely on undifferen tiated token-level entropy regularization, which easily leads to entropy collapse or model degradation under sparse token rewards. In this work, we propose TEPO, a novel token-level framework that (1) leverages sequence-level likelihood to link group-level rewards with individual tokens via token-level aggregation, and (2) introduces a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. Experiments demonstrate that TEPO not only achieves state-of-the-art performance on mathematical reasoning benchmarks but also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.
Xingyu Lin, Yilin Wen, Du Su +5
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. Yet long-horizon autoregressive rollouts introduce a fundamental limitation: local policy mismatch can compound into severe state distribution shift over trajectories. Existing LLM RL methods, such as GRPO and GSPO, control policy mismatch at the token or sequence level, but overlook this trajectory-induced shift, causing training instability and eventual collapse as the horizon grows. To close the state distribution gap, we propose Trajectory-Aware Policy Optimization (TAPO), a principled framework that theoretically derives the state-distribution correction from the off-policy policy gradient and incorporates it into the RL objective. TAPO further employs a geometric-mean estimator for variance reduction, together with a length-aware dynamic clipping that prevents overly deviated updates across the trajectory. Extensive experiments demonstrate that TAPO significantly surpasses existing LLM RL algorithms across eight competition-level benchmarks, achieving up to 48% and 12% improvement over GRPO on mathematical reasoning and algorithmic coding, with substantially enhanced training stability.
Zhicheng Cai, Weinan Dai, Yuyan Zhou +3
Institute for AI Industry Research (AIR), Tsinghua University SIA-Lab of Tsinghua AIR and ByteDance Seed