Authors: Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko
Organizations: Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.
Figures & tables
Figure 1: Illustration of the candidate SSPO gate families and their induced weighting behavior as the importance ratio departs from the on-policy point ρ=1 . The heavier-tailed arctan -based gates preserve more signal away from the trust region, while sigmoid-style gates attenuate more aggressively.
Figure 2: Sequence reward curves for all six μ=4 transfer settings arranged as a 2×3 grid: Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B on math and coding. Thick curves denote method means, while thin curves show the three per-setting seeds without exponential smoothing. On the math tasks, reward stays within the natural [0,2] range induced by answer and format scoring.
Figure 3: Token entropy curves for the same six μ=4 transfer settings, again arranged as a 2×3 grid. The entropy panels make the contrast in collapse behavior visible: GRPO shows the sharpest entropy loss, GSPO stabilizes hard sequence-level clipping, and SSPO preserves high-reward exploration in most settings without flattening the trajectories into unrealistically smooth traces.
Model
GSM8K
M500
A25
Δ Avg
Qwen2.5-7B-Instruct
77.9
56.6
10.0
–
+ GRPO
84.5
64.2
10.0
+4.7
+ GMPO
88.8
65.0
13.3
+7.5
+ SAPO
82.8
58.6
10.0
+2.3
+ SSPO ( fexpsig )
89.2
64.8
16.7
+8.7
+ SSPO ( fexpatan )
89.6
65.8
16.7
+9.2
Table 1: Pass@1 on mathematical reasoning benchmarks. The best results are shown in bold, the second-best are underlined and the strongest baseline in italics. Δ Avg reports the average absolute improvement over the base model across the three benchmarks. A25 denotes AIME25; M500 denotes MATH-500.
Model
Method
A24
A25
M500
CodeAct
Qwen3-8B
Base
17.8
13.2
56.8
18.4
GRPO
23.9
18.7
61.2
19.3
GSPO
26.1
20.8
63.8
24.0
SAPO
25.4
21.5
62.9
22.4
SSPO
28.3
22.6
64.6
27.0
Qwen3-14B
Base
21.4
15.8
60.7
21.8
Table 2: Benchmark matrix for the transfer settings. A24/A25 denote AIME24/AIME25 Pass@1; M500 denotes MATH-500 Pass@1; CodeAct denotes end-to-end solved rate.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Training curves for reward and entropy. Values are reported after exponential smoothing with α=0.02 for best configurations of each objective selected by grid search.
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2