Reinforcement learning improves the reasoning capability of large language models through verifiable rewards for mathematical and coding tasks. In token-level methods such as GRPO, clipping acts on individual token ratios while the reward and advantage apply to a complete response. Recent sequence-level methods align these units but still use fixed clipping bands, whose tail probabilities change as the score distribution evolves. We propose Conformal Prediction-Enabled Sequence-level Policy Optimization (CSPO), which adaptively calibrates sequence-level clipping bounds using recent training statistics. The resulting band targets a specified marginal tail budget and adapts to policy drift; calibration conditional on response length requires additional assumptions. We bound the probability that a fresh score falls outside the band by the target miscoverage level plus explicit CDF discrepancy and drift terms. Under additional moment conditions, this bound also controls the objective and gradient error relative to the unclipped GSPO power surrogate. Experiments with the Qwen3-1.7B series show improvements over GSPO on difficult mathematical reasoning benchmarks and ALFWorld, with clipping endpoints that adapt during training.
Figures & tables
Method
ALFWorld
Pick
Look
Clean
Heat
Cool
Pick2
All
Qwen3-1.7B
0.00
40.00
3.23
0.00
0.00
0.00
3.91
GSPO
58.06
66.67
45.00
61.54
22.22
20.83
31.25
GRPO
67.86
66.67
56.52
61.54
55.17
45.45
52.34
DAPO
76.92
70.00
72.00
75.00
56.52
33.33
53.12
CSPO
93.33
70.00
68.00
75.00
60.00
44.00
64.06
Table 1: Performance on ALFWorld.
Method
AIME24
AIME25
AMC
MATH500
Minerva
Oly.
Average
Qwen3-1.7B-Base
5.42
3.54
25.38
54.43
13.72
22.47
20.83
GRPO
11.40
5.65
39.43
70.49
21.90
33.27
30.36
DAPO
11.05
5.72
41.56
73.04
22.82
34.32
31.42
GSPO
15.08
8.33
42.86
73.36
22.68
37.83
33.36
CSPO
16.87
8.57
43.36
74.22
23.19
38.44
34.11
Table 2: Pass@1 performance on difficult mathematical reasoning benchmarks.
Figure 1: Learning curves and evaluation results on ALFWorld.
Figure 2: Learning curves and evaluation results on DAPO-Math-17k.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Update element or score
Band/control construction
Primary controlled object
PPO / GRPO
Token ratio
Fixed numerical endpoints
Pessimistic token surrogate
GSPO
Response W=eS/L
Fixed response-level endpoints
Sequence-coherent power surrogate
FSPO
Sequence log-ratio
Fixed L -scaled geometry
Length reweighting and gradient direction
CTPO ( Zhang et al., 2026 )
Prefix/cumulative ratio
Position-dependent scaling
Prefix correction and position scaling
BandPO ( Li et al., 2026 )
Probability ratio
Divergence-derived probability-aware bounds
Projected trust-region geometry
BAPO ( Xi et al., 2025 )
Ratio contributions
Advantage/entropy-driven asymmetry
Contribution balance and entropy
Appendix
Table 3: Comparison with representative clipping and conformal-control methods. “Controlled object” denotes the stated construction or guarantee, not an empirical performance claim.
Figure 3: Validation results on ALFWorld.
Figure 4: Pass@1 on various benchmarks.
Method
AIME24
AIME25
AMC
MATH500
Minerva
Oly.
Average
CSPO- α -0.1
16.87
8.57
43.36
74.22
23.19
38.44
34.11
CSPO- α -0.2
14.85
8.58
42.86
74.49
23.02
37.23
33.50
CSPO- α -0.4
13.52
7.40
41.50
72.98
22.47
36.34
32.37
Appendix
Table 5: Ablations on miscoverage level α .
Figure 5: Ablation studies on DAPO-Math-17k.
Figure 6: Computational efficiency on DAPO-Math-17k.
Figure 7: Policy update stability on DAPO-Math-17k.
Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores, in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment, in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Contrastive Sequence-level Policy Optimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks. Code will be released upon paper acceptance.
Feng Zhang, Xinhong Ma, Ziqiang Dong +5
Beijing Institute of Technology · Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen +1
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.
Tianyi Wang, Yixia Li, Long Li +6
Southern University of Science and Technology · Beijing University of Posts and Telecommunications · INFLY TECH +3
A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.
Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia