Existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, such as GRPO, rely on rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of Large Language Models (LLMs). In this paper, we identify three critical challenges in these methods: (1) inefficient gradient utilization caused by the binary cutoff of hard clipping, (2) insensitive probability mass arising from uniform ratio constraints that ignore the token distribution, and (3) asymmetric signal reliability stemming from the disparate credit assignment ambiguity between positive and negative samples. To bridge these gaps, we propose Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions. MASPO integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence. Extensive evaluations demonstrate that MASPO serves as a robust, all-in-one RLVR solution, significantly outperforming baselines. Our code is available at: https://github.com/FlyTune/MASPO-RL.
Figures & tables
Figure 1: MASPO resolves three core limitations of GRPO: (1) inefficient gradient utilization caused by hard boundaries that indiscriminately discard valid gradients; (2) insensitive probability mass , where uniform clipping overlooks the head/tail mass disparity; and (3) asymmetric signal reliability , which neglects the inherent noise differences between verified positive and ambiguous negative signals.
Figure 2: Overview of the MASPO framework. The architecture integrates a Mass-Adaptive Limiter to scale constraints inversely with token probability and an Asymmetric Risk Controller to modulate update magnitude based on advantage signals, unified via a differentiable Soft Gaussian Gating mechanism.
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Average
Method
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
Backbone: DeepSeek-R1-Distill-Qwen-1.5B
GRPO
33.2
71.8
27.7
49.9
79.5
94.8
77.6
90.8
26.1
48.8
46.3
64.7
48.4
70.1
Clip Higher
36.6
70.1
30.1
55.8
82.8
94.9
71.7
88.6
24.8
49.3
42.0
61.6
48.0
70.1
DAC
40.0
73.6
28.7
51.2
80.2
95.0
78.1
92.5
27.4
54.0
46.3
64.5
50.1
71.8
Entropy Adv.
29.6
67.8
23.9
44.2
77.1
94.1
78.5
89.7
25.2
48.6
46.4
64.3
46.8
68.1
Table 1: Main results on mathematical reasoning benchmarks. We report Avg@32 (A@32) and Pass@32 (P@32). Bold indicates the best performance, and underlined represents the second-best.
Figure 3: Evolution of Training Dynamics and Performance across Model Scales. Top row: 1.5B model; Bottom row: 7B model. MASPO demonstrates superior convergence, achieving higher performance ceilings.
Figure 4: Hyperparameter sensitivity and scalability analysis. Top: MASPO demonstrates robustness across a range of α and β values, outperforming the GRPO baseline. Bottom: Training dynamics on the larger 14B model, demonstrating MASPO’s scalability by maintaining higher entropy and superior performance compared to GRPO.
Scale
GRPO
MASPO
Gain ↑
A
P
A
P
A
P
1.5B
48.4
70.1
51.4
72.9
+3.0
+2.8
7B
58.9
77.2
61.8
79.8
+2.9
+2.6
14B
53.6
67.4
56.4
71.1
+2.8
+3.7
Table 2: Scalability analysis of average Avg@32 (A) and Pass@32 (P) across 1.5B, 7B, and 14B models.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Evaluation Focus & Characteristics
AIME 2024
Focuses on arithmetic precision and robustness against contamination. Requires integer outputs (000–999).
AIME 2025
A strictly held-out set representing the frontier of current reasoning capabilities. Tests generalization to unseen, complex logical chains.
AMC 2023
Covers foundational algebra, geometry, and counting. Serves as a baseline for competitive mathematical proficiency.
MATH-500
A representative subset of the MATH dataset, spanning 7 domains including Calculus. Heavily tests LaTeX parsing and symbolic manipulation.
Minerva
Derived from scientific literature, testing higher-order reasoning and domain-specific notation understanding.
OlympiadBench
An aggregate of international competitions (IMO, CMO), representing the upper bound of cross-lingual mathematical logic.
Appendix
Table 3: Summary of the evaluation benchmarks used to assess model performance.
Figure 5: Supplementary training dynamics for additional baselines (Clip Higher, Advantage Reweighting, etc.).
Figure 6: Ablation study of the SAPO gating mechanism on the 1.5B model. The unilateral design (similar to MASPO) prevents the training collapse observed in the original bilateral SAPO.
Hyperparameters
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg.
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
α=0.1 , β=0.03
53.1
78.4
41.3
72.0
89.5
97.5
84.1
90.8
38.1
55.5
56.2
70.6
60.4
77.5
α=0.3 , β=0
54.8
79.8
39.8
58.7
90.3
96.6
84.7
92.9
38.2
55.2
57.3
73.6
60.9
76.1
α=0.3 , β=0.03
53.2
82.4
42.9
73.2
91.4
95.0
86.0
94.7
39.3
58.6
58.0
74.9
61.8
79.8
α=0.3 , β=0.1
55.3
85.6
40.1
66.3
90.9
98.7
84.1
92.9
38.2
55.0
55.8
71.4
60.7
78.3
α=0.5 , β=0
51.5
81.8
41.6
64.6
90.5
97.2
84.4
91.6
38.8
55.9
56.5
71.6
60.6
77.1
Appendix
Table 5: Full hyperparameters comparison on DeepSeek-R1-Distill-Qwen-7B.
Figure 7: Complete hyperparameter sweep results on the DeepSeek-R1-Distill-Qwen-7B, confirming the robustness of MASPO across a wider range of hyperparameter settings.
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Average
Method
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
GRPO
56.6
82.9
40.5
67.8
92.2
97.4
66.5
70.4
22.6
35.3
43.2
50.7
53.6
67.4
MASPO
63.5
86.1
45.3
72.4
93.4
99.9
66.8
73.3
24.6
41.4
45.0
53.7
56.4
71.1
Appendix
Table 6: Performance results on the DeepSeek-R1-Distill-Qwen-14B model. We report Avg@32 and Pass@32.
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · *Work done during internship at Microsoft Research Asia. · 2Microsoft Research +1