Recent advances in large language models (LLMs) highlight the importance of post training techniques for improving reasoning and mathematical ability. Group Relative Policy Optimization (GRPO) has shown promise in this domain by combining group relative advantage estimation, PPO style clipping, and KL regularization. However, its complexity raises the question of whether all components are necessary for fostering reasoning behaviors. We conduct a systematic analysis of GRPO and identify two key findings: (1) incorporating negative feedback is essential training solely on actions above a baseline limits learning; and (2) PPO style constraints, such as policy ratio clipping, are not required to improve mathematical reasoning or performance. Building on these insights, we propose REINFORCE with Group Relative Advantage (RGRA), a simplified variant that retains group relative advantage estimation but removes PPO style clipping and policy ratio terms. Experiments across standard mathematical benchmarks indicate that RGRA has the potential to achieve stronger performance than GRPO. Our results suggest that simpler REINFORCE based approaches can effectively enhance reasoning in LLMs, offering a more transparent and efficient alternative to GRPO.
Figures & tables
Math-English Benchmarks
Model
Method
GSM8K
MATH
Gaokao2023 Math-En
Olympiad Bench
AMC23
Avg
Qwen2.5-0.5-it
-
41.5
22.6
21.0
6.2
7.5
19.8
Qwen2.5-1.5-it
-
61.1
38.9
35.1
11.4
17.5
32.8
Llama3.2-1.0-it
-
37.9
18.9
14.5
4.6
10.0
13.4
Qwen2.5-0.5-it
GRPO
50.9
30.3
30.4
8.9
7.5
25.6
Qwen2.5-1.5-it
GRPO
71.0
44.2
38.7
12.6
20.0
37.3
Table 1: Performance of trained models on English Math benchmarks. All models are trained using gsm8k Dataset.
Chinese Math Benchmarks
Model
Method
CMATH
CN-Middle-School
Avg
Qwen2.5-0.5-it
-
34.3
42.6
38.5
Qwen2.5-1.5-it
-
52.3
51.5
51.9
Llama3.2-1.0-it
-
29.5
24.8
27.2
Qwen2.5-0.5-it
GRPO
51.2
51.5
51.4
Qwen2.5-1.5-it
GRPO
75.0
56.4
65.7
Table 2: Performance of trained models on Chinese Math benchmarks. All models are trained using gsm8k Dataset.
STEM Benchmarks
Model
Method
English (MMLU-STEM)
Chinese (Gaokao2024)
Avg
Qwen2.5-0.5-it
-
40.6
24.4
32.5
Qwen2.5-1.5-it
-
59.2
31.5
45.4
Llama3.2-1.0-it
-
31.7
13.7
22.7
Qwen2.5-0.5-it
GRPO
41.3
21.2
31.3
Qwen2.5-1.5-it
GRPO
58.7
32.6
45.7
Table 3: Performance of trained models on STEM benchmarks (English: MMLU, Chinese: Gaokao2024). All models are trained using gsm8k Dataset.
Figure 1: Training metrics (10-step running average).
Figure 2: Comparison of answers with and without reasoning traces.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Experiment
Hyper-parameter
Value
GRPO
Batch size
24
Learning rate
1×10−5
Group size G
8
Temperature
1
KL coefficient
0.005
Max new tokens
512
Appendix
Table 4: Hyper-parameters used for training on GSM8K.
Mike · School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019 USA · Intelligent Neuromorphic and Quantum Understanding for Innovative Research and Engineering (INQUIRE) Laboratory, University of Oklahoma, Norman, OK, USA