Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Authors: Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li, Xialiang Tong, Tao Zhong, Mingxuan Yuan, Zhichao Lu, +2 more
Organizations: Southern University of Science and Technology · National University of Singapore · Harbin Institute of Technology, Weihai · Huawei Noah’s Ark Lab · City University of Hong Kong
Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.
Figures & tables
Figure 1: Overview of the three research questions and main findings. Panel (a) contrasts ES and GRPO post-training behavior; Panel (b) shows that keeping larger ES updates preserves target-task performance and reports held-out Maj@32 changes; Panel (c) summarizes normalization, population-size, and estimator choices.
Aspect
ES
GRPO
Signal path
Ri→zi→θ
ri→Ai→θ
Update rule
Standardized perturbation average
Clipped policy-gradient surrogate
Execution
Perturbed-policy rollouts; no backprop
Policy rollouts and backprop
Backward state
Not retained
Retained on GPU
Table 1: Update and efficiency comparison of ES and GRPO.
Figure 2: Comparison of ES and GRPO during training and testing. During GSM8K post-training, GPQA token-level entropy drops substantially under GRPO but remains largely stable under ES; GRPO finishes below the base model on Pass@16 and Pass@32, whereas ES finishes above it. On the representative task pairs, GRPO raises Pass@1 but lowers Pass@16 and Pass@32, whereas ES improves all three metrics.
Pass Metrics
GSM8K
CSQA
HotpotQA
Countdown
GPQA
MBPP
Average
Method
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
Qwen2.5-1.5B-Instruct
Base
70.3
94.8
96.4
60.6
93.4
96.1
27.6
46.6
50.3
7.5
46.9
56.9
21.4
87.1
94.4
58.6
83.8
87.2
41.0
75.4
80.2
GRPO
78.1
93.9
95.3
63.7
93.2
95.5
25.0
44.3
48.1
8.5
49.5
58.8
22.2
87.3
96.0
60.0
82.4
86.0
42.9
75.1
79.9
ES
73.2
94.9
96.6
60.7
94.1
96.8
24.9
45.8
49.9
8.8
49.4
59.5
22.9
88.5
96.5
58.7
83.0
86.4
41.5
76.0
80.9
Table 2: Pass@ K results ( ×100 ) in the Easy Setting after two epochs of GSM8K post-training. Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.
Pass Metrics
AIME24
AIME25
AMC23
MATH500
Average
Method
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
DeepSeek-R1-Distill-Qwen-1.5B
Base
23.4
59.9
66.7
22.6
43.8
50.0
64.1
93.6
95.0
80.9
96.7
98.0
47.7
73.5
77.4
GRPO
27.6
62.2
66.7
25.4
45.8
53.3
73.9
94.7
95.0
84.8
96.1
96.8
52.9
74.7
78.0
ES
22.8
60.3
70.0
23.2
46.7
50.0
70.5
96.0
97.5
83.2
97.2
98.2
49.9
75.0
78.9
Table 3: Pass@ K results ( ×100 ) on mathematical benchmarks in the Hard Setting. Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.
Figure 3: Representative Pass@1–Pass@ K Pareto fronts across models and tasks. The math average is computed equally over AIME24, AIME25, AMC23, and MATH-500; higher values are better on both axes. Gray dashed lines connect the reference Pareto fronts formed by Base, GRPO, and ES. Solid dark lines trace the full Pareto fronts after adding the endpoints from the two sequential compositions. Faded markers are dominated in the corresponding pairwise comparison. The two sequential compositions add non-dominated trade-off points in all three settings.
Model
GRPO
Full ES
Magnitude-Thresholded ES
τ=1.0×10−3
τ=1.5×10−3
τ=2.0×10−3
Qwen2.5-1.5B-Instruct
0.0475
1.933 (40.7 × )
1.615 (79.11%)
1.242 (92.47%)
0.871 ( 97.57% )
Llama-3.2-3B-Instruct
0.0954
4.185 (43.9 × )
3.482 (78.08%)
2.589 (92.64%)
1.707 ( 97.89% )
Qwen2.5-7B-Instruct
0.0831
3.664 (44.1 × )
3.032 (78.41%)
2.218 (93.02%)
1.427 ( 98.11% )
DeepSeek-R1-Distill-Qwen-1.5B
0.0548
2.338 (42.7 × )
2.201 (62.18%)
2.017 (77.61%)
1.772 ( 87.29% )
Table 4: Relative whole-model L2 distance ( ×10−2 ) and the distribution of nonzero ES updates across magnitude thresholds. Parentheses in the Full ES column report the distance relative to GRPO; parentheses in the magnitude-thresholded columns report sτ , the percentage of nonzero updates with magnitudes in (0,τ] .
Figure 4: Target-task Pass@1 across update-sparsity levels in the Easy and Hard Settings. The dashed lines indicate previously recorded Base Model performance, which also forms the connected endpoint at 100% update sparsity.
Pass Metrics
GPQA
MBPP
CSQA
Countdown
Average
Method
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
@1
@16
@32
DeepSeek-R1-Distill-Qwen-1.5B
Base
24.2
80.8
91.4
62.9
85.6
87.6
46.8
89.2
93.7
37.5
79.9
85.7
42.9
83.9
89.6
GRPO
31.8
86.4
91.9
66.6
87.0
88.3
47.5
88.9
93.6
36.6
77.5
82.2
45.6
85.0
89.0
ES
28.0
84.6
92.4
63.5
85.9
87.2
46.1
88.9
93.4
38.5
79.4
84.4
44.0
84.7
89.3
Table 5: Pass@ K results ( ×100 ) on four held-out benchmarks in the one-epoch Hard Setting. Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.
Qwen2.5-0.5B-Instruct
Qwen2.5-1.5B-Instruct
Qwen2.5-3B-Instruct
Update
N=8
N=16
N=32
N=64 (ref.)
N=8
N=16
N=32
N=64 (ref.)
N=8
N=16
N=32
N=64 (ref.)
100
0.4782
0.5021
0.5100
0.5085
0.8211
0.8250
0.8259
0.8244
0.8912
0.8945
0.8960
0.8957
200
0.4614
0.4995
0.5180
0.5201
0.8242
0.8314
0.8366
0.8352
0.8903
0.8970
0.8966
0.8992
300
0.4421
0.4935
0.5253
0.5287
0.8276
0.8387
0.8449
0.8438
0.8901
0.9001
0.9004
0.9031
Table 6: Smoothed GSM8K rewards across model and population sizes. Shading marks reduced-population results within 0.01 of the N=64 reference; values use debiased exponential smoothing with weight 0.99 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
GSM8K
CSQA
HotpotQA
Countdown
GPQA
MBPP
Average
Method
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
Qwen2.5-1.5B-Instruct
Base
82.6
84.3
67.8
68.8
31.1
31.1
21.7
27.0
20.7
20.2
61.9
61.5
47.6
48.8
GRPO
84.2
84.8
70.0
71.0
28.8
29.0
27.8
31.5
22.7
22.7
60.3
61.9
49.0
50.2
ES
82.3
83.9
67.2
68.9
29.8
29.8
27.6
33.1
25.3
24.2
58.4
62.3
48.4
50.4
ES → GRPO
83.9
84.2
69.2
69.6
28.8
29.1
26.2
31.6
20.7
18.7
61.5
62.7
48.4
49.3
Appendix
Table 7: Maj@ K results ( ×100 ) in the Easy Setting after two epochs of GSM8K post-training, corresponding to the Pass@ K results in Table 2 . Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.
AIME24
AIME25
AMC23
MATH500
Average
Method
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
DeepSeek-R1-Distill-Qwen-1.5B
Base
36.7
40.0
33.3
33.3
82.5
82.5
90.0
91.0
60.6
61.7
GRPO
43.3
46.7
30.0
33.3
85.0
87.5
91.2
91.8
62.4
64.8
ES
36.7
43.3
36.7
33.3
90.0
92.5
91.8
92.2
63.8
65.3
ES → GRPO
46.7
53.3
33.3
33.3
85.0
92.5
91.6
92.6
64.2
67.9
Appendix
Table 8: Maj@ K results ( ×100 ) on mathematical benchmarks in the Hard Setting, corresponding to the Pass@ K results in Table 3 . Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.
GPQA
MBPP
CSQA
Countdown
Average
Method
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
M@16
M@32
DeepSeek-R1-Distill-Qwen-1.5B
Base
33.3
33.3
69.7
70.4
54.1
54.1
71.1
75.1
57.0
58.2
GRPO
35.9
35.4
69.7
70.8
53.7
52.7
68.5
72.8
56.9
57.9
ES
39.4
39.4
70.8
69.7
52.7
53.6
71.7
76.2
58.6
59.7
ES → GRPO
36.4
37.4
71.6
72.0
52.4
53.0
70.9
73.6
57.8
59.0
Appendix
Table 9: Maj@ K results ( ×100 ) on four held-out benchmarks in the one-epoch Hard Setting, corresponding to the Pass@ K results in Table 5 . Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.
Pass changes
Majority-vote changes
Method
Δ @1
Δ @16
Δ @32
Δ M@16
Δ M@32
Qwen2.5-1.5B-Instruct
GRPO
+0.7
−0.2
−0.1
+1.3
+1.5
ES
+0.0
+0.6
+0.8
+1.0
+2.0
Llama-3.2-3B-Instruct
GRPO
+2.3
−1.3
−1.6
+1.3
+0.6
Appendix
Table 10: Held-out performance changes for matched Easy Setting runs, averaged equally over five non-GSM8K tasks and reported in percentage points. M@16 and M@32 denote Maj@16 and Maj@32. Positive values denote gains.
Threshold τ
Update sparsity (%)
Pass@1 (%)
Δ Full ES (%)
Qwen2.5-1.5B-Instruct
Full ES
0.00
73.351
+0.000
2.5×10−4
27.82
73.330
−0.021
5.0×10−4
50.77
73.102
−0.249
1.0×10−3
79.11
73.000
−0.351
1.5×10−3
92.47
72.091
−1.260
Appendix
Table 11: Complete GSM8K Pass@1 results for magnitude-thresholded ES in the Easy Setting. Base Model values are included as absolute references.
Threshold τ
Update sparsity (%)
Pass@1 (%)
Δ Full ES (%)
Full ES
0.00
82.844
+0.000
2.5×10−4
17.80
83.112
+0.269
5.0×10−4
38.32
82.919
+0.075
1.0×10−3
62.18
83.075
+0.231
1.5×10−3
77.61
83.013
+0.169
2.0×10−3
87.29
82.394
−0.450
Appendix
Table 12: Complete MATH-500 Pass@1 results for magnitude-thresholded ES with DeepSeek-R1-Distill-Qwen-1.5B in the Hard Setting. The Base Model value is included as an absolute reference.
Objective
Evaluation coupling
Corr(X+,X0)
κbase
Corr(X+,X−)
κpair
GSM8K regenerated-rollout reward
Common seed
0.2777
1.4445
0.1304
1.9870
GSM8K regenerated-rollout reward
Independent seed
0.0561
1.3577
0.0224
2.7847
SST-2 supervised CE
Fixed batch
0.9776
0.0447
0.9196
0.1557
Mean token log-probability
Fixed rollout
0.9939
0.0125
0.9857
0.0285
Policy sequence loss
Fixed rollout
0.9939
0.0167
0.9903
0.0194
GRPO surrogate
Fixed rollout
0.9998
0.0006
0.9997
0.0007
Appendix
Table 13: Local subtraction diagnostic at the Qwen2.5-1.5B-Instruct base checkpoint. κbase and κpair are the raw scalar variance ratios in Equation ( 51 ); lower is better and values below 1 indicate variance reduction. Panel (a) compares objectives at σ=10−3 , and Panel (b) sweeps the scale for GSM8K regenerated-rollout reward.
Figure 5: Full GSM8K population-size trajectories across three model scales. Lines show debiased exponential smoothing with weight 0.99 . Same-color shading shows the local standard deviation of raw-minus-smoothed residuals; it describes within-trajectory fluctuation. Model-specific vertical ranges expose within-model differences.
Table 14: Training settings used for the reported comparisons. Hard GRPO is the released discrete-token GRPO checkpoint. Its training parameters are transcribed from Appendix D.1 of the SofT-GRPO paper ( Zheng et al., 2026b ) .
Parameter
Easy
Hard
Samples per problem
32
32
Temperature
0.6
0.6
Maximum new tokens
2,048
8,192
Reported K
16, 32
16, 32
Execution
Single GPU
Single GPU
Appendix
Table 15: Offline sampled-evaluation settings.
Setting
Role
Dataset / split
Purpose
Easy
Post-training
GSM8K train
Standard grade-school mathematical word problems with rule-verifiable numeric answers.
In-domain test
GSM8K test
Measures transfer to unseen problems from the post-training task family.
Mathematical RL training data used in the DeepScaleR line of work.
Mathematical test
AIME 2024; AIME 2025; AMC 2023; MATH-500
Competition and advanced mathematical reasoning benchmarks commonly used for 1.5B reasoning-RL evaluation.
Held-out test
GPQA Diamond; MBPP sanitized test; CommonsenseQA validation; Countdown test
Tests whether mathematical post-training transfers to or degrades non-training task families.
Appendix
Table 16: Datasets and splits used in the reported comparisons. No held-out dataset contributes post-training rewards.
Dataset
Distributed source used for provenance
License declared by the source
GSM8K
openai/gsm8k
MIT
CommonsenseQA
tau/commonsense_qa
MIT
HotpotQA
Official HotpotQA dataset release
CC BY-SA 4.0
Countdown
HuggingFaceTB/Countdown-Task-GOLD ; the fixed test construction corresponds to Jiayi-Pan/Countdown-Tasks-3to4
No license specified in either dataset card
GPQA
Idavidrein/gpqa
CC BY 4.0
MBPP
google-research-datasets/mbpp
CC BY 4.0
Appendix
Table 17: Licenses declared by the dataset distributions used in this work.
Figure 6: Pass@ K profiles of individual post-training methods and sequential compositions. For individual post-training, GRPO leads at K=1 , whereas ES overtakes it at larger K while remaining above the Base Model throughout. Among the two sequential compositions, ES → GRPO gives the strongest AIME24 profile, while GRPO → ES leads on AIME25 for K≥2 , showing complementary but order-dependent benefits.
Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning. However, directly applying ES to billion-parameter LLMs is highly ineffective. In such high-dimensional parameter spaces, most random perturbations are nearly orthogonal to useful update directions, leading to unstable optimization. We propose Hyper-ES, a subspace-based ES framework that avoids the weakness of ES in full-parameter search while exploiting its strength in low-dimensional optimization. Instead of asking ES to discover useful directions from random perturbations in the LLM parameter space, Hyper-ES first performs a small number of inexpensive gradient-based fine-tuning runs to obtain descent directions. Although each direction may provide only a limited improvement on its own, their span forms a compact adaptation subspace that captures useful reasoning updates. Hyper-ES then applies CMA-ES to optimize layer-wise DARE-TIES merging coefficients within this subspace, allowing ES to search over combinations of meaningful descent directions rather than over arbitrary full-model perturbations. We evaluate Hyper-ES on three Qwen2.5-Instruct and DeepSeek-R1-Distill backbones across six mathematical reasoning datasets. Results show that Hyper-ES consistently outperforms GRPO-LoRA by 1% while requiring 10% fewer space-consuming gradient updates. Code at https://github.com/kuangrepi/Hyper-ES.
Yu Gu, Zhi Zheng, Yunpeng Ba +3
School of Intelligence Science and Technology, Nanjing University, China · School of Computing, National University of Singapore, Singapore · School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, China +1
Evolution Strategies (ES) have emerged as a scalable gradient-free alternative to reinforcement learning based LLM fine-tuning, but it remains unclear whether comparable task performance implies comparable solutions in parameter space. We compare ES and Group Relative Policy Optimization (GRPO) across four tasks in both single-task and sequential continual-learning settings. ES matches or exceeds GRPO in single-task accuracy and remains competitive sequentially when its iteration budget is controlled. Despite this similarity in task performance, the two methods produce markedly different model updates: ES makes much larger changes and induces broader off-task KL drift, whereas GRPO makes smaller, more localized updates. Strikingly, the ES and GRPO solutions are linearly connected with no loss barrier, even though their update directions are nearly orthogonal. We develop an analytical theory of ES that explains all these phenomena within a unified framework, showing how ES can accumulate large off-task movement on weakly informative directions while still making enough progress on the task to match gradient-based RL in downstream accuracy. These results show that gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation. The source code is publicly available: https://github.com/Bhoy1/ESvsGRPO.
William Hoy, Binxu Wang, Xu Pan
Department of Industrial and Systems Engineering, University of Miami · Kempner Institute, Harvard University · Harvard University
Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive. Evolution strategies (ES) enable memory-efficient full-parameter post-training without backpropagation and can eventually match the performance of gradient-based reinforcement learning (RL). However, resource-constrained settings typically offer only a few GPUs, so the high GPU-hour requirements of ES translate into prohibitively long training times. To address this, we introduce Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that decomposes the full parameter space into lower-dimensional subspaces and searches over them cooperatively to improve optimization efficiency. We post-train a Qwen3.5-4B tool-using agent for the math task and evaluate it on five benchmarks of varying difficulty. Under the GPU-hour budget of full-parameter GRPO's best validation checkpoint, CoPES recovers 92% of GRPO's validation-accuracy gain, versus 67% for standard ES, while its theoretical GPU memory requirement is less than one-eighth that of full-parameter GRPO. It consistently outperforms standard ES and LoRA-based GRPO on all evaluated pass@k metrics across the five benchmarks. Additional experiments further show the advantage of CoPES on the question-answering task. These results demonstrate an improved trade-off between memory requirements and training time for agentic LLM post-training under resource constraints. The code is open-sourced in https://github.com/MetaronWang/CoPES
Zhiyuan Wang, Shengcai Liu, Jiahao Wu +5
Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology · Hong Kong Polytechnic University · Hong Kong University of Science and Technology +2