Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

Yunpeng Ba; Zhi Zheng; Yue Xie; Jiaqing Li; Xialiang Tong; Tao Zhong; Mingxuan Yuan; Zhichao Lu

cs.LGAug 27, 2026

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

Authors: Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li, Xialiang Tong, Tao Zhong, Mingxuan Yuan, Zhichao Lu, +2 more

Organizations: Southern University of Science and Technology · National University of Singapore · Harbin Institute of Technology, Weihai · Huawei Noah’s Ark Lab · City University of Hong Kong

Abstract

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.

Figures & tables

Figure 1: Overview of the three research questions and main findings. Panel (a) contrasts ES and GRPO post-training behavior; Panel (b) shows that keeping larger ES updates preserves target-task performance and reports held-out Maj@32 changes; Panel (c) summarizes normalization, population-size, and estimator choices.

Aspect	ES	GRPO
Signal path	$R_{i}\!\rightarrow\!z_{i}\!\rightarrow\!\theta$	$r_{i}\!\rightarrow\!\widehat{A}_{i}\!\rightarrow\!\theta$
Update rule	Standardized perturbation average	Clipped policy-gradient surrogate
Execution	Perturbed-policy rollouts; no backprop	Policy rollouts and backprop
Backward state	Not retained	Retained on GPU

Table 1: Update and efficiency comparison of ES and GRPO.

Panel of Figure 2 — Figure 2: Comparison of ES and GRPO during training and testing. During GSM8K post-training, GPQA token-level entropy drops substantially under GRPO but remains largely stable under ES; GRPO finishes below the base model on Pass@16 and Pass@32, whereas ES finishes above it. On the representative task pairs, GRPO raises Pass@1 but lowers Pass@16 and Pass@32, whereas ES improves all three metrics.

Pass Metrics
	GSM8K			CSQA			HotpotQA			Countdown			GPQA			MBPP			Average
Method	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32
Qwen2.5-1.5B-Instruct
Base	70.3	94.8	96.4	60.6	93.4	96.1	27.6	46.6	50.3	7.5	46.9	56.9	21.4	87.1	94.4	58.6	83.8	87.2	41.0	75.4	80.2
GRPO	78.1	93.9	95.3	63.7	93.2	95.5	25.0	44.3	48.1	8.5	49.5	58.8	22.2	87.3	96.0	60.0	82.4	86.0	42.9	75.1	79.9
ES	73.2	94.9	96.6	60.7	94.1	96.8	24.9	45.8	49.9	8.8	49.4	59.5	22.9	88.5	96.5	58.7	83.0	86.4	41.5	76.0	80.9

Table 2: Pass@

K

results (

\times 100

) in the Easy Setting after two epochs of GSM8K post-training. Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.

Pass Metrics
	AIME24			AIME25			AMC23			MATH500			Average
Method	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32
DeepSeek-R1-Distill-Qwen-1.5B
Base	23.4	59.9	66.7	22.6	43.8	50.0	64.1	93.6	95.0	80.9	96.7	98.0	47.7	73.5	77.4
GRPO	27.6	62.2	66.7	25.4	45.8	53.3	73.9	94.7	95.0	84.8	96.1	96.8	52.9	74.7	78.0
ES	22.8	60.3	70.0	23.2	46.7	50.0	70.5	96.0	97.5	83.2	97.2	98.2	49.9	75.0	78.9

Table 3: Pass@

K

results (

\times 100

) on mathematical benchmarks in the Hard Setting. Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.

Panel of Figure 3 — Figure 3: Representative Pass@1–Pass@ $K$ Pareto fronts across models and tasks. The math average is computed equally over AIME24, AIME25, AMC23, and MATH-500; higher values are better on both axes. Gray dashed lines connect the reference Pareto fronts formed by Base, GRPO, and ES. Solid dark lines trace the full Pareto fronts after adding the endpoints from the two sequential compositions. Faded markers are dominated in the corresponding pairwise comparison. The two sequential compositions add non-dominated trade-off points in all three settings.

Model	GRPO	Full ES	Magnitude-Thresholded ES
Model	GRPO	Full ES	$\tau=1.0\times 10^{-3}$	$\tau=1.5\times 10^{-3}$	$\tau=2.0\times 10^{-3}$
Qwen2.5-1.5B-Instruct	0.0475	1.933 (40.7 $\times$ )	1.615 (79.11%)	1.242 (92.47%)	0.871 ( 97.57% )
Llama-3.2-3B-Instruct	0.0954	4.185 (43.9 $\times$ )	3.482 (78.08%)	2.589 (92.64%)	1.707 ( 97.89% )
Qwen2.5-7B-Instruct	0.0831	3.664 (44.1 $\times$ )	3.032 (78.41%)	2.218 (93.02%)	1.427 ( 98.11% )
DeepSeek-R1-Distill-Qwen-1.5B	0.0548	2.338 (42.7 $\times$ )	2.201 (62.18%)	2.017 (77.61%)	1.772 ( 87.29% )

Table 4: Relative whole-model

L_{2}

distance (

\times 10^{-2}

) and the distribution of nonzero ES updates across magnitude thresholds. Parentheses in the Full ES column report the distance relative to GRPO; parentheses in the magnitude-thresholded columns report

s_{\tau}

, the percentage of nonzero updates with magnitudes in

(0,\tau]

Panel of Figure 4 — Figure 4: Target-task Pass@1 across update-sparsity levels in the Easy and Hard Settings. The dashed lines indicate previously recorded Base Model performance, which also forms the connected endpoint at 100% update sparsity.

Pass Metrics
	GPQA			MBPP			CSQA			Countdown			Average
Method	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32	@1	@16	@32
DeepSeek-R1-Distill-Qwen-1.5B
Base	24.2	80.8	91.4	62.9	85.6	87.6	46.8	89.2	93.7	37.5	79.9	85.7	42.9	83.9	89.6
GRPO	31.8	86.4	91.9	66.6	87.0	88.3	47.5	88.9	93.6	36.6	77.5	82.2	45.6	85.0	89.0
ES	28.0	84.6	92.4	63.5	85.9	87.2	46.1	88.9	93.4	38.5	79.4	84.4	44.0	84.7	89.3

Table 5: Pass@

K

results (

\times 100

) on four held-out benchmarks in the one-epoch Hard Setting. Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.

	Qwen2.5-0.5B-Instruct				Qwen2.5-1.5B-Instruct				Qwen2.5-3B-Instruct
Update	$N=8$	$N=16$	$N=32$	$N=64$ (ref.)	$N=8$	$N=16$	$N=32$	$N=64$ (ref.)	$N=8$	$N=16$	$N=32$	$N=64$ (ref.)
100	0.4782	0.5021	0.5100	0.5085	0.8211	0.8250	0.8259	0.8244	0.8912	0.8945	0.8960	0.8957
200	0.4614	0.4995	0.5180	0.5201	0.8242	0.8314	0.8366	0.8352	0.8903	0.8970	0.8966	0.8992
300	0.4421	0.4935	0.5253	0.5287	0.8276	0.8387	0.8449	0.8438	0.8901	0.9001	0.9004	0.9031

Table 6: Smoothed GSM8K rewards across model and population sizes. Shading marks reduced-population results within

0.01

of the

N=64

reference; values use debiased exponential smoothing with weight

0.99

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	GSM8K		CSQA		HotpotQA		Countdown		GPQA		MBPP		Average
Method	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32
Qwen2.5-1.5B-Instruct
Base	82.6	84.3	67.8	68.8	31.1	31.1	21.7	27.0	20.7	20.2	61.9	61.5	47.6	48.8
GRPO	84.2	84.8	70.0	71.0	28.8	29.0	27.8	31.5	22.7	22.7	60.3	61.9	49.0	50.2
ES	82.3	83.9	67.2	68.9	29.8	29.8	27.6	33.1	25.3	24.2	58.4	62.3	48.4	50.4
ES $\rightarrow$ GRPO	83.9	84.2	69.2	69.6	28.8	29.1	26.2	31.6	20.7	18.7	61.5	62.7	48.4	49.3

Appendix

Table 7: Maj@

K

results (

\times 100

) in the Easy Setting after two epochs of GSM8K post-training, corresponding to the Pass@

K

results in Table 2 . Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.

	AIME24		AIME25		AMC23		MATH500		Average
Method	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32
DeepSeek-R1-Distill-Qwen-1.5B
Base	36.7	40.0	33.3	33.3	82.5	82.5	90.0	91.0	60.6	61.7
GRPO	43.3	46.7	30.0	33.3	85.0	87.5	91.2	91.8	62.4	64.8
ES	36.7	43.3	36.7	33.3	90.0	92.5	91.8	92.2	63.8	65.3
ES $\rightarrow$ GRPO	46.7	53.3	33.3	33.3	85.0	92.5	91.6	92.6	64.2	67.9

Appendix

Table 8: Maj@

K

results (

\times 100

) on mathematical benchmarks in the Hard Setting, corresponding to the Pass@

K

results in Table 3 . Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.

	GPQA		MBPP		CSQA		Countdown		Average
Method	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32	M@16	M@32
DeepSeek-R1-Distill-Qwen-1.5B
Base	33.3	33.3	69.7	70.4	54.1	54.1	71.1	75.1	57.0	58.2
GRPO	35.9	35.4	69.7	70.8	53.7	52.7	68.5	72.8	56.9	57.9
ES	39.4	39.4	70.8	69.7	52.7	53.6	71.7	76.2	58.6	59.7
ES $\rightarrow$ GRPO	36.4	37.4	71.6	72.0	52.4	53.0	70.9	73.6	57.8	59.0

Appendix

Table 9: Maj@

K

results (

\times 100

) on four held-out benchmarks in the one-epoch Hard Setting, corresponding to the Pass@

K

results in Table 5 . Underlining marks the best Base/GRPO/ES result in each task cell; bold and shading mark the best and second-best Average cells across all five methods.

	Pass changes			Majority-vote changes
Method	$\Delta$ @1	$\Delta$ @16	$\Delta$ @32	$\Delta$ M@16	$\Delta$ M@32
Qwen2.5-1.5B-Instruct
GRPO	$+0.7$	$-0.2$	$-0.1$	$+1.3$	$+1.5$
ES	$+0.0$	$+0.6$	$+0.8$	$+1.0$	$+2.0$
Llama-3.2-3B-Instruct
GRPO	$+2.3$	$-1.3$	$-1.6$	$+1.3$	$+0.6$

Appendix

Table 10: Held-out performance changes for matched Easy Setting runs, averaged equally over five non-GSM8K tasks and reported in percentage points. M@16 and M@32 denote Maj@16 and Maj@32. Positive values denote gains.

Threshold $\tau$	Update sparsity (%)	Pass@1 (%)	$\Delta$ Full ES (%)
Qwen2.5-1.5B-Instruct
Full ES	0.00	73.351	$+0.000$
$2.5\times 10^{-4}$	27.82	73.330	$-0.021$
$5.0\times 10^{-4}$	50.77	73.102	$-0.249$
$1.0\times 10^{-3}$	79.11	73.000	$-0.351$
$1.5\times 10^{-3}$	92.47	72.091	$-1.260$

Appendix

Table 11: Complete GSM8K Pass@1 results for magnitude-thresholded ES in the Easy Setting. Base Model values are included as absolute references.

Threshold $\tau$	Update sparsity (%)	Pass@1 (%)	$\Delta$ Full ES (%)
Full ES	0.00	82.844	$+0.000$
$2.5\times 10^{-4}$	17.80	83.112	$+0.269$
$5.0\times 10^{-4}$	38.32	82.919	$+0.075$
$1.0\times 10^{-3}$	62.18	83.075	$+0.231$
$1.5\times 10^{-3}$	77.61	83.013	$+0.169$
$2.0\times 10^{-3}$	87.29	82.394	$-0.450$

Appendix

Table 12: Complete MATH-500 Pass@1 results for magnitude-thresholded ES with DeepSeek-R1-Distill-Qwen-1.5B in the Hard Setting. The Base Model value is included as an absolute reference.

Objective	Evaluation coupling	$\operatorname{Corr}(X_{+},X_{0})$	$\kappa_{\mathrm{base}}$	$\operatorname{Corr}(X_{+},X_{-})$	$\kappa_{\mathrm{pair}}$
GSM8K regenerated-rollout reward	Common seed	0.2777	1.4445	0.1304	1.9870
GSM8K regenerated-rollout reward	Independent seed	0.0561	1.3577	0.0224	2.7847
SST-2 supervised CE	Fixed batch	0.9776	0.0447	0.9196	0.1557
Mean token log-probability	Fixed rollout	0.9939	0.0125	0.9857	0.0285
Policy sequence loss	Fixed rollout	0.9939	0.0167	0.9903	0.0194
GRPO surrogate	Fixed rollout	0.9998	0.0006	0.9997	0.0007

Appendix

Table 13: Local subtraction diagnostic at the Qwen2.5-1.5B-Instruct base checkpoint.

\kappa_{\mathrm{base}}

and

\kappa_{\mathrm{pair}}

are the raw scalar variance ratios in Equation ( 51 ); lower is better and values below

1

indicate variance reduction. Panel (a) compares objectives at

\sigma=10^{-3}

, and Panel (b) sweeps the scale for GSM8K regenerated-rollout reward.

Panel of Figure 5 — Figure 5: Full GSM8K population-size trajectories across three model scales. Lines show debiased exponential smoothing with weight $0.99$ . Same-color shading shows the local standard deviation of raw-minus-smoothed residuals; it describes within-trajectory fluctuation. Model-specific vertical ranges expose within-model differences.

Parameter	Easy ES	Easy GRPO	Hard ES	Hard GRPO
Model	Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct	Same as Easy ES	DeepSeek-R1-Distill- Qwen-1.5B	Same as Hard ES
Training data	GSM8K	GSM8K	DeepScaleR	DeepScaleR
Epochs	2	2	1	1
Global train batch	64	64	64	64
Population / rollout group	32 directions	8 responses	32 directions	8 responses
Learning scale	$\sigma=1.5{\times}10^{-3}$ , $\alpha=2.5{\times}10^{-4}$	$1.0{\times}10^{-6}$ learning rate	$\sigma=1.5{\times}10^{-3}$ , $\alpha=2.5{\times}10^{-4}$	$1.0{\times}10^{-6}$ learning rate

Appendix

Table 14: Training settings used for the reported comparisons. Hard GRPO is the released discrete-token GRPO checkpoint. Its training parameters are transcribed from Appendix D.1 of the SofT-GRPO paper ( Zheng et al., 2026b ) .

Parameter	Easy	Hard
Samples per problem	32	32
Temperature	0.6	0.6
Maximum new tokens	2,048	8,192
Reported $K$	16, 32	16, 32
Execution	Single GPU	Single GPU

Appendix

Table 15: Offline sampled-evaluation settings.

Setting	Role	Dataset / split	Purpose
Easy	Post-training	GSM8K train	Standard grade-school mathematical word problems with rule-verifiable numeric answers.
	In-domain test	GSM8K test	Measures transfer to unseen problems from the post-training task family.
	Held-out test	CommonsenseQA validation; HotpotQA distractor validation; Countdown test; GPQA Diamond; MBPP sanitized test	Covers commonsense choice, multi-hop QA, arithmetic planning, scientific reasoning, and code generation.
Hard	Post-training	DeepScaleR Preview	Mathematical RL training data used in the DeepScaleR line of work.
	Mathematical test	AIME 2024; AIME 2025; AMC 2023; MATH-500	Competition and advanced mathematical reasoning benchmarks commonly used for 1.5B reasoning-RL evaluation.
	Held-out test	GPQA Diamond; MBPP sanitized test; CommonsenseQA validation; Countdown test	Tests whether mathematical post-training transfers to or degrades non-training task families.

Appendix

Table 16: Datasets and splits used in the reported comparisons. No held-out dataset contributes post-training rewards.

Dataset	Distributed source used for provenance	License declared by the source
GSM8K	openai/gsm8k	MIT
CommonsenseQA	tau/commonsense_qa	MIT
HotpotQA	Official HotpotQA dataset release	CC BY-SA 4.0
Countdown	HuggingFaceTB/Countdown-Task-GOLD ; the fixed test construction corresponds to Jiayi-Pan/Countdown-Tasks-3to4	No license specified in either dataset card
GPQA	Idavidrein/gpqa	CC BY 4.0
MBPP	google-research-datasets/mbpp	CC BY 4.0

Appendix

Table 17: Licenses declared by the dataset distributions used in this work.

Panel of Figure 6 — Figure 6: Pass@ $K$ profiles of individual post-training methods and sequential compositions. For individual post-training, GRPO leads at $K=1$ , whereas ES overtakes it at larger $K$ while remaining above the Base Model throughout. Among the two sequential compositions, ES $\rightarrow$ GRPO gives the strongest AIME24 profile, while GRPO $\rightarrow$ ES leads on AIME25 for $K\geq 2$ , showing complementary but order-dependent benefits.

Explore similar work

Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging
Aug 6, 2026Yu Gu, Zhi Zheng, Yunpeng Ba +3LLM Fine-Tuning
School of Intelligence Science and Technology, Nanjing University, China · School of Computing, National University of Singapore, Singapore · School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, China +1
Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training
Apr 2, 2026William Hoy, Binxu Wang, Xu PanEvolution Strategies RL Post-Training
Department of Industrial and Systems Engineering, University of Miami · Kempner Institute, Harvard University · Harvard University
Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training
Aug 3, 2026Zhiyuan Wang, Shengcai Liu, Jiahao Wu +5Evolutionary Optimization Evolution Strategies
Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology · Hong Kong Polytechnic University · Hong Kong University of Science and Technology +2

Aug 6, 2026J/K move · Enter open · S save

Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

Yu Gu, Zhi Zheng, Yunpeng Ba, Xialiang Tong+2

School of Intelligence Science and Technology, Nanjing University, China · School of Computing, National University of Singapore, Singapore · School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, China+1

LLM Fine-Tuning

Open paper AskPDF arXiv

Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning. However, directly applying ES to billion-parameter LLMs is highly ineffective. In such high-dimensional parameter spaces, most random perturbations are nearly orthogonal to useful update directions, leading to unstable optimization. We propose Hyper-ES, a subspace-based ES framework that avoids the weakness of ES in full-parameter search while exploiting its strength in low-dimensional optimization. Instead of asking ES to discover useful directions from random perturbations in the LLM parameter space, Hyper-ES first performs a small number of inexpensive gradient-based fine-tuning runs to obtain descent directions. Although each direction may provide only a limited improvement on its own, their span forms a compact adaptation subspace that captures useful reasoning updates. Hyper-ES then applies CMA-ES to optimize layer-wise DARE-TIES merging coefficients within this subspace, allowing ES to search over combinations of meaningful descent directions rather than over arbitrary full-model perturbations. We evaluate Hyper-ES on three Qwen2.5-Instruct and DeepSeek-R1-Distill backbones across six mathematical reasoning datasets. Results show that Hyper-ES consistently outperforms GRPO-LoRA by 1% while requiring 10% fewer space-consuming gradient updates. Code at https://github.com/kuangrepi/Hyper-ES.

Abstract

Explore similar work

Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training

Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training