Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training

William Hoy; Binxu Wang; Xu Pan

cs.LGApr 2, 2026

Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training

Authors: William Hoy, Binxu Wang, Xu Pan

Organizations: Department of Industrial and Systems Engineering, University of Miami · Kempner Institute, Harvard University · Harvard University

Abstract

Evolution Strategies (ES) have emerged as a scalable gradient-free alternative to reinforcement learning based LLM fine-tuning, but it remains unclear whether comparable task performance implies comparable solutions in parameter space. We compare ES and Group Relative Policy Optimization (GRPO) across four tasks in both single-task and sequential continual-learning settings. ES matches or exceeds GRPO in single-task accuracy and remains competitive sequentially when its iteration budget is controlled. Despite this similarity in task performance, the two methods produce markedly different model updates: ES makes much larger changes and induces broader off-task KL drift, whereas GRPO makes smaller, more localized updates. Strikingly, the ES and GRPO solutions are linearly connected with no loss barrier, even though their update directions are nearly orthogonal. We develop an analytical theory of ES that explains all these phenomena within a unified framework, showing how ES can accumulate large off-task movement on weakly informative directions while still making enough progress on the task to match gradient-based RL in downstream accuracy. These results show that gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation. The source code is publicly available: https://github.com/Bhoy1/ESvsGRPO.

Figures & tables

Task	Base	ES (iter=100)	ES (iter=300)	GRPO
Countdown	20.5	75.1	78.6	74.5
Math	61.8	72.4	74.0	68.8
Chemistry	63.4	68.1	76.5	74.9
BoolQ	83.7	87.2	88.1	86.6

Table 1: Accuracy (%) for single task training.

Figure 1: Accuracy (%) on each task throughout sequential training. The shaded region marks the stage at which that task was trained. ES (300) shows clear degradation on earlier tasks as training progresses, while ES (100) and GRPO remain comparatively stable.

Stage	ES (MMLU)	GRPO (MMLU)	ES (IFEval)	GRPO (IFEval)
Base model	77.5	77.5	79.7	79.7
After Task 1 (Countdown)	77.0	77.0	80.0	80.6
After Task 2 (Math)	76.2	77.5	82.6	80.6
After Task 3 (Chemistry)	74.6	78.0	82.3	81.7
After Task 4 (BoolQ)	73.8	78.3	81.9	78.4
$\Delta$ (base $\to$ final)	$-$ 3.7	$+$ 0.8	$+$ 2.2	$-$ 1.3

Table 2: Holdout benchmark accuracy (%) throughout sequential training.

Figure 2: Incremental KL divergence per training step. Each cell shows $\text{KL}(\pi_{\text{after}}\|\pi_{\text{before}})$ for a given evaluation task (row) after a given training step (column). Both panels share the same color scale. Across three seeds there is large variance for some blocks. We have included median, min and max in Appendix C.4 so the complete picture can be seen

Checkpoint	ES norm	GRPO norm	Ratio
After Task 1 (Countdown)	87.28	1.00	87 $\times$
After Task 2 (Math)	122.76	1.15	107 $\times$
After Task 3 (Chemistry)	149.99	1.65	91 $\times$
After Task 4 (BoolQ)	173.00	1.84	94 $\times$

Table 3: Weight update norms (

\|\Delta\theta\|_{2}

) at each sequential checkpoint, measured from the base model.

Stage	Sparsity ES	Sparsity GRPO	Stable Rank ES	Stable Rank GRPO
T0 Countdown	3.8%	88.0%	1100	60
T1 Math	2.9%	88.0%	1090	70
T2 Chemistry	2.4%	87.7%	1084	89
T3 BoolQ	2.1%	87.7%	1078	103

Table 4: Sparsity (fraction of entries with

|\Delta\theta_{i}|\leq\varepsilon=1\times 10^{-5}

) and stable rank (

\|\Delta W\|_{F}^{2}/\|\Delta W\|_{2}^{2}

, averaged across attention QKVO projections, and MLP up/down/gate projections) of ES and GRPO weight updates at each sequential checkpoint.

Figure 3: Per-task accuracy along the linear interpolation path between the final ES and GRPO checkpoints. The dashed gray line shows the linear interpolation between endpoint accuracies. Across all tasks, the interpolated model remains close to or above the linear baseline with no catastrophic accuracy drop, indicating that the two solutions lie in the same basin of the loss landscape.

Figure 4: Task accuracy when moving from the base model along ES (blue), GRPO (red), and random (gray) directions at increasing magnitudes. Dashed vertical lines mark $\|\Delta_{\text{GRPO}}\|$ (red) and $\|\Delta_{\text{ES}}\|$ (blue). Dotted horizontal lines show base and checkpoint accuracies.

Figure 5: Holdout task accuracy (IFEval, MMLU) when perturbing along ES (blue) and GRPO (red) directions from the final BoolQ checkpoint.

Figure 6: ES fine-tuning drifts as a random walk through the parameter space; GRPO does not. A. Total drift $\|\Delta\theta_{T}\|^{2}$ for Qwen and Llama, each divided by its own random-walk scaling $\alpha^{2}d/N$ so that the theory becomes the identity (dashed). Inset: GRPO with the same normalisation. B. Individual Qwen tensors against the same scaling (dashed): qkv_proj follows it for the whole run, post_ln saturates away from it. C. Effective dimension ratio $\rho=d_{\mathrm{eff}}/d_{\mathrm{total}}$ against fit quality $R^{2}$ for every tensor. All weight matrices collapse onto the ideal scaling $R^{2}=1,\rho=1$ , the norm vectors do not. D. GRPO moves along the low-dimensional task-relevant subspace (red). ES accumulates both a task-relevant component and a large off-manifold random walk (green), yielding a much larger displacement nearly orthogonal to GRPO. The two solutions are linearly connected because the off-manifold component is loss-invariant.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Countdown	Math	SciK	BoolQ
Base	20.78 $\pm$ 0.20	61.40 $\pm$ 0.35	63.77 $\pm$ 0.32	83.97 $\pm$ 0.23
After Countdown	74.52 $\pm$ 0.70	65.20 $\pm$ 0.72	64.35 $\pm$ 0.80	83.43 $\pm$ 0.95
After Math	75.40 $\pm$ 0.31	71.73 $\pm$ 0.42	69.45 $\pm$ 0.46	84.20 $\pm$ 1.18
After Chemistry	72.67 $\pm$ 1.03	71.40 $\pm$ 1.40	74.85 $\pm$ 1.01	84.03 $\pm$ 2.29
After BoolQ	64.87 $\pm$ 8.78	69.93 $\pm$ 1.01	73.80 $\pm$ 1.59	87.53 $\pm$ 0.60

Appendix

Table 5: Qwen3-4B-Instruct: accuracy (%, mean

\pm

std over 3 seeds) throughout sequential training with ES (iter=100). Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
Base	17.22 $\pm$ 2.11	66.20 $\pm$ 4.01	62.77 $\pm$ 0.68	85.20 $\pm$ 1.39
After Countdown	69.17 $\pm$ 4.86	67.27 $\pm$ 6.47	60.58 $\pm$ 4.85	85.52 $\pm$ 1.78
After Math	70.32 $\pm$ 3.49	69.80 $\pm$ 5.89	61.63 $\pm$ 2.03	85.57 $\pm$ 1.69
After Chemistry	70.65 $\pm$ 4.01	71.67 $\pm$ 5.03	64.47 $\pm$ 7.34	82.10 $\pm$ 8.00
After BoolQ	70.50 $\pm$ 3.77	74.27 $\pm$ 5.80	66.08 $\pm$ 7.05	89.80 $\pm$ 1.31

Appendix

Table 6: Qwen3-4B-Instruct: accuracy (%, mean

\pm

std over 3 seeds) throughout sequential training with GRPO. Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
Base	2.4	38.2	60.6	73.4
After Countdown	37.4	41.2	60.9	75.8
After Math	34.9	41.8	64.0	71.6
After Chemistry	29.3	41.0	65.9	68.8
After BoolQ	20.3	38.2	65.7	81.9

Appendix

Table 7: Llama-3.2-3B-Instruct: accuracy (%) throughout sequential training with ES (iter=100). Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
Base	1.8	39.4	61.5	73.4
After Countdown	37.8	42.2	62.6	75.3
After Math	40.8	42.0	62.4	75.9
After Chemistry	40.4	44.2	65.8	72.2
After BoolQ	40.3	46.4	67.0	83.0

Appendix

Table 8: Llama-3.2-3B-Instruct: accuracy (%) throughout sequential training with GRPO. Bold indicates the task just trained at that stage.

Config	$\sigma$	$\alpha$
1	0.0005	0.00025
2	0.001	0.0005
3	0.0015	0.00075

Appendix

Table 9: ES hyperparameter configurations. Config 3 selected.

Config	Learning rate
1	$1\times 10^{-6}$
2	$3\times 10^{-6}$
3	$1\times 10^{-5}$
4	$3\times 10^{-5}$

Appendix

Table 10: GRPO learning rate configurations. Config 3 selected.

Eval. task	After Countdown	After Math	After Chemistry	After BoolQ
Countdown	0.200 [0.187, 0.232]	0.162 [0.124, 0.221]	0.075 [0.074, 0.083]	0.086 [0.047, 0.086]
Math	0.062 [0.054, 0.071]	0.080 [0.067, 0.088]	0.041 [0.033, 0.042]	0.043 [0.036, 0.054]
Chemistry	0.137 [0.116, 0.143]	0.313 [0.272, 1.276]	1.100 [0.106, 3.062]	0.103 [0.090, 0.649]
BoolQ	0.135 [0.126, 0.137]	0.170 [0.151, 0.227]	0.096 [0.086, 0.127]	0.102 [0.090, 0.122]

Appendix

Table 11: ES: median [min, max] incremental test KL divergence across 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.

Eval. task	After Countdown	After Math	After Chemistry	After BoolQ
Countdown	0.381 [0.293, 0.454]	0.008 [0.006, 0.012]	0.012 [0.004, 0.030]	0.017 [0.005, 0.029]
Math	0.050 [0.048, 0.056]	0.009 [0.003, 0.010]	0.010 [0.007, 0.016]	0.006 [0.005, 0.011]
Chemistry	0.012 [0.000, 0.115]	0.010 [0.000, 0.019]	0.117 [0.072, 3.738]	0.056 [0.017, 0.105]
BoolQ	0.051 [0.035, 0.061]	0.010 [0.003, 0.010]	0.038 [0.028, 4.431]	0.239 [0.094, 0.322]

Appendix

Table 12: GRPO: median [min, max] incremental test KL divergence across 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
After Countdown	0.206 $\pm$ 0.023	0.062 $\pm$ 0.008	0.132 $\pm$ 0.014	0.133 $\pm$ 0.006
After Math	0.436 $\pm$ 0.032	0.178 $\pm$ 0.021	0.755 $\pm$ 0.477	0.355 $\pm$ 0.079
After Chemistry	0.536 $\pm$ 0.059	0.250 $\pm$ 0.033	2.541 $\pm$ 2.390	0.545 $\pm$ 0.211
After BoolQ	0.551 $\pm$ 0.112	0.288 $\pm$ 0.028	2.013 $\pm$ 1.781	0.545 $\pm$ 0.158

Appendix

Table 13: Qwen3-4B-Instruct: cumulative KL divergence from base model, ES (iter=100). Mean

\pm

std over 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
After Countdown	0.376 $\pm$ 0.081	0.052 $\pm$ 0.004	0.040 $\pm$ 0.066	0.049 $\pm$ 0.013
After Math	0.420 $\pm$ 0.113	0.069 $\pm$ 0.018	0.087 $\pm$ 0.129	0.072 $\pm$ 0.025
After Chemistry	0.435 $\pm$ 0.143	0.093 $\pm$ 0.038	1.297 $\pm$ 2.120	1.904 $\pm$ 3.140
After BoolQ	0.435 $\pm$ 0.115	0.091 $\pm$ 0.024	1.282 $\pm$ 2.062	0.194 $\pm$ 0.063

Appendix

Table 14: Qwen3-4B-Instruct: cumulative KL divergence from base model, GRPO. Mean

\pm

std over 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
After Countdown	0.069	0.020	0.074	0.236
After Math	0.071	0.038	0.127	0.536
After Chemistry	0.095	0.058	0.396	1.644
After BoolQ	0.170	0.066	1.362	1.788

Appendix

Table 15: Llama-3.2-3B-Instruct: cumulative KL divergence from base model, ES (iter=100). Single seed. Bold indicates the task just trained at that stage.

Stage	Countdown	Math	SciK	BoolQ
After Countdown	0.072	0.011	0.094	0.312
After Math	0.081	0.012	0.107	0.347
After Chemistry	0.091	0.022	2.306	1.411
After BoolQ	0.099	0.023	2.330	1.717

Appendix

Table 16: Llama-3.2-3B-Instruct: cumulative KL divergence from base model, GRPO. Single seed. Bold indicates the task just trained at that stage.

Figure 7: GRPO single-task training reward across 20 epochs for each task.

Figure 8: ES single-task training reward across 300 steps for each task.

ES
Noise scale ( $\sigma$ )	0.0015
Step size ( $\alpha$ )	0.00075
Population size	30
Batch size	200

Appendix

Table 17: Training hyperparameters for ES (left) and GRPO (right).

Method	FLOPs	vs. GRPO
GRPO ( $T=80$ )	$8(80)(63)(8)\,PL=322{,}560\,PL$	$1.00\times$
ES (100 iter)	$2(100)(30)(200)\,PL=1{,}200{,}000\,PL$	$3.72\times$
ES (300 iter)	$2(300)(30)(200)\,PL=3{,}600{,}000\,PL$	$11.16\times$

Appendix

Table 18: Compute per task, relative to GRPO.

Task	Answer format	$\|\mathcal{D}_{\text{test}}\|$	Answer extraction and scoring
Countdown	\boxed{expr}	2,000	Extract last \boxed{} via regex. Verify the expression contains only digits and $+,-,*,/$ ; each input number is used exactly once; and the evaluated result equals the target (tolerance $<10^{-5}$ ).
MATH	\boxed{ans}	500	Extract last \boxed{} with nested-brace handling. Normalize both prediction and gold by stripping LaTeX wrappers ( \text , \dfrac , currency/degree symbols). Compare as strings; on mismatch, convert fractions and decimals to floats and compare numerically (tolerance $<10^{-6}$ ).
SciKnowEval Chem.	"answer":"X"	2,000	Extract answer letter via regex cascade: (1) JSON-style "answer":"X" , (2) unquoted answer: X , (3) last standalone A/B/C/D token. Use the last match found.
BoolQ	\boxed{yes/no}	2,000	Extract last \boxed{} , normalize to lowercase. Accept {yes, true, 1} as positive and {no, false, 0} as negative.
Holdout tasks (not trained on; used to measure forgetting)
MMLU	"answer":"X"	14,042	Same regex cascade as SciKnowEval Chemistry.

Appendix

Table 19: Evaluation protocol per task. All tasks use greedy decoding (temperature

=0

), 1,024 max generated tokens, and binary scoring.

Figure 9: Total weight drift under ES follows the random-walk law; under GRPO it does not. Squared displacement of the full parameter vector

\|\Delta\theta_{T}\|^{2}

against training step

T

for Qwen ( A ,

d=4.02\times 10^{9}

) and Llama ( B ,

d=3.21\times 10^{9}

). Solid blue, ES; dashed black, the prediction

\alpha^{2}Td/N

with

\alpha=\sigma/2

\sigma=0.0015

and

N=30

. The

R^{2}

in the legend is measured against that prediction, which has no fitted parameter; the annotation gives instead the

R^{2}

of a freely fitted slope and the effective dimension

\rho=d_{\mathrm{eff}}/d_{\mathrm{total}}

that slope implies. Insets show GRPO on its own axes over its full

79

steps, annotated with the ES/GRPO displacement ratio at the matched step

T=79

. Values and interpretation in the text.

Figure 10: Per-parameter effective dimension and agreement with the random-walk theory, Llama vs Qwen. For every parameter tensor we fit

\|\Delta\theta_{T}\|^{2}=\alpha^{2}Td_{\mathrm{eff}}/N

through the origin over

500

ES steps and report

\rho=d_{\mathrm{eff}}/d_{\mathrm{total}}

( A , B ) and the

R^{2}

of that fit ( C , D ), for the large weight matrices (left) and the RMSNorm gain vectors (right). Faint dots are individual tensors —

28

per per-layer family in Llama,

36

in Qwen — and solid markers give the median with the interquartile range. Two reference levels are drawn, distinguished by colour: the dashed black line is the null model in which a tensor random-walks through its full parameter space,

\rho=1

; the dotted grey line is a perfect fit,

R^{2}=1

. Under each name is the tensor’s shape in each model (blue, Llama; red, Qwen); shading marks the QK-norm families, present only in Qwen. The fit has no intercept, so

R^{2}

is free to be negative, as it is throughout much of D ; the axis in C is correspondingly expanded, spanning only the top

2\times 10^{-4}

of the

R^{2}

range. The large matrices cluster tightly at

\rho\approx 1

with

R^{2}

indistinguishable from unity at ordinary resolution, while the norm parameters scatter across the full range of both quantities; the weight-growth curves underlying the latter are shown in Figure 11 . Qwen’s layers.34.input_layernorm is excluded throughout (

n=35

for that family; see text).

Figure 11: Per-parameter weight drift under ES: where the random-walk law holds and where it breaks. Squared displacement

\|\Delta\theta_{T}\|^{2}

against ES step

T

for individual parameter tensors of Llama ( A – D , blue) and Qwen ( E – H , red), one parameter family per column. Dashed lines give the random-walk prediction

\alpha^{2}Td_{p}/N

, where

d_{p}

is the number of entries in the tensor (

\alpha=\sigma/2

\sigma=0.0015

N=30

); it is common to every curve in a panel, since all tensors in a family share a shape. Curves are individual tensors shaded light to dark by layer depth — three evenly spaced layers for qkv_proj , six for input_ln and post_ln , and the single final norm for norm . Annotations give the median

R^{2}

and median

\rho=d_{\mathrm{eff}}/d_{\mathrm{total}}

over the entire family, not only the curves drawn;

R^{2}

is that of a fit through the origin, as the law has no intercept, and may therefore be negative. The large weight matrices ( A , E ) lie on the predicted line for the whole run, while the RMSNorm gains rise steeply and then saturate — the departure that the negative

R^{2}

of a through-origin fit records. Values in the text.

Stage	Qwen3-4B $\cos$	Llama-3.2-3B $\cos$
T0 Countdown	$1.0\times 10^{-4}$	$0.8\times 10^{-4}$
T1 Math	$1.6\times 10^{-4}$	$1.0\times 10^{-4}$
T2 Chemistry	$2.2\times 10^{-4}$	$1.0\times 10^{-4}$
T3 BoolQ	$2.7\times 10^{-4}$	$1.3\times 10^{-4}$

Appendix

Table 20: Empirical cosine similarity between ES and GRPO update directions at each sequential checkpoint.

Panel of Figure 12 — Figure 12: Wick contraction classes for $\mathbb{E}[(\epsilon^{\top}Q\epsilon)^{2}\,\epsilon_{e}\epsilon_{f}]$ , with external loose ends $e,f$ shown as crosses.

Explore similar work

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Aug 27, 2026Yunpeng Ba, Zhi Zheng, Yue Xie +7Evolution Strategies LLM Post-Training
Southern University of Science and Technology · National University of Singapore · Harbin Institute of Technology, Weihai +2
DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies
Oct 4, 2026Zhishen Sun, Hongzhan Wang, Sizhe Dang +2LLM Fine-Tuning Evolution Strategies
Xi’an Jiaotong University · SGIT AI Lab
Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
Sep 29, 2025Xin Qiu, Yulu Gan, Conor F. Hayes +6Fine-Tuning Evolutionary Optimization
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1

Aug 27, 2026J/K move · Enter open · S save

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li+6

Southern University of Science and Technology · National University of Singapore · Harbin Institute of Technology, Weihai+2

Evolution Strategies LLM Post-Training RL for Language Model Reasoning Reasoning Diversity

Open paper AskPDF arXiv

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.

Abstract

Explore similar work

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning