Evolution Strategies (ES) have emerged as a scalable gradient-free alternative to reinforcement learning based LLM fine-tuning, but it remains unclear whether comparable task performance implies comparable solutions in parameter space. We compare ES and Group Relative Policy Optimization (GRPO) across four tasks in both single-task and sequential continual-learning settings. ES matches or exceeds GRPO in single-task accuracy and remains competitive sequentially when its iteration budget is controlled. Despite this similarity in task performance, the two methods produce markedly different model updates: ES makes much larger changes and induces broader off-task KL drift, whereas GRPO makes smaller, more localized updates. Strikingly, the ES and GRPO solutions are linearly connected with no loss barrier, even though their update directions are nearly orthogonal. We develop an analytical theory of ES that explains all these phenomena within a unified framework, showing how ES can accumulate large off-task movement on weakly informative directions while still making enough progress on the task to match gradient-based RL in downstream accuracy. These results show that gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation. The source code is publicly available: https://github.com/Bhoy1/ESvsGRPO.
Figures & tables
Task
Base
ES (iter=100)
ES (iter=300)
GRPO
Countdown
20.5
75.1
78.6
74.5
Math
61.8
72.4
74.0
68.8
Chemistry
63.4
68.1
76.5
74.9
BoolQ
83.7
87.2
88.1
86.6
Table 1: Accuracy (%) for single task training.
Figure 1: Accuracy (%) on each task throughout sequential training. The shaded region marks the stage at which that task was trained. ES (300) shows clear degradation on earlier tasks as training progresses, while ES (100) and GRPO remain comparatively stable.
Stage
ES (MMLU)
GRPO (MMLU)
ES (IFEval)
GRPO (IFEval)
Base model
77.5
77.5
79.7
79.7
After Task 1 (Countdown)
77.0
77.0
80.0
80.6
After Task 2 (Math)
76.2
77.5
82.6
80.6
After Task 3 (Chemistry)
74.6
78.0
82.3
81.7
After Task 4 (BoolQ)
73.8
78.3
81.9
78.4
Δ (base → final)
− 3.7
+ 0.8
+ 2.2
− 1.3
Table 2: Holdout benchmark accuracy (%) throughout sequential training.
Figure 2: Incremental KL divergence per training step. Each cell shows KL(πafter∥πbefore) for a given evaluation task (row) after a given training step (column). Both panels share the same color scale. Across three seeds there is large variance for some blocks. We have included median, min and max in Appendix C.4 so the complete picture can be seen
Checkpoint
ES norm
GRPO norm
Ratio
After Task 1 (Countdown)
87.28
1.00
87 ×
After Task 2 (Math)
122.76
1.15
107 ×
After Task 3 (Chemistry)
149.99
1.65
91 ×
After Task 4 (BoolQ)
173.00
1.84
94 ×
Table 3: Weight update norms ( ∥Δθ∥2 ) at each sequential checkpoint, measured from the base model.
Stage
Sparsity ES
Sparsity GRPO
Stable Rank ES
Stable Rank GRPO
T0 Countdown
3.8%
88.0%
1100
60
T1 Math
2.9%
88.0%
1090
70
T2 Chemistry
2.4%
87.7%
1084
89
T3 BoolQ
2.1%
87.7%
1078
103
Table 4: Sparsity (fraction of entries with ∣Δθi∣≤ε=1×10−5 ) and stable rank ( ∥ΔW∥F2/∥ΔW∥22 , averaged across attention QKVO projections, and MLP up/down/gate projections) of ES and GRPO weight updates at each sequential checkpoint.
Figure 3: Per-task accuracy along the linear interpolation path between the final ES and GRPO checkpoints. The dashed gray line shows the linear interpolation between endpoint accuracies. Across all tasks, the interpolated model remains close to or above the linear baseline with no catastrophic accuracy drop, indicating that the two solutions lie in the same basin of the loss landscape.
Figure 4: Task accuracy when moving from the base model along ES (blue), GRPO (red), and random (gray) directions at increasing magnitudes. Dashed vertical lines mark ∥ΔGRPO∥ (red) and ∥ΔES∥ (blue). Dotted horizontal lines show base and checkpoint accuracies.
Figure 5: Holdout task accuracy (IFEval, MMLU) when perturbing along ES (blue) and GRPO (red) directions from the final BoolQ checkpoint.
Figure 6: ES fine-tuning drifts as a random walk through the parameter space; GRPO does not. A. Total drift ∥ΔθT∥2 for Qwen and Llama, each divided by its own random-walk scaling α2d/N so that the theory becomes the identity (dashed). Inset: GRPO with the same normalisation. B. Individual Qwen tensors against the same scaling (dashed): qkv_proj follows it for the whole run, post_ln saturates away from it. C. Effective dimension ratio ρ=deff/dtotal against fit quality R2 for every tensor. All weight matrices collapse onto the ideal scaling R2=1,ρ=1 , the norm vectors do not. D. GRPO moves along the low-dimensional task-relevant subspace (red). ES accumulates both a task-relevant component and a large off-manifold random walk (green), yielding a much larger displacement nearly orthogonal to GRPO. The two solutions are linearly connected because the off-manifold component is loss-invariant.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Countdown
Math
SciK
BoolQ
Base
20.78 ± 0.20
61.40 ± 0.35
63.77 ± 0.32
83.97 ± 0.23
After Countdown
74.52 ± 0.70
65.20 ± 0.72
64.35 ± 0.80
83.43 ± 0.95
After Math
75.40 ± 0.31
71.73 ± 0.42
69.45 ± 0.46
84.20 ± 1.18
After Chemistry
72.67 ± 1.03
71.40 ± 1.40
74.85 ± 1.01
84.03 ± 2.29
After BoolQ
64.87 ± 8.78
69.93 ± 1.01
73.80 ± 1.59
87.53 ± 0.60
Appendix
Table 5: Qwen3-4B-Instruct: accuracy (%, mean ± std over 3 seeds) throughout sequential training with ES (iter=100). Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
Base
17.22 ± 2.11
66.20 ± 4.01
62.77 ± 0.68
85.20 ± 1.39
After Countdown
69.17 ± 4.86
67.27 ± 6.47
60.58 ± 4.85
85.52 ± 1.78
After Math
70.32 ± 3.49
69.80 ± 5.89
61.63 ± 2.03
85.57 ± 1.69
After Chemistry
70.65 ± 4.01
71.67 ± 5.03
64.47 ± 7.34
82.10 ± 8.00
After BoolQ
70.50 ± 3.77
74.27 ± 5.80
66.08 ± 7.05
89.80 ± 1.31
Appendix
Table 6: Qwen3-4B-Instruct: accuracy (%, mean ± std over 3 seeds) throughout sequential training with GRPO. Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
Base
2.4
38.2
60.6
73.4
After Countdown
37.4
41.2
60.9
75.8
After Math
34.9
41.8
64.0
71.6
After Chemistry
29.3
41.0
65.9
68.8
After BoolQ
20.3
38.2
65.7
81.9
Appendix
Table 7: Llama-3.2-3B-Instruct: accuracy (%) throughout sequential training with ES (iter=100). Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
Base
1.8
39.4
61.5
73.4
After Countdown
37.8
42.2
62.6
75.3
After Math
40.8
42.0
62.4
75.9
After Chemistry
40.4
44.2
65.8
72.2
After BoolQ
40.3
46.4
67.0
83.0
Appendix
Table 8: Llama-3.2-3B-Instruct: accuracy (%) throughout sequential training with GRPO. Bold indicates the task just trained at that stage.
Config
σ
α
1
0.0005
0.00025
2
0.001
0.0005
3
0.0015
0.00075
Appendix
Table 9: ES hyperparameter configurations. Config 3 selected.
Table 11: ES: median [min, max] incremental test KL divergence across 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.
Eval. task
After Countdown
After Math
After Chemistry
After BoolQ
Countdown
0.381 [0.293, 0.454]
0.008 [0.006, 0.012]
0.012 [0.004, 0.030]
0.017 [0.005, 0.029]
Math
0.050 [0.048, 0.056]
0.009 [0.003, 0.010]
0.010 [0.007, 0.016]
0.006 [0.005, 0.011]
Chemistry
0.012 [0.000, 0.115]
0.010 [0.000, 0.019]
0.117 [0.072, 3.738]
0.056 [0.017, 0.105]
BoolQ
0.051 [0.035, 0.061]
0.010 [0.003, 0.010]
0.038 [0.028, 4.431]
0.239 [0.094, 0.322]
Appendix
Table 12: GRPO: median [min, max] incremental test KL divergence across 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
After Countdown
0.206 ± 0.023
0.062 ± 0.008
0.132 ± 0.014
0.133 ± 0.006
After Math
0.436 ± 0.032
0.178 ± 0.021
0.755 ± 0.477
0.355 ± 0.079
After Chemistry
0.536 ± 0.059
0.250 ± 0.033
2.541 ± 2.390
0.545 ± 0.211
After BoolQ
0.551 ± 0.112
0.288 ± 0.028
2.013 ± 1.781
0.545 ± 0.158
Appendix
Table 13: Qwen3-4B-Instruct: cumulative KL divergence from base model, ES (iter=100). Mean ± std over 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
After Countdown
0.376 ± 0.081
0.052 ± 0.004
0.040 ± 0.066
0.049 ± 0.013
After Math
0.420 ± 0.113
0.069 ± 0.018
0.087 ± 0.129
0.072 ± 0.025
After Chemistry
0.435 ± 0.143
0.093 ± 0.038
1.297 ± 2.120
1.904 ± 3.140
After BoolQ
0.435 ± 0.115
0.091 ± 0.024
1.282 ± 2.062
0.194 ± 0.063
Appendix
Table 14: Qwen3-4B-Instruct: cumulative KL divergence from base model, GRPO. Mean ± std over 3 seeds (42, 52, 62). Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
After Countdown
0.069
0.020
0.074
0.236
After Math
0.071
0.038
0.127
0.536
After Chemistry
0.095
0.058
0.396
1.644
After BoolQ
0.170
0.066
1.362
1.788
Appendix
Table 15: Llama-3.2-3B-Instruct: cumulative KL divergence from base model, ES (iter=100). Single seed. Bold indicates the task just trained at that stage.
Stage
Countdown
Math
SciK
BoolQ
After Countdown
0.072
0.011
0.094
0.312
After Math
0.081
0.012
0.107
0.347
After Chemistry
0.091
0.022
2.306
1.411
After BoolQ
0.099
0.023
2.330
1.717
Appendix
Table 16: Llama-3.2-3B-Instruct: cumulative KL divergence from base model, GRPO. Single seed. Bold indicates the task just trained at that stage.
Figure 7: GRPO single-task training reward across 20 epochs for each task.
Figure 8: ES single-task training reward across 300 steps for each task.
ES
Noise scale ( σ )
0.0015
Step size ( α )
0.00075
Population size
30
Batch size
200
Appendix
Table 17: Training hyperparameters for ES (left) and GRPO (right).
Method
FLOPs
vs. GRPO
GRPO ( T=80 )
8(80)(63)(8)PL=322,560PL
1.00×
ES (100 iter)
2(100)(30)(200)PL=1,200,000PL
3.72×
ES (300 iter)
2(300)(30)(200)PL=3,600,000PL
11.16×
Appendix
Table 18: Compute per task, relative to GRPO.
Task
Answer format
∣Dtest∣
Answer extraction and scoring
Countdown
\boxed{expr}
2,000
Extract last \boxed{} via regex. Verify the expression contains only digits and +,−,∗,/ ; each input number is used exactly once; and the evaluated result equals the target (tolerance <10−5 ).
MATH
\boxed{ans}
500
Extract last \boxed{} with nested-brace handling. Normalize both prediction and gold by stripping LaTeX wrappers ( \text , \dfrac , currency/degree symbols). Compare as strings; on mismatch, convert fractions and decimals to floats and compare numerically (tolerance <10−6 ).
SciKnowEval Chem.
"answer":"X"
2,000
Extract answer letter via regex cascade: (1) JSON-style "answer":"X" , (2) unquoted answer: X , (3) last standalone A/B/C/D token. Use the last match found.
BoolQ
\boxed{yes/no}
2,000
Extract last \boxed{} , normalize to lowercase. Accept {yes, true, 1} as positive and {no, false, 0} as negative.
Holdout tasks (not trained on; used to measure forgetting)
MMLU
"answer":"X"
14,042
Same regex cascade as SciKnowEval Chemistry.
Appendix
Table 19: Evaluation protocol per task. All tasks use greedy decoding (temperature =0 ), 1,024 max generated tokens, and binary scoring.
Figure 9: Total weight drift under ES follows the random-walk law; under GRPO it does not. Squared displacement of the full parameter vector ∥ΔθT∥2 against training step T for Qwen ( A , d=4.02×109 ) and Llama ( B , d=3.21×109 ). Solid blue, ES; dashed black, the prediction α2Td/N with α=σ/2 , σ=0.0015 and N=30 . The R2 in the legend is measured against that prediction, which has no fitted parameter; the annotation gives instead the R2 of a freely fitted slope and the effective dimension ρ=deff/dtotal that slope implies. Insets show GRPO on its own axes over its full 79 steps, annotated with the ES/GRPO displacement ratio at the matched step T=79 . Values and interpretation in the text.
Figure 10: Per-parameter effective dimension and agreement with the random-walk theory, Llama vs Qwen. For every parameter tensor we fit ∥ΔθT∥2=α2Tdeff/N through the origin over 500 ES steps and report ρ=deff/dtotal ( A , B ) and the R2 of that fit ( C , D ), for the large weight matrices (left) and the RMSNorm gain vectors (right). Faint dots are individual tensors — 28 per per-layer family in Llama, 36 in Qwen — and solid markers give the median with the interquartile range. Two reference levels are drawn, distinguished by colour: the dashed black line is the null model in which a tensor random-walks through its full parameter space, ρ=1 ; the dotted grey line is a perfect fit, R2=1 . Under each name is the tensor’s shape in each model (blue, Llama; red, Qwen); shading marks the QK-norm families, present only in Qwen. The fit has no intercept, so R2 is free to be negative, as it is throughout much of D ; the axis in C is correspondingly expanded, spanning only the top 2×10−4 of the R2 range. The large matrices cluster tightly at ρ≈1 with R2 indistinguishable from unity at ordinary resolution, while the norm parameters scatter across the full range of both quantities; the weight-growth curves underlying the latter are shown in Figure 11 . Qwen’s layers.34.input_layernorm is excluded throughout ( n=35 for that family; see text).
Figure 11: Per-parameter weight drift under ES: where the random-walk law holds and where it breaks. Squared displacement ∥ΔθT∥2 against ES step T for individual parameter tensors of Llama ( A – D , blue) and Qwen ( E – H , red), one parameter family per column. Dashed lines give the random-walk prediction α2Tdp/N , where dp is the number of entries in the tensor ( α=σ/2 , σ=0.0015 , N=30 ); it is common to every curve in a panel, since all tensors in a family share a shape. Curves are individual tensors shaded light to dark by layer depth — three evenly spaced layers for qkv_proj , six for input_ln and post_ln , and the single final norm for norm . Annotations give the median R2 and median ρ=deff/dtotal over the entire family, not only the curves drawn; R2 is that of a fit through the origin, as the law has no intercept, and may therefore be negative. The large weight matrices ( A , E ) lie on the predicted line for the whole run, while the RMSNorm gains rise steeply and then saturate — the departure that the negative R2 of a through-origin fit records. Values in the text.
Stage
Qwen3-4B cos
Llama-3.2-3B cos
T0 Countdown
1.0×10−4
0.8×10−4
T1 Math
1.6×10−4
1.0×10−4
T2 Chemistry
2.2×10−4
1.0×10−4
T3 BoolQ
2.7×10−4
1.3×10−4
Appendix
Table 20: Empirical cosine similarity between ES and GRPO update directions at each sequential checkpoint.
Figure 12: Wick contraction classes for E[(ϵ⊤Qϵ)2ϵeϵf] , with external loose ends e,f shown as crosses.
Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.
Yunpeng Ba, Zhi Zheng, Yue Xie +7
Southern University of Science and Technology · National University of Singapore · Harbin Institute of Technology, Weihai +2
Evolution Strategies (ES) enable memory efficient full parameter fine-tuning of large language models (LLMs) using only forward computation. However, standard ES uniformly averages rewards across problems and compresses problem level population feedback into a single scalar, making it difficult to capture how the learning value of each problem changes with model capability. To address this limitation, we propose Difficulty-Aware Reweighting and Targeted Replay for Evolution Strategies (DART-ES). DART-ES estimates the local solvability of each problem from its pass rate across the perturbation population and aggregates historical observations to construct a dynamic difficulty state. This shared state jointly guides continuous difficulty reweighting and rare solvable sample replay, thereby improving perturbation direction evaluation and training data allocation without introducing an additional difficulty model or backpropagation. Extensive experiments show that DART-ES achieves good fine-tuning performance. DART-ES outperforms ES on all five base models and improves the average accuracy from 72.07% to 73.53%, exceeding the 73.26% achieved by GRPO on GSM8K. Across five challenging mathematical reasoning benchmarks, DART-ES achieves an average accuracy of 49.20%, compared with 48.34% for ES and remains competitive with strong 7B models trained with RL. Further experiments show consistent gains in instruction tuning, code generation and the Countdown task with a 14B model, demonstrating strong generalization across tasks and scalability to larger models. Beyond performance gains, DART-ES also shows clear advantages in system efficiency. It reduces runtime per step by 15.2%--50.2% and peak memory usage per GPU by 21.1%--51.1% compared with GRPO. Despite performing full parameter fine-tuning, DART-ES also requires less runtime and GPU memory than GRPO+LoRA.
Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.
Xin Qiu, Yulu Gan, Conor F. Hayes +6
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1