The Evolution Strategies Paradox: Better Models from Worse Perturbations

Roberto Dailey; Kajetan Schweighofer; Jamieson Warner; Conor F. Hayes; Elliot Meyerson; Babak Hodjat; Risto Miikkulainen; Xin Qiu

cs.AINov 3, 2026

The Evolution Strategies Paradox: Better Models from Worse Perturbations

Authors: Roberto Dailey, Kajetan Schweighofer, Jamieson Warner, Conor F. Hayes, Elliot Meyerson, Babak Hodjat, Risto Miikkulainen, Xin Qiu

Organizations: Cognizant AI Lab, San Francisco · The University of Texas at Austin, Austin

Abstract

How is it possible for perturbation-based methods like Evolution Strategies (ES) to optimize billion-parameter large language models (LLMs)? Classical search methods are highly dependent on finding good candidates within generated populations. However, when using ES to fine-tune LLMs, a phenomenon can occur where all offspring models consistently perform worse than the parent model, yet during training, the parent model consistently improves. This paradox occurs across models, tasks, and even modalities. This paper characterizes the paradox, then examines how the local reward landscape and properties of the ES estimator contribute to it: in high-dimensional spaces, exploration leads to worse performance, yet still highlights good directions; aggregating those directions then allows for making local improvements. The result can be seen as a new optimization paradigm where a principled interaction of noise and curvature produces the paradox as a byproduct of effective optimization. These findings provide guidance for taking advantage of perturbation-based methods in LLM fine-tuning and in high-dimensional optimization in general.

Figures & tables

Figure 1: The Evolution Strategies Paradox across three examples: OLMo 2 7B fine-tuned on Sudoku, Qwen 2.5 7B fine-tuned on Countdown, and a 60K parameter CNN on MNIST. Left plots show the fraction of offspring models better than the parent model; right plots show the training curves (i.e. the performance of the parent models over time). In each case, offspring models perform worse than the parent model for nearly all of training, while the parent models consistently improve.

Panel of Figure 2 — Figure 2: Tracking the ES paradox: (a) A visual representation of the ES paradox over iterations in a training run. At each iteration, offspring are generated which are all worse than the parent model in measured reward. (b) Fraction of perturbations better than the parent in the Sudoku fine-tuning task over iterations in the training run. Larger batch size leads to less noise in the reward landscape, and makes the paradox more prominent.

Figure 3: Reward distributions at different perturbation magnitudes. Checkpoints in Countdown and Sudoku training runs were perturbed 300 times at each specified magnitude, and distributions of rewards were plotted. As the magnitudes increase, the means decrease, and the variance increases. Thus, the offspring are likely to perform worse than the parent.

Panel of Figure 4 — Figure 4: Effect of perturbation magnitude: a) A comparison in fine-tuning performance on Sudoku with perturbation magnitudes where the paradox occurs ( $\sigma=1e-3$ ) vs perturbation magnitudes where offspring produce viable improvement ( $\sigma\in\{1e-4,2.5e-4\}$ ). b) The fraction of better offspring compared to the parent under different perturbation magnitudes. Even though the smaller perturbation magnitudes produce better candidates, the larger perturbations produce better directions and thus a better overall training run.

Figure 5: Correlation between rewards at various perturbation magnitudes $\sigma$ at iterations 200 and 500 iterations in the Sudoku Fine-Tuning task. The $\sigma$ range at which the paradox occurs, i.e. the best of the 30 offspring has a lower reward than the parent, is labeled with red arrows. The rewards at small and large $\sigma$ correlate significantly. Thus, the same good directions can be identified at various perturbation magnitudes, even at which all the perturbations are worse than the parent.

Panel of Figure 6 — Figure 6: Best-of-30 selection ablation: (a) Training performance of scaled best-of-30 perturbation selection compared with the actual aggregated ES update. (b) Fraction of the perturbations with the best-of-30 method that are better than the parent. When scaled from a size where the best perturbation performs worse to the size of a typical ES update, the model is able to improve. However, the aggregated ES update direction still significantly outperforms this best-of-30 direction, suggesting that aggregation contributes further to performance.

Figure 7: Average accuracy of Qwen 2.5 7B on Sudoku task along a random direction from $-2\sigma$ to $+2\sigma$ . Left: Landscape on a set of 800 problems. Right: Landscape on a set of 50 problems. A significant portion of the variance occurs at frequencies much higher than the perturbation size used in ES, and can thus be treated as observation noise.

Panel of Figure 8 — Figure 8: Update reward improvement and update magnitude vs. population size. (a) Distributions of reward improvements from ES updates for different population sizes for CNN on MNIST. (b) Magnitude of ES updates in MNIST. (c) Mean and confidence intervals of improvement from ES updates at different population sizes for the Sudoku LLM fine-tuning task. (d) Magnitude of ES update for various population sizes. Distributions were wide, so for clarity, only means and CIs are shown. The model was Qwen 2.5 7B. In both settings, update reward improvement increases with population size and then plateaus, while the norm of the ES update decreases with population size.

Panel of Figure 9 — Figure 9: (a): A visual representation of reward vs. perturbation magnitude, demonstrating why the offspring are worse than the parent. Perturbations happen at a fixed size; if they are too small, observation noise drowns out differences between good and bad directions, and if they are too big, the correlation between perturbation reward and direction quality breaks down. The paradox occurs when the magnitude is at a distance where all perturbations have a reward worse than the current model, but they still differentiate the good directions from the bad. (b) A demonstration of the tradeoff in perturbation size empirically, with performance of Qwen 2.5 7B on the Sudoku task after 300 ES iterations varying $\sigma$ , holding all else constant. Results illustrate observation noise deteriorating performance at small values of $\sigma$ , and correlations breaking down at large values, demonstrating the tradeoff visualized by (a).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Landscape	Signal	Noise	Expected reward change
Linear, exact rewards	$\alpha$	$\alpha\sqrt{(d-1)/N}$	$\alpha\\|g\\|$
Linear, observation noise	$\alpha\rho$	$\alpha\sqrt{(d-1)/N}$	$\alpha\rho\\|g\\|$
Quadratic, observation noise	$\alpha\rho$	$\alpha\sqrt{(d-1)/N}\,(1+\delta_{Q})^{1/2}$	$\alpha\rho\\|g\\|-\dfrac{\alpha^{2}}{2N}\operatorname{tr}[Q]-\dfrac{\alpha^{2}\rho^{2}}{2}\lambda_{g}$

Appendix

Table 1: Per-update signal

\mathbb{E}[\hat{g}^{\!\top}\Delta\theta]

, noise

\sqrt{\mathbb{E}\|\Delta\theta_{\perp}\|^{2}}

, and expected reward change

\mathbb{E}[\Delta R]

under the three landscape models, from equation 31 –equation 33 , equation 37 –equation 40 , and equation 44 –equation 49 . Throughout,

\rho=\sigma\|g\|/\sigma_{R}

is the gradient correlation of the population rewards, equal to

1

and

c

in the two linear cases;

\lambda_{g}=\hat{g}^{\!\top}Q\hat{g}

; and

\delta_{Q}=2\sigma^{4}\operatorname{tr}[P_{\perp}Q^{2}P_{\perp}]/((d-1)\sigma_{R}^{2})\leq 4/(d-1)

. The

O(1/N)

and

\operatorname{tr}[Q^{3}]

terms of equation 49 are omitted.

Evolution Strategies

Evolution strategies are gradient-free, population-based methods that optimize black-box objectives through parameter perturbations and selection.

See all 35 papers

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
Sep 29, 2025Xin Qiu, Yulu Gan, Conor F. Hayes +6Fine-Tuning Evolutionary Optimization
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1
DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies
Oct 4, 2026Zhishen Sun, Hongzhan Wang, Sizhe Dang +2LLM Fine-Tuning Evolution Strategies
Xi’an Jiaotong University · SGIT AI Lab
Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training
Apr 2, 2026William Hoy, Binxu Wang, Xu PanEvolution Strategies RL Post-Training
Department of Industrial and Systems Engineering, University of Miami · Kempner Institute, Harvard University · Harvard University

Sep 29, 2025J/K move · Enter open · S save

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang+5

Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA+1

Fine-Tuning Evolutionary Optimization LLM Fine-Tuning Evolution Strategies RL Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Abstract

Evolution Strategies

Explore similar work

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies

Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training