How is it possible for perturbation-based methods like Evolution Strategies (ES) to optimize billion-parameter large language models (LLMs)? Classical search methods are highly dependent on finding good candidates within generated populations. However, when using ES to fine-tune LLMs, a phenomenon can occur where all offspring models consistently perform worse than the parent model, yet during training, the parent model consistently improves. This paradox occurs across models, tasks, and even modalities. This paper characterizes the paradox, then examines how the local reward landscape and properties of the ES estimator contribute to it: in high-dimensional spaces, exploration leads to worse performance, yet still highlights good directions; aggregating those directions then allows for making local improvements. The result can be seen as a new optimization paradigm where a principled interaction of noise and curvature produces the paradox as a byproduct of effective optimization. These findings provide guidance for taking advantage of perturbation-based methods in LLM fine-tuning and in high-dimensional optimization in general.
Figures & tables
Figure 1: The Evolution Strategies Paradox across three examples: OLMo 2 7B fine-tuned on Sudoku, Qwen 2.5 7B fine-tuned on Countdown, and a 60K parameter CNN on MNIST. Left plots show the fraction of offspring models better than the parent model; right plots show the training curves (i.e. the performance of the parent models over time). In each case, offspring models perform worse than the parent model for nearly all of training, while the parent models consistently improve.
Figure 2: Tracking the ES paradox: (a) A visual representation of the ES paradox over iterations in a training run. At each iteration, offspring are generated which are all worse than the parent model in measured reward. (b) Fraction of perturbations better than the parent in the Sudoku fine-tuning task over iterations in the training run. Larger batch size leads to less noise in the reward landscape, and makes the paradox more prominent.
Figure 3: Reward distributions at different perturbation magnitudes. Checkpoints in Countdown and Sudoku training runs were perturbed 300 times at each specified magnitude, and distributions of rewards were plotted. As the magnitudes increase, the means decrease, and the variance increases. Thus, the offspring are likely to perform worse than the parent.
Figure 4: Effect of perturbation magnitude: a) A comparison in fine-tuning performance on Sudoku with perturbation magnitudes where the paradox occurs ( σ=1e−3 ) vs perturbation magnitudes where offspring produce viable improvement ( σ∈{1e−4,2.5e−4} ). b) The fraction of better offspring compared to the parent under different perturbation magnitudes. Even though the smaller perturbation magnitudes produce better candidates, the larger perturbations produce better directions and thus a better overall training run.
Figure 5: Correlation between rewards at various perturbation magnitudes σ at iterations 200 and 500 iterations in the Sudoku Fine-Tuning task. The σ range at which the paradox occurs, i.e. the best of the 30 offspring has a lower reward than the parent, is labeled with red arrows. The rewards at small and large σ correlate significantly. Thus, the same good directions can be identified at various perturbation magnitudes, even at which all the perturbations are worse than the parent.
Figure 6: Best-of-30 selection ablation: (a) Training performance of scaled best-of-30 perturbation selection compared with the actual aggregated ES update. (b) Fraction of the perturbations with the best-of-30 method that are better than the parent. When scaled from a size where the best perturbation performs worse to the size of a typical ES update, the model is able to improve. However, the aggregated ES update direction still significantly outperforms this best-of-30 direction, suggesting that aggregation contributes further to performance.
Figure 7: Average accuracy of Qwen 2.5 7B on Sudoku task along a random direction from −2σ to +2σ . Left: Landscape on a set of 800 problems. Right: Landscape on a set of 50 problems. A significant portion of the variance occurs at frequencies much higher than the perturbation size used in ES, and can thus be treated as observation noise.
Figure 8: Update reward improvement and update magnitude vs. population size. (a) Distributions of reward improvements from ES updates for different population sizes for CNN on MNIST. (b) Magnitude of ES updates in MNIST. (c) Mean and confidence intervals of improvement from ES updates at different population sizes for the Sudoku LLM fine-tuning task. (d) Magnitude of ES update for various population sizes. Distributions were wide, so for clarity, only means and CIs are shown. The model was Qwen 2.5 7B. In both settings, update reward improvement increases with population size and then plateaus, while the norm of the ES update decreases with population size.
Figure 9: (a): A visual representation of reward vs. perturbation magnitude, demonstrating why the offspring are worse than the parent. Perturbations happen at a fixed size; if they are too small, observation noise drowns out differences between good and bad directions, and if they are too big, the correlation between perturbation reward and direction quality breaks down. The paradox occurs when the magnitude is at a distance where all perturbations have a reward worse than the current model, but they still differentiate the good directions from the bad. (b) A demonstration of the tradeoff in perturbation size empirically, with performance of Qwen 2.5 7B on the Sudoku task after 300 ES iterations varying σ , holding all else constant. Results illustrate observation noise deteriorating performance at small values of σ , and correlations breaking down at large values, demonstrating the tradeoff visualized by (a).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Landscape
Signal
Noise
Expected reward change
Linear, exact rewards
α
α(d−1)/N
α∥g∥
Linear, observation noise
αρ
α(d−1)/N
αρ∥g∥
Quadratic, observation noise
αρ
α(d−1)/N(1+δQ)1/2
αρ∥g∥−2Nα2tr[Q]−2α2ρ2λg
Appendix
Table 1: Per-update signal E[g^⊤Δθ] , noise E∥Δθ⊥∥2 , and expected reward change E[ΔR] under the three landscape models, from equation 31 –equation 33 , equation 37 –equation 40 , and equation 44 –equation 49 . Throughout, ρ=σ∥g∥/σR is the gradient correlation of the population rewards, equal to 1 and c in the two linear cases; λg=g^⊤Qg^ ; and δQ=2σ4tr[P⊥Q2P⊥]/((d−1)σR2)≤4/(d−1) . The O(1/N) and tr[Q3] terms of equation 49 are omitted.
Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.
Xin Qiu, Yulu Gan, Conor F. Hayes +6
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1
Evolution Strategies (ES) enable memory efficient full parameter fine-tuning of large language models (LLMs) using only forward computation. However, standard ES uniformly averages rewards across problems and compresses problem level population feedback into a single scalar, making it difficult to capture how the learning value of each problem changes with model capability. To address this limitation, we propose Difficulty-Aware Reweighting and Targeted Replay for Evolution Strategies (DART-ES). DART-ES estimates the local solvability of each problem from its pass rate across the perturbation population and aggregates historical observations to construct a dynamic difficulty state. This shared state jointly guides continuous difficulty reweighting and rare solvable sample replay, thereby improving perturbation direction evaluation and training data allocation without introducing an additional difficulty model or backpropagation. Extensive experiments show that DART-ES achieves good fine-tuning performance. DART-ES outperforms ES on all five base models and improves the average accuracy from 72.07% to 73.53%, exceeding the 73.26% achieved by GRPO on GSM8K. Across five challenging mathematical reasoning benchmarks, DART-ES achieves an average accuracy of 49.20%, compared with 48.34% for ES and remains competitive with strong 7B models trained with RL. Further experiments show consistent gains in instruction tuning, code generation and the Countdown task with a 14B model, demonstrating strong generalization across tasks and scalability to larger models. Beyond performance gains, DART-ES also shows clear advantages in system efficiency. It reduces runtime per step by 15.2%--50.2% and peak memory usage per GPU by 21.1%--51.1% compared with GRPO. Despite performing full parameter fine-tuning, DART-ES also requires less runtime and GPU memory than GRPO+LoRA.
Evolution Strategies (ES) have emerged as a scalable gradient-free alternative to reinforcement learning based LLM fine-tuning, but it remains unclear whether comparable task performance implies comparable solutions in parameter space. We compare ES and Group Relative Policy Optimization (GRPO) across four tasks in both single-task and sequential continual-learning settings. ES matches or exceeds GRPO in single-task accuracy and remains competitive sequentially when its iteration budget is controlled. Despite this similarity in task performance, the two methods produce markedly different model updates: ES makes much larger changes and induces broader off-task KL drift, whereas GRPO makes smaller, more localized updates. Strikingly, the ES and GRPO solutions are linearly connected with no loss barrier, even though their update directions are nearly orthogonal. We develop an analytical theory of ES that explains all these phenomena within a unified framework, showing how ES can accumulate large off-task movement on weakly informative directions while still making enough progress on the task to match gradient-based RL in downstream accuracy. These results show that gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation. The source code is publicly available: https://github.com/Bhoy1/ESvsGRPO.
William Hoy, Binxu Wang, Xu Pan
Department of Industrial and Systems Engineering, University of Miami · Kempner Institute, Harvard University · Harvard University