Evolution Strategies for Deep RL pretraining

Adrian Martínez; Ananya Gupta; Hanka Goralija; Mario Rico; Saúl Fenollosa; Tamar Alphaidze

cs.LGMar 31, 2026

Evolution Strategies for Deep RL pretraining

Authors: Adrian Martínez, Ananya Gupta, Hanka Goralija, Mario Rico, Saúl Fenollosa, Tamar Alphaidze

Organizations: École Polytechnique Fédérale de Lausanne (EPFL)

Abstract

Although Deep Reinforcement Learning has proven highly effective for complex decision-making problems, it demands significant computational resources and careful parameter adjustment in order to develop successful strategies. Evolution strategies offer a more straightforward, derivative-free approach that is less computationally costly and simpler to deploy. However, ES generally do not match the performance levels achieved by DRL, which calls into question their suitability for more demanding scenarios. This study examines the performance of ES and DRL across tasks of varying difficulty, including Flappy Bird, Breakout and Mujoco environments, as well as whether ES could be used for initial training to enhance DRL algorithms. The results indicate that ES do not consistently train faster than DRL. When used as a preliminary training step, they only provide benefits in less complex environments (Flappy Bird) and show minimal or no improvement in training efficiency or stability across different parameter settings when applied to more sophisticated tasks (Breakout and MuJoCo Walker).

Figures & tables

Figure 1: Smoothed learning curves for ES, DQN, and ES-pretrained DQN versus cumulative training time in Flappy Bird environment

Figure 2: Smoothed learning curves for ES and DQN versus cumulative training time in Breakout environment

Panel of Figure 3 — Figure 3: Performance comparison of ES, PPO, and ES pretraining across various Mujoco environments.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
Sep 29, 2025Xin Qiu, Yulu Gan, Conor F. Hayes +6Fine-Tuning Evolutionary Optimization
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1
DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies
Oct 4, 2026Zhishen Sun, Hongzhan Wang, Sizhe Dang +2LLM Fine-Tuning Evolution Strategies
Xi’an Jiaotong University · SGIT AI Lab
Combining Trained Models in Reinforcement Learning
May 4, 2026Ujjwal Patil, Javad GhofraniReinforcement Learning Knowledge Distillation
Department of Computer Science Hochschule Bonn-Rhein-Sieg (H-BRS) Sankt Augustin, Germany

Sep 29, 2025J/K move · Enter open · S save

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang+5

Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA+1

Fine-Tuning Evolutionary Optimization LLM Fine-Tuning Evolution Strategies RL Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Abstract

Reinforcement Learning

Explore similar work

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies

Combining Trained Models in Reinforcement Learning