Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models

Alkis Sygkounas; Amy Loutfi; Andreas Persson

cs.LGMar 30, 2026

Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models

Authors: Alkis Sygkounas, Amy Loutfi, Andreas Persson

Organizations: Machine Perception and Interaction Lab, Örebro University, Sweden

Abstract

Reinforcement learning algorithms are defined by their learning update rules, which are typically hand-designed and fixed. We present an evolutionary framework for discovering reinforcement learning algorithms by searching directly over executable update rules that implement complete training procedures. The approach builds on REvolve, an evolutionary system that uses large language models as generative variation operators, and extends it from reward-function discovery to algorithm discovery. To promote the emergence of nonstandard learning rules, the search excludes canonical mechanisms such as actor--critic structures, temporal-difference losses, and value bootstrapping. Because reinforcement learning algorithms are highly sensitive to internal scalar parameters, we introduce a post-evolution refinement stage in which a large language model proposes feasible hyperparameter ranges for each evolved update rule. Evaluated end-to-end by full training runs on multiple Gymnasium benchmarks, the discovered algorithms achieve competitive performance relative to established baselines, including SAC, PPO, DQN, and A2C.

Figures & tables

Figure 1. Illustrative overview of the proposed method. A population of candidate algorithms is iteratively evolved. In each generation (solid arrows), a large language model proposes coherent variants ( $\mathcal{L}_{f_{1}},\mathcal{L}_{f_{2}},\ldots,\mathcal{L}_{f_{n}}$ ), which are evaluated via training in Gymnasium environments to obtain fitness scores ( $\bar{F}^{(g)}_{k}$ ) that are subsequently used for selection and population updates. Post-evolution (dashed arrows), a hyperparameter setting ( $\beta_{i}^{\star}$ ), selected via LLM-guided optimization, optimizes the resulting best update rule ( $\mathcal{L}^{\star}$ ), which is then advanced for final evaluation.

Panel of Figure 2 — Figure 2. Representative Gymnasium environments used for training (top row) and evaluation (bottom row).

Figure 3. Evolution of maximum population fitness across generations for GPT-5.2 and Claude 4.5 Opus. Curves show the mean across two evolutionary seeds, with shaded regions indicating the standard deviation across seeds.

Environment	PPO	A2C	DQN	SAC	CG-FPD	DF-CWP-CP
CartPole	$\mathbf{500.0\pm 0.0}$	$\mathbf{500.0\pm 0.0}$	$\mathbf{500.0\pm 0.0}$	–	$\mathbf{500.0\pm 0.0}$	$\mathbf{500.0\pm 0.0}$
LunarLander	$246.60\pm 30.9$	$246.60\pm 13.4$	$250.10\pm 4.10$	–	$241.20\pm 11.0$	$\mathbf{260.60\pm 19.12}$
MountainCar	$-128.12\pm 44.64$	$-134.20\pm 8.70$	$-147.80\pm 40.40$	–	$\mathbf{-105.80\pm 10.51}$	$-108.67\pm 7.34$
Acrobot	$-63.5\pm 0.8$	$-213.6\pm 202.5$	$\mathbf{-61.90\pm 0.10}$	–	$-90.6\pm 5.86$	$-78.67\pm 8.72$
HalfCheetah	$1579.13\pm 643.80$	$795.91\pm 147.70$	–	$\mathbf{4988.5\pm 2667.70}$	$2407.88\pm 311.90$	$2103.80\pm 246.70$
Reacher	$-3.15\pm 0.14$	$-6.48\pm 0.57$	–	$\mathbf{-2.05\pm 0.19}$	$-2.67\pm 0.15$	$-5.43\pm 0.44$

Table 1. Mean

\pm

standard deviation of the maximum evaluation return per environment. Each algorithm is trained with five seeds; for each seed, the checkpoint with the highest evaluation return is evaluated over 100 episodes, and results report the mean and standard deviation across seeds. Dashes (–) indicate inapplicability (e.g., SAC for discrete environments, DQN for continuous control). PPO and A2C support both action types and are evaluated alongside the evolved methods ( CG-FPD and DF-CWP-CP) on all compatible environments.

Panel of Figure 4 — Figure 4. Seed-averaged evaluation learning curves for the two evolved algorithms across ten environments. Top: CG-FPD . Bottom: DF-CWP-CP. Curves show evaluation return smoothed with a moving average; shaded regions indicate one standard deviation across seeds. CartPole, InvertedPendulum, and MountainCar are trained for 500k steps, while all other environments are trained for 1M steps. Peak evaluation returns observed at the per-seed level are attenuated in the seed-averaged smoothed curves.

Figure 5. Ablation of the Levenshtein regularization weight

\alpha

in the evolutionary objective (Eq. 3.3 ). Curves show best population fitness per generation for

\alpha=0

and

\alpha=1

, averaged across evolutionary seeds.

Environment	CG-FPD (baseline)	CG-FPD + value bootstrap
MountainCar-v0	-105.80 ± 10.51	-122.12 ± 7.32
LunarLander-v3	241.20 ± 11.00	194.68 ± 7.04
Reacher	-2.67 ± 0.15	-3.27 ± 0.11

Table 2. Ablation of terminal value bootstrap added to CG-FPD . Results report mean

\pm

standard deviation of peak evaluation return over random seeds.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	State Dim.	Action Space	Action Type
CartPole-v1	4	$\{0,1\}$	Discrete
MountainCar-v0	2	$\{0,1,2\}$	Discrete
LunarLander-v3	8	$\{0,1,2,3\}$	Discrete
Acrobot-v1	6	$\{0,1,2\}$	Discrete
InvertedPendulum-v4	4	$\mathbb{R}^{1}$	Continuous
HalfCheetah-v5	17	$\mathbb{R}^{6}$	Continuous

Appendix

Table 3. Observation dimensionality and action space characteristics of the Gymnasium environments.

Environment	Reward
CartPole-v1	$+1$ per timestep the pole remains upright.
MountainCar-v0	$-1$ per timestep until the goal is reached.
LunarLander-v3	Shaped reward on position, velocity, and angle; fuel penalties; $+100$ for landing, $-100$ for crash.
Acrobot-v1	$-1$ per timestep until the upright terminal state is reached.
InvPendulum-v4	$+1$ per timestep while upright; termination on instability.
HalfCheetah-v5	Forward velocity minus control cost.

Appendix

Table 4. Reward definitions of the Gymnasium benchmark environments.

Automated Algorithm Discovery

Automated algorithm discovery develops methods that generate and test executable algorithms for optimization, science, software, and machine learning.

See all 165 papers

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +14Automated Algorithm Discovery Evolutionary RL
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +4
Discovering Multiagent Learning Algorithms with Large Language Models
Feb 18, 2026Zun Li, John Schultz, Daniel Hennes +1Counterfactual Regret Minimization Multi-Agent Reinforcement Learning
Google DeepMind
AGAR: a reinforcement learning substrate for LLM program evolution
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +11LLM-Based Program Synthesis RL for Code Generation
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +2

Oct 6, 2026J/K move · Enter open · S save

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

Haoran Li, Zengle Ge, Xiaomin Yuan, Yui Lo+13

University of the Chinese Academy of Sciences · Baidu · Tsinghua University+4

Automated Algorithm Discovery Evolutionary RL Model-Free RL Self-Evolving Agents Large Language Model-Guided Optimization Genetic Programming

Open paper AskPDF arXiv

LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is difficult to scale: simultaneous changes can disrupt learning, while isolated changes overlook their dependencies. Evaluating candidate algorithms also requires costly training, with fitness remaining uncertain across random seeds. We introduce RLDiscover, a framework for the self-evolution of model-free deep RL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation. Experiments across SAC, PPO, and DQN on four benchmark suites show substantial improvements in mean return, with per-family median gains of 32%-84% and a peak return ratio of approximately 363x over a near-zero baseline. These gains include transitions from failed learning to successful task completion, and improvements persist when evolution starts from stronger open-source implementations. On measured SAC locomotion runs, evaluation uses approximately one-fifteenth the estimated compute required to fully evaluate the same candidate pool. Remarkably, independent searches repeatedly discover interpretable combinations of adaptive robust losses, progress-dependent value targets, and running statistics, with selected programs transferring to unseen tasks. These findings point toward a broader role for self-evolution in AI: discovering interpretable algorithms that improve how agents learn.

Abstract

Automated Algorithm Discovery

Explore similar work

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

Discovering Multiagent Learning Algorithms with Large Language Models

AGAR: a reinforcement learning substrate for LLM program evolution