RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

Haoran Li; Zengle Ge; Xiaomin Yuan; Yui Lo; Songlin Zhou; Jiahua Ying; Haoxin Li; Qianhui Liu

cs.AIOct 6, 2026

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

Authors: Haoran Li, Zengle Ge, Xiaomin Yuan, Yui Lo, Songlin Zhou, Jiahua Ying, Haoxin Li, Qianhui Liu, +9 more

Organizations: University of the Chinese Academy of Sciences · Baidu · Tsinghua University · Institute of Automation, Chinese Academy of Sciences · Peking University · Shanghai University · University of Bristol

Abstract

LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is difficult to scale: simultaneous changes can disrupt learning, while isolated changes overlook their dependencies. Evaluating candidate algorithms also requires costly training, with fitness remaining uncertain across random seeds. We introduce RLDiscover, a framework for the self-evolution of model-free deep RL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation. Experiments across SAC, PPO, and DQN on four benchmark suites show substantial improvements in mean return, with per-family median gains of 32%-84% and a peak return ratio of approximately 363x over a near-zero baseline. These gains include transitions from failed learning to successful task completion, and improvements persist when evolution starts from stronger open-source implementations. On measured SAC locomotion runs, evaluation uses approximately one-fifteenth the estimated compute required to fully evaluate the same candidate pool. Remarkably, independent searches repeatedly discover interpretable combinations of adaptive robust losses, progress-dependent value targets, and running statistics, with selected programs transferring to unseen tasks. These findings point toward a broader role for self-evolution in AI: discovering interpretable algorithms that improve how agents learn.

Figures & tables

Figure 1: RLDiscover overview. Left : 5 algorithm components of RL algorithms (state representation, value estimation, critic learning, policy optimization, exploration). Center : PCE evolves components independently (Phase 1, UCB1 selection), then co-evolves them jointly (Phase 2). Right : PPE filters candidates through L0 $\to$ L1 $\to$ L2, so only candidates that survive cheap screening consume a full-length training run. Top : the LLM mutation engine generates code variants guided by performance feedback.

SAC, own-init (17 tasks): evolved vs. our own SAC baseline; evolved § — median $+39.8\%$
Task	Init	Baseline Return	Evolved Return	Fitness	Improvement
dm-cheetah-run	Ours	302.3 $\,\pm$ 37.3	646.9 $\,\pm$ 67.4	0.647	+114.0%
dm-walker-walk	Ours	679.6 $\,\pm$ 41.0	971.0 $\,\pm$ 3.3 †	0.971	+42.9%
dm-walker-run	Ours	487.7 $\,\pm$ 56.3	585.7 $\,\pm$ 7.1	0.586	+20.1%
dm-quadruped-walk	Ours	931.6 $\,\pm$ 12.5	939.8 $\,\pm$ 4.1 †	0.940	+0.9%
gym-HalfCheetah-v5	Ours	5093.6 $\,\pm$ 209.3	9053.7 $\,\pm$ 1554.1	0.604	+77.7%

Table 1: Full per-task results across the three algorithm families, all at L2 with 3 seeds

\{0,42,123\}

(protocol in Appendix D ; MountainCar-v0 is a 10-seed exception). Every block compares an evolved program against the baseline it was seeded from: our own SAC baseline (17 tasks), an open-source SOTA seed on all 9 locomotion tasks with an external reference (Section 4.2 ; no task is filtered out), and the standard PPO / DQN implementation of our framework. The SOTA-init block re-evolves the same 9 locomotion tasks and is not counted again in the 26. Fitness is the task-normalized return; Improvement is computed on raw returns; bold = improvement. Markers

\dagger

, § and ¶ are explained in Appendix F .

Panel of Figure 2 — Figure 2: Evolution finds algorithmic improvements rather than per-task overfits. (a) Frequency of independently discovered innovations across all 17 champions, coloured by the component each modifies. (b) Fitness $\Delta$ from running a source champion (column) on a target task (row); green = improvement, red = regression; boxed cells marked $*$ are the source task itself. Criteria, the full $17\times 6$ matrix and per-task values: Appendices I and H .

		Ablating PCE		Single-fidelity PPE
	Full	w/o decomp.	w/o Phase 1	L0	L2	L1
(1)	0.784 $\,\pm$ 0.09	0.762 $\,\pm$ 0.13	0.758 $\,\pm$ 0.14	0.728 $\,\pm$ 0.17	0.700 $\,\pm$ 0.08	0.691 $\,\pm$ 0.05
(2)	0.731 $\,\pm$ 0.23	0.687 $\,\pm$ 0.25	0.704 $\,\pm$ 0.24	0.687 $\,\pm$ 0.25	0.700 $\,\pm$ 0.21	0.694 $\,\pm$ 0.22
(3)	0.789 $\,\pm$ 0.08	0.731 $\,\pm$ 0.17	0.748 $\,\pm$ 0.13	0.727 $\,\pm$ 0.15	0.728 $\,\pm$ 0.10	0.736 $\,\pm$ 0.08

Table 2: Component ablation across the three algorithm families: audited L2 champion fitness, mean

\pm

cross-task std over 3 independent evolution runs per cell (

8

pairs

\times

6 conditions

\times

3 runs

=144

runs); per-task matrix in Table 10 , per-cell evolution spreads in Appendix N . The

\pm

is the standard deviation across tasks within each row.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	encode	value target	critic loss	policy loss	exploration
DQN	MLP	TD target	MSE	greedy ∗	$\epsilon$ -greedy
Rainbow	MLP	$n$ -step	distributional	greedy ∗	NoisyNet
DDPG / TD3	MLP	twin TD target	MSE	DPG	OU / Gaussian
A2C / PPO	MLP	GAE	MSE	clipped surrogate	entropy
SAC	MLP	soft Bellman	MSE	reparam. + entropy	entropy

Appendix

Table 3: Standard model-free algorithms populate the same five slots; the decomposition is a template rather than a SAC-specific factoring. ∗ For value-based methods the policy slot is degenerate , not absent: the policy is the greedy

\arg\max

over

Q

, so the slot is fixed rather than searched (we therefore exclude it from the search for DQN). We evolve the three families marked in bold.

Level	Steps	Seeds	Eval Eps	Measured wall-clock	Purpose
L0	50K	1	5	8–11 min (9.3)	Rapid screening
L1	200K	2	10	60–79 min (78)	Moderate validation
L2	1M	3	20	4.1–9.6 h (5.0)	Final confirmation

Appendix

Table 4: PPE evaluation levels. Only programs passing lower levels are promoted.

Task	Seed gap	own gain	SOTA gain	Final gap	Outcome
dm-cheetah-run	$+57.6\%$	$+114.0\%$	$+20.0\%$	$+24.4\%$	SOTA-init
dm-walker-walk	$+29.8\%$	$+42.9\%$	$+0.5\%$	$+0.2\%$	tie
dm-walker-run	$+35.2\%$	$+20.1\%$	$+2.6\%$	$+24.2\%$	SOTA-init
dm-quadruped-walk	$-0.7\%$	$+0.9\%$	$+1.7\%$	$+0.1\%$	tie
gym-HalfCheetah-v5	$+53.7\%$	$+77.7\%$	$+0.6\%$	$+18.2\%$	SOTA-init
gym-Hopper-v5	$-4.8\%$	$+12.4\%$	$+20.0\%$	$+1.8\%$	SOTA-init

Appendix

Table 5: The nine locomotion tasks both SAC batches were evolved on with 3 seeds. Seed gap : the SOTA-init baseline relative to the own-init baseline. own gain / SOTA gain : each batch’s L2 champion relative to its own seed, i.e. what that batch’s search contributed. Final gap and outcome : the two champions against each other, with the SOTA-init champion as denominator throughout. All figures are on raw returns, since fitness is normalised per task and the batches share a normalisation only within a task; a positive value always means the SOTA-init side is ahead. The final-gap column disaggregates the

23.4\%\to 13.1\%

mean narrowing; the mean of its absolute values is

13.1\%

. Note that “SOTA-init” names the batch, not a claim that its seed is stronger on every task—it starts behind on quadruped-walk and Hopper.

Task	Source	Obs Dim	Act Dim	Norm Bound
dm-cheetah-run	DMControl	17	6	1000
dm-walker-walk	DMControl	24	6	1000
dm-walker-run	DMControl	24	6	1000
dm-quadruped-walk	DMControl	78	12	1000
gym-HalfCheetah-v5	Gymnasium	17	6	15000
gym-Hopper-v5	Gymnasium	11	3	4000

Appendix

Table 6: SAC benchmark tasks with observation/action dimensions and normalization bounds. All 17 are evolved in the own-init batch; the 9 locomotion tasks (all DMControl and all Gymnasium entries) additionally have an external SOTA reference and were also run in the SOTA-init batch, all 9 of which are reported in Table 1 . The cross-algorithm experiments additionally use gym-Hopper/HalfCheetah/Ant/Walker2d/Humanoid-v5 for PPO and CartPole-v1, MountainCar-v0, Acrobot-v1, LunarLander-v3 for DQN.

	Source program ( $\Delta$ fitness)
Task	walker-walk*	quadruped*	walker2d*	Humanoid*	Baseline Fitness
dm-cheetah-run	+0.265	+0.269	$-$ 0.129	+0.313	0.302
dm-walker-walk	+0.292 *	+0.290	+0.243	+0.221	0.680
dm-walker-run	+0.091	+0.025	$-$ 0.319	+0.142	0.488
dm-quadruped-walk	+0.019	+0.006*	$-$ 0.669	$-$ 0.082	0.932
gym-HalfCheetah-v5	+0.058	+0.154	$-$ 0.148	+0.141	0.340

Appendix

Table 7: Cross-task L2 generalization, full

9\times 4

matrix: fitness improvement (

\Delta

) over the fixed L2 baseline when a champion evolved on one task is re-evaluated on each target. Source task marked with *; the summary rows exclude source cells. Bold = best source program for that target. Provenance. The matrix was evaluated as a single batch using the champions available at that time. The gym-Walker2d-v5 own-init champion was subsequently improved by resumed evolution (fitness

0.760\rightarrow 0.905

), so the entire walker2d column describes the earlier program. We did not substitute the newer champion into individual cells: only its own-task value was re-measured, so patching the diagonal alone would leave a column mixing two different programs and therefore not internally comparable. Two consequences follow. The walker2d diagonal here (

0.647+0.113=0.760

) is below the gym-Walker2d-v5 own-init row of Table 1 (

0.905

); and the weak transfer of that column is a property of the earlier program, not an established property of the champion reported in Table 1 .

Fraction of generated candidate programs discarded as AST-duplicates before screening;
Metric	Mean	Min	Max
L0 evaluations per run	230	230	230
L0 $\to$ L1 promotion rate	7.6%	4.8%	12.2%
Code deduplication rate	18%	2.5%	30%
the denominator is the $B{=}240$ budget, so $\approx$ 230 survive to L0.
L1 evaluations per run	17.5	11	28

Appendix

Table 8: PPE filtering efficiency across 6 locomotion tasks. Statistics are per run, over the full

B{=}240

candidate budget.

Figure 3: PPE filtering efficiency (per-run averages). (a) Program filtering funnel: from 230 L0-evaluated candidates to 5 L2-validated finalists. L0 filters 92% of programs at minimal cost. (b) GPU compute distribution: total

\sim

87 GPU-hours per run versus

\sim

1330 for evaluating all candidates at L2, a

\sim

\times

reduction.

SAC — DMControl
	Baseline			Evolved
Task	Seed 0	Seed 42	Seed 123	Seed 0	Seed 42	Seed 123
dm-cheetah-run	344.0	309.4	253.5	612.4	741.1	587.3
dm-walker-walk	626.0	687.4	725.5	974.3	972.4	966.5
dm-walker-run	553.1	415.6	494.4	583.7	595.2	578.1
dm-quadruped-walk	936.2	914.5	944.0	938.2	935.8	945.4

Appendix

Table 9: Per-seed L2 returns for all 17 SAC own-init tasks and all 9 PPO/DQN tasks (training seeds

\{0,42,123\}

; SAC/PPO 1M steps and 20 evaluation episodes, DQN 500K steps and 30 episodes). These are the values summarized in Table 1 .

\lx@paragraphsign

MountainCar-v0 is the one exception: because its outcome is bimodal (solve at

\approx-130

or fail at

-200

), Table 1 reports a 10-seed re-evaluation rather than a 3-seed one. The three seeds shown here are the

\{0,42,123\}

subset of that draw; the full 10 evolved returns are

-120.7

-141.2

-200.0

-153.9

-124.8

-200.0

-128.8

-124.4

-134.7

-120.6

(mean

-144.9

), and the baseline returns exactly

-200.0

on all 10. The 3-seed subset therefore does not average to the reported mean.

SAC
		Ablating PCE		Ablating PPE
Task	Full	w/o decomp.	w/o Phase 1	L0-only	L2-only	L1-only
dm-cheetah-run	0.685	0.639	0.622	0.556	0.613	0.636
gym-HalfCheetah-v5	0.763	0.702	0.694	0.682	0.649	0.671
md-100envs	0.907	0.945	0.954	0.957	0.754	0.728
mw-drawer-open-v3	0.780	0.763	0.763	0.718	0.782	0.730

Appendix

Table 10: Per-task component ablation: audited L2 champion fitness, mean over 3 independent evolution runs per cell (evolution seeds 42/123/2026). Best per row in bold.

Ablating PCE
Condition	dm-cheetah-run	HalfCheetah-v5	md-100envs	mw-drawer-open	Avg
Full method	0.685 $\pm$ 0.025	0.763 $\pm$ 0.048	0.907 $\pm$ 0.095	0.780 $\pm$ 0.027	0.784
w/o decomposition	0.639 $\pm$ 0.001	0.702 $\pm$ 0.083	0.945 $\pm$ 0.017	0.763 $\pm$ 0.051	0.762
w/o Phase 1	0.622 $\pm$ 0.018	0.694 $\pm$ 0.109	0.954 $\pm$ 0.011	0.763 $\pm$ 0.029	0.758
Ablating PPE
L0-only selection	0.556 $\pm$ 0.059	0.682 $\pm$ 0.071	0.957 $\pm$ 0.044	0.718 $\pm$ 0.035	0.728

Appendix

Table 11: SAC component ablation with per-cell evolution variance: audited L2 champion fitness, mean

\pm

std over 3 independent evolution runs (evolution seeds 42/123/2026). The

\pm

here is evolution variance, not the training variance reported in Table 1 ; the two are different quantities and should not be compared in magnitude.

\ddagger

The baseline was never evolved, so no evolution variance exists for it; the spread on that row is its training variance over the three seeds

\{0,42,123\}

and is not comparable to the evolution variance above it. Best per column in bold.

Task	Full	w/o Phase 1	w/o decomp.
LunarLander-v3	0.752 $\pm$ 0.027	0.638 $\pm$ 0.064	0.465 $\pm$ 0.084
MountainCar-v0 (saturated)	0.908 $\pm$ 0.006	0.948 $\pm$ 0.009	0.909 $\pm$ 0.049

Appendix

Table 12: DQN component ablation (3 independent evolution runs per cell; mean

\pm

std of the audited L2 champion fitness). LunarLander replicates the SAC ordering full

>

w/o Phase 1

>

w/o decomposition directionally. MountainCar is saturated—all conditions within 0.908–0.948—and is not read; no mean is taken across the two tasks. Best per row in bold.

Task	Encoding	Value Target	Critic Loss	Policy Loss	Exploration
DeepMind Control
dm-cheetah-run	$\mathrm{arcsinh}(x)$ , no rescale	Fixed $0.95\min$ $+\,0.05\max$	Huber ( $\beta{=}1$ ) $+\,10^{-5}Q^{2}$	Standard SAC $+\,10^{-3}$ mean L2	$5\times$ reward, EMA curiosity, action L1
dm-walker-walk	$5\,\mathrm{arcsinh}(x/5)$ , clip $[-25,25]$	Sine anneal mean $\to$ min, then variance penalty	Asym. Huber, EMA-adaptive $\delta$ , $\tau$ 0.55–0.75	Cosine min/max blend; asym. $\alpha$ LR $5\times$ / $1\times$	Dual-timescale EMA novelty, cosine decay
dm-walker-run	$5\,\mathrm{arcsinh}(x/5)$ + decaying noise	Per-sample min/mean from Q-disagreement	Asym. Huber, quantile-EMA $\delta$	Softmin Q ( $\beta$ 2 $\to$ 10), PI $\alpha$ controller	Convexified reward + movement synergy
dm-quadruped-walk	Clip $[-20,20]$ , then symlog	Cosine-annealed max-Q weight $0.1\to 0$	Scaled log-cosh ( $\beta{=}2$ ) $+\,10^{-5}Q^{2}$	Standard SAC $+\,10^{-3}$ action L2	$10\times$ reward + decaying movement bonus
Gym MuJoCo

Appendix

Table 13: Component implementations of the champion programs, DeepMind Control and Gym locomotion (9 tasks). “Standard SAC” and “unchanged” mark slots that are not meaningfully evolved away from the seed program;

p

denotes training progress in

[0,1]

Task	Encoding	Value Target	Critic Loss	Policy Loss	Exploration
Meta-World
mw-push-v3	Unused dims $\to$ distance/velocity features, clip $[-10,10]$	Softmin blend, $\kappa$ $5.0\to 0.5$ , $w\leq 0.4$	Asym. Huber, EMA $\delta$ , $\tau$ $0.8\to 0.5$	Batch-normalized Q + UCB disagreement bonus	Potential shaping ( $c{=}50/200$ ) + EMA curiosity
mw-drawer-open-v3	Inject distance/contact/alignment scalars, then symlog	Variance-aware min/mean, pessimism $1.0\to 0.2$	Quantile-EMA Huber (0.85), TD reweighting	Min-Q $-$ decaying disagreement; action penalties	Adaptive potential shaping, decay floor 0.2
mw-door-open-v3	Velocity rebuild + proximity features, scaled symlog	Min-Q $+\,0.25$ cosine-decayed optimism	Log-cosh Huber, EMA $\delta$ , $+\,10^{-4}Q^{2}$	$0.95\min$ $+\,0.05\max$ ; 0.05 disagreement penalty	Potential shaping + object motion + grasp term
mw-peg-insert-side-v3	History dims $\to$ velocity/direction features	Min-Q $+\,\beta(\max{-}\min)$ , $\beta$ $0.15\to 0$	Standard Huber, MAD-derived EMA $\delta$	Standard SAC (+ inactive saturation hinge)	Potential shaping (inverse/exp/grasp), $0.30\to 0.05$
mw-pick-place-v3	Overwrite dims 18–29 with TCP/object/goal vectors	Min-Q $+\leq 15\%$ disagreement-gated optimism	Huber, $\delta{=}$ EMA 80th-pct TD error	Standard SAC + mean L2 annealed to 0	Potential shaping on reach/place/ $z$ distances

Appendix

Table 14: Component implementations of the champion programs, Meta-World manipulation and MetaDrive driving (8 tasks). On these benchmarks the encoder is where most of the evolved content sits: every champion rewrites the raw observation vector into task-specific geometric features.

Task	Encoding	Value Target	Critic Loss	Policy Loss	Exploration
gym-Hopper-v5	Piecewise symlog above $\|x\|{>}5$ , clip $[-15,15]$	Cosine-annealed GAE $\lambda$ $0.98\to 0.90$ + MC-return blend (weight $0.1p$ )	Smooth-L1, $\beta$ $1.0\to 0.2$ , $+\,10^{-4}V^{2}$	Dual clipping ( $c{=}3$ ) + multiplicatively adapted KL coef. (target 0.015) + decaying entropy §	EMA-normalized state-difference novelty $-$ action L2 + survival bonus
gym-HalfCheetah-v5	Symlog $\mathrm{sign}(x)\log(1{+}\|x\|)$	Linear GAE $\lambda$ $0.98\to 0.90$	Huber ( $\beta{=}1$ ) $+$ decaying $5{\times}10^{-5}V^{2}$	Clip anneal $0.2\to 0.1$ , adv. clamp $\pm 5$ , KL penalty $0.02(1{-}p)$ , entropy	$-0.005\,\mathbb{E}[a^{2}]$ + decaying state-magnitude bonus
gym-Ant-v5	$5\tanh(x/5)$ + decaying Gaussian noise	Linear GAE $\lambda$ $0.95\to 0.85$ + advantage mean-centering	Adaptive Huber, $\beta$ $2.0\to 0.5$ , $+\,10^{-4}V^{2}$	Dual clipping ( $c{=}1.5$ ) + KL penalty 0.005 + entropy $0.01e^{-3p}$	Dimension-invariant action penalty + decaying survival bonus
gym-Walker2d-v5	Symmetric clip $[-10,10]$	Linear GAE $\lambda$ $0.95\to 0.90$	Adaptive- $\delta$ Huber, $\delta{=}$ batch median $\|$ TD $\|$ , clamp $[0.1,10]$	Dual clipping ( $c{=}3$ ) + hinge KL penalty $\mathrm{relu}(\mathrm{KL}{-}0.015)$ , entropy	Decaying action-energy penalty $-0.005\Sigma a^{2}(1{-}p)$
gym-Humanoid-v5	Piecewise log above $\|x\|{>}2$ , linear below	Dual- $\lambda$ GAE : parallel $\lambda{=}0.97$ and $0.90$ streams, averaged 50/50	Huber, fixed $\beta{=}10$ (MSE-like in range, linear on outliers)	Clip anneal $0.2\to 0.05$ + dual clipping + KL-mask pseudo early-stopping + KL penalty	Decaying L1 and L2 action penalties, clip $\pm 10$

Appendix

Table 15: Component implementations of the 5 PPO champion programs.

p

denotes training progress in

[0,1]

. Every champion inherits an unreachable off-policy (twin-

Q

) branch from the shared template, which is ignored here. § Entropy term with a sign opposite to our trainer’s convention: see the caveats below.

Task	Encoding	Value Target	Critic Loss	Policy Loss	Exploration
CartPole-v1	Welford online standardization $\to$ symlog $\to$ clip $\pm 20$	Unchanged $r{+}\gamma(1{-}d)\max_{a}Q^{\prime}$ [dead: $0.8\min{+}0.2\max$ aggregation]	$0.5\times$ MSE—unchanged up to a constant [dead: 2nd-critic term]	Never called	Welford-normalized transition distance, positive part, $0.01(1{-}p)$
MountainCar-v0	Symlog $\to$ EMA mean/var ( $\alpha{=}0.005$ ) $\to$ clip $\pm 5$	Unchanged [dead: uncertainty-weighted min/max mixing]	Adaptive expectile regression, weights $0.7/0.3\to 0.5/0.5$	Never called	Online RND with random Fourier features, $0.2\,\mathrm{relu}(z)(1{-}p)$
LunarLander-v3	Symmetric clip $[-20,20]$	Unchanged [dead: adaptive min/mean blend and the advertised adaptive- $\gamma$ warm-up]	Smooth-L1 ( $\beta{=}1$ )—a genuine live change from seed MSE	Never called	State-difference bonus (L2, cap 5.0), first 20% of training only, linearly decayed
Acrobot-v1	Clip $\pm 10$ + negligible decaying noise	Unchanged + live reward clamp $\pm 1000$ (inert for Acrobot) [dead: optimistic min/max mixing]	Asymmetric Huber, under/over weights $1.2/0.8$ , $+\,10^{-5}Q^{2}$	Never called	Online RND (RFF) + NLMS predictor, EMA novelty, $e^{-4p}$ decay

Appendix

Table 16: Component implementations of the 4 DQN champion programs. Bracketed [dead] annotations name mechanisms present in the file but unreachable under our DQN trainer; they are excluded from all counts. The policy-loss slot is never called at all, so no DQN champion can innovate there.

Explore similar work

AGAR: a reinforcement learning substrate for LLM program evolution
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +11LLM-Based Program Synthesis RL for Code Generation
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +2
SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolution
Apr 27, 2026Sichun Luo, Yi Huang, Haochen Luo +7Automated Algorithm Discovery Evolutionary Optimization
1The University of Hong Kong · 2JIUTIAN Research, China Mobile · 3City University of Hong Kong
Discovered, Not Designed: Population Evolution for Collaborative and Compute-Intensive Model Discovery
Oct 5, 2026Bo Peng, Lizhu Zhang, Yuhang Zhou +5Automated Algorithm Discovery Evolutionary Optimization
Meta

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

AGAR: a reinforcement learning substrate for LLM program evolution

Haoran Li, Zengle Ge, Xiaomin Yuan, Yui Lo+10

University of the Chinese Academy of Sciences · Baidu · Tsinghua University+2

LLM-Based Program Synthesis RL for Code Generation Reinforcement Learning Credit Assignment in RL

Open paper AskPDF arXiv

Given a task and an evaluator, a language model can rewrite a candidate program while a search loop decides which rewrites survive, offering a practical route to algorithm discovery. But that loop is governed by five constants set by hand: which parent to select, how hard to mutate, how to keep diversity, what to remember, and a scalar score that never says which part of the program earned it. Reinforcement learning already has an estimator for each. The obstacle is that program evolution is not usually written down as a decision process. We formalize it as a Markov decision process whose action is the modular prefix the model is conditioned on, rather than the program it emits. Credit assignment, value estimation, adaptive exploration, and experience memory can then attach to distinct components. AGAR (Algorithm Generation As RL) provides the resulting substrate: any estimator can be replaced or switched off without changing the controller, making the transfer auditable one mechanism at a time, with no gradient training of the backend model. Across 19 tasks, two backends, and three seeds under one harness, AGAR improves on the stronger of two published baselines on most tasks, with gains concentrated in the competitive-programming family. The formalization also yields a checkable reading of prior work: these systems are implicitly zero-discount, not by choice, but because fitness is exogenous to an individual rather than a return over successors, leaving a discount factor nothing to act on.

Abstract

Explore similar work

AGAR: a reinforcement learning substrate for LLM program evolution

SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolution

Discovered, Not Designed: Population Evolution for Collaborative and Compute-Intensive Model Discovery