EliteTable: Self-Improving Program Search for Executable Control Policies

Stuart Farmer

cs.AIOct 8, 2026

EliteTable: Self-Improving Program Search for Executable Control Policies

Authors: Stuart Farmer

Organizations: Casa Dilauro Independent Research (casadilauro.com)

Abstract

EliteTable is a self-improving program-search algorithm guided by large language models (LLMs) that uses a fixed-capacity table of high-scoring solutions to guide new proposals, edits of existing solutions, and remixes of several solutions. Performance was evaluated across six Gymnasium control tasks, each with ten generations and fifty candidate slots per generation. Each task was evaluated across ten independently initialized searches. Candidate-score medians rose between the first and final generations in all six tasks, and selected-winner search-score medians rose in five. In CartPole-v1, the selected winner reached the search-score ceiling in the first generation of all ten searches. All ten paired held-out endpoints improved in four of the six tasks. BipedalWalker-v3 and CarRacing-v3 improved without reaching their reporting targets. Six of ten Acrobot-v1 searches declined on held-out episodes, and the median paired change was negative. The results document improvement during program search and the transfer of selected-winner gains to new episodes in some tasks. Comparisons and ablations are needed to establish EliteTable's relative performance and the contributions of its components.

Figures & tables

Panel of Figure 1 — Figure 1: EliteTable’s evaluation–revision loop. All candidate slots finish before the table is updated. Held-out scores do not enter the loop.

Environment	Observation/action interface	Step cap	Target	Task Settings
CartPole-v1	State vector; discrete	500	475	Recorded environment profile.
MountainCar-v0	State vector; discrete	200	$-110$	Recorded environment profile.
Acrobot-v1	State vector; discrete	500	$-100$	Recorded environment profile.
LunarLander-v3	State vector; discrete	1000	200	Gravity $-10$ ; wind disabled.
BipedalWalker-v3	State vector; continuous	1600	300	Standard, non-hardcore task.
CarRacing-v3	$96\times 96$ RGB; continuous	1000	900	Continuous control; lap-completion fraction 0.95; domain randomization disabled. Policies derive control directly from native RGB observations, so this task is interpreted separately from state-vector tasks.

Table 1: Task suite, settings and descriptive reporting targets. Other constructor settings follow the recorded environment profiles. RGB denotes red, green, and blue image channels.

Figure 2: Candidate search-return distributions across generations, pooled over ten searches per task. Solid blue lines show pooled candidate medians; faint lines show medians within individual searches. Violin widths show smoothed density, not operator membership or independent replication.

Figure 3: Search and held-out returns over ten generations. Dashed blue lines show median search return; solid orange lines show median held-out return, with interquartile ranges (IQRs) shaded and individual search trajectories drawn faintly. Dotted lines indicate the reporting threshold. Each environment contains ten eligible searches. MountainCar-v0 generation 6 has nine valid held-out measurements; all other environment-generation combinations have ten. Shading represents between-search dispersion rather than a confidence interval.

Environment	First	Final	Change [95% CI]	Improved	Target	$p_{\mathrm{Holm}}$
CartPole-v1	500.00	500.00	$0.00\;[0.00,\,0.00]$	0/10	10/10	—
MountainCar-v0	$-119.18$	$-107.07$	$+7.41\;[2.56,\,22.47]$	10/10	6/10	0.01172
Acrobot-v1	$-94.83$	$-101.53$	$-6.38\;[-18.64,\,7.98]$	4/10	3/10	1.00000
LunarLander-v3	$-39.04$	214.04	$+189.31\;[109.25,\,260.20]$	10/10	6/10	0.01172
BipedalWalker-v3	$-39.30$	$-7.65$	$+42.27\;[5.89,\,55.94]$	10/10	0/10	0.01172
CarRacing-v3	206.66	691.81	$+507.11\;[421.20,\,629.13]$	10/10	0/10	0.01172

Table 3: Held-out performance across ten searches per task. First and Final are medians of the generation-one and generation-ten mean returns. Change is the median paired change with a descriptive 95% bootstrap confidence interval (CI). Improved counts positive paired changes; Target counts final winners meeting the reporting threshold. Exploratory two-sided sign-test

p

-values are Holm-adjusted across all six tasks. A dash denotes ten ties and no informative test.

Program Synthesis

Program synthesis automatically constructs executable programs from examples, specifications, demonstrations, or natural-language descriptions.

See all 181 papers

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +14Automated Algorithm Discovery Evolutionary RL
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +4
AGAR: a reinforcement learning substrate for LLM program evolution
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +11LLM-Based Program Synthesis RL for Code Generation
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +2
SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolution
Apr 27, 2026Sichun Luo, Yi Huang, Haochen Luo +7Automated Algorithm Discovery Evolutionary Optimization
1The University of Hong Kong · 2JIUTIAN Research, China Mobile · 3City University of Hong Kong

Oct 6, 2026J/K move · Enter open · S save

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

Haoran Li, Zengle Ge, Xiaomin Yuan, Yui Lo+13

University of the Chinese Academy of Sciences · Baidu · Tsinghua University+4

Automated Algorithm Discovery Evolutionary RL Model-Free RL Self-Evolving Agents Large Language Model-Guided Optimization Genetic Programming

Open paper AskPDF arXiv

LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is difficult to scale: simultaneous changes can disrupt learning, while isolated changes overlook their dependencies. Evaluating candidate algorithms also requires costly training, with fitness remaining uncertain across random seeds. We introduce RLDiscover, a framework for the self-evolution of model-free deep RL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation. Experiments across SAC, PPO, and DQN on four benchmark suites show substantial improvements in mean return, with per-family median gains of 32%-84% and a peak return ratio of approximately 363x over a near-zero baseline. These gains include transitions from failed learning to successful task completion, and improvements persist when evolution starts from stronger open-source implementations. On measured SAC locomotion runs, evaluation uses approximately one-fifteenth the estimated compute required to fully evaluate the same candidate pool. Remarkably, independent searches repeatedly discover interpretable combinations of adaptive robust losses, progress-dependent value targets, and running statistics, with selected programs transferring to unseen tasks. These findings point toward a broader role for self-evolution in AI: discovering interpretable algorithms that improve how agents learn.

Abstract

Program Synthesis

Explore similar work

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

AGAR: a reinforcement learning substrate for LLM program evolution

SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolution