Simple Baselines are Competitive with Code Evolution

Yonatan Gideoni; Sebastian Risi; Yarin Gal

cs.AIFeb 18, 2026

Simple Baselines are Competitive with Code Evolution

Authors: Yonatan Gideoni, Sebastian Risi, Yarin Gal

Organizations: University of Oxford · Sakana AI and ITU Copenhagen

Abstract

Code evolution is a family of techniques that rely on large language models to search through possible computer programs by evolving existing code. While code evolution pipelines have shown impressive performance across domains, many are highly complex and are typically not compared to simpler alternatives. To prevent bad comparisons from stagnating the field, we propose two simple baselines and compare them to popular pipelines across several domains: finding better mathematical bounds, designing agentic scaffolds, developing heuristic combinatorial optimization algorithms, and machine learning competitions. Surprisingly, the baselines match or outperform existing methods across all domains, prompting a closer look at which factors drive performance in different settings. For mathematical bounds, we find that the search space matters far more than the exact search algorithm, with both simple and complex methods performing similarly given the same search space. Thus, expanding the search space is what matters for improving performance, and does so across all pipelines. Moreover, different domain knowledge embedded in prompts can significantly affect the search's efficiency, potentially leading to overestimating a pipeline's sample efficiency. For automated agentic scaffold design, we show that typical high-variance evaluations lead to all automated search methods picking subpar scaffolds relative to a simple majority vote. To mitigate this, we propose improved evaluation procedures that reduce stochasticity while keeping the search economically feasible. Overall, these results indicate that further improving code evolution's performance depends more on the setup surrounding the search pipeline than on the pipeline itself. We hope these insights will enable developing new code evolution methods and help spur meaningful progress in the field.

Figures & tables

Panel of Figure 1 — Figure 1: (a) For AlphaEvolve-style mathematical bounds each problem effectively defines a function that gets as input a list of numbers, possibly with some extra structure, and outputs a mathematical bound that should be maximized or minimized. (b, c) The two baselines, (b) randomly sampling a set of programs IID from an LLM and picking the best one and (c) generating a set of programs, evaluating them, and generating a new set conditioned on some of the programs that ran successfully.

			Baselines (ours)
Problem	AlphaEvolve	ShinkaEvolve	IID RS	SCS
# problems $\geq$ ShinkaEvolve	7/9		4/9	6/9
# problems $\geq$ AlphaEvolve		3/9	2/9	4/9
Average rank ( $\downarrow$ )		1.9	2.3	1.8
First autocorr. ineq. ( $\downarrow$ )	1.505	1.522	1.535	1.519
Second autocorr. ineq. ( $\uparrow$ )	0.8962	0.8955	0.8739	0.8795

Table 1: Best bounds found using code evolution and baselines. Best results are bolded, second best are underlined. AlphaEvolve results are excluded from comparisons but included for reference as it uses an unknown but likely much higher budget and more domain knowledge. Sequential conditioned sampling (SCS) matches or exceeds ShinkaEvolve on most problems, with IID random sampling (IID RS) also exhibiting competitive performance, occasionally outperforming other methods. Arrows denote whether higher or lower is better. # problems

\geq

M

is the number of problems for which a method matches or exceeds method

M

. More significant digits are used when discovered bounds are close. Each result is achieved using a $20 budget.

Figure 3: Average probability of matching or exceeding ShinkaEvolve at a given budget over the 9 problems for the two baselines. Colored numbers on the right are the probabilities at the max budget of $20. Both baselines perform well over different budgets, with sequential conditioned sampling (SCS) generally outperforming ShinkaEvolve. Shaded regions are asymmetric 95% confidence intervals, see Appendix G for details and Appendix H for per-problem breakdowns.

Figure 4: AIME 2024 and 2025 accuracies for different methods. Majority@5/10 indicate manually designed majority vote scaffolds. (Left) Validation accuracies are those measured while evolving different scaffolds, where both validation and test accuracies are based on 3 evaluations over the dataset. ShinkaEvolve numbers are from Lange et al. [20] . IID RS seemingly performs best out of the search methods, although all do worse than majority vote on the test set, with large drops in accuracy. (Right) Results when re-evaluating the scaffolds 10 times, with whiskers denoting 95% confidence intervals. Validation accuracies are lower for all automated search methods as their scaffolds are seemingly selected more so due to stochasticity in the evaluations than them achieving good performance. Unlike when evaluating only 3 times, it is apparent that there is no clear difference between ShinkaEvolve and IID RS, with the probability of improvement being

P(\text{Shinka}>\text{IID RS})=0.49

. Using an evaluation cascade with IID RS results in selecting a robustly higher-performing scaffold.

Figure 5: Empirical distributions of majority vote@5 accuracies for AIME 2025 when evaluating each question a different number of times. Even when evaluating 10 times there is a standard deviation of more than 1% in the accuracy. Distributions were calculated by sampling 100 answers to each question and bootstrapping.

Competition	AIDE	IID RS	SCS
# competitions $\geq$ AIDE		2/10	6/10
# competitions $\geq$ Median	8	4	8
# competitions $\geq$ Bronze	5	3	5
# competitions $\geq$ Silver	3	3	5
# competitions $\geq$ Gold	2	2	2
Insult Detection ( $\uparrow$ )	0.9135 ×	0.9056 ×	0.8460 ×

Table 2: Results for different methods on the Kaggle competitions.

{\color[rgb]{0.75,0.5,0.25}*},{\color[rgb]{0.5,0.5,0.5}+},{\color[rgb]{0.72,0.53,0.04}\times}

denote scores that would have gotten bronze, silver, or gold medals respectively. Best and second-best method results per competition are respectively shown in bold and underlined. For unabbreviated competition names and human baseline metrics, see Appendix M .

Figure 6: Relative performance of different methods on ALE-Bench LITE. IID RS and SCS outperform ShinkaEvolve on 8/10 problems and have a higher average score. Scores are the median of three evaluations of the best discovered program. Means are over the three evaluations, with uncertainties denoting standard errors.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Problem	Input size	Pre-AE Bound	AE Bound	AE Appendix
First autocorrelation inequality ( $\downarrow$ )	Unbounded, step function heights	1.5098	1.5053	B.1
Second autocorrelation inequality ( $\uparrow$ )	Unbounded, step function heights	0.88922	0.8962	B.2
Uncertainty inequality ( $\downarrow$ )	3 coefficients of a Hermite polynomial	0.3523	0.3521	B.4
Erdős’ minimum overlap ( $\downarrow$ )	Unbounded, step function heights	0.380927	0.380924	B.5
Sums vs. differences of finite sets ( $\uparrow$ )	Unbounded, set $U\subset\mathbb{Z}_{\geq 0}$ fulfilling some properties	1.14465	1.1584	B.6
Max–min distance ratio for 16 2D points ( $\downarrow$ )	32 coordinates (16×2)	12.890	12.88927	B.8

Appendix

Table 3: Bounds of AlphaEvolve (AE) problems studied here, divided into analysis (top), combinatorics (middle), and geometry (bottom). The arrow next to the problem name indicates whether it is an upper bound, so lower results are tighter and hence better (

\downarrow

), or a lower bound so higher is tighter and thus better (

\uparrow

). All numbers are from Novikov et al. [23] . “Pre-AE” are the best bounds from before Novikov et al. [23] .

Problem	OpenEvolve
First autocorr. ineq. ( $\downarrow$ )	–
Second autocorr. ineq. ( $\uparrow$ )	0.8903
Uncertainty ineq. ( $\downarrow$ )	0.3521
Erdős’ min. overlap ( $\downarrow$ )	–
Sums/differences of sets ( $\uparrow$ )	1.1095
Max–min dist. ratio ( $\downarrow$ )	–

Appendix

Table 4: Best bounds found by OpenEvolve given 300 generations using GPT-5.2 and GPT-5 mini. “–” denotes a failed run. Arrows denote whether higher or lower is better.

Figure 7: Per-problem probability of matching/exceeding ShinkaEvolve for the two baselines. Shaded regions are 95% confidence intervals.

Figure 8: For each baseline, the average probability of matching or exceeding ShinkaEvolve across the 9 problems, as a function of the number of evaluated programs. The

x

axis is the percent of evaluated programs out of the maximum in the corresponding ShinkaEvolve run. Shaded regions are 95% confidence intervals.

Figure 9: For each baseline, the per-problem probability of matching or exceeding ShinkaEvolve across the 9 problems, as a function of the number of evaluated programs. Shaded regions are 95% confidence intervals.

Abbreviation	Full Competition Name
Insult Detection	Detecting Insults in Social Commentary
Dec. 2021 Tabular	Tabular Playground Series – December 2021
Trans. Conductors	Nomad2018: Predict Transparent Conductors
English Text Norm.	Text Normalization Challenge – English Language
May 2022 Tabular	Tabular Playground Series – May 2022
Random Pizza	Random Acts of Pizza

Appendix

Table 5: Competition abbreviations and full names.

	Human baseline metrics
Competition	Median	Bronze ∗	Silver +	Gold ×
Insult Detection ( $\uparrow$ )	0.778	0.791	0.823	0.833
Dec. 2021 Tabular ( $\uparrow$ )	0.953	0.956	0.956	0.956
Trans. Conductors ( $\downarrow$ )	0.069	0.065	0.062	0.055
English Text Norm. ( $\uparrow$ )	0.990	0.990	0.991	0.997
May 2022 Tabular ( $\uparrow$ )	0.972	0.998	0.998	0.998

Appendix

Table 6: Human baseline metrics on the Kaggle competitions. Median is the median score in the competition, with bronze, silver, and gold denoting the cutoff score to receive each medal. Metrics are from Schmidgall et al. [26] .

LLMs for Evolutionary Optimization

See all 28 papers

What Do Evolutionary Coding Agents Evolve?
May 19, 2026Nico Pelleriti, Sree Harsha Nelaturu, Zhanke Zhou +4Evolutionary Optimization AI Coding Agents
1Zuse Institute Berlin · 2Technical University of Berlin · 3Hong Kong Baptist University +1
ImprovEvolve: Basin-Hopping Meets LLM-Guided Evolutionary Search
Feb 10, 2026Alexey Kravatskiy, Valentin Khrulkov, Ivan OseledetsLarge Language Model-Guided Scientific Discovery Evolutionary Optimization
1MIRIAI, Russia · 2FusionBrain Lab, Russia · 3AXXX, Russia +1
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
May 31, 2026Yangzhen Wu, Aaron J. Li, Wenjie Ma +10Synthetic Benchmark Generation Code Generation Evaluation
University of California, Berkeley · Institute for Interdisciplinary Information Sciences, Tsinghua University

May 19, 2026J/K move · Enter open · S save

What Do Evolutionary Coding Agents Evolve?

Nico Pelleriti, Sree Harsha Nelaturu, Zhanke Zhou, Zongze Li+3

1Zuse Institute Berlin · 2Technical University of Berlin · 3Hong Kong Baptist University+1

Evolutionary Optimization AI Coding Agents AI Agent Evaluation Code Generation Evaluation

Open paper AskPDF arXiv

Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of evolutionary coding traces spanning four evolutionary frameworks, reasoning and non-reasoning models, and 16 tasks across mathematics and algorithm design. To analyze these traces, we develop EvoReplay, a replay-based methodology that reconstructs the local search states behind high-scoring solutions and tests controlled interventions, including adjusting constants, removing program components and substituting models or prompting contexts. We annotate every code edit in EvoTrace with one of nine recurring edit types using an LLM-as-judge pipeline validated against blind human re-annotation. Across EvoTrace, most score gains come from a small subset of these edit types. We further find a deterministic cycling pattern: about 30% of code lines added during search are byte-identical re-introductions of previously-deleted lines, present throughout nearly every run. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure. EvoTrace enables more diagnostic evaluation of evolutionary coding agents beyond final benchmark scores.

Abstract

LLMs for Evolutionary Optimization

Explore similar work

What Do Evolutionary Coding Agents Evolve?

ImprovEvolve: Basin-Hopping Meets LLM-Guided Evolutionary Search

BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution