LLM-driven evolutionary search is a promising paradigm for scientific discovery, but existing methods remain costly and slow, often applying expensive reasoning uniformly while underusing evidence from failed, redundant, and saturated trials. We introduce Speculative Evolution (SpecEvo), a multi-LLM framework that coordinates lightweight and frontier models through continual consolidation of search evidence. Lightweight models power both parallel Speculators, which generate and refine candidate programs, and an Advisor, which distills accumulated successes, failures, and saturated directions into persistent guidance for subsequent exploration. At sparse checkpoints, a frontier-model Navigator inspects the evolving population and intervenes through synthesis, surgical refinement, or reframing according to search stagnation. This speculate-then-consolidate workflow delegates routine exploration and evidence consolidation to inexpensive models while reserving frontier reasoning for strategic interventions whose benefits extend across many subsequent low-cost evaluations. Across 176 tasks spanning four scientific and algorithmic discovery domains, SpecEvo matches or outperforms frontier-model baselines while reducing API cost by up to 80%, achieves up to 1.5x greater evaluation efficiency than cost-efficient baselines on equation discovery, and improves over the strongest CO-Bench baseline by 3.8%. These results support trajectory-conditioned model coordination as an effective approach to cost-efficient scientific discovery.
Figures & tables
Figure 1: Top: Performance vs. API cost on mathematical discovery tasks; SpecEvo forms the Pareto frontier. Bottom: Best-so-far NMSE on LSR-Synth Biology and Chemistry over 500 evaluations.
Figure 2: Overview of SpecEvo . Initialization (left) generates M seed programs, expands each executable seed into V lightweight variants, and embeds the evaluated population to initialize the archive. During evolution (right), W lightweight-model Speculators continuously explore the archive. The frontier-model Navigator (top) intervenes every ΔN completed evaluations, while the lightweight-model Advisor (bottom) updates reusable guidance every ΔA evaluations.
Figure 3: Performance under a fixed three-hour wall-clock budget. SpecEvo attains the highest performance in both families while reducing API cost by roughly 50%–80%, remaining stable across model configurations.
Figure 4
Figure 6: Performance comparison under $10 API cost.
Figure 6
Figure 10: Best solutions generated by different methods for the Circle Packing Rectangle task.
Table 2: Performance on LSR-Synth under 500 evaluations. The final row reports the total evaluations required for each baseline to match SpecEvo @500.
Table 3: Ablation study on representative tasks. Relative decreases in mean score from the full SpecEvo configuration.
Table 4: Mode-specific context provided to the Navigator.
Problem
Configuration
Objective and task definition
Circle Packing (Square)
N=26
Pack N pairwise-disjoint circles inside a unit square and maximize the sum of their radii.
Circle Packing (Rectangle)
N=21
Pack N pairwise-disjoint circles inside a rectangle with perimeter 4 and maximize the sum of their radii.
Heilbronn Problem (Triangle)
N=11
Place N points on or inside a triangle of unit area so as to maximize the minimum area of any triangle formed by three selected points.
Heilbronn Problem (Convex Region)
N=13
Place N points on or inside a convex region of unit area so as to maximize the minimum area of any triangle formed by three selected points.
MinMax Distance
(N,d)=(16,2)
Find N points in Rd that maximize the ratio between the minimum and maximum pairwise Euclidean distances.
MinMax Distance
(N,d)=(14,3)
Optimize the same MinMax-distance objective for 14 points in three-dimensional Euclidean space.
Appendix
Table 5: Mathematical optimization benchmarks used in our evaluation.
Task
Objective
Problem description
Expert Parallelism Load Balancer (EPLB)
Balance expert-parallel workloads across GPUs
Given expert load statistics for a mixture-of-experts model, determine the number and placement of expert replicas across physical GPU slots. The objective is to reduce computational skew and stragglers while satisfying placement constraints and preserving inference throughput.
Model Placement (PRISM)
Optimize model-to-GPU placement cost
Assign multiple models or model components to a collection of GPUs under memory and capacity constraints. The placement should minimize global execution cost or peak resource pressure while maintaining feasibility and service-level requirements.
LLM-SQL
Maximize prefix-cache efficiency
Reorder table rows, columns, or fields so that serialized SQL inputs share longer prefixes and obtain higher prefix-cache reuse. The transformation must preserve the original table semantics and query results.
Transaction Scheduling (TXN)
Minimize transaction-execution makespan
Determine an execution order for dependent or conflicting database transactions. The scheduler seeks to reduce total completion time while respecting dependencies and avoiding excessive conflicts, aborts, or waiting time.
Appendix
Table 6: Systems optimization benchmarks used in our evaluation.
Category
#
Included problems
Packing
9
Bin packing; Multi-Demand Multidimensional Knapsack problem; Multidimensional knapsack problem; Container loading; Container loading with weight restrictions; Packing unequal circles; Packing unequal rectangles and squares number / area.
Table 7: The 36 combinatorial optimization problems in CO-Bench, grouped into eight categories.
Chemistry
Biology
Physics
Material
Overall
# Problems
36
24
44
25
129
Avg. Datapoints (k)
5.0
5.0
5.0
5.0
5.0
Avg. Variants
3.0
3.0
3.9
3.0
3.3
Appendix
Table 8: Statistics of the LSR-Synth problems used in our evaluation.
Model
Price per 1M Tokens (USD)
Input
Output
Frontier Models
GPT-5
1.25
10.00
Kimi-K2-Thinking
0.60
2.50
Small Models
Qwen3-30B-A3B-Instruct-2507
0.10
0.30
Appendix
Table 9: OpenRouter API pricing for the models used in our experiments, reported in USD per one million tokens (accessed August 1, 2026). Kimi-K2 in the main text, figures, and tables refers to Kimi-K2-Thinking.
Figure 13: Input and output token usage across six representative tasks using GPT-5 as the frontier-model backbone. Total token consumption is reported in millions (M) for each method.
Strategy
EPLB (↑)
LLM-SQL (↑)
Transaction (↑)
PRISM (↑)
Backbone: GPT-5
OpenEvolve
7.69 ± 0.36
5.16 ± 0.95
2.48 ± 0.25
6.38 ± 0.25
GEPA
10.04 ± 1.26
5.59 ± 0.43
2.38 ± 0.52
4.03 ± 0.37
AdaEvolve
8.26 ± 0.11
6.44 ± 0.40
2.46 ± 0.39
6.22 ± 0.19
EvoX
9.52 ± 0.57
5.19 ± 0.23
2.47 ± 0.31
5.97 ± 0.22
LEVI
2.90 ± 0.25
1.65 ± 0.12
1.20 ± 0.21
2.42 ± 0.98
Appendix
Table 11: API cost in USD under the fixed three-hour wall-clock budget on systems optimization tasks. Each entry reports the mean and standard deviation over three independent runs. Lower is better.
Strategy
EPLB (↑)
LLM-SQL (↑)
Transaction (↑)
PRISM (↑)
Avg
Best
Avg
Best
Avg
Best
Avg
Best
Backbone: GPT-5
OpenEvolve
0.1398 ± 0.011
0.1459
0.7152 ± 0.024
0.7289
4006.0 ± 65.47
4081.6
26.242 ± 0.008
26.247
GEPA
0.1394 ± 0.010
0.1453
0.7242 ± 0.006
0.7315
4078.2 ± 213.73
4201.6
26.235 ± 0.002
26.236
AdaEvolve
0.1462 ± 0.000
0.1464
0.7287 ± 0.004
0.7307
3958.7 ± 195.14
4184.1
26.256 ± 0.000
26.256
EvoX
0.1448 ± 0.001
0.1454
0.6936 ± 0.031
0.7298
3827.8 ± 354.60
4237.3
26.256 ± 0.000
26.256
Appendix
Table 13: Performance comparison on systems optimization tasks.
Figure 14: Category-level performance on CO-Bench. Each bar represents the average normalized score over all problems within the corresponding category. Higher is better.
Figure 15: Per-task score differences on the 36 CO-Bench tasks.
Baseline
W / T / L
Mean Δ
95% CI
LEVI
21 / 7 / 8
+0.0332
[+0.0070,+0.0668]
EvoX
18 / 6 / 12
+0.0460
[+0.0073,+0.0906]
AdaEvolve
21 / 4 / 11
+0.0395
[−0.0076,+0.0908]
GEPA
18 / 5 / 13
+0.0393
[+0.0020,+0.0841]
OpenEvolve
21 / 4 / 11
+0.0500
[+0.0091,+0.0985]
Appendix
Table 15: Paired statistical comparison of SpecEvo against baselines on the 36 CO-Bench tasks.
Figure 16: Per-task score differences on the 129 symbolic regression tasks.
Split
Baseline
W / T / L
Mean Δ
95% CI
ID
EvoX
97 / 2 / 30
+0.91
[+0.45,+1.37]
ID
AdaEvolve
95 / 1 / 33
+0.92
[+0.45,+1.38]
ID
GEPA
109 / 1 / 19
+1.64
[+1.20,+2.06]
ID
OpenEvolve
96 / 4 / 29
+0.87
[+0.44,+1.30]
OOD
EvoX
81 / 1 / 47
+0.93
[+0.40,+1.47]
OOD
AdaEvolve
85 / 0 / 44
+1.14
[+0.49,+1.79]
Appendix
Table 17: Paired comparison of SpecEvo against baselines on LSR-Synth under the ID and OOD evaluation settings.
Improve a candidate algorithm: preserve the parent’s useful structure but make one concrete algorithmic change likely to raise the score; no renaming, reformatting, comment edits, random-seed changes, or isolated constant retuning; treat inspirations as high-level ideas. Components (3–5 main phases); Strengths; Weaknesses (most likely bottleneck, citing a code element); Plan (one sentence).
Focused single-weakness repair
Make a surgical improvement to an already functional candidate: fix exactly one weakness while preserving the parent’s algorithmic identity and every unrelated component; no unrelated improvements, constant retuning, seed-only changes, or unnecessary helpers. Tightest constraint; Fix (minimum meaningful change); Preserved.
Mechanism replacement
Replace one interchangeable local mechanism without changing the overall algorithm (e.g., uniform with stratified initialization, a fixed step with an adaptive step or line search, greedy with Metropolis acceptance); preserve the signature, I/O contract, and parent structure. Mechanism identified; Old versus new; Risk (main failure mode and guard).
Analysis-targeted mutation
Additional input: a cached analysis of the parent (parent-analysis prompt below). Implement exactly the one suggested change most likely to improve the score, without inventing another direction. Chosen bottleneck; Implementation plan; What stays unchanged.
Structural crossover
Two parents (A and B): build a coherent hybrid from the strongest compatible mechanism of each parent within a single data flow, objective, and constraint-handling logic, rather than pasting blocks together. Component table (initialization, optimization core, constraint handling; each from A, B, or a hybrid); Compatibility note; Expected improvement.
Appendix
Table 18: Operator-specific instructions inserted into the shared Speculator template, and the auxiliary Speculator prompts. Unless noted otherwise, outputs follow the three-section format ( ## Analysis , ## Description , ## Code ); the required ## Analysis parts are listed in italics.
Class
What goes wrong
Feasibility
Two circles intersect after the radii are inflated or the layout is rescaled, but the program returns without rechecking pairwise feasibility
Output shape
The returned array does not contain 21 rows with three values each, usually because a helper or fallback path uses a different representation
Type
Lists and arrays are combined in arithmetic, or a coordinate is missing or infinite
Name
A variable or import is used before being defined, often within a nested helper or early return branch
Runtime
The program exhausts the evaluation time limit instead of stopping after convergence
Numeric
An array is used directly in a truth test, or a reduction is applied to an empty array
Appendix
Table 19: Failure classes among rejected candidates on Circle Packing Rectangle, ordered from most to least common. Each rejected candidate belongs to exactly one class.
Role
Mechanism
Prevalence and remarks
Construction
Structured lattice or patterned row initialization
Ubiquitous; hexagonal rows with an explicit pattern rather than random points
Search over the container aspect ratio
Common; the objective couples container shape with row occupancy
Refinement
Simulated annealing with stochastic acceptance
Common; uses a cooling schedule over perturbed configurations
Continuous penalty descent
Common; penalizes intersections smoothly rather than forbidding them
Pairwise separation relaxation
Common; applies damped opposing displacements to overlapping pairs
Enlargement with backtracking for each circle
Occasional; halves the step until an individual radius fits
Appendix
Table 20: Algorithmic mechanisms observed on Circle Packing Rectangle, grouped by their role. A program typically combines mechanisms from every group.
Class
What goes wrong
Indexing
A physical or logical index derived from capacity arithmetic exceeds a tensor bound
Name
A module or attribute is used without being imported or defined, most often the priority queue module
Type
A tensor has the wrong element type, or an integer computation exceeds its range
Feasibility
Replica counts do not sum to the budget for every layer, or the placement exhausts the available capacity
Runtime
The placement loop exhausts the evaluation time limit
Shape
A tensor dimension does not match the extent expected by the evaluator
Appendix
Table 21: Failure classes among rejected candidates on EPLB, ordered from most to least common. Each rejected candidate belongs to exactly one class.
Decision
Mechanism
Prevalence and remarks
Structure
Independent solution for each layer
Ubiquitous; imposed by the interface
Explicit treatment of the group, node, and GPU hierarchy
Ubiquitous; implements placement as a nested traversal of the hierarchy
Replicas
Allocation by marginal gain or apportionment
Ubiquitous; assigns the next replica where it most reduces the load per replica
Water-filling threshold found by bisection
Common; gives each expert enough replicas to fall below a searched load threshold
Huntington–Hill apportionment by name
Occasional; implements the classical rule explicitly
Placement
Modular or rotated round-robin assignment
Common; disperses the replicas of each expert across devices
Appendix
Table 22: Algorithmic mechanisms observed on EPLB, grouped by the design decision they resolve. A program typically combines mechanisms from every group.
Evolutionary coding agents such as AlphaEvolve are one of the notable successes in using Large Language Models (LLMs) to build AI Scientists. These agents tackle open-ended scientific problems by iteratively improving and evolving programs, leveraging the prior knowledge and reasoning capabilities of LLMs. Despite their success, mutations over programs are mostly low-level upon surface heuristics, without a persistent unit of control. In contrast, science compounds through ideas. In this work, we propose ExpEvolve, which injects idea-level extraction and control into evolution: it discovers and leverages the high-level factors from programs that explain scoring differences among different programs. We characterize theoretical conditions on when discovery with idea factors improves sample efficiency. We also build a practical framework to show that the extracted high-level idea factors are helpful for guiding exploitation from experience and exploration when discovery comes to a plateau. Across 11 open-ended mathematical tasks, ExpEvolve significantly improves the final performance and sample efficiency up to 5.0x. The improvements also transfer to 4 other LLMs and to 6 machine learning tasks from MLS-Bench. Analysis shows that ExpEvolve can discover idea-level factors to provide better exploration and exploitation for evolutionary discovery.
Yongqiang Chen, Chenxi Liu, Deyu Zou +4
MBZUAI · Carnegie Mellon University · TMLR Group, Hong Kong Baptist University +2
Evolutionary search with large language models (LLMs) can stall when progress requires external knowledge the model lacks. Supplying relevant documents helps, but simply adding web search tool can keep returning the same pages as solutions change. We introduce EvoDuet, a bi-level optimization method that co-evolves solutions and search queries with fixed model parameters. At each iteration, a retrieval gate lets the LLM assess its knowledge gap and choose to retrieve new documents, reuse stored ones, or proceed without them. An inner loop refines queries and ranks documents by the solution scores they are predicted to yield; an outer loop generates candidates in parallel from these documents and records the evaluated outcomes for later searches. Across 21 optimization tasks with one candidate per iteration, EvoDuet raises OpenEvolve's normalized discovery gain from 74.1% to 78.0% with GPT-5.6-Luna and from 61.3% to 82.3% with Gemini-3.8-Flash, whereas Qwen3.5-9B does not benefit. Our best runs surpass the previously reported best scores on eight tasks, including Swap Reduction on Q20 and Rosetta, and match them on three more. EvoDuet also improves with other scaffolds (e.g., Top-K, EvoX) on Sums/Diffs and Denoising, demonstrating its applicability across evolutionary search scaffolds.
Young-Jun Lee, Jinheon Baek, Soyeong Jeong +5
University of Minnesota · KAIST · Seoul National University +1
LLM-guided evolutionary methods such as AlphaEvolve have proven effective in domains like math, systems research, and algorithmic discovery, but their reliance on frontier models makes each run expensive. We argue this is largely an artifact of how existing frameworks allocate search: archives that fail to preserve solution diversity force compensation through stronger mutation models; blind model use spends frontier dollars on local edits a smaller model could handle; and full-set evaluation wastes rollouts on redundant examples. We introduce LEVI, a harness-first evolutionary framework built on the bet that stronger search architectures can substitute for or even outperform larger LLMs in evolutionary search. LEVI improves on three core components of evolutionary search: a solution database that establishes diversity from the beginning, and then maintains it throughout the run; a smarter mutation router that plays into the strengths of large and small LLMs; and a rank-preserving proxy benchmark for rollout-heavy settings. Across systems-research benchmarks LEVI attains the highest score on a budget 3.3-6.7x smaller than the published frontier-model runs of existing frameworks like ShinkaEvolve, GEPA, and AdaEvolve; on one problem, LEVI matches the existing best at a 35x lower cost. On prompt optimization, LEVI matches or exceeds GEPA at less than half of its rollout budget on four different benchmarks. LEVI is available as an open-source framework at https://github.com/ttanv/levi.