cs.AIOct 20, 2026
SaveSpecEvo: Speculative Evolution with Large Language Models for Cost-Efficient Scientific Discovery
Organizations: Hanoi University of Science and Technology · National University of Singapore
Abstract
LLM-driven evolutionary search is a promising paradigm for scientific discovery, but existing methods remain costly and slow, often applying expensive reasoning uniformly while underusing evidence from failed, redundant, and saturated trials. We introduce Speculative Evolution (SpecEvo), a multi-LLM framework that coordinates lightweight and frontier models through continual consolidation of search evidence. Lightweight models power both parallel Speculators, which generate and refine candidate programs, and an Advisor, which distills accumulated successes, failures, and saturated directions into persistent guidance for subsequent exploration. At sparse checkpoints, a frontier-model Navigator inspects the evolving population and intervenes through synthesis, surgical refinement, or reframing according to search stagnation. This speculate-then-consolidate workflow delegates routine exploration and evidence consolidation to inexpensive models while reserving frontier reasoning for strategic interventions whose benefits extend across many subsequent low-cost evaluations. Across 176 tasks spanning four scientific and algorithmic discovery domains, SpecEvo matches or outperforms frontier-model baselines while reducing API cost by up to 80%, achieves up to 1.5x greater evaluation efficiency than cost-efficient baselines on equation discovery, and improves over the strongest CO-Bench baseline by 3.8%. These results support trajectory-conditioned model coordination as an effective approach to cost-efficient scientific discovery.
Figures & tables
Figure 1: Top: Performance vs. API cost on mathematical discovery tasks; SpecEvo forms the Pareto frontier. Bottom: Best-so-far NMSE on LSR-Synth Biology and Chemistry over 500 evaluations.
Figure 2: Overview of SpecEvo . Initialization (left) generates seed programs, expands each executable seed into lightweight variants, and embeds the evaluated population to initialize the archive. During evolution (right), lightweight-model Speculators continuously explore the archive. The frontier-model Navigator (top) intervenes every completed evaluations, while the lightweight-model Advisor (bottom) updates reusable guidance every evaluations.
Figure 3: Performance under a fixed three-hour wall-clock budget. SpecEvo attains the highest performance in both families while reducing API cost by roughly 50%–80%, remaining stable across model configurations.
Figure 6: Performance comparison under $10 API cost.
Figure 10: Best solutions generated by different methods for the Circle Packing Rectangle task.
| Table 2: Performance on LSR-Synth under 500 evaluations. The final row reports the total evaluations required for each baseline to match SpecEvo @500. | Table 3: Ablation study on representative tasks. Relative decreases in mean score from the full SpecEvo configuration. |
| Domain Split OE GEPA AdaEvo EvoX RelayEvo LEVI SpecEvo Chemistry ID 5.27E-06 1.72E-05 2.48E-06 6.87E-06 2.59E-06 5.80E-06 4.57E-07 OOD 4.99E-02 2.69E-01 6.05E-02 8.68E-02 1.30E-01 1.17E-01 1.68E-02 Biology ID 6.33E-05 8.79E-05 9.92E-06 3.79E-05 1.01E-05 3.87E-06 3.34E-07 OOD 7.48E-01 1.92E+00 1.28E-01 5.83E-01 1.56E-01 1.96E-02 3.74E-03 Materials ID 5.92E-08 1.86E-07 6.32E-07 9.05E-08 5.79E-08 5.73E-07 4.84E-08 OOD 2.44E-06 7.57E-05 1.18E-04 1.63E-05 6.72E-06 8.36E-06 1.68E-06 Physics ID 3.29E-04 1.53E-03 2.79E-04 6.66E-04 1.81E-04 1.38E-05 2.92E-06 OOD 6.93E-03 2.82E-02 1.32E-02 5.32E-02 3.30E-03 2.23E-03 4.75E-04 Evals. to Beat SpecEvo 764 824 715 780 768 752 – | Ablation setting Circle Packing Rectangle Heilbronn Triangle w/o AST features 0.38% 7.76% w/o description embedding 0.13% 6.37% w/o refit 1.65% 4.99% Single operator prompt 2.17% 12.74% w/o Navigator 2.25% 9.42% Random mode 2.60% 7.47% Synthesis mode only 2.71% 0.83% Surgical mode only 0.49% 12.47% Reframe mode only 1.97% 7.76% w/o Advisor 3.82% 6.09% Fixed guidance 3.46% 8.31% |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Mode | Full-code anchors | Description-only inspirations | Requested move |
| Synthesis | top 3 niche elites | next 5 niche elites | combine mechanisms across anchors |
| Surgical | global champion only | next 5 niche elites | one local structural correction |
| Reframe | top 2 niche elites | next 5 niche elites | introduce an absent strategy family |
Table 4: Mode-specific context provided to the Navigator.
| Problem | Configuration | Objective and task definition |
| Circle Packing (Square) | Pack pairwise-disjoint circles inside a unit square and maximize the sum of their radii. | |
| Circle Packing (Rectangle) | Pack pairwise-disjoint circles inside a rectangle with perimeter and maximize the sum of their radii. | |
| Heilbronn Problem (Triangle) | Place points on or inside a triangle of unit area so as to maximize the minimum area of any triangle formed by three selected points. | |
| Heilbronn Problem (Convex Region) | Place points on or inside a convex region of unit area so as to maximize the minimum area of any triangle formed by three selected points. | |
| MinMax Distance | Find points in that maximize the ratio between the minimum and maximum pairwise Euclidean distances. | |
| MinMax Distance | Optimize the same MinMax-distance objective for points in three-dimensional Euclidean space. |
Table 5: Mathematical optimization benchmarks used in our evaluation.
| Task | Objective | Problem description |
| Expert Parallelism Load Balancer (EPLB) | Balance expert-parallel workloads across GPUs | Given expert load statistics for a mixture-of-experts model, determine the number and placement of expert replicas across physical GPU slots. The objective is to reduce computational skew and stragglers while satisfying placement constraints and preserving inference throughput. |
| Model Placement (PRISM) | Optimize model-to-GPU placement cost | Assign multiple models or model components to a collection of GPUs under memory and capacity constraints. The placement should minimize global execution cost or peak resource pressure while maintaining feasibility and service-level requirements. |
| LLM-SQL | Maximize prefix-cache efficiency | Reorder table rows, columns, or fields so that serialized SQL inputs share longer prefixes and obtain higher prefix-cache reuse. The transformation must preserve the original table semantics and query results. |
| Transaction Scheduling (TXN) | Minimize transaction-execution makespan | Determine an execution order for dependent or conflicting database transactions. The scheduler seeks to reduce total completion time while respecting dependencies and avoiding excessive conflicts, aborts, or waiting time. |
Table 6: Systems optimization benchmarks used in our evaluation.
| Category | # | Included problems |
| Packing | 9 | Bin packing; Multi-Demand Multidimensional Knapsack problem; Multidimensional knapsack problem; Container loading; Container loading with weight restrictions; Packing unequal circles; Packing unequal rectangles and squares number / area. |
| Cutting | 4 | Assortment problem; Constrained / unconstrained guillotine cutting; Constrained non-guillotine cutting. |
| Facility location | 4 | Capacitated / Uncapacitated warehouse location; Capacitated / Uncapacitated p-median problem. |
| Scheduling | 7 | Aircraft landing; Crew scheduling; Common due date scheduling; Flow shop scheduling; Hybrid Reentrant Shop Scheduling; Job shop scheduling; Open shop scheduling. |
| Routing | 3 | Traveling salesman problem; Period vehicle routing problem; Resource constrained shortest path. |
| Assignment | 2 | Constrained / unconstrained assignment. |
Table 7: The 36 combinatorial optimization problems in CO-Bench, grouped into eight categories.
| Chemistry | Biology | Physics | Material | Overall | |
| # Problems | 36 | 24 | 44 | 25 | 129 |
| Avg. Datapoints (k) | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 |
| Avg. Variants | 3.0 | 3.0 | 3.9 | 3.0 | 3.3 |
Table 8: Statistics of the LSR-Synth problems used in our evaluation.
| Model | Price per 1M Tokens (USD) | |
| Input | Output | |
| Frontier Models | ||
| GPT-5 | ||
| Kimi-K2-Thinking | ||
| Small Models | ||
| Qwen3-30B-A3B-Instruct-2507 | ||
Table 9: OpenRouter API pricing for the models used in our experiments, reported in USD per one million tokens (accessed August 1, 2026). Kimi-K2 in the main text, figures, and tables refers to Kimi-K2-Thinking.
Figure 13: Input and output token usage across six representative tasks using GPT-5 as the frontier-model backbone. Total token consumption is reported in millions (M) for each method.
| Strategy | EPLB | LLM-SQL | Transaction | PRISM |
| Backbone: GPT-5 | ||||
| OpenEvolve | 7.69 0.36 | 5.16 0.95 | 2.48 0.25 | 6.38 0.25 |
| GEPA | 10.04 1.26 | 5.59 0.43 | 2.38 0.52 | 4.03 0.37 |
| AdaEvolve | 8.26 0.11 | 6.44 0.40 | 2.46 0.39 | 6.22 0.19 |
| EvoX | 9.52 0.57 | 5.19 0.23 | 2.47 0.31 | 5.97 0.22 |
| LEVI | 2.90 0.25 | 1.65 0.12 | 1.20 0.21 | 2.42 0.98 |
Table 11: API cost in USD under the fixed three-hour wall-clock budget on systems optimization tasks. Each entry reports the mean and standard deviation over three independent runs. Lower is better.
| Strategy | EPLB | LLM-SQL | Transaction | PRISM | ||||
| Avg | Best | Avg | Best | Avg | Best | Avg | Best | |
| Backbone: GPT-5 | ||||||||
| OpenEvolve | 0.1398 0.011 | 0.1459 | 0.7152 0.024 | 0.7289 | 4006.0 65.47 | 4081.6 | 26.242 0.008 | 26.247 |
| GEPA | 0.1394 0.010 | 0.1453 | 0.7242 0.006 | 0.7315 | 4078.2 213.73 | 4201.6 | 26.235 0.002 | 26.236 |
| AdaEvolve | 0.1462 0.000 | 0.1464 | 0.7287 0.004 | 0.7307 | 3958.7 195.14 | 4184.1 | 26.256 0.000 | 26.256 |
| EvoX | 0.1448 0.001 | 0.1454 | 0.6936 0.031 | 0.7298 | 3827.8 354.60 | 4237.3 | 26.256 0.000 | 26.256 |
Table 13: Performance comparison on systems optimization tasks.
Figure 14: Category-level performance on CO-Bench. Each bar represents the average normalized score over all problems within the corresponding category. Higher is better.
Figure 15: Per-task score differences on the 36 CO-Bench tasks.
| Baseline | W / T / L | Mean | 95% CI |
| LEVI | 21 / 7 / 8 | ||
| EvoX | 18 / 6 / 12 | ||
| AdaEvolve | 21 / 4 / 11 | ||
| GEPA | 18 / 5 / 13 | ||
| OpenEvolve | 21 / 4 / 11 |
Table 15: Paired statistical comparison of SpecEvo against baselines on the 36 CO-Bench tasks.
Figure 16: Per-task score differences on the 129 symbolic regression tasks.
| Split | Baseline | W / T / L | Mean | 95% CI |
| ID | EvoX | 97 / 2 / 30 | ||
| ID | AdaEvolve | 95 / 1 / 33 | ||
| ID | GEPA | 109 / 1 / 19 | ||
| ID | OpenEvolve | 96 / 4 / 29 | ||
| OOD | EvoX | 81 / 1 / 47 | ||
| OOD | AdaEvolve | 85 / 0 / 44 |
Table 17: Paired comparison of SpecEvo against baselines on LSR-Synth under the ID and OOD evaluation settings.
| Prompt | Role and instruction |
| Variation operators (sampled uniformly; shared template) | |
| General mutation | Improve a candidate algorithm: preserve the parent’s useful structure but make one concrete algorithmic change likely to raise the score; no renaming, reformatting, comment edits, random-seed changes, or isolated constant retuning; treat inspirations as high-level ideas. Components (3–5 main phases); Strengths; Weaknesses (most likely bottleneck, citing a code element); Plan (one sentence). |
| Focused single-weakness repair | Make a surgical improvement to an already functional candidate: fix exactly one weakness while preserving the parent’s algorithmic identity and every unrelated component; no unrelated improvements, constant retuning, seed-only changes, or unnecessary helpers. Tightest constraint; Fix (minimum meaningful change); Preserved. |
| Mechanism replacement | Replace one interchangeable local mechanism without changing the overall algorithm (e.g., uniform with stratified initialization, a fixed step with an adaptive step or line search, greedy with Metropolis acceptance); preserve the signature, I/O contract, and parent structure. Mechanism identified; Old versus new; Risk (main failure mode and guard). |
| Analysis-targeted mutation | Additional input: a cached analysis of the parent (parent-analysis prompt below). Implement exactly the one suggested change most likely to improve the score, without inventing another direction. Chosen bottleneck; Implementation plan; What stays unchanged. |
| Structural crossover | Two parents (A and B): build a coherent hybrid from the strongest compatible mechanism of each parent within a single data flow, objective, and constraint-handling logic, rather than pasting blocks together. Component table (initialization, optimization core, constraint handling; each from A, B, or a hybrid); Compatibility note; Expected improvement. |
Table 18: Operator-specific instructions inserted into the shared Speculator template, and the auxiliary Speculator prompts. Unless noted otherwise, outputs follow the three-section format ( ## Analysis , ## Description , ## Code ); the required ## Analysis parts are listed in italics.
| Class | What goes wrong |
| Feasibility | Two circles intersect after the radii are inflated or the layout is rescaled, but the program returns without rechecking pairwise feasibility |
| Output shape | The returned array does not contain 21 rows with three values each, usually because a helper or fallback path uses a different representation |
| Type | Lists and arrays are combined in arithmetic, or a coordinate is missing or infinite |
| Name | A variable or import is used before being defined, often within a nested helper or early return branch |
| Runtime | The program exhausts the evaluation time limit instead of stopping after convergence |
| Numeric | An array is used directly in a truth test, or a reduction is applied to an empty array |
Table 19: Failure classes among rejected candidates on Circle Packing Rectangle, ordered from most to least common. Each rejected candidate belongs to exactly one class.
| Role | Mechanism | Prevalence and remarks |
| Construction | Structured lattice or patterned row initialization | Ubiquitous; hexagonal rows with an explicit pattern rather than random points |
| Search over the container aspect ratio | Common; the objective couples container shape with row occupancy | |
| Refinement | Simulated annealing with stochastic acceptance | Common; uses a cooling schedule over perturbed configurations |
| Continuous penalty descent | Common; penalizes intersections smoothly rather than forbidding them | |
| Pairwise separation relaxation | Common; applies damped opposing displacements to overlapping pairs | |
| Enlargement with backtracking for each circle | Occasional; halves the step until an individual radius fits |
Table 20: Algorithmic mechanisms observed on Circle Packing Rectangle, grouped by their role. A program typically combines mechanisms from every group.
| Class | What goes wrong |
| Indexing | A physical or logical index derived from capacity arithmetic exceeds a tensor bound |
| Name | A module or attribute is used without being imported or defined, most often the priority queue module |
| Type | A tensor has the wrong element type, or an integer computation exceeds its range |
| Feasibility | Replica counts do not sum to the budget for every layer, or the placement exhausts the available capacity |
| Runtime | The placement loop exhausts the evaluation time limit |
| Shape | A tensor dimension does not match the extent expected by the evaluator |
Table 21: Failure classes among rejected candidates on EPLB, ordered from most to least common. Each rejected candidate belongs to exactly one class.
| Decision | Mechanism | Prevalence and remarks |
| Structure | Independent solution for each layer | Ubiquitous; imposed by the interface |
| Explicit treatment of the group, node, and GPU hierarchy | Ubiquitous; implements placement as a nested traversal of the hierarchy | |
| Replicas | Allocation by marginal gain or apportionment | Ubiquitous; assigns the next replica where it most reduces the load per replica |
| Water-filling threshold found by bisection | Common; gives each expert enough replicas to fall below a searched load threshold | |
| Huntington–Hill apportionment by name | Occasional; implements the classical rule explicitly | |
| Placement | Modular or rotated round-robin assignment | Common; disperses the replicas of each expert across devices |
Table 22: Algorithmic mechanisms observed on EPLB, grouped by the design decision they resolve. A program typically combines mechanisms from every group.
More in this topic
Large Language Model-Guided Scientific Discovery
Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.
See all 274 papers