Advancing Automated Algorithm Design via Evolutionary Stagewise Design with LLMs

Chen Lu; Ke Xue; Chengrui Gao; Yunqi Shi; Siyuan Xu; Mingxuan Yuan; Chao Qian; Zhi-Hua Zhou

cs.AIMar 9, 2026

Advancing Automated Algorithm Design via Evolutionary Stagewise Design with LLMs

Authors: Chen Lu, Ke Xue, Chengrui Gao, Yunqi Shi, Siyuan Xu, Mingxuan Yuan, Chao Qian, Zhi-Hua Zhou

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Huawei Noah’s Ark Lab, China

Abstract

With the rapid advancement of human science and technology, problems in industrial scenarios are becoming increasingly challenging, bringing significant challenges to traditional algorithm design. Automated algorithm design with LLMs emerges as a promising solution, but the currently adopted black-box modeling deprives LLMs of any awareness of the intrinsic mechanism of the target problem, leading to hallucinated designs. In this paper, we introduce Evolutionary Stagewise Algorithm Design (EvoStage), a novel evolutionary paradigm that bridges the gap between the rigorous demands of industrial-scale algorithm design and the LLM-based algorithm design methods. Drawing inspiration from CoT, EvoStage decomposes the algorithm design process into sequential, manageable stages and integrates real-time intermediate feedback to iteratively refine algorithm design directions. To further reduce the algorithm design space and avoid falling into local optima, we introduce a multi-agent system and a "global-local perspective" mechanism. We apply EvoStage to the design of two types of common optimizers: designing parameter configuration schedules of the Adam optimizer for chip placement, and designing acquisition functions of Bayesian optimization for black-box optimization. Experimental results across open-source benchmarks demonstrate that EvoStage outperforms human-expert designs and existing LLM-based methods within only a couple of evolution steps, even achieving the historically state-of-the-art half-perimeter wire-length results on every tested chip case. Furthermore, when deployed on a commercial-grade 3D chip placement tool, EvoStage significantly surpasses the original performance metrics, achieving record-breaking efficiency. We hope EvoStage can significantly advance automated algorithm design in the real world, helping elevate human productivity.

Figures & tables

Figure 1: The overview of EvoStage. The upper left part is the intuitive illustration of the local perspective operator (i.e., the Stagewise-Design operator), where the algorithm design task is decomposed automatically by the LLM coordinator agent, with the coordinator reflecting on the intermediate information (denoted as "Inter layout", using chip placement as an example) and giving guidance for the next stage, and $N$ LLM coder agents producing $N$ algorithm components (one coder responsible for one respective component) according to the guidance from the coordinator stage-by-stage. The lower left part is the intuitive illustration of the two global perspective operators. The Global-Explore operator is to prompt the coder to produce a novel multi-stage heuristic in one shot with a selected reference, while the Global-Enhance operator is to prompt the coder to produce a better multi-stage heuristic in one shot with a selected reference. The right part is the multi-stage heuristic population, which maintains $M$ algorithms. Each algorithm is divided into $N$ components, each of which is implemented as a multi-stage heuristic. When selecting an algorithm from the population, we select the corresponding $N$ multi-stage heuristics for the $N$ components and assemble a complete algorithm from them. EvoStage performs an iterative process of selecting parent algorithms from the population, reproducing new offspring algorithms by operators, and updating the population to keep better algorithms, with the goal of iteratively improving the algorithms maintained in the population and finally generating a high-performing algorithm.

Figure 2: The intuitive comparison between the traditional black-box modeling of algorithm design (upper) and the proposed Stagewise Design paradigm (lower).

Figure 3: One detailed example of the reflection reasoning of the coordinator between stages in the Stagewise Design paradigm.

Figure 4: The example prompt of the local perspective operator Stagewise-Design.

Panel of Figure 5 — Figure 5: The example prompt of the two global perspective operators: Global-Explore and Global-Enhance.

Chip case	adaptec1	adaptec2	adaptec3	adaptec4	bigblue1	bigblue2	bigblue3	bigblue4	Avg
DREAMPlace-Nesterov [ 20 ]	70.28	79.23	185.74	168.81	87.36	131.10	291.62	725.47	1.009
Xplace-NN [ 21 ]	71.34	80.35	195.62	175.53	87.34	135.53	296.01	734.29	1.033
EvoStage (ours)	69.76	78.59	183.50	166.67	86.76	130.75	286.23	720.69	1.000
Chip case	superblue1	superblue3	superblue4	superblue5	superblue7	superblue10	superblue16	superblue18	Avg
DREAMPlace-Nesterov [ 20 ]	390.49	441.02	291.23	460.48	553.56	868.69	403.17	222.36	1.019
Xplace-NN [ 21 ]	388.12	444.84	289.76	456.79	548.96	867.67	408.34	220.34	1.017

Table 1: GP HPWL results (

\times 10^{6}

) of the ISPD 2005 benchmark and the ICCAD 2015 benchmark achieved by two state-of-the-art GP placers (DREAMPlace-Nesterov [ 20 ] and Xplace-NN [ 21 ] ) and our method EvoStage. Results in bold denote the current best results (the historically state-of-the-art results, to the best of our knowledge) that surpass the previous best results ever achieved. The average ratio takes the results of the proposed EvoStage as baselines.

Chip case	DREAMPlace-Adam [ 20 ]	EoH [ 4 ]		AlphaEvolve [ 3 ]		EvoStage-single (ours)		EvoStage (ours)
Chip case	HPWL	HPWL	Pass Rate (%)	HPWL	Pass Rate (%)	HPWL	Pass Rate (%)	HPWL	Pass Rate (%)
adaptec1	73.27	75.84	20.00	73.46	24.00	70.40	60.00	69.76	80.00
adaptec2	82.28	148.00	8.00	87.78	16.00	80.17	64.00	78.59	80.00
adaptec3	189.17	189.44	4.00	187.78	20.00	186.70	64.00	183.50	68.00
adaptec4	172.15	169.83	36.00	173.38	16.00	168.50	64.00	166.67	72.00
bigblue1	88.93	92.64	44.00	95.45	32.00	87.60	100.00	86.76	88.00

Table 2: GP HPWL results (

\times 10^{6}

) of the ISPD 2005 benchmark. For all LLM-based methods, only the learning rate schedule is designed (i.e., for EoH, AlphaEvolve, and EvoStage-single), except for EvoStage (it designs both the learning rate and the optimization step schedule); a maximum budget of 25 evaluations is used. Pass Rate measures the basic quality of the codes generated by LLM-based methods. One designed code is counted as passed if it is legal and achieves the target overflow with reasonable HPWL (i.e., less than 1e9). The average ratio takes the DREAMPlace-Adam’s results as baselines. The results in bold denote the best results when only designing the learning rate schedule, and the results in bold with underline denote the better results achieved by designing both the learning rate schedule and the optimization step schedule.

Chip case	DREAMPlace-Adam [ 20 ]	EoH [ 4 ]		AlphaEvolve [ 3 ]		EvoStage-single (ours)		EvoStage (ours)
Chip case	HPWL	HPWL	Pass Rate (%)	HPWL	Pass Rate (%)	HPWL	Pass Rate (%)	HPWL	Pass Rate (%)
superblue1	390.08	386.45	84.00	388.87	48.00	386.13	100.00	381.98	100.00
superblue3	462.22	582.64	4.00	462.31	4.00	449.44	68.00	433.81	84.00
superblue4	292.11	289.62	80.00	291.62	44.00	289.13	88.00	287.01	80.00
superblue5	452.34	453.57	28.00	452.31	16.00	451.16	84.00	448.32	92.00
superblue7	556.69	554.63	24.00	554.18	32.00	549.40	72.00	544.00	84.00

Table 3: GP HPWL results (

\times 10^{6}

) of the ICCAD 2015 benchmark. For all LLM-based methods, only the learning rate schedule is designed (i.e., for EoH, AlphaEvolve, and EvoStage-single), except for EvoStage (it designs both the learning rate and the optimization step schedule); a maximum budget of 25 evaluations is used. Pass Rate measures the basic quality of the codes generated by LLM-based methods. One designed code is counted as passed if it is legal and achieves the target overflow with reasonable HPWL (i.e., less than 1e9). The average ratio takes the DREAMPlace-Adam’s results as baselines. The results in bold denote the best results when only designing the learning rate schedule, and the results in bold with underline denote the better results achieved by designing both the learning rate schedule and the optimization step schedule.

Figure 6: The HPWL comparison of EvoStage with or without the Stagewise-Design operator over evaluations on 4 chip cases, i.e., Adaptec1–Adaptec4. The red $\times$ denotes a failed run of placement which does not achieve the target overflow.

Figure 7: An intuitive illustration of the 3D memory-on-logic stacking, where on the top die, the macro cells with large sizes (e.g., the memories) are placed (denoted in green), and the standard cells with smaller sizes (e.g., the logic gates) are placed on the bottom die (denoted in gray). Note that some macro cells densely interconnected with the standard cells are also allowed to be placed on the bottom die.

	HPWL-logic	HPWL-mem	Overflow-logic	Overflow-mem	Iteration
Base 3D placer	1.00	1.00	1.00	1.00	1.00
EvoStage	0.91	0.92	1.00	0.99	0.48
Improvement (%)	9.24	7.51	0.00	0.52	52.21

Table 4: Desensitized GP results (HPWL and overflow) of the placement of a real-world industrial 3D chip case. The base 3D placer is a commercial-grade 3D chip placement tool. To improve it, EvoStage simultaneously designs the learning rate schedule of the gradient optimizer Adam and the density weight (i.e., the Lagrangian multiplier of each Lagrangian-relaxation problem) schedule. HPWL-logic and HPWL-mem denote the half-perimeter wirelength of the logic die and the memory die, respectively; Overflow-logic and Overflow-mem measure the overlap issues of the logic die and the memory die, respectively; and Iteration denotes the total number of gradient descent optimization steps. Note that we desensitize the results by computing the ratios that take the base 3D placer’s results as baselines.

Panel of Figure 8 — Figure 8: Visualization of the 3D global placement results (on both the logic die and the memory die) of the base 3D placer (left, a commercial-grade 3D chip placement tool) and EvoStage (right). Cells belonging to the same partition share the same color, and superior cohesion within clusters composed of cells of the same color indicates a higher-quality layout.

Task	UCB [ 51 ]	EI [ 50 ]	EoH [ 4 ]		AlphaEvolve [ 3 ]		EvoStage (ours)
Task	Optimal Gap	Optimal Gap	Optimal Gap	Pass Rate(%)	Optimal Gap	Pass Rate(%)	Optimal Gap	Pass Rate(%)
Ackley 2D	1.509e+00	8.459e-01	2.575e-01	66.67	9.325e-01	55.56	2.401e-01	100.00
Rastrigin 2D	9.811e+00	7.441e+00	2.733e+00	66.67	2.311e+00	11.11	1.927e+00	100.00
Griewank 2D	1.370e+00	6.617e-01	4.349e-01	77.78	1.337e-01	55.56	1.211e-01	88.89
Levy 2D	3.035e+00	7.125e-03	3.755e-02	44.44	7.200e-03	22.22	1.228e-03	44.44
SliceLocalization	1.527e-03	4.154e-03	1.089e-03	55.56	2.000e-03	44.44	2.389e-04	88.89

Table 5: Optimization results obtained by popular acquisition functions (UCB [ 51 ] and EI [ 50 ] ) and those designed automatically by LLM-based methods (EoH [ 4 ] , AlphaEvolve [ 3 ] and our EvoStage) on different benchmarks. The upper 4 benchmarks are synthetic functions, and the bottom 4 benchmarks are HPO-NAS problems. The optimal gap is the gap between the produced result and the optimal result, which is the smaller the better. The metric "Pass Rate" (i.e., rate of legal codes) measures the code generation quality of each LLM-based method. The last row shows the average rank of each method across all benchmarks with respect to the metric Optimal Gap or Pass Rate. Numbers in bold denote the best results achieved on each benchmark.

Figure 9: The optimal gap curves of different acquisition functions (UCB [ 51 ] and EI [ 50 ] ) and the online designing of our method EvoStage on 4 NAS problems. In the ProteinStructure problem, the curve of UCB coincides with that of EI. The points on the curve of EvoStage indicate the current best optimal gap achieved by the best acquisition design so far.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: The initial prompt for the learning rate schedule design of the Adam optimizer for global placement.

Figure 11: The initial prompt for the optimization step schedule design of the Adam optimizer for global placement.

Figure 12: Prompt to elicit the reflection reasoning of the coordinator.

Parameter	Value	Parameter	Value
LLM settings
LLM coder model	gpt-4o	LLM coordinator model	gpt-4o
LLM coder temperature	0.2	LLM coordinator temperature	0.7
Evolution settings
Number of generations	5	Population size	5
Selection number	2	Number of stages	4

Appendix

Table 6: The hyperparameter settings of EvoStage for designing parameter configuration schedules of the Adam optimizer for chip placement.

Figure 13: The initial prompt for the acquisition function design of Bayesian optimization.

Parameter	Value	Parameter	Value
LLM settings
LLM coder model	gpt-4o	LLM coordinator model	gpt-4o
LLM coder temperature	0.2	LLM coordinator temperature	0.7
Evolution settings
Number of generations	3	Population size	3
Selection number	2	Number of stages	3

Appendix

Table 7: The hyperparameter settings of EvoStage for designing acquisition functions of Bayesian optimization.

Figure 14: The curve of the learning rate schedules designed by EvoStage, compared to those of the two most widely-used schedules (i.e., exponential decay and constant schedules) on the two chip cases, adaptec4 and superblue7.

Panel of Figure 15 — Figure 15: Examples of the algorithm codes generated by our proposed EvoStage.

Panel of Figure 16 — Figure 16: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on adaptec4, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.

Panel of Figure 17 — Figure 17: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on bigblue3, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.

Panel of Figure 18 — Figure 18: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on superblue3, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.

Panel of Figure 19 — Figure 19: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on superblue5, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.

Automated Algorithm Discovery

Automated algorithm discovery develops methods that generate and test executable algorithms for optimization, science, software, and machine learning.

See all 169 papers

Agentic Algorithm Design: LLMs that Propose, Formalise, and Prove Runtime Bounds for Evolutionary Algorithms in Lean
Oct 26, 2026Per Kristian LehreAutomated Algorithm Discovery Black-Box Optimization
School of Computer Science, University of Birmingham, UK
Online Automated Algorithm Design with Large Language Models
Sep 21, 2026Zhiyao Zhang, Yichen Li, Xingyu Wu +2Automated Algorithm Discovery Black-Box Optimization
Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
AdaEva: Accelerating LLM-Driven Algorithm Design with Adaptive Partial Evaluation
Oct 2, 2026Tai Nguyen, Fei Liu, Phong Le +2Large Language Model-Guided Optimization Automated Algorithm Discovery
University of St Andrews, United Kingdom · Sorbonne Universit´e, CNRS, LIP6, France · University of Zurich and ETH Zurich, Switzerland

Oct 26, 2026J/K move · Enter open · S save

Agentic Algorithm Design: LLMs that Propose, Formalise, and Prove Runtime Bounds for Evolutionary Algorithms in Lean

Per Kristian Lehre

School of Computer Science, University of Birmingham, UK

Automated Algorithm Discovery Black-Box Optimization LLMs for Evolutionary Optimization Runtime Analysis of Evolutionary Algorithms

Open paper AskPDF arXiv

LLMs now prove hard theorems in Lean. Can they also design algorithms and prove their correctness and efficiency? LLM-guided design of optimisation heuristics (such as AlphaEvolve) is purely empirical. Automated theorem provers with LLMs prove given statements. So far, there has been little evidence that LLMs can do both: design an algorithm and guarantee its efficiency by a mathematical proof. We present a staged pipeline modelled after research methodology in design and analysis of evolutionary algorithms. The agent proposes black-box optimisation algorithms, and Lean guards filter out flawed designs. The agent then defines a potential function and proves drift bounds that guarantee an expected runtime improvement over the baseline algorithm. At no point in the pipeline are the algorithms compared empirically. The agent generates a black-box optimisation algorithm with expected runtime

O(n^2)

on a CycleUncover problem, where the classical (1+1) EA needs runtime

e^{Ω(n)}

. The pipeline also designs and analyses a black-box optimisation algorithm for integer-valued submodular minimisation with expected runtime

O(n^{2+2c})

, and it matches the optimal order

O(n^2)

on LeadingOnes. Across 8 problem classes, 12 of 15 runs with frontier models complete the pipeline with a machine-checked expected-runtime bound. An ablation study shows that removing the staging and asking the agent to zero-shot an algorithmic design with a runtime guarantee leads to a copy of the seeded baseline algorithm (random search) or to incomplete proofs. Our study suggests that machine-checked specifications in Lean can replace empirical evaluation in open-ended algorithm discovery. To the best of our knowledge, this is the first example of an automated pipeline that designs efficient black-box optimisation algorithms with machine-checked runtime bounds in Lean.

Abstract

Automated Algorithm Discovery

Explore similar work

Agentic Algorithm Design: LLMs that Propose, Formalise, and Prove Runtime Bounds for Evolutionary Algorithms in Lean

Online Automated Algorithm Design with Large Language Models

AdaEva: Accelerating LLM-Driven Algorithm Design with Adaptive Partial Evaluation