Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Huawei Noah’s Ark Lab, China
With the rapid advancement of human science and technology, problems in industrial scenarios are becoming increasingly challenging, bringing significant challenges to traditional algorithm design. Automated algorithm design with LLMs emerges as a promising solution, but the currently adopted black-box modeling deprives LLMs of any awareness of the intrinsic mechanism of the target problem, leading to hallucinated designs. In this paper, we introduce Evolutionary Stagewise Algorithm Design (EvoStage), a novel evolutionary paradigm that bridges the gap between the rigorous demands of industrial-scale algorithm design and the LLM-based algorithm design methods. Drawing inspiration from CoT, EvoStage decomposes the algorithm design process into sequential, manageable stages and integrates real-time intermediate feedback to iteratively refine algorithm design directions. To further reduce the algorithm design space and avoid falling into local optima, we introduce a multi-agent system and a "global-local perspective" mechanism. We apply EvoStage to the design of two types of common optimizers: designing parameter configuration schedules of the Adam optimizer for chip placement, and designing acquisition functions of Bayesian optimization for black-box optimization. Experimental results across open-source benchmarks demonstrate that EvoStage outperforms human-expert designs and existing LLM-based methods within only a couple of evolution steps, even achieving the historically state-of-the-art half-perimeter wire-length results on every tested chip case. Furthermore, when deployed on a commercial-grade 3D chip placement tool, EvoStage significantly surpasses the original performance metrics, achieving record-breaking efficiency. We hope EvoStage can significantly advance automated algorithm design in the real world, helping elevate human productivity.
Figures & tables
Figure 1: The overview of EvoStage. The upper left part is the intuitive illustration of the local perspective operator (i.e., the Stagewise-Design operator), where the algorithm design task is decomposed automatically by the LLM coordinator agent, with the coordinator reflecting on the intermediate information (denoted as "Inter layout", using chip placement as an example) and giving guidance for the next stage, and N LLM coder agents producing N algorithm components (one coder responsible for one respective component) according to the guidance from the coordinator stage-by-stage. The lower left part is the intuitive illustration of the two global perspective operators. The Global-Explore operator is to prompt the coder to produce a novel multi-stage heuristic in one shot with a selected reference, while the Global-Enhance operator is to prompt the coder to produce a better multi-stage heuristic in one shot with a selected reference. The right part is the multi-stage heuristic population, which maintains M algorithms. Each algorithm is divided into N components, each of which is implemented as a multi-stage heuristic. When selecting an algorithm from the population, we select the corresponding N multi-stage heuristics for the N components and assemble a complete algorithm from them. EvoStage performs an iterative process of selecting parent algorithms from the population, reproducing new offspring algorithms by operators, and updating the population to keep better algorithms, with the goal of iteratively improving the algorithms maintained in the population and finally generating a high-performing algorithm.
Figure 2: The intuitive comparison between the traditional black-box modeling of algorithm design (upper) and the proposed Stagewise Design paradigm (lower).
Figure 3: One detailed example of the reflection reasoning of the coordinator between stages in the Stagewise Design paradigm.
Figure 4: The example prompt of the local perspective operator Stagewise-Design.
Figure 5: The example prompt of the two global perspective operators: Global-Explore and Global-Enhance.
Chip case
adaptec1
adaptec2
adaptec3
adaptec4
bigblue1
bigblue2
bigblue3
bigblue4
Avg
DREAMPlace-Nesterov [ 20 ]
70.28
79.23
185.74
168.81
87.36
131.10
291.62
725.47
1.009
Xplace-NN [ 21 ]
71.34
80.35
195.62
175.53
87.34
135.53
296.01
734.29
1.033
EvoStage (ours)
69.76
78.59
183.50
166.67
86.76
130.75
286.23
720.69
1.000
Chip case
superblue1
superblue3
superblue4
superblue5
superblue7
superblue10
superblue16
superblue18
Avg
DREAMPlace-Nesterov [ 20 ]
390.49
441.02
291.23
460.48
553.56
868.69
403.17
222.36
1.019
Xplace-NN [ 21 ]
388.12
444.84
289.76
456.79
548.96
867.67
408.34
220.34
1.017
Table 1: GP HPWL results ( ×106 ) of the ISPD 2005 benchmark and the ICCAD 2015 benchmark achieved by two state-of-the-art GP placers (DREAMPlace-Nesterov [ 20 ] and Xplace-NN [ 21 ] ) and our method EvoStage. Results in bold denote the current best results (the historically state-of-the-art results, to the best of our knowledge) that surpass the previous best results ever achieved. The average ratio takes the results of the proposed EvoStage as baselines.
Chip case
DREAMPlace-Adam [ 20 ]
EoH [ 4 ]
AlphaEvolve [ 3 ]
EvoStage-single (ours)
EvoStage (ours)
HPWL
HPWL
Pass Rate (%)
HPWL
Pass Rate (%)
HPWL
Pass Rate (%)
HPWL
Pass Rate (%)
adaptec1
73.27
75.84
20.00
73.46
24.00
70.40
60.00
69.76
80.00
adaptec2
82.28
148.00
8.00
87.78
16.00
80.17
64.00
78.59
80.00
adaptec3
189.17
189.44
4.00
187.78
20.00
186.70
64.00
183.50
68.00
adaptec4
172.15
169.83
36.00
173.38
16.00
168.50
64.00
166.67
72.00
bigblue1
88.93
92.64
44.00
95.45
32.00
87.60
100.00
86.76
88.00
Table 2: GP HPWL results ( ×106 ) of the ISPD 2005 benchmark. For all LLM-based methods, only the learning rate schedule is designed (i.e., for EoH, AlphaEvolve, and EvoStage-single), except for EvoStage (it designs both the learning rate and the optimization step schedule); a maximum budget of 25 evaluations is used. Pass Rate measures the basic quality of the codes generated by LLM-based methods. One designed code is counted as passed if it is legal and achieves the target overflow with reasonable HPWL (i.e., less than 1e9). The average ratio takes the DREAMPlace-Adam’s results as baselines. The results in bold denote the best results when only designing the learning rate schedule, and the results in bold with underline denote the better results achieved by designing both the learning rate schedule and the optimization step schedule.
Chip case
DREAMPlace-Adam [ 20 ]
EoH [ 4 ]
AlphaEvolve [ 3 ]
EvoStage-single (ours)
EvoStage (ours)
HPWL
HPWL
Pass Rate (%)
HPWL
Pass Rate (%)
HPWL
Pass Rate (%)
HPWL
Pass Rate (%)
superblue1
390.08
386.45
84.00
388.87
48.00
386.13
100.00
381.98
100.00
superblue3
462.22
582.64
4.00
462.31
4.00
449.44
68.00
433.81
84.00
superblue4
292.11
289.62
80.00
291.62
44.00
289.13
88.00
287.01
80.00
superblue5
452.34
453.57
28.00
452.31
16.00
451.16
84.00
448.32
92.00
superblue7
556.69
554.63
24.00
554.18
32.00
549.40
72.00
544.00
84.00
Table 3: GP HPWL results ( ×106 ) of the ICCAD 2015 benchmark. For all LLM-based methods, only the learning rate schedule is designed (i.e., for EoH, AlphaEvolve, and EvoStage-single), except for EvoStage (it designs both the learning rate and the optimization step schedule); a maximum budget of 25 evaluations is used. Pass Rate measures the basic quality of the codes generated by LLM-based methods. One designed code is counted as passed if it is legal and achieves the target overflow with reasonable HPWL (i.e., less than 1e9). The average ratio takes the DREAMPlace-Adam’s results as baselines. The results in bold denote the best results when only designing the learning rate schedule, and the results in bold with underline denote the better results achieved by designing both the learning rate schedule and the optimization step schedule.
Figure 6: The HPWL comparison of EvoStage with or without the Stagewise-Design operator over evaluations on 4 chip cases, i.e., Adaptec1–Adaptec4. The red × denotes a failed run of placement which does not achieve the target overflow.
Figure 7: An intuitive illustration of the 3D memory-on-logic stacking, where on the top die, the macro cells with large sizes (e.g., the memories) are placed (denoted in green), and the standard cells with smaller sizes (e.g., the logic gates) are placed on the bottom die (denoted in gray). Note that some macro cells densely interconnected with the standard cells are also allowed to be placed on the bottom die.
HPWL-logic
HPWL-mem
Overflow-logic
Overflow-mem
Iteration
Base 3D placer
1.00
1.00
1.00
1.00
1.00
EvoStage
0.91
0.92
1.00
0.99
0.48
Improvement (%)
9.24
7.51
0.00
0.52
52.21
Table 4: Desensitized GP results (HPWL and overflow) of the placement of a real-world industrial 3D chip case. The base 3D placer is a commercial-grade 3D chip placement tool. To improve it, EvoStage simultaneously designs the learning rate schedule of the gradient optimizer Adam and the density weight (i.e., the Lagrangian multiplier of each Lagrangian-relaxation problem) schedule. HPWL-logic and HPWL-mem denote the half-perimeter wirelength of the logic die and the memory die, respectively; Overflow-logic and Overflow-mem measure the overlap issues of the logic die and the memory die, respectively; and Iteration denotes the total number of gradient descent optimization steps. Note that we desensitize the results by computing the ratios that take the base 3D placer’s results as baselines.
Figure 8: Visualization of the 3D global placement results (on both the logic die and the memory die) of the base 3D placer (left, a commercial-grade 3D chip placement tool) and EvoStage (right). Cells belonging to the same partition share the same color, and superior cohesion within clusters composed of cells of the same color indicates a higher-quality layout.
Task
UCB [ 51 ]
EI [ 50 ]
EoH [ 4 ]
AlphaEvolve [ 3 ]
EvoStage (ours)
Optimal Gap
Optimal Gap
Optimal Gap
Pass Rate(%)
Optimal Gap
Pass Rate(%)
Optimal Gap
Pass Rate(%)
Ackley 2D
1.509e+00
8.459e-01
2.575e-01
66.67
9.325e-01
55.56
2.401e-01
100.00
Rastrigin 2D
9.811e+00
7.441e+00
2.733e+00
66.67
2.311e+00
11.11
1.927e+00
100.00
Griewank 2D
1.370e+00
6.617e-01
4.349e-01
77.78
1.337e-01
55.56
1.211e-01
88.89
Levy 2D
3.035e+00
7.125e-03
3.755e-02
44.44
7.200e-03
22.22
1.228e-03
44.44
SliceLocalization
1.527e-03
4.154e-03
1.089e-03
55.56
2.000e-03
44.44
2.389e-04
88.89
Table 5: Optimization results obtained by popular acquisition functions (UCB [ 51 ] and EI [ 50 ] ) and those designed automatically by LLM-based methods (EoH [ 4 ] , AlphaEvolve [ 3 ] and our EvoStage) on different benchmarks. The upper 4 benchmarks are synthetic functions, and the bottom 4 benchmarks are HPO-NAS problems. The optimal gap is the gap between the produced result and the optimal result, which is the smaller the better. The metric "Pass Rate" (i.e., rate of legal codes) measures the code generation quality of each LLM-based method. The last row shows the average rank of each method across all benchmarks with respect to the metric Optimal Gap or Pass Rate. Numbers in bold denote the best results achieved on each benchmark.
Figure 9: The optimal gap curves of different acquisition functions (UCB [ 51 ] and EI [ 50 ] ) and the online designing of our method EvoStage on 4 NAS problems. In the ProteinStructure problem, the curve of UCB coincides with that of EI. The points on the curve of EvoStage indicate the current best optimal gap achieved by the best acquisition design so far.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: The initial prompt for the learning rate schedule design of the Adam optimizer for global placement.
Figure 11: The initial prompt for the optimization step schedule design of the Adam optimizer for global placement.
Figure 12: Prompt to elicit the reflection reasoning of the coordinator.
Parameter
Value
Parameter
Value
LLM settings
LLM coder model
gpt-4o
LLM coordinator model
gpt-4o
LLM coder temperature
0.2
LLM coordinator temperature
0.7
Evolution settings
Number of generations
5
Population size
5
Selection number
2
Number of stages
4
Appendix
Table 6: The hyperparameter settings of EvoStage for designing parameter configuration schedules of the Adam optimizer for chip placement.
Figure 13: The initial prompt for the acquisition function design of Bayesian optimization.
Parameter
Value
Parameter
Value
LLM settings
LLM coder model
gpt-4o
LLM coordinator model
gpt-4o
LLM coder temperature
0.2
LLM coordinator temperature
0.7
Evolution settings
Number of generations
3
Population size
3
Selection number
2
Number of stages
3
Appendix
Table 7: The hyperparameter settings of EvoStage for designing acquisition functions of Bayesian optimization.
Figure 14: The curve of the learning rate schedules designed by EvoStage, compared to those of the two most widely-used schedules (i.e., exponential decay and constant schedules) on the two chip cases, adaptec4 and superblue7.
Figure 15: Examples of the algorithm codes generated by our proposed EvoStage.
Figure 16: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on adaptec4, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.
Figure 17: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on bigblue3, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.
Figure 18: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on superblue3, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.
Figure 19: The global placement plot comparison of EvoStage and DREAMPlace-Nesterov [ 20 ] on superblue5, where movable standard cells are denoted in blue and fixed macro cells are denoted in red. The critical areas, where EvoStage produces better layouts (e.g., neater boundaries or stronger cohesion), are highlighted in yellow circles.
LLMs now prove hard theorems in Lean. Can they also design algorithms and prove their correctness and efficiency? LLM-guided design of optimisation heuristics (such as AlphaEvolve) is purely empirical. Automated theorem provers with LLMs prove given statements. So far, there has been little evidence that LLMs can do both: design an algorithm and guarantee its efficiency by a mathematical proof. We present a staged pipeline modelled after research methodology in design and analysis of evolutionary algorithms. The agent proposes black-box optimisation algorithms, and Lean guards filter out flawed designs. The agent then defines a potential function and proves drift bounds that guarantee an expected runtime improvement over the baseline algorithm. At no point in the pipeline are the algorithms compared empirically. The agent generates a black-box optimisation algorithm with expected runtime O(n2) on a CycleUncover problem, where the classical (1+1) EA needs runtime eΩ(n). The pipeline also designs and analyses a black-box optimisation algorithm for integer-valued submodular minimisation with expected runtime O(n2+2c), and it matches the optimal order O(n2) on LeadingOnes. Across 8 problem classes, 12 of 15 runs with frontier models complete the pipeline with a machine-checked expected-runtime bound. An ablation study shows that removing the staging and asking the agent to zero-shot an algorithmic design with a runtime guarantee leads to a copy of the seeded baseline algorithm (random search) or to incomplete proofs. Our study suggests that machine-checked specifications in Lean can replace empirical evaluation in open-ended algorithm discovery. To the best of our knowledge, this is the first example of an automated pipeline that designs efficient black-box optimisation algorithms with machine-checked runtime bounds in Lean.
Per Kristian Lehre
School of Computer Science, University of Birmingham, UK
Large language models (LLMs) enable automated algorithm design (AAD) through reasoning and code synthesis. However, most existing LLM-based AAD methods separate algorithm design from target optimization, deploying a fixed design even as the optimization state evolves. Conventional adaptive optimizers can respond to such changes, but their adjustments remain confined to predefined parameters, operators, or strategies. To address these limitations, we introduce online LLM-based AAD, a novel optimization paradigm that treats the algorithm itself as a state-dependent decision variable. At each stage, LLM agents synthesize an algorithm with new behavior logic from the current optimization state. Executing the generated algorithm advances the search and provides feedback for subsequent designs, coupling algorithm design with target optimization without requiring a separate offline algorithm pretraining stage. To implement this paradigm, we propose OnDesign, a multi-agent framework that reconciles competing design perspectives to synthesize executable algorithms and uses execution feedback to refine how runtime evidence is interpreted for subsequent designs. We evaluate OnDesign across two mainstream black-box optimization paradigms on three scenarios: Bayesian optimization, evolutionary continuous optimization, and evolutionary mixed-variable optimization. Extensive experiments on six benchmark suites and one engineering problem across multiple problem dimensions demonstrate superior overall performance over conventional optimizers and offline LLM-based AAD methods.
Zhiyao Zhang, Yichen Li, Xingyu Wu +2
Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
Large Language Models (LLMs) are increasingly used for automated algorithm design. However the computational cost of evaluating the generated algorithms can be excessive. We consider the common LLM-driven automated algorithm design (LLM4AD) setting in which a candidate algorithm is evaluated by aggregating its performance over a shared set of training instances. This instance-wise structure raises a natural question: must every candidate be evaluated on the entire instance set before deciding whether it remains competitive? Taking inspiration from algorithm configuration, we introduce AdaEva, a drop-in adaptive partial-evaluation framework that progressively evaluates candidates on larger subsets of the same instance pool and eliminates unpromising candidates as evidence accumulates. Importantly, AdaEva leaves the underlying LLM4AD procedure and per-instance evaluator unchanged and requires no prior knowledge about instance difficulty. We instantiate this idea using successive halving (AdaEva-S) and statistical racing (AdaEva-R), and evaluate both mechanisms across three representative LLM4AD frameworks, multiple LLM backbones, and optimization domains spanning combinatorial and continuous black-box optimization. Under matched evaluation budgets, AdaEva more reliably balances evaluation effort across candidates than fixed partial-evaluation strategies, yielding strong search efficiency and anytime performance together with improved held-out generalization across the evaluated settings.
Tai Nguyen, Fei Liu, Phong Le +2
University of St Andrews, United Kingdom · Sorbonne Universit´e, CNRS, LIP6, France · University of Zurich and ETH Zurich, Switzerland