An important Bottleneck in LLM Optimizers: Principle-Level Local Optima

Nian Ran; Tianhe Jia; Zhengyang Mu; Yue Wang; Sebastian Brulin; Markus Olhofer; Mauricio A. Alvarez; Yang Liu

cs.LGOct 16, 2026

An important Bottleneck in LLM Optimizers: Principle-Level Local Optima

Authors: Nian Ran, Tianhe Jia, Zhengyang Mu, Yue Wang, Sebastian Brulin, Markus Olhofer, Mauricio A. Alvarez, Yang Liu,

Organizations: The University of Manchester · UKRI AI Centre for Doctoral Training in Decision Making for Complex Systems · University of Chinese Academy of Sciences · Zhongguancun Academy ·

Abstract

Feedback-driven LLM optimizers are increasingly used for evolutionary search, Bayesian optimization, heuristic design, and scientific discovery. Yet rapid initial gains often give way to prolonged plateaus despite substantial remaining evaluation budgets. We investigate principle-level search collapse, where an optimizer's search becomes confined to a narrow set of design principles, even though it may continue to generate distinct candidates and achieve small improvements. We assemble a diagnostic benchmark spanning molecular optimization, network sentinel-node selection, and GPU-kernel optimization, and introduce a decision-tree protocol combining principle narrowing, prolonged performance plateaus, and gaps to attainable reference solutions. Across 161 assessable runs on the three problem domains, 78.3% receive support for collapse, including 55.3% with strong support. Candidate-level variation can nevertheless conceal restricted search at the principle level. We then test diversity injection, a common response to search stagnation, such as higher t, multiple LLMs, and web search, and show that it provides limited and inconsistent benefits: among 65 molecular runs, still 73.8% and 53.8% receive support and strong support respectively. We finally test reference-principle interventions and find that they improve some runs, but their benefits vary across models and tasks: access to a better principle does not guarantee its effective use. Our benchmark and diagnostic protocol provide a basis for evaluating future optimizers on principle-level collapse. Our findings highlight the need for online collapse detection and better mechanisms for generating and maintaining alternative principles during optimization.

Figures & tables

Figure 1: Overview of the collapse diagnosis. Narrowing may be supported by structural metrics or an eligible audit of principle texts. A long plateau leaves at least

30\%

of planned budget. The lower branch summarizes limited narrowing evidence. Text-audit eligibility and exceptions are specified in the full protocol (Appendix C , Figure 6 ).

	GPU kernel	Molecular optimization	Sentinel-node selection
Candidate $x$	one PyTorch/Triton source file	SMILES string	node subset of size $r$ (4 or 5)
Utility $f$	$-\ln$ (geometric-mean latency)	scalarized five-objective score	$-\ln$ (approximation error)
Descriptor $\phi$	executed route, blind LLM annotation against a per-problem rubric (6–9 leaf routes + families)	Bemis–Murcko scaffold	node incidence in selected sets
Diagnostic set $\mathcal{A}$	fastest 10 valid kernels	top 50 distinct molecules	top 50 distinct node sets
Candidate-level diversity	5-gram Jaccard distance between source files	fingerprint diversity, uniqueness	set uniqueness
Reference $x_{\mathrm{ref}}$	FlashInfer library kernel, same GPU	best known molecule, $F=4.6985$	exhaustive optimum (Dolphin); published optimized set (residence hall)

Table 1: Instantiation of the framework on the three tasks. Task-specific structural descriptors and references complement the graded decision tree in Figure 1 .

Panel of Figure 2 — Figure 2: Best performance and effective count: molecules (left), residence hall (middle) and GPU kernels (right; the four operators pooled). Solid/dashed lines use the left/right axes. Stars mark mean plateau onset. The molecular reference is $4.699$ ; the dashed line in the network is the lower bound of $N_{\mathrm{eff}}$ ; the kernel panel shows expert latency divided by best-so-far latency (expert $=1$ ) as the geometric mean across runs, and $N_{\mathrm{eff}}$ in the diagnostic set of the ten fastest kernels; per-operator curves are in Figure 5 .

Figure 3: Molecular early/late diagnostics. Hollow/filled bars are early/final values. Uniqueness uses the first/last 1,000 generated candidates. Other metrics use reconstructed 50-candidate sets. Effective and unique scaffold counts are divided by 50.

Figure 4: Diversity-injection diagnostics, with the definitions and scaling of Figure 3 . ExLLM variants average backbones within each seed. Two LLMs and MCCE are independent five-seed comparisons. Broader generation does not reliably preserve a broad selected scaffold distribution.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Kernel performance and route concentration. Solid lines show expert latency divided by best-so-far latency (left, log scale; higher is better); dashed lines show effective route count in the top 10 (right). Stars mark mean plateau onset. RMSNorm is a negative control.

Figure 6: Full diagnostic protocol. Structural evidence, eligibility for textual auditing, and the text audit are all parts of assessing principle narrowing. Text auditing requires explicit principle-space modeling covering the actual search directions. In the final 1,000 generated attempts,

T_{1}

requires dominant strategy share

\geq 50\%

T_{2}

allows at most five minor revisions without a major change, and

T_{3}

requires no new proposed direction.

Method	Condition	S	L	N
PiEvo ( $\sigma=0.9$ )	Sigma 0.9	1	1	3
PiEvo	Original	2	3	0
ExLLM(Sonnet 5)	Explore (7:2:1)	5	0	0
ExLLM(Sonnet 5)	Original	3	2	0
ExLLM(Sonnet 5)	Web search	3	2	0
ExLLM(Sonnet 5 Thinking)	Original	5	0	0

Appendix

Table 2: Molecules: current non-intervention diagnostic counts.

	Dolphin			Residence hall
Method	S	L	N	S	L	N
PiEvo	0	0	5	4	1	0
ExLLM(Sonnet 5)	0	0	5	3	0	2
ExLLM(DeepSeek)	0	0	5	5	0	0
ExLLM(Qwen)	0	0	5	0	4	1
ExLLM(GPT-5.4)	0	0	5	1	1	3

Appendix

Table 3: Non-intervention diagnostic counts on both networks. S/L/N denote strong support, limited support, and collapse not established. All rows use the original optimizer setting.

Method / condition	Top-1	Top-10	HV (50)	$U$	$D$	$b_{\mathrm{plat}}$	Remaining (%)
ExLLM(GPT-5.4)	$4.047\pm 0.118$	$4.027\pm 0.110$	$0.551\pm 0.199$	$0.326\pm 0.067$	$0.507\pm 0.040$	$2762.800\pm 829.500$	$44.744\pm 16.590$
ExLLM(Sonnet 5)	$3.975\pm 0.008$	$3.962\pm 0.012$	$0.374\pm 0.022$	$0.122\pm 0.011$	$0.533\pm 0.050$	$1172.400\pm 459.202$	$76.552\pm 9.184$
ExLLM(DeepSeek)	$3.998\pm 0.049$	$3.986\pm 0.043$	$0.419\pm 0.093$	$0.312\pm 0.072$	$0.463\pm 0.032$	$2064.000\pm 802.579$	$58.720\pm 16.052$
ExLLM(Qwen)	$4.091\pm 0.109$	$4.047\pm 0.077$	$0.659\pm 0.220$	$0.338\pm 0.094$	$0.589\pm 0.025$	$3079.200\pm 1415.091$	$38.416\pm 28.302$
PiEvo	$4.100\pm 0.200$	$4.057\pm 0.189$	$0.610\pm 0.363$	$0.559\pm 0.303$	$0.532\pm 0.170$	$3214.800\pm 1509.945$	$35.704\pm 30.199$
MCCE	$4.288\pm 0.103$	$4.220\pm 0.059$	$0.860\pm 0.080$	$0.656\pm 0.020$	$0.501\pm 0.030$	$4255.600\pm 534.429$	$14.888\pm 10.689$

Appendix

Table 4: Molecular baseline performance. HV and diversity use the reconstructed 50-candidate set. Remaining budget is planned budget after the retrospective plateau onset, not measured wasted evaluations.

Method / condition	$U_{e}$	$U_{l}$	$D_{e}$	$D_{l}$	$N_{\mathrm{eff},e}$	$N_{\mathrm{eff},l}$	$S_{e}$	$S_{l}$	$m_{e}$	$m_{l}$
ExLLM(GPT-5.4)	$0.717\pm 0.058$	$0.206\pm 0.082$	$0.849\pm 0.008$	$0.507\pm 0.040$	$40.117\pm 4.846$	$1.908\pm 1.116$	$42.800\pm 3.564$	$3.400\pm 1.817$	$0.068\pm 0.018$	$0.820\pm 0.215$
ExLLM(Sonnet 5)	$0.495\pm 0.067$	$0.067\pm 0.013$	$0.842\pm 0.006$	$0.533\pm 0.050$	$38.842\pm 3.583$	$2.577\pm 1.312$	$41.800\pm 2.775$	$3.400\pm 2.302$	$0.064\pm 0.009$	$0.664\pm 0.204$
ExLLM(DeepSeek)	$0.886\pm 0.032$	$0.098\pm 0.058$	$0.862\pm 0.007$	$0.463\pm 0.032$	$43.106\pm 6.482$	$1.241\pm 0.332$	$46.000\pm 3.391$	$1.600\pm 0.894$	$0.084\pm 0.054$	$0.936\pm 0.092$
ExLLM(Qwen)	$0.675\pm 0.047$	$0.390\pm 0.103$	$0.845\pm 0.008$	$0.589\pm 0.025$	$42.094\pm 4.131$	$1.755\pm 0.730$	$45.000\pm 1.871$	$3.200\pm 1.924$	$0.076\pm 0.050$	$0.808\pm 0.209$
PiEvo	$0.824\pm 0.129$	$0.415\pm 0.363$	$0.863\pm 0.005$	$0.532\pm 0.170$	$48.276\pm 1.257$	$30.685\pm 11.115$	$48.800\pm 0.837$	$35.400\pm 9.813$	$0.040\pm 0.014$	$0.132\pm 0.067$
MCCE	$0.770\pm 0.047$	$0.646\pm 0.022$	$0.852\pm 0.004$	$0.501\pm 0.030$	$39.926\pm 4.504$	$1.659\pm 0.547$	$43.400\pm 2.966$	$2.400\pm 1.140$	$0.088\pm 0.036$	$0.804\pm 0.177$

Appendix

Table 5: Molecular concentration. Early/late denote first-offspring/final reconstructed sets; uniqueness instead uses first/last 1,000 generated candidates. These are top-50 measurements, not the earlier top-10 window-based verdicts.

Dolphin
Method / condition	Top-1 error	Top-10 error	$U$	$N_{\mathrm{eff},e}$	$N_{\mathrm{eff},l}$	$b_{\mathrm{plat}}$	Remaining (%)
ExLLM(GPT-5.4)	$4.071\times 10^{-3}\pm 2.406\times 10^{-5}$	$4.147\times 10^{-3}\pm 2.617\times 10^{-5}$	$0.228\pm 0.028$	$45.780\pm 2.351$	$13.971\pm 1.419$	$852.400\pm 276.991$	$72.503\pm 8.935$
ExLLM(Sonnet 5)	$4.115\times 10^{-3}\pm 7.582\times 10^{-5}$	$4.189\times 10^{-3}\pm 8.557\times 10^{-5}$	$0.420\pm 0.035$	$44.787\pm 4.002$	$13.990\pm 0.630$	$929.000\pm 570.675$	$70.032\pm 18.409$
ExLLM(DeepSeek)	$4.216\times 10^{-3}\pm 3.604\times 10^{-5}$	$4.320\times 10^{-3}\pm 3.408\times 10^{-5}$	$0.271\pm 0.061$	$48.251\pm 2.670$	$14.214\pm 1.602$	$657.400\pm 309.967$	$78.794\pm 9.999$
ExLLM(Qwen)	$4.234\times 10^{-3}\pm 1.073\times 10^{-4}$	$4.373\times 10^{-3}\pm 1.501\times 10^{-4}$	$0.482\pm 0.115$	$50.305\pm 1.465$	$21.705\pm 7.702$	$1291.200\pm 334.970$	$58.348\pm 10.805$
PiEvo	$4.098\times 10^{-3}\pm 7.122\times 10^{-5}$	$4.176\times 10^{-3}\pm 1.034\times 10^{-4}$	$0.099\pm 0.052$	$33.441\pm 12.692$	$14.108\pm 1.878$	$962.600\pm 453.699$	$68.948\pm 14.635$

Appendix

Table 6: Network baseline performance and effective node pool. Effective nodes summarize node usage, not semantic principles; lower errors are better.

Dolphin
Method / condition	$U_{e}$	$U_{l}$	$C_{5,e}$	$C_{5,l}$	$R_{2,e}$	$R_{2,l}$	$R_{3,e}$	$R_{3,l}$
ExLLM(GPT-5.4)	$0.482\pm 0.021$	$0.169\pm 0.041$	$0.270\pm 0.021$	$0.742\pm 0.080$	$0.088\pm 0.011$	$0.324\pm 0.134$	$0.052\pm 0.011$	$0.128\pm 0.056$
ExLLM(Sonnet 5)	$0.652\pm 0.132$	$0.282\pm 0.047$	$0.294\pm 0.024$	$0.745\pm 0.041$	$0.084\pm 0.009$	$0.372\pm 0.070$	$0.056\pm 0.009$	$0.156\pm 0.038$
ExLLM(DeepSeek)	$0.546\pm 0.079$	$0.154\pm 0.027$	$0.244\pm 0.017$	$0.703\pm 0.114$	$0.072\pm 0.011$	$0.360\pm 0.235$	$0.044\pm 0.009$	$0.108\pm 0.048$
ExLLM(Qwen)	$0.658\pm 0.040$	$0.389\pm 0.172$	$0.248\pm 0.019$	$0.617\pm 0.178$	$0.064\pm 0.009$	$0.360\pm 0.258$	$0.036\pm 0.009$	$0.132\pm 0.097$
PiEvo	$0.447\pm 0.129$	$0.039\pm 0.029$	$0.404\pm 0.126$	$0.693\pm 0.050$	$0.216\pm 0.109$	$0.328\pm 0.076$	$0.080\pm 0.037$	$0.124\pm 0.022$

Appendix

Table 7: Network node concentration.

C_{5}

is normalized top-five coverage;

R_{2},R_{3}

are maximum pair/triple reuse. Higher values indicate concentration;

e/l

denote early/late.

Panel of Figure 7 — Figure 7: Additional molecular trajectories: dominant scaffold share and unique scaffold count (left), fingerprint diversity and cumulative uniqueness (right). Final values are carried forward to budget 5,000.

Panel of Figure 8 — Figure 8: Network early/late bars and supplementary trajectories. Dolphin’s concentration is interpreted against its exhaustive optimum, not as failure on its own. Pair/triple reuse measures repeated node combinations; effective-node count measures spread of node usage.

Panel of Figure 9 — Figure 9: Negative controls in the layout of Figure 2 : Dolphin (left; the reference is the exhaustive optimum) and RMSNorm (right; the reference is the expert kernel, ratio $=1$ ). Solid/dashed lines use the left/right axes; stars mark mean plateau onset. Both tasks plateau early with concentrated pools, but near or beyond the reference, so the Headroom condition fails and collapse is not established. The RMSNorm curve is the geometric mean across seeds; $N_{\mathrm{eff}}$ uses the diagnostic set of the ten fastest kernels, smoothed as in Figure 5 ; the GPT-4o control is included.

Method / condition	Top-1	Top-10	HV (50)	$U$	$D$	$b_{\mathrm{plat}}$	Remaining (%)
ExLLM(Baseline)	$4.028\pm 0.011$	$4.005\pm 0.007$	$0.501\pm 0.039$	$0.274\pm 0.044$	$0.523\pm 0.021$	$2269.600\pm 439.577$	$54.608\pm 8.792$
ExLLM(Explore)	$4.048\pm 0.079$	$4.021\pm 0.070$	$0.503\pm 0.115$	$0.263\pm 0.025$	$0.520\pm 0.007$	$2262.700\pm 554.565$	$54.746\pm 11.091$
ExLLM(Web search)	$4.108\pm 0.121$	$4.067\pm 0.091$	$0.627\pm 0.174$	$0.301\pm 0.005$	$0.519\pm 0.031$	$2591.850\pm 594.545$	$48.163\pm 11.891$
ExLLM(Temp 1.3)	$4.086\pm 0.116$	$4.046\pm 0.087$	$0.577\pm 0.148$	$0.400\pm 0.031$	$0.532\pm 0.029$	$2425.200\pm 1042.968$	$51.496\pm 20.859$
Two LLMs	$4.165\pm 0.304$	$4.141\pm 0.274$	$0.667\pm 0.356$	$0.636\pm 0.106$	$0.485\pm 0.075$	$2032.200\pm 949.502$	$59.356\pm 18.990$
MCCE	$4.288\pm 0.103$	$4.220\pm 0.059$	$0.860\pm 0.080$	$0.656\pm 0.020$	$0.501\pm 0.030$	$4255.600\pm 534.429$	$14.888\pm 10.689$

Appendix

Table 8: Diversity injection, average backbones within each seed, then report mean of five seed averages. Temperature uses three backbones; baseline/explore/web use four. Two LLMs and MCCE each use five independent runs.

Method / condition	$U_{e}$	$U_{l}$	$D_{e}$	$D_{l}$	$N_{\mathrm{eff},e}$	$N_{\mathrm{eff},l}$	$S_{e}$	$S_{l}$	$m_{e}$	$m_{l}$
ExLLM(Baseline)	$0.693\pm 0.018$	$0.190\pm 0.033$	$0.850\pm 0.006$	$0.523\pm 0.021$	$41.040\pm 4.320$	$1.871\pm 0.230$	$43.900\pm 2.626$	$2.900\pm 0.720$	$0.073\pm 0.026$	$0.807\pm 0.038$
ExLLM(Explore)	$0.646\pm 0.030$	$0.183\pm 0.007$	$0.849\pm 0.008$	$0.520\pm 0.007$	$40.740\pm 1.241$	$2.366\pm 1.209$	$43.400\pm 0.894$	$3.550\pm 1.866$	$0.067\pm 0.009$	$0.766\pm 0.096$
ExLLM(Web search)	$0.644\pm 0.032$	$0.248\pm 0.036$	$0.839\pm 0.008$	$0.519\pm 0.031$	$42.475\pm 2.613$	$4.727\pm 3.496$	$44.800\pm 1.681$	$7.200\pm 3.475$	$0.064\pm 0.016$	$0.730\pm 0.116$
ExLLM(Temp 1.3)	$0.735\pm 0.027$	$0.326\pm 0.055$	$0.851\pm 0.006$	$0.532\pm 0.029$	$43.091\pm 1.725$	$1.774\pm 0.442$	$45.200\pm 1.121$	$2.733\pm 0.596$	$0.061\pm 0.014$	$0.827\pm 0.096$
Two LLMs	$0.903\pm 0.019$	$0.556\pm 0.136$	$0.848\pm 0.003$	$0.485\pm 0.075$	$39.731\pm 1.984$	$1.298\pm 0.461$	$42.400\pm 1.517$	$1.600\pm 0.894$	$0.064\pm 0.009$	$0.908\pm 0.148$
MCCE	$0.770\pm 0.047$	$0.646\pm 0.022$	$0.852\pm 0.004$	$0.501\pm 0.030$	$39.926\pm 4.504$	$1.659\pm 0.547$	$43.400\pm 2.966$	$2.400\pm 1.140$	$0.088\pm 0.036$	$0.804\pm 0.177$

Appendix

Table 9: Diversity-injection concentration under the same aggregation as Table 8 .

ExLLM(GPT-5.4)
Method / condition	Top-1	Top-10	HV (50)	$U$	$D$	$b_{\mathrm{plat}}$	Remaining (%)
ExLLM(GPT-5.4) / Explore (7:2:1)	$3.993\pm 0.117$	$3.978\pm 0.106$	$0.429\pm 0.216$	$0.367\pm 0.038$	$0.532\pm 0.020$	$2094.600\pm 1632.767$	$58.108\pm 32.655$
ExLLM(GPT-5.4)	$4.047\pm 0.118$	$4.027\pm 0.110$	$0.551\pm 0.199$	$0.326\pm 0.067$	$0.507\pm 0.040$	$2762.800\pm 829.500$	$44.744\pm 16.590$
ExLLM(GPT-5.4) / Temperature 1.3	$4.204\pm 0.366$	$4.140\pm 0.295$	$0.656\pm 0.476$	$0.342\pm 0.048$	$0.482\pm 0.028$	$2880.600\pm 1771.310$	$42.388\pm 35.426$
ExLLM(GPT-5.4) / Web search	$4.062\pm 0.271$	$4.018\pm 0.210$	$0.559\pm 0.356$	$0.310\pm 0.061$	$0.523\pm 0.058$	$2781.000\pm 1498.193$	$44.380\pm 29.964$
ExLLM(Sonnet 5)

Appendix

Table 10: Per-backbone molecular diversity-injection comparisons. Sonnet has no temperature-1.3 arm.

Panel of Figure 10 — Figure 10: Per-backbone diversity injection: GPT-5.4 and Sonnet. Sonnet has no temperature-1.3 arm. Two LLMs and MCCE are fixed independent comparators, not variants of the named backbone.

Panel of Figure 11 — Figure 11: Per-backbone diversity injection: DeepSeek and Qwen. These panels expose heterogeneity that a pooled mean can conceal; they do not establish a uniform effect size across backbones.

Method / condition	Top-1	Top-10	$U$	$U_{e}$	$U_{l}$	$N_{\mathrm{eff},l}$	Top-10 scaffolds
ExLLM(GPT-5.4)	$4.047\pm 0.118$	$4.027\pm 0.110$	$0.326\pm 0.067$	$0.717\pm 0.058$	$0.206\pm 0.082$	$1.908\pm 1.116$	$1.800\pm 0.447$
ExLLM(Sonnet 5)	$3.975\pm 0.008$	$3.962\pm 0.012$	$0.122\pm 0.011$	$0.495\pm 0.067$	$0.067\pm 0.013$	$2.577\pm 1.312$	$2.400\pm 1.342$
ExLLM(DeepSeek)	$3.998\pm 0.049$	$3.986\pm 0.043$	$0.312\pm 0.072$	$0.886\pm 0.032$	$0.098\pm 0.058$	$1.241\pm 0.332$	$1.200\pm 0.447$
ExLLM(Qwen)	$4.091\pm 0.109$	$4.047\pm 0.077$	$0.338\pm 0.094$	$0.675\pm 0.047$	$0.390\pm 0.103$	$1.755\pm 0.730$	$1.800\pm 1.304$
ExLLM(GPT-4o)	$4.190\pm 0.007$	$4.170\pm 0.006$	$0.833\pm 0.061$	$0.988\pm 0.007$	$0.682\pm 0.075$	$1.057\pm 0.083$	$1.000\pm 0.000$
ExLLM(Sonnet thinking)	$3.972\pm 0.004$	$3.963\pm 0.011$	$0.203\pm 0.038$	$0.575\pm 0.062$	$0.094\pm 0.018$	$1.665\pm 0.770$	$1.600\pm 0.894$

Appendix

Table 11: Older-model and thinking controls. Model training differences are not experimentally isolated.

Method / condition	Updates early	Updates late	Anomalies all	Anomalies early	Anomalies late	$T_{\mathrm{eff},e}$	$T_{\mathrm{eff},l}$
PiEvo	$13.600\pm 2.966$	$1.000\pm 2.236$	$19.600\pm 13.903$	$3.600\pm 2.966$	$1.000\pm 2.236$	$2.675\pm 0.951$	$2.229\pm 1.191$
PiEvo / Sigma 0.9	$11.000\pm 0.707$	$0.400\pm 0.894$	$3.800\pm 2.168$	$1.000\pm 0.707$	$0.400\pm 0.894$	$8.213\pm 0.843$	$9.053\pm 1.411$

Appendix

Table 12: PiEvo on molecules. Early/late are the first/last 1,000 generated attempts after initialization; updates include warm-up. Text usage and posterior summaries are attempt-weighted; events are counted at their recorded batch boundaries.

Method / problem	Updates early	Updates late	Anomalies all	Anomalies early	Anomalies late	$T_{\mathrm{eff},e}$	$T_{\mathrm{eff},l}$
PiEvo /dolphin	$11.600\pm 1.342$	$0.000\pm 0.000$	$4.200\pm 2.950$	$2.600\pm 1.817$	$0.000\pm 0.000$	$5.623\pm 3.048$	$2.826\pm 1.487$
PiEvo /residence hall	$11.800\pm 1.304$	$0.000\pm 0.000$	$6.400\pm 2.702$	$2.200\pm 1.095$	$0.000\pm 0.000$	$7.429\pm 1.589$	$6.382\pm 2.348$

Appendix

Table 13: PiEvo on residence hall. Early/late are the first/last 1,000 generated attempts after initialization; updates include warm-up. Text usage and posterior summaries are attempt-weighted; events are counted at their recorded batch boundaries.

Method / condition	Top-1	Top-10	$H^{\mathrm{belief}}_{e}$	$H^{\mathrm{belief}}_{l}$	Scaffold $N_{\mathrm{eff},l}$	$b_{\mathrm{plat}}$
PiEvo	$4.100\pm 0.200$	$4.057\pm 0.189$	$0.010\pm 0.007$	$0.000\pm 0.000$	$30.685\pm 11.115$	$3214.800\pm 1509.945$
PiEvo / Sigma 0.9	$4.068\pm 0.156$	$4.040\pm 0.152$	$0.115\pm 0.026$	$0.008\pm 0.004$	$18.972\pm 9.671$	$3595.200\pm 970.716$

Appendix

Table 14: PiEvo molecular noise sensitivity. Belief entropies are attempt-weighted over the first/last 1,000 generated attempts. They measure posterior uncertainty, not scaffold coverage.

Method / condition	Top-1	Top-10	$U$	$N_{\mathrm{eff},l}$	$m_{l}$
ExLLM(GPT-5.4) / Hint from 40%	$4.021\pm 0.090$	$3.992\pm 0.095$	$0.396\pm 0.038$	$3.717\pm 1.083$	$0.544\pm 0.190$
ExLLM(GPT-5.4) / Hint from start	$4.264\pm 0.098$	$4.201\pm 0.088$	$0.315\pm 0.040$	$1.541\pm 0.511$	$0.856\pm 0.172$
ExLLM(GPT-5.4)	$4.047\pm 0.118$	$4.027\pm 0.110$	$0.326\pm 0.067$	$1.908\pm 1.116$	$0.820\pm 0.215$
ExLLM(Sonnet 5) / Hint from 40%	$4.485\pm 0.298$	$4.394\pm 0.254$	$0.151\pm 0.016$	$2.440\pm 1.292$	$0.636\pm 0.234$
ExLLM(Sonnet 5) / Hint from start	$4.617\pm 0.183$	$4.513\pm 0.163$	$0.170\pm 0.033$	$2.162\pm 0.314$	$0.572\pm 0.064$
ExLLM(Sonnet 5)	$3.975\pm 0.008$	$3.962\pm 0.012$	$0.122\pm 0.011$	$2.577\pm 1.312$	$0.664\pm 0.204$

Appendix

Table 15: Molecular L2 timing comparison. Performance improvements, not exact reference identity, define intervention benefit.

Method / condition	Top-1 error	Top-10 error	$U$	$N_{\mathrm{eff},l}$
ExLLM(GPT-5.4) / Hint from 40%	$4.077\times 10^{-3}\pm 3.422\times 10^{-5}$	$4.144\times 10^{-3}\pm 2.249\times 10^{-5}$	$0.213\pm 0.031$	$13.947\pm 1.283$
ExLLM(GPT-5.4)	$4.071\times 10^{-3}\pm 2.406\times 10^{-5}$	$4.147\times 10^{-3}\pm 2.617\times 10^{-5}$	$0.228\pm 0.028$	$13.971\pm 1.419$
ExLLM(Sonnet 5) / Hint from 40%	$4.115\times 10^{-3}\pm 7.580\times 10^{-5}$	$4.196\times 10^{-3}\pm 7.556\times 10^{-5}$	$0.391\pm 0.061$	$14.292\pm 0.928$
ExLLM(Sonnet 5)	$4.115\times 10^{-3}\pm 7.582\times 10^{-5}$	$4.189\times 10^{-3}\pm 8.557\times 10^{-5}$	$0.420\pm 0.035$	$13.990\pm 0.630$
ExLLM(DeepSeek) / Hint from 40%	$4.089\times 10^{-3}\pm 5.906\times 10^{-5}$	$4.172\times 10^{-3}\pm 4.753\times 10^{-5}$	$0.336\pm 0.079$	$15.383\pm 1.991$
ExLLM(DeepSeek)	$4.216\times 10^{-3}\pm 3.604\times 10^{-5}$	$4.320\times 10^{-3}\pm 3.408\times 10^{-5}$	$0.271\pm 0.061$	$14.214\pm 1.602$

Appendix

Table 16: Dolphin delayed L2 intervention. This is a continuation comparison, not a test of reproducing the reference node set.

Method / condition	Top-1 error	Top-10 error	$U$	$N_{\mathrm{eff},l}$
ExLLM(GPT-5.4) / Hint from 40%	$5.112\times 10^{-7}\pm 3.635\times 10^{-7}$	$1.152\times 10^{-6}\pm 2.242\times 10^{-7}$	$0.395\pm 0.065$	$27.815\pm 2.855$
ExLLM(GPT-5.4)	$7.005\times 10^{-7}\pm 5.689\times 10^{-7}$	$1.352\times 10^{-6}\pm 6.547\times 10^{-7}$	$0.378\pm 0.109$	$26.752\pm 3.587$
ExLLM(Sonnet 5) / Hint from 40%	$1.579\times 10^{-6}\pm 1.759\times 10^{-6}$	$3.600\times 10^{-6}\pm 2.861\times 10^{-6}$	$0.341\pm 0.111$	$20.962\pm 1.078$
ExLLM(Sonnet 5)	$1.514\times 10^{-6}\pm 1.084\times 10^{-6}$	$4.434\times 10^{-6}\pm 2.171\times 10^{-6}$	$0.303\pm 0.109$	$18.709\pm 1.172$
ExLLM(DeepSeek) / Hint from 40%	$9.447\times 10^{-7}\pm 5.885\times 10^{-7}$	$2.660\times 10^{-6}\pm 1.405\times 10^{-6}$	$0.360\pm 0.087$	$25.469\pm 5.719$
ExLLM(DeepSeek)	$1.402\times 10^{-6}\pm 8.028\times 10^{-7}$	$3.846\times 10^{-6}\pm 1.887\times 10^{-6}$	$0.255\pm 0.049$	$23.065\pm 5.774$

Appendix

Table 17: Residence hall delayed L2 intervention. This is a continuation comparison, not a test of reproducing the reference node set.

Panel of Figure 12 — Figure 12: Intervention concentration diagnostics. Within each backbone: baseline early, baseline late, from-start late when available, and delayed-hint late. Improved performance does not require an increase in principle diversity; a run may move to a better concentrated region.

Task	$n$	Collapse	Late plateau	$G<2$ , concentrated	Not concentrated
GQA decode	20	5	5	10	0
MLA decode	20	9	6	5	0
MLA prefill	20	19	1	0	0
MoE FP8	20	15	4	0	1
RMSNorm	20	0	0	19	1

Appendix

Table 18: Joint kernel diagnostics for the four main ExLLM backbones, five seeds each. “Late plateau” means concentrated with

G\geq 2

but less than 30% of budget remaining. The

G<2

column does not imply matching the expert.

Method	Task	$n$	$G$ (latency / expert)	Late $N_{\mathrm{eff}}$	Collapse / assessed
ExLLM(GPT-5.4)	GQA decode	5	$28.015\pm 15.670$	$1.000\pm 0.000$	3/5
ExLLM(GPT-5.4)	MLA decode	5	$22.140\pm 2.674$	$1.000\pm 0.000$	4/5
ExLLM(GPT-5.4)	MLA prefill	5	$33.396\pm 2.389$	$1.000\pm 0.000$	5/5
ExLLM(GPT-5.4)	MoE FP8	5	$37.548\pm 1.011$	$1.260\pm 0.355$	5/5
ExLLM(GPT-5.4)	RMSNorm	5	$0.467\pm 0.013$	$1.399\pm 0.298$	0/5
ExLLM(Claude Sonnet 5)	GQA decode	5	$0.762\pm 0.062$	$1.207\pm 0.298$	0/5

Appendix

Table 19: Kernel performance and final route concentration. ExLLM uses five seeds per task; PiEvo uses three. Late effective counts and collapse denominators include only runs with at least 20 valid kernels; latency summaries include runs with a recorded latency even when they fail that coverage requirement.

0/0

means unassessable, not no collapse. GPT-4o is an additional control. No PiEvo RMSNorm results are supplied.

Figure 13: Kernel early/late diagnostics in the fastest-ten sets. Empty/filled bars denote initial/final sets. Code diversity is mean pairwise token 5-gram Jaccard distance, not the fraction of distinct candidates. Fine-grained routes and their broader families are both shown.

Figure 14: The same kernel early/late diagnostics with paired individual runs. Differences across problems and seeds remain visible rather than being inferred solely from pooled means.

Timing	Hint	Pairs / planned	Better	Worse	Within band	Geom. gain
Start	L1	48/48	28	12	8	1.169
Start	L2	46/48	14	21	11	0.833
Start	L3	46/48	20	15	11	1.067
Onset	L1	41/41	12	10	19	1.110
Onset	L2	41/41	13	9	19	1.010
Onset	L3	41/41	16	8	17	0.986

Appendix

Table 20: Paired kernel intervention outcomes across the four main backbones. Geometric-mean gain above one favors the hint. Missing final latencies are excluded, not assigned neutral gains.

Figure 15: Kernel principle interventions, paired with each run’s baseline on seeds 42–44. Left: reference-route share in the final fastest-ten set (runs with at least 20 valid kernels). Right: baseline/hinted latency, with geometric means and individual runs; values above one indicate benefit. Top/bottom rows inject from the start/at detected narrowing. Adoption and performance need not improve together. Exact coverage and gain-band definitions are in Table 20 .

Figure 16: Best achieved latency by executed route in baselines and L2/L3 interventions. Route membership does not ensure an expert-quality implementation.

Figure 17: L1 from the start: hinted runs (solid) and their baselines (dotted), matched by seed and color. Latency is divided by expert latency; lower is better.

Figure 18: L1 at detected narrowing. Dashed vertical lines mark injection; other conventions match the from-start comparison.

Figure 19: L2 from the start, paired with baseline trajectories.

Figure 20: L2 at detected narrowing, paired with baseline trajectories.

Figure 21: L3 from the start, paired with baseline trajectories.

Figure 22: L3 at detected narrowing, paired with baseline trajectories.

Figure 23: Kernel first-attempt quality versus subsequent development. (a) All 186 events, grouped by the ratio of the route’s first kernel to the run’s best latency at that moment. Dots are events coloured by backbone (filled: at least 50 attempts left, hollow: fewer); boxes span the interquartile range; the solid line is the median and the dashed line the median of the control. The vertical axis is logarithmic in $1+N_{\mathrm{after}}$ so that zero can be shown. (b) The 18 first appearances of the expert’s route; stars mark routes that later produced the run’s fastest kernel.

First kernel of the route vs. the run’s best so far	events	never developed again	median $N_{\mathrm{after}}$	IQR	events with $\geq 50$ attempts left	their median $N_{\mathrm{after}}$
First kernel faster than the incumbent	73	3%	59	15–96	71	64
Up to 2x slower	69	23%	5	1–28	67	5
More than 2x slower	44	36%	1	0–4	36	1.5
All events	186	18%	9.5	1–78.5	174	14

Appendix

Table 21: Further development of a route by the quality of its first kernel (186 events, 80 baseline runs, five seeds). IQR is the interquartile range of

N_{\mathrm{after}}

; the last two columns restrict to events with at least 50 attempts left.

Problem	Backbone	seed	attempt	$r$	$N_{\mathrm{after}}$	best on the route / expert
GQA decode	Sonnet 5	42	22	1.35	14	0.98
GQA decode	Sonnet 5	43	15	3.39	37	0.77 $\star$
GQA decode	Sonnet 5	44	14	3.95	63	0.66 $\star$
GQA decode	Sonnet 5	45	19	2.99	13	1.01
GQA decode	Sonnet 5	46	13	1.55	18	0.79 $\star$
MLA decode	Sonnet 5	43	16	3.27	151	0.86 $\star$

Appendix

Table 22: The 18 first appearances of the expert’s route in the baseline runs: evaluation attempt of the first kernel, its ratio

r

to the run’s best latency at that moment, further kernels on the route, and the best latency reached on the route relative to the expert (

\star

: the route produced the run’s fastest kernel).

Task	Hint
Molecular (L2)	Search around compact heteroaryl-amide chemotypes rather than only refining generic benzamide or para-methoxy aryl-amide families. Favor low-synthetic-complexity molecules with a small sulfur-containing or heteroaromatic ring, a simple aryl/benzyl partner, and light substituent tuning to keep QED high while driving the unwanted target probabilities near zero.
Dolphin (L2)	A direction is a heterogeneous degree profile: include one low-degree or leaf-like sentinel and combine it with low-to-mid and moderate/high-role nodes whose errors complement each other as a set.
Residence hall (L2)	Look for five sentinels that jointly cover different network roles across degree and neighborhood structure. Try heterogeneous complementarity, try a mixed low/mid/moderately-high profile and test replacements that preserve that role coverage.
Kernel, all problems (L1)	Optimization trajectories on this problem typically settle early on one overall design approach whose latency stays far above what the hardware allows, and further refinement within that approach does not close the gap. Whenever the implementations shown share the same overall algorithmic decomposition, do not keep tuning it; propose a candidate whose decomposition of the computation is fundamentally different.
GQA / MLA paged decode (L2)	Instead of one program per (sequence, head) that walks the whole KV sequence serially, split each sequence’s KV range into several chunks processed by different programs in parallel, each producing partial attention statistics (running max, normalizer, accumulated output), and merge the chunks afterwards with log-sum-exp rescaling. Pack the heads that share the same KV as the row dimension of a matrix product so that scores and outputs run on tensor cores and each KV block is read once.
MLA paged prefill (L2)	Treat the computation as a two-dimensional tiling—a block of query tokens (with all heads, which share the same compressed KV) against a block of KV tokens—so that score computation and output accumulation are matrix products on tensor cores, with an online softmax carried across KV blocks and the causal mask applied only where a block crosses the diagonal. Long sequences can additionally be split along the KV dimension across programs and merged with log-sum-exp rescaling. Avoid host-side loops over sequences.

Appendix

Table 23: Oracle-principle hints used in the interventions.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 608 papers

Apr 21, 2026

What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search

Recent work has demonstrated the promise of orchestrating large language models (LLMs) within evolutionary and agentic optimization systems. However, the mechanisms driving these optimization gains remain poorly understood. In this work, we present a large-scale study of LLM-guided evolutionary search, collecting optimization trajectories for 15 LLMs across 8 tasks. Although zero-shot problem-solving ability correlates with final optimization outcomes, it explains only part of the variance: models with similar initial capability often induce dramatically different search trajectories and outcomes. By analyzing these trajectories, we find that strong LLM optimizers behave as local refiners, producing frequent incremental improvements while progressively localizing the search in semantic space. Conversely, weaker optimizers exhibit large semantic drift, with sporadic breakthroughs followed by stagnation. Notably, various measures of solution novelty do not predict final performance; novelty is beneficial only when the search remains sufficiently localized around high-performing regions of the solution space. Our results highlight the importance of trajectory analysis for understanding and improving LLM-based optimization systems and provide actionable insights for their design and training.

Xinhao Zhang, Xi Chen, François Portet +1

Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

Evolutionary Optimization Large Language Model-Guided Optimization LLMs for Evolutionary Optimization

Ask PDF

Sep 30, 2026

Initialization Improves LLM-Driven Discovery

Large Language Models (LLMs) have been used for novel discovery of algorithms, theorems, drugs, and other tasks through the use of harnesses that prompt an LLM to iteratively optimize an objective. In this work, we study the relationship between the population of previous iterates and eventual discovery success. We generalize past work on harness design to develop a suite of 12 harnesses called 'Modular' and characterize their performance across 5 diverse discovery tasks, finding that discovery success is brittle and sensitive to harness design. We uncover mode collapse, characterized by a dramatic drop in the diversity of iterates, as a common failure mode. We find that popular state-of-the-art harnesses and diversity-inducing harness interventions, which aim to prolong this collapse, yield inconsistent gains. Our results instead uncover that the performance of early discoveries is predictive of eventual success. We therefore propose a universally applicable intervention that performs an initial stage of parallel exploration in order to initialize subsequent iterative optimization. Our method provides consistent gains across many harnesses and target applications, confirming the importance of initialization in LLM-driven discovery.

Mansi Sakarvadia, Marco Ciccone, Colin Raffel

University of Chicago · Vector Institute · University of Toronto

Large Language Model-Guided Optimization Large Language Model-Guided Scientific Discovery

Ask PDF

Sep 29, 2026

OptiCom : A Unified Framework for State-Conditioned Composition in LLM-Driven Optimization

Large language models (LLMs) are increasingly deployed to solve complex scientific and practical problems via iterative optimization. However, dynamically coordinating diverse search mechanisms as candidate quality, failure modes, and resource budgets evolve remains a critical open challenge. Targeted empirical diagnostics reveal that mechanism effectiveness is highly state-dependent. Motivated by this, we analyze how individual decisions drive final outcomes, decomposing the expected terminal improvement under a shared budget into cumulative decision opportunities minus cumulative selection losses. Guided by this opportunity-loss theoretical foundation, we propose OptiCom, a unified framework that represents LLM-driven optimizers within a shared configuration space: C=(A,Q,O,E,M,S), corresponding to artifact, query, operator, evaluation, memory, and strategy. Operating within this space, a fast LLM-based Optimization Controller dynamically composes immediate mechanisms through structured Action Packages, while a slower Strategy Adapter refines long-term selection preferences, operator weights, and templates based on accumulated trajectory feedback. Comprehensive evaluations across 32 benchmark groups demonstrate the superiority of framework: OptiCom achieves an average Max-score rank of 1.72 among 14 evaluated configurations, securing the top score in 23 groups. Ultimately, these results highlight the broad applicability and high extensibility of OptiCom as a general-purpose paradigm for robust LLM test-time scaling.

Chenxing Wei, Sichen Liu, Lizhao Liu +6

School of Computing and Data Science, The University of Hong Kong · Shenzhen Loop Area Institute · ByteDance +3

Large Language Model-Guided Optimization Test-Time Scaling

Ask PDF

Abstract

Large Language Model-Guided Optimization

Explore similar work

What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search

Initialization Improves LLM-Driven Discovery

OptiCom : A Unified Framework for State-Conditioned Composition in LLM-Driven Optimization