From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors

Qi Huang; Furong Ye; Ananta Shahane; Thomas Bäck; Niki van Stein

cs.LGMar 3, 2026

From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors

Authors: Qi Huang, Furong Ye, Ananta Shahane, Thomas Bäck, Niki van Stein

Organizations: Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2333 CC Leiden, The Netherlands

Abstract

Large Language Models (LLMs) have already been widely adopted for automated algorithm design, demonstrating strong abilities in generating and evolving algorithms across various fields. Existing work has largely focused on examining their effectiveness in solving specific problems, with search strategies primarily guided by adaptive prompt designs. In this paper, through investigating the token-wise attribution of the prompts to LLM-generated algorithmic codes, we show that providing high-quality algorithmic code examples can substantially improve the performance of the LLM-driven optimization. Building upon this insight, we propose leveraging prior benchmark algorithms to guide LLM-driven optimization and demonstrate superior performance on two black-box optimization benchmarks: the pseudo-Boolean optimization suite (pbo) and the black-box optimization suite (bbob). Our findings highlight the value of integrating benchmarking studies to enhance both efficiency and robustness of the LLM-driven black-box optimization methods.

Figures & tables

Fig. 1: The workflow of LLM-driven optimization approaches

Fig. 2: The heatmap of the token-wise relevance of a given prompt to its corresponding newly generated algorithmic code. The result is obtained on an instruction-tuned 27b Gemma 3 LLM using the AttnLRP explainer. Darker shading indicates higher aggregated relevance scores (

R_{i}\in[0,1]

in Equation 3 ), thus more important.

Model	Task Description	Strategy	Expected Output	Note	Parent Heuristic(s)	Code Description	Linguistic Description
Qwen3 8b	1.171 $\pm$ 0.374	1.090 $\pm$ 0.452	0.929 $\pm$ 0.389	0.146 $\pm$ 0.036	1.138 $\pm$ 0.636	1.232 $\pm$ 0.672	0.539 $\pm$ 0.145
Qwen3 14b	0.688 $\pm$ 0.096	1.068 $\pm$ 0.300	0.893 $\pm$ 0.125	0.144 $\pm$ 0.034	0.863 $\pm$ 0.203	0.901 $\pm$ 0.231	0.594 $\pm$ 0.080
Qwen3 32b	1.139 $\pm$ 0.250	1.103 $\pm$ 0.327	1.013 $\pm$ 0.319	0.210 $\pm$ 0.043	1.184 $\pm$ 0.499	1.273 $\pm$ 0.517	0.703 $\pm$ 0.166
Gemma3 4b	1.142 $\pm$ 0.757	0.790 $\pm$ 0.312	0.672 $\pm$ 0.402	0.135 $\pm$ 0.050	1.547 $\pm$ 0.588	1.684 $\pm$ 0.555	0.519 $\pm$ 0.385
Gemma3 12b	0.500 $\pm$ 0.285	1.501 $\pm$ 0.266	0.481 $\pm$ 0.137	0.129 $\pm$ 0.059	0.819 $\pm$ 0.215	0.875 $\pm$ 0.228	0.355 $\pm$ 0.056
Gemma3 27b	0.507 $\pm$ 0.130	2.531 $\pm$ 0.539	0.611 $\pm$ 0.120	0.292 $\pm$ 0.093	0.954 $\pm$ 0.181	1.019 $\pm$ 0.215	0.554 $\pm$ 0.104

TABLE I: Mean and standard deviation of relevance scores (

\times 10^{-3}

) for different parts of prompts. The reported mean and standard deviation are aggregated over the conversations corresponding to five problems of the pbo suite for each LLM model. Relevance scores are normalized such that the sum of all token-level scores within a conversation equals 1. Consequently, the reported means are averages over tokens and do not sum to 1 across prompt parts. For each model, the most relevant prompt component is shown in bold , and the second most relevant is underlined .

Panel of Figure 4 — Fig. 3: Convergence process of refinement-only LHNS and LLaMEA methods for the OneMax problem. $x$ -axis represents the number of algorithms generated by the LLM, and $y$ -axis indicates the best-so-far AUC value. Results are from using Gemini, GPT, and Qwen, respectively (from Left to Right).

Results on 23 pbo benchmark problems
LLM	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	ShinkaEvolve
Gemini	$\mathbf{0.967_{\pm 0.043}(2.125)}$	$0.925_{\pm 0.076}(3.167)$	$0.711_{\pm 0.224}(6.250)$	$0.923_{\pm 0.070}(3.500)$	$0.869_{\pm 0.087}(5.042)$	$0.920_{\pm 0.126}(3.042)$	$0.852_{\pm 0.144}(4.792)$
GPT	$0.918_{\pm 0.088}(3.292)$	$\mathbf{0.924_{\pm 0.086}(3.125)}$	$0.780_{\pm 0.206}(5.292)$	$0.897_{\pm 0.112}(3.625)$	$0.861_{\pm 0.206}(4.250)$	$0.892_{\pm 0.117}(3.875)$	$0.860_{\pm 0.153}(4.542)$
Qwen	$\mathbf{0.950_{\pm 0.083}(2.542)}$	$0.943_{\pm 0.068}(2.958)$	$0.823_{\pm 0.164}(5.917)$	$0.921_{\pm 0.076}(3.958)$	$0.912_{\pm 0.094}(4.125)$	$0.882_{\pm 0.103}(5.042)$	$0.914_{\pm 0.157}(3.417)$
Results on 24 bbob benchmark problems
LLM	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	ShinkaEvolve

TABLE II: Normalized best-achieved AUC results on 23 pbo problems and 24 bbob problems. We report the mean

\pm

standard deviation (average rank).The best-performing entries in the ranking are shown in bold.

Panel of Figure 6 — Fig. 4: Boxplots of the best normalized AUC values obtained by the seven approaches on 23 problems of pbo (Top) and 24 problems of bbob (Bottom). The LLM-driven approaches have been tested on Gemini, GPT, and Qwen, respectively (from Left to Right). For conciseness, MCTS-AHD and ShinkaEvolve are abbreviated as MCTS and Shinka , respectively.

Fig. 5: An illustration of the contribution of queries refining prior benchmark algorithms in BAG. The corresponding obtained results are marked by stars, and green indicates triggering improvements. Results are from using Gemini.

Fig. 6: CodeBLEU similarity scores between ordered pairs of generated algorithms (see Equation 2 ). Axes present time (i.e., the number of algorithms generated). Each column shows the relevance scores between a newly generated algorithm and all previous ones. Larger scores indicate higher relevance.

Fig. 7: Boxplots of the best normalized AUC values obtained by BAG with each of the top-

k

example benchmark algorithms (

\mathbf{A}_{bench}

) removed . The results are aggregated across three LLMs and six bbob problems. The leftmost box-plot (

k

=0) corresponds to the default setting, where no benchmark algorithms are removed.

Fig. 8: Boxplots of the best normalized AUC values obtained by BAG with different frequency factor

q

. The results are aggregated across three LLMs and six bbob problems.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Open-source models used in Section V . Models acquired from Hugging Face.
Name	Version	Provider
Gemma 3	google/gemma-3-4b-it	Google DeepMind
	google/gemma-3-12b-it
	google/gemma-3-27b-it
Qwen 3	Qwen/Qwen3-8b	Alibaba Cloud
	Qwen/Qwen3-14b

Appendix

TABLE III: The used LLMs

Suite	Dimensionality	Budget	Evaluation timeout per instance
pbo	100	1,000,000	600 seconds
bbob	5	10,000	600 seconds

Appendix

TABLE IV: Setups of the pbo and bbob suites

Fig. 9: Average performance of BAG with different frequency factor

q

. Results are aggregated across three LLMs and seven pbo benchmarks.

Fig. 10: Average performance of BAG with different frequency factor

q

. The results are aggregated across three LLMs and six bbob problems.

Fig. 11: Average performance of BAG with each of the top-

k

example benchmark algorithms (

\mathbf{A}_{bench}

) removed . The results are aggregated across three LLMs and seven pbo problems. The leftmost box-plot (

k

=0) corresponds to the default setting, where no benchmark algorithms are removed.

Fig. 12: Average performance of BAG with each of the top-

k

example benchmark algorithms (

\mathbf{A}_{bench}

) removed . The results are aggregated across three LLMs and six bbob problems. The leftmost box-plot (

k

=0) corresponds to the default setting, where no benchmark algorithms are removed.

Variant	Minimal	Single LLM	Native
Backend models	1	1	Multi-model ensemble
LLM identity	Single LLM	Single LLM	Gemini/GPT/Qwen
# Islands	1	2	2
Bandit model selection	No	Yes, one model	Yes, multi-model
Text embeddings	No	Yes, OpenAI text-embedding-3-small
Novelty LLM judge	No	Yes, same model	Yes, GPT 5 Nano

Appendix

TABLE V: Configuration Summary of the Three ShinkaEvolve Variants

Fig. 13: Performance comparison of the three ShinkaEvolve variants by benchmark (top) and by backend LLM (bottom).

Variant	Avg. Rank	Median Rank	# Wins	Mean Raw AUC	Std. Raw AUC
Minimal	1.947	2.000	50/141	0.497	0.256
Single LLM	2.092	2.000	36/141	0.483	0.264
Native	1.961	2.000	54/141	0.509	0.260

Appendix

TABLE VI: Overall Performance of ShinkaEvolve Variants

Function ID	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	Shinka
F1	0.983 (3)	1.000 (1)	0.764 (7)	0.963 (4)	0.946 (6)	0.957 (5)	0.990 (2)
F2	0.852 (5)	0.832 (6)	0.995 (3)	0.995 (3)	0.728 (7)	1.000 (1)	0.999 (2)
F3	0.959 (3)	0.959 (4)	0.927 (6)	0.967 (2)	0.956 (5)	1.000 (1)	0.885 (7)
F4	0.981 (2)	0.979 (4)	0.824 (7)	0.976 (5)	0.961 (6)	1.000 (1)	0.981 (2)
F5	0.954 (2)	0.933 (3)	0.623 (7)	0.900 (6)	0.900 (5)	1.000 (1)	0.919 (4)
F6	0.965 (3)	0.966 (2)	0.763 (7)	0.956 (4)	0.937 (5)	1.000 (1)	0.913 (6)

Appendix

TABLE VII: The best normalized (higher is better) AUC achieved by four LLM-driven approaches on 23 pbo problems. Results are obtained using Gemini 2.5 Flash Lite. Each normalized AUC value is followed by its corresponding rank in brackets. The best entries are underlined.

Function ID	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	Shinka
F1	0.988 (4)	0.992 (2)	0.534 (7)	1.000 (1)	0.939 (6)	0.992 (3)	0.946 (5)
F2	0.860 (2)	0.674 (6)	0.704 (4)	0.725 (3)	0.620 (7)	0.702 (5)	1.000 (1)
F3	0.995 (2)	0.971 (4)	0.959 (6)	1.000 (1)	0.967 (5)	0.980 (3)	0.934 (7)
F4	0.937 (4)	0.922 (6)	0.939 (3)	1.000 (1)	0.922 (5)	0.956 (2)	0.911 (7)
F5	0.928 (5)	0.978 (2)	1.000 (1)	0.900 (7)	0.939 (4)	0.962 (3)	0.927 (6)
F6	0.970 (3)	0.993 (2)	0.820 (7)	0.968 (4)	0.967 (5)	0.951 (6)	1.000 (1)

Appendix

TABLE VIII: The best normalized (higher is better) AUC achieved by four LLM-driven approaches on 23 pbo problems. Results are obtained using GPT 5 Nano. Each normalized AUC value is followed by its corresponding rank in brackets. The best entries are underlined.

Function ID	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	Shinka
F1	0.973 (3)	1.000 (1)	0.877 (7)	0.970 (4)	0.977 (2)	0.944 (5)	0.926 (6)
F2	0.773 (5)	0.830 (2)	0.580 (7)	0.811 (4)	0.824 (3)	0.591 (6)	1.000 (1)
F3	0.970 (4)	0.977 (2)	0.860 (7)	0.977 (2)	0.964 (5)	0.940 (6)	1.000 (1)
F4	0.980 (3)	0.986 (2)	0.911 (7)	0.932 (6)	1.000 (1)	0.938 (5)	0.965 (4)
F5	0.990 (2)	0.971 (4)	0.904 (7)	0.943 (5)	0.989 (3)	0.922 (6)	1.000 (1)
F6	0.968 (5)	0.973 (4)	0.924 (7)	0.957 (6)	0.982 (3)	0.986 (2)	1.000 (1)

Appendix

TABLE IX: The best normalized (higher is better) AUC achieved by four LLM-driven approaches on 23 pbo problems. Results are obtained using Qwen3 Coder Flash. Each normalized AUC value is followed by its corresponding rank in brackets. The best entries are underlined.

Function ID	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	Shinka
F1	0.994 (2)	0.991 (3)	0.976 (5)	0.904 (6)	0.990 (4)	1.000 (1)	0.603 (7)
F2	0.876 (3)	0.881 (2)	0.025 (7)	1.000 (1)	0.787 (5)	0.805 (4)	0.042 (6)
F3	0.608 (2)	0.559 (3)	0.151 (6)	1.000 (1)	0.452 (5)	0.493 (4)	0.135 (7)
F4	0.240 (5)	0.404 (3)	0.118 (7)	1.000 (1)	0.670 (2)	0.352 (4)	0.187 (6)
F5	0.996 (5)	1.000 (1)	0.936 (7)	0.998 (4)	0.995 (6)	0.999 (2)	0.998 (3)
F6	0.960 (5)	1.000 (1)	0.757 (6)	0.972 (4)	0.972 (3)	0.990 (2)	0.520 (7)

Appendix

TABLE X: The best normalized (higher is better) AUC achieved by four LLM-driven approaches on 24 bbob problems. Results are obtained using Gemini 2.5 Flash Lite. Each normalized AUC value is followed by its corresponding rank in brackets. The best entries are underlined.

Function ID	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	Shinka
F1	0.985 (4)	1.000 (1)	0.987 (2)	0.986 (3)	0.985 (4)	0.977 (6)	0.930 (7)
F2	1.000 (1)	0.955 (4)	0.000 (7)	0.944 (5)	0.771 (6)	0.988 (2)	0.975 (3)
F3	0.760 (3)	0.505 (4)	0.117 (7)	0.413 (5)	0.790 (2)	0.330 (6)	1.000 (1)
F4	0.862 (2)	0.704 (3)	0.180 (7)	1.000 (1)	0.552 (4)	0.182 (6)	0.250 (5)
F5	0.999 (4)	1.000 (3)	0.987 (7)	0.999 (5)	0.995 (6)	1.000 (1)	1.000 (1)
F6	0.987 (4)	0.996 (3)	0.689 (7)	0.999 (2)	0.839 (5)	1.000 (1)	0.832 (6)

Appendix

TABLE XI: The best normalized (higher is better) AUC achieved by four LLM-driven approaches on 24 bbob problems. Results are obtained using GPT 5 Nano. Each normalized AUC value is followed by its corresponding rank in brackets. The best entries are underlined.

Function ID	BAG	EoH	LHNS	LLaMEA	MCTS-AHD	ReEvo	Shinka
F1	0.986 (3)	0.994 (2)	0.956 (6)	1.000 (1)	0.475 (7)	0.968 (5)	0.979 (4)
F2	1.000 (1)	0.936 (5)	0.937 (4)	0.965 (3)	0.270 (7)	0.992 (2)	0.310 (6)
F3	1.000 (1)	0.769 (3)	0.518 (4)	0.970 (2)	0.182 (5)	0.109 (7)	0.146 (6)
F4	0.946 (3)	0.566 (5)	0.425 (7)	0.883 (4)	0.553 (6)	1.000 (1)	0.994 (2)
F5	0.999 (2)	0.999 (3)	0.988 (7)	0.999 (4)	0.997 (5)	1.000 (1)	0.996 (6)
F6	1.000 (1)	0.820 (3)	0.866 (2)	0.746 (5)	0.540 (7)	0.698 (6)	0.749 (4)

Appendix

TABLE XII: The best normalized (higher is better) AUC achieved by four LLM-driven approaches on 24 bbob problems. Results are obtained using Qwen3 Coder Flash. Each normalized AUC value is followed by its corresponding rank in brackets. The best entries are underlined.