Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation

Zhongwei Yu; Sourabh Roy; Bin Cao; Xue Yan; Anjie Liu; Jun Wang

cs.LGSep 29, 2026

Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation

Authors: Zhongwei Yu, Sourabh Roy, Bin Cao, Xue Yan, Anjie Liu, Jun Wang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · University College London (UCL) · Institute of Automation, Chinese Academy of Sciences

Abstract

Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian optimisation (BO). Recently, practitioners have started to use large language models (LLMs) to generate and execute BO programs through coding harnesses. In such emerging practices, the posterior belief is shaped not only by Bayesian inference but also by LLM-generated model and data artefacts, offering a flexible route for task context to enter BO as executable code. To study whether and how LLMs can be harnessed to compile diverse contextual signals for BO, we formulate LLM-compiled BO as generalised-context decision making. We propose HarBO, a BO-specialised harness that compiles generalised context into the core artefacts of standard BO through a validated multi-stage workflow. Our theory analyses the regret under imperfect compilation and the effect of adding new context. Across synthetic functions and real-world benchmarks, we find that LLM harnesses can effectively compile context into standard BO, achieving competitive performance with specialised LLM-embedding-based and direct LLM-in-the-loop BO methods. General coding harnesses can be effective in familiar domains such as hyperparameter optimisation, but fall short in unfamiliar, context-rich domains. Together, these results establish LLM harnesses as a promising, but not automatically reliable, route for making rich task context usable in BO.

Figures & tables

Figure 1: From human-specified BO to LLM-compiled BO, where the domain practitioner delegates the optimisation task to a harness-based coding agent through interactive conversation.

Figure 2: The staged workflow of HarBO. The context is decomposed into epistemic

c^{ep}

, aleatory

c_{t}^{al}

, history

\mathcal{H}_{t}

, and optional instruction

\delta_{t}

. These components are compiled through the belief channel (stages Z, D, GP, and R) and the utility channel (stage Acq), and then a decision

x_{t}

is selected by the core Bayesian engine.

Figure 3: In-process comparison against baselines.

Figure 4: Context ablations in the nine in-process environments.

Figure 5: Agentic campaigns compared with in-process HarBO and Vanilla GP-UCB.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	A	B	C	D	E	H	HM	Dock	O-Suzuki
Metric	$\hat{y}^{*}$	$\hat{y}^{*}$	$\hat{y}^{*}$	$R_{T}/T$	$R_{T}/T$	$100\hat{y}^{*}$	$100(R_{T}/T)$	$\hat{y}^{*}$	$\hat{y}^{*}$
HarBO	$13.83\pm 0.03$	$\mathbf{14.07\pm 0.13}$	$\mathbf{14.00\pm 0.18}$	$\mathbf{6.08\pm 1.16}$	$\mathbf{7.44\pm 0.87}$	$\mathbf{-5.34\pm 0.02}$	$2.09\pm 0.58$	$\mathbf{11.06\pm 0.51}$	$\mathbf{98.41\pm 1.96}$
HarBO (transfer)	–	–	–	–	–	$-5.47\pm 0.14$	–	–	–
HarBO (no prior)	$12.60\pm 1.58$	$12.81\pm 2.69$	$7.75\pm 4.00$	$8.39\pm 1.54$	$9.51\pm 1.06$	$-9.57\pm 5.17$	$1.98\pm 0.58$	$10.28\pm 1.31$	$97.47\pm 0.93$
HarBO (no category mapping)	–	–	–	–	–	–	–	–	$97.24\pm 0.83$
HarBO (no pilots)	–	$13.90\pm 0.12$	–	–	$7.57\pm 1.23$	–	–	–	–

Appendix

Table 1: In-process results across the nine environments.

Method	A	B	C	D	E	H	HM	Dock	O-Suzuki
HarBO (DSV4F high)	$318\pm 82$	$648\pm 474$	$612\pm 141$	$848\pm 120$	$459\pm 105$	$537\pm 123$	$609\pm 220$	$2190\pm 836$	$278\pm 78$
HarBO (DSV4F low)	$128\pm 8$	$139\pm 22$	$179\pm 7$	$200\pm 18$	$198\pm 26$	$85\pm 18$	$223\pm 23$	$2354\pm 903$	$71\pm 10$
HarBO (DSV4F none)	$78\pm 7$	$104\pm 20$	$133\pm 2$	$125\pm 2$	$176\pm 24$	$74^{\dagger}$	$138^{\dagger}$	$1612^{\dagger}$	$36\pm 5$
HarBO (GPT-4o-mini none)	$106\pm 19$	$87^{\dagger}$	$190^{\dagger}$	$238^{\dagger}$	–	–	–	–	$32\pm 3$
CAKE	$726\pm 128$	$783\pm 64$	$854\pm 198$	$893\pm 186$	$531\pm 92$	$2420\pm 462$	$1109\pm 344$	–	$787\pm 73$
LGBO	$111\pm 2$	$129\pm 3$	$136\pm 10$	$156\pm 8$	$124\pm 4$	$127\pm 4$	$135\pm 16$	–	$114\pm 3$

Appendix

Table 2: Wall time by environment and method (seconds).

Method	A	B	C	D	E	H	HM	Dock	O-Suzuki
HarBO (DSV4F high)	166,905	363,010	375,473	431,451	374,427	331,125	435,997	268,672	197,710
HarBO (DSV4F low)	58,173	61,119	114,476	185,467	198,700	67,949	201,925	61,407	75,547
HarBO (DSV4F none)	14,264	28,677	28,837	58,208	102,611	6,559	56,286	8,826	42,806
HarBO (GPT-4o-mini none)	23,819	14,003	20,958	39,745	–	–	–	–	38,166
CAKE	639,458	707,457	643,056	653,209	777,828	864,220	928,093	–	881,789
LGBO	403,837	449,885	406,256	416,420	488,461	548,720	601,924	–	604,647

Appendix

Table 3: Aggregate LLM token consumption across successful campaigns by environment and method.

Method	Reasoning	LLM calls	Total tokens	Wall time (s)	R ${}_{T}/T$
HarBO (ours)	high	53	94,701	624	$8.36$
CAKE	–	80	155,877	495	$13.44$
CAKE	high	81	600,079	4,742	$14.08$
CAKE	low	80	473,696	3,766	$13.54$
LGBO	–	47	97,425	119	$13.04$
LGBO	high	48	371,182	2,587	$12.46$

Appendix

Table 4: Reasoning-cost comparison on scenario E (

T=50

, seed 0).

Method / variant	A	D	E	H	HM	Dock
Metric	$\hat{y}^{*}$	$R_{T}/T$	$R_{T}/T$	$100\hat{y}^{*}$	$R_{T}/T$	$\hat{y}^{*}$
Agentic (skill)	$14.02\pm 0.10$	$5.33\pm 1.09$	$6.73\pm 0.76$	$-5.41\pm 0.16$	$0.037\pm 0.012$	$11.30\pm 0.48$
Agentic (noskill)	$13.70\pm 0.43$	$7.69\pm 0.34$	$8.24\pm 0.33$	$-5.37\pm 0.03$	$0.036\pm 0.007$	$10.72\pm 0.59$
Agentic (transfer)	–	–	–	$-5.39\pm 0.02$	–	–
Agentic (no-ref)	–	–	–	–	–	$11.08\pm 0.21$
Agentic (no-ref, noskill)	–	–	–	–	–	$10.86\pm 0.64$

Appendix

Table 5: Agentic campaigns and in-process references.

Model	Thinking	A	B	C	D	E	H	HM	Dock	O-Suzuki
DeepSeek-V4-Flash-0731	high	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$
DeepSeek-V4-Flash-0731	low	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$0/5$	$1/5$	$0/5$
DeepSeek-V4-Flash-0731	none	$2/5$	$0/5$	$3/5$	$3/5$	$2/5$	$4/5$	$4/5$	$4/5$	$2/5$
gpt-4o-mini	none	$2/5$	$4/5$	$4/5$	$4/5$	$5/5$	$5/5$	$5/5$	$5/5$	$2/5$

Appendix

Table 6: Actual terminal failures by model and thinking configuration.

Model	Thinking	A	B	C	D	E	H	HM	Dock	O-Suzuki
DeepSeek-V4-Flash-0731	high	$0/5$	$1/5$	$2/5$	$1/5$	$0/5$	$1/5$	$3/5$	$1/5$	$0/5$
DeepSeek-V4-Flash-0731	low	$2/5$	$1/5$	$3/5$	$3/5$	$2/5$	$1/5$	$4/5$	$3/5$	$2/5$
DeepSeek-V4-Flash-0731	none	$2/5$	$0/5$	$3/5$	$3/5$	$4/5$	$4/5$	$5/5$	$5/5$	$5/5$
gpt-4o-mini	none	$5/5$	$5/5$	$5/5$	$5/5$	$5/5$	$5/5$	$5/5$	$5/5$	$5/5$

Appendix

Table 7: Runs containing at least one validation error by model and thinking configuration.

Model	Thinking	A	B	C	D	E	H	HM	Dock	O-Suzuki
DeepSeek-V4-Flash-0731	high	0	2	2	1	0	1	7	1	0
DeepSeek-V4-Flash-0731	low	3	1	4	5	2	1	8	5	2
DeepSeek-V4-Flash-0731	none	4	0	6	6	6	8	10	8	7
gpt-4o-mini	none	7	14	10	19	16	10	10	10	8

Appendix

Table 8: Total validation errors by model and thinking configuration.

Scenario	DSV4F (high)	DSV4F (low)	DSV4F (none)	GPT-4o-mini (none)
A	$13.83$ (5/5)	$13.97$ (5/5)	$14.04$ (3/5)	$13.84$ (3/5)
B	$14.07$ (5/5)	$13.42$ (5/5)	$13.53$ (5/5)	$13.82$ (1/5)
C	$14.00$ (5/5)	$12.03$ (5/5)	$13.75$ (2/5)	$14.11$ (1/5)
D ( $R_{T}/T$ )	$6.08$ (5/5)	$7.17$ (5/5)	$7.69$ (2/5)	$14.07$ (1/5)
E ( $R_{T}/T$ )	$7.44$ (5/5)	$7.79$ (5/5)	$9.57$ (3/5)	– (0/5)
H	$-0.053$ (5/5)	$-0.088$ (5/5)	$-0.061$ (1/5)	– (0/5)

Appendix

Table 9: Successful-seed performance by model and reasoning setting.

Figure 6: Model and reasoning-intensity ablation trajectories.

Bayesian Optimization

Bayesian optimization uses probabilistic models and sequential experiments to find strong solutions when evaluating a black-box objective is costly.

See all 240 papers

Bayesian Optimization with Rich Auxiliary Information via LLMs
Sep 16, 2026Efe Mert Karagözlü, Rohit Sonker, Tejus Gupta +2Bayesian Optimization Large Language Model-Guided Optimization
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213
FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization
Oct 15, 2026Hao Chen, Ziyang Ling, Ronald X. Xu +1Large Language Model-Guided Optimization Bayesian Optimization
School of Biomedical Engineering, University of Science and Technology of China, Suzhou, China · School of Engineering Science, University of Science and Technology of China, Suzhou, China
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
May 16, 2026Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1Bayesian Optimization Benchmark Design
National University of Singapore

Sep 16, 2026J/K move · Enter open · S save

Bayesian Optimization with Rich Auxiliary Information via LLMs

Efe Mert Karagözlü, Rohit Sonker, Tejus Gupta, Barnabás Póczos+1

School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213

Bayesian Optimization Large Language Model-Guided Optimization

Open paper AskPDF arXiv

Bayesian Optimization (BO) is widely used for optimizing expensive black-box functions, yet it typically reduces each expensive experiment to an input and its objective value, discarding much of the information the experiment produces. Such auxiliary information can include training curves in hyperparameter optimization, expert notes and images in scientific experimentation, or known facts about the specific optimization problem. Leveraging the ability of large language models (LLMs) to process diverse and unstructured information, we introduce two methods: AuxBO-Evolve, which uses auxiliary information to evolve beliefs over the location of the optimum, and AuxBO-Vicinity, which uses it to locally guide the acquisition function. Across synthetic tasks, hyperparameter optimization benchmarks, and a nuclear fusion task with unstructured scientist-written logs, our methods consistently improve optimization performance and outperform standard BO and existing LLM-based approaches. We further find that LLMs provide more effective probabilistic guidance when modeling likely maximizer locations rather than objective values pointwise. Overall, our results demonstrate that exploiting rich information beyond standard

(\mathbf{x}, y)

interactions can substantially improve BO performance.

Abstract

Bayesian Optimization

Explore similar work

Bayesian Optimization with Rich Auxiliary Information via LLMs

FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization

BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks