FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization

FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization

Abstract

Figures & tables

Large Language Model-Guided Optimization

Bayesian Optimization with Rich Auxiliary Information via LLMs

Abstract

Large Language Model-Guided Optimization

Explore similar work

Appendix

Bayesian Optimization with Rich Auxiliary Information via LLMs

Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation

LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentation

Hao Chen; Ziyang Ling; Ronald X. Xu; Mingzhai Sun

cs.CLOct 15, 2026

Authors: Hao Chen, Ziyang Ling, Ronald X. Xu, Mingzhai Sun

Organizations: School of Biomedical Engineering, University of Science and Technology of China, Suzhou, China · School of Engineering Science, University of Science and Technology of China, Suzhou, China

Large language model (LLM)-guided Bayesian optimization (BO) uses past observations to guide search, but proposed directions and their revision can remain implicit. We propose Feedback-Guided Search State (FGSS), a structured hypothesis-management interface connecting language-based guidance to observed evidence, executable parameter interventions, and comparative support updates. An LLM reflector proposes and revises hypotheses; a deterministic curator validates them and maintains identities and test records. The evolving state guides candidate construction and priorities. Instantiated on LLAMBO, FGSS-BO retains its surrogate and default expected improvement selection, with a bounded testing policy. On two retrospective protocol/media tasks with nearest-neighbor response oracles, it attains higher sample mean final best and search AUC than the matched LLAMBO baseline and evaluated non-LLM baselines. Prose-reflection and EI-only controls show mixed component effects. Six direct-training hyperparameter-optimization tasks show task-dependent transfer, strongest on diabetes SVM; exact-grid and reaction controls examine other settings. FGSS exposes hypotheses, their execution, and cross-round revisions for inspection while supporting search gains in selected settings.

Figure 1: Overview of FGSS-BO. The extractor, LLM reflector, and deterministic curator maintain evidence-linked hypotheses; a regime gate controls candidate allocation. LLAMBO retains its history-conditioned surrogate and default EI selection, with a bounded exception for eligible matched tests. Solid arrows show current-round flow; dashed arrows show cross-round memory and eligible comparative support updates.

Task	Metric	Random Search	HEBO	GP-EI	LLAMBO	FGSS-BO
iPSC-RPE	Final best	$0.780\pm 0.078$	$0.569\pm 0.169$	$0.793\pm 0.026$	$0.691\pm 0.117$	$0.813\pm 0.023$
iPSC-RPE	Search AUC	$0.557\pm 0.150$	$0.465\pm 0.100$	$0.505\pm 0.145$	$0.494\pm 0.057$	$0.640\pm 0.117$
C2C12	Final best	$0.790\pm 0.340$	$1.026\pm 0.623$	$1.516\pm 0.904$	$1.281\pm 0.220$	$1.725\pm 0.411$
C2C12	Search AUC	$0.552\pm 0.289$	$0.722\pm 0.404$	$0.868\pm 0.653$	$1.025\pm 0.264$	$1.167\pm 0.256$

Table 1: Primary offline protocol/media search over six paired runs: mean

\pm

sample standard deviation, higher is better. All methods share ten low-score initial observations and twenty search evaluations under the same task-specific nearest-neighbor response oracle. LLAMBO uses the shared feasibility-preserving interface.

Figure 2: Primary-task mean best-so-far objective over six paired runs. The dashed line separates ten shared low-score initialization observations from twenty search evaluations; shading shows one sample standard deviation. LLAMBO and FGSS-BO share the feasible-candidate interface and nearest-neighbor oracle. C2C12 scores refer to the retained configuration–source query task.

Method	Final best	Search AUC
LLAMBO	$.691\pm.117$	$.494\pm.057$
Reflective Summary	$.785\pm.061$	$.646\pm.054$
FGSS w/o Tests	$.804\pm.070$	$.668\pm.051$
FGSS No-Override	$.802\pm.110$	$.690\pm.098$
Full FGSS	$.813\pm.023$	$.640\pm.117$

Table 2: iPSC component comparison, six paired runs; mean

\pm

sample standard deviation. All variants share the low-start nearest-neighbor protocol. No-Override retains test candidates and support updates but always selects by EI. LLAMBO uses the shared feasibility-preserving interface.

Figure 3: Proposal-level candidate diagnostics from earlier runs, comparing free-form LLM proposals with feasibility-preserving construction. Severe iPSC-RPE duplication motivates a shared feasible-candidate interface.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Setting	Value	Scope/provenance
Initialization/search evaluations	Primary/HPO $10/20$ ; Summit $8/20$	Six paired seeds, batch size one
Candidate pool; templates; generations	$10$ ; $2$ ; $10$	Shared main-run configuration
Reflector output/temperature/retry	$1500$ tokens; $0$ ; $1$	FGSS extra call; retries logged
Shared refiner output/temperature	$1000$ tokens; $0$	Top five plus recent five observations
Evidence top/bottom/recent	$3/3/5$	Outcome-ranked top/bottom
Stagnation windows and thresholds	$3/5$ ; $0.01/0.02$	Short/long progress heuristics

Appendix

Table 3: Implementation settings inherited from development defaults and used in the recorded main runs, without a hyperparameter sweep.

Regime	Global	Guided	Local	Recovery
Explore	.55	.25	.10	.10
Mixed	.40	.30	.20	.10
Exploit	.25	.30	.35	.10
Recover	.45	.15	.15	.25

Appendix

Table 4: Nominal candidate-allocation presets, not fixed realized ratios. The gate preserves a global floor of

.30

(

.55

before reliable support,

.45

under weak support); plateau recovery enforces global/recovery floors

.45/.25

and guided cap

.20

Task	Cutoff	Configuration source
iPSC-RPE	$0.12$	Recorded CLI override
C2C12	$0.05$	Task-runner default
SnAr	$0.10$	Summit task preset
Reizman 1–4	$50$	Summit task preset
HPLC	$1000$	Task-runner default
HPO	$0$	Task-bridge setting

Appendix

Table 5: Absolute regime-gate cutoffs and their configuration sources. iPSC uses the reported override rather than the generic runner default of

0.01

Appendix

Table 6: Actual selection changes and EI-selected matched tests; each run has twenty search evaluations. EI tests receive a recorded support update without changing EI selection.

Task	Variable	Type	Admissible domain
SnAr	Residence time	Continuous	$[0.5,2.0]$ min
	Pyrrolidine equivalents	Continuous	$[1,5]$ equivalents
	DFNB concentration	Continuous	$[0.1,0.5]$ mol/L
	Temperature	Continuous	$[30,120]$ degrees Celsius
Reizman 1–4	Residence time	Continuous	$[60,600]$ s
	Temperature	Continuous	$[30,110]$ degrees Celsius

Appendix

Table 7: Domains used by the actual adapters. Summit bounds and catalyst levels follow benchmark definitions ( Felton et al., 2021 ) ; HPLC bounds follow the Olympus dataset configuration ( Häse et al., 2021 ) . iPSC concentration levels follow the original task-code definition; the adapter domains are not assumed exhaustive physical domains. C2C12 coordinates follow its adapter preprocessing.

Task	Seed	Prompt tokens	Completion tokens	Elapsed (s)
iPSC	0	347576	10226	177.5
	1	332143	10664	144.0
	2	368290	10319	144.7
	3	346650	10845	145.7
	4	346886	10709	127.8
	5	337694	9570	113.4

Appendix

Table 8: Recorded full-FGSS reflector usage per primary-task run, with twenty final call records each. Elapsed is the sum of saved reflector-call latencies, not total optimizer wall-clock time.

Figure 4: Five-method primary-task trajectories, six paired runs with ten shared low-score initialization observations and twenty search evaluations. Lines are means and bands are sample standard deviations. All methods use the same nearest-neighbor response oracle; bands are not confidence intervals. LLAMBO uses the shared feasibility-preserving interface.

Task	Reference	Metric	Mean difference	Bootstrap 95% interval
iPSC-RPE	GP-EI	Final best	$0.020$	$[-0.009,0.050]$
iPSC-RPE	LLAMBO	Final best	$0.123$	$[0.046,0.205]$
iPSC-RPE	GP-EI	Search AUC	$0.135$	$[0.014,0.254]$
iPSC-RPE	LLAMBO	Search AUC	$0.147$	$[0.054,0.252]$
C2C12	GP-EI	Final best	$0.210$	$[-0.271,0.691]$
C2C12	LLAMBO	Final best	$0.445$	$[0.110,0.959]$

Appendix

Table 9: Paired primary-task differences, FGSS-BO minus the reference; positive favors FGSS-BO. LLAMBO uses the shared feasibility-preserving interface.

Task	Metric	Random Search	HEBO	GP-EI	LLAMBO	FGSS-BO
SnAr	Final best	$0.593\pm 0.074$	$1.051\pm 0.005$	$1.058\pm 0.001$	$0.641\pm 0.107$	$0.680\pm 0.057$
SnAr	Search AUC	$0.572\pm 0.078$	$0.967\pm 0.048$	$0.990\pm 0.049$	$0.579\pm 0.101$	$0.573\pm 0.123$
Reizman Case 1	Final best	$67.505\pm 10.443$	$86.480\pm 6.887$	$91.424\pm 0.019$	$60.071\pm 15.792$	$69.751\pm 14.764$
Reizman Case 1	Search AUC	$64.033\pm 9.672$	$74.673\pm 9.317$	$83.315\pm 5.291$	$55.087\pm 13.751$	$65.738\pm 15.484$
Reizman Case 2	Final best	$21.602\pm 6.882$	$37.151\pm 6.817$	$40.085\pm 0.000$	$20.910\pm 7.124$	$22.286\pm 9.016$
Reizman Case 2	Search AUC	$19.916\pm 6.719$	$32.166\pm 6.307$	$32.762\pm 4.777$	$19.164\pm 6.721$	$21.262\pm 8.662$

Appendix

Table 10: Cross-domain Summit results: five methods over six paired runs, with eight initialization and twenty search evaluations. Values are mean

\pm

sample standard deviation; higher is better. LLAMBO uses the shared feasibility-preserving interface.

Contrast	Metric	Mean difference	Bootstrap 95% interval
Full FGSS $-$ Reflective Summary	Final best	$0.0285$	$[0.0050,0.0617]$
	Search AUC	$-0.0061$	$[-0.0702,0.0605]$
Full FGSS $-$ FGSS w/o Tests	Final best	$0.0088$	$[-0.0370,0.0730]$
	Search AUC	$-0.0275$	$[-0.1264,0.0651]$
FGSS w/o Tests $-$ Reflective Summary	Final best	$0.0197$	$[-0.0527,0.0910]$
	Search AUC	$0.0214$	$[-0.0310,0.0822]$

Appendix

Table 11: Paired iPSC component differences over six runs. Positive favors the first method.

Condition	Final best	Search AUC
True	$.8132\pm.0225$	$.6401\pm.1175$
Permuted	$.7857\pm.0726$	$.6751\pm.0654$

Appendix

Table 12: iPSC-RPE state-feedback control: mean

\pm

sample standard deviation over six paired runs.

Figure 5: iPSC-RPE with true or permuted state feedback, alongside the LLAMBO reference. Lines show mean best-so-far true objective over six paired runs; shading shows one sample standard deviation. The dashed boundary separates ten initialization observations from twenty search evaluations. Only FGSS-BO state feedback is permuted; the surrogate retains true observation history.

Method	Final best	Search AUC	Selection changes	Direct support updates
LLAMBO	$0.6905\pm 0.1168$	$0.4936\pm 0.0571$	0	0
Full FGSS	$0.8132\pm 0.0225$	$0.6401\pm 0.1175$	18	37
FGSS No-Override	$0.8015\pm 0.1103$	$0.6896\pm 0.0980$	0	22

Appendix

Table 13: iPSC selection-policy control over six paired runs: mean

\pm

sample standard deviation. Counts are totals over 120 search evaluations per method. Selection changes mean departures from the recorded EI choice, not tests naturally selected by EI. Full FGSS and LLAMBO reuse the audited primary references and shared feasibility-preserving interface.

Figure 6: iPSC No-Override ablation, all six paired seeds. Curves and bars show means; shading and bar error bars show sample standard deviations. The right panel reports full FGSS minus No-Override: numbered points are individual seeds, and red intervals are exploratory paired bootstrap 95% intervals. No-Override retains hypothesis-test generation and support updates but always selects by EI. LLAMBO uses the shared feasibility-preserving interface.

Figure 7: Immediate opportunity costs of all 18 full-FGSS selection changes under the fixed iPSC oracle. The left panel compares the alternative response scores; the right reports EI minus test response and one-step incumbent gaps, indexed by seed and one-based search round. Positive gaps favor EI. Each comparison uses the same recorded history and candidate pool.

Figure 8: Candidate-interface controls on both primary tasks over all six paired seeds. Lines are means; shading is sample standard deviation. The dashed line separates ten shared low-score initial observations from twenty search evaluations. iPSC Freeform uses recovery fallback in all 120 search rounds, producing six identical evaluation trajectories; C2C12 has no recovery fallback. Both methods omit FGSS state.

Task	Direction	Final best		Search AUC
		LLAMBO	FGSS-BO	LLAMBO	FGSS-BO
Breast RF	Accuracy $\uparrow$	$.93187\pm.00340$	$.93187\pm.00368$	$.93161\pm.00365$	$.93152\pm.00388$
Breast SVM	Accuracy $\uparrow$	$.94286\pm.00197$	$.94432\pm.00266$	$.94282\pm.00202$	$.94370\pm.00256$
Digits RF	Accuracy $\uparrow$	$.53550\pm.25607$	$.52727\pm.29339$	$.46033\pm.30376$	$.47376\pm.31506$
Digits SVM	Accuracy $\uparrow$	$.99246\pm.00052$	$.99258\pm.00036$	$.99236\pm.00045$	$.99235\pm.00036$
Diabetes RF	MSE $\downarrow$	$.58558\pm.03102$	$.57109\pm.04270$	$.61024\pm.04163$	$.61335\pm.07394$

Appendix

Table 14: Complete six-task HPO transfer comparison: means

\pm

sample standard deviations over six paired runs per task. Accuracy is maximized; both diabetes metrics are minimized. RF denotes RandomForest. Raw accuracy and MSE are not aggregated. LLAMBO uses the shared feasibility-preserving interface.

Parameter	Domain	Scale
RandomForest
Max. depth	$[1,15]$	Linear
Max. feature fraction	$[.01,.99]$	Logit
Min. impurity decrease	$[0,.5]$	Linear
Min. leaf sample fraction	$[.01,.49]$	Logit
Min. split sample fraction	$[.01,.99]$	Logit

Appendix

Table 15: HPO domains and sampling scales inherited from LLAMBO’s Bayesmark configuration, shared by both methods. Maximum depth is integer; all other parameters are continuous.

Task	Metric	Gain	Bootstrap 95% interval
Breast RF	Final	$0.00000$	$[-.00110,.00110]$
	AUC	$-.00009$	$[-.00066,.00038]$
Breast SVM	Final	$.00147$	$[.00037,.00293]$
	AUC	$.00088$	$[-.00051,.00242]$
Digits RF	Final	$-.00824$	$[-.13083,.09440]$
	AUC	$.01343$	$[-.06695,.07959]$

Appendix

Table 16: Paired HPO mean gains use unrounded per-seed metrics. Final denotes final best; AUC denotes search AUC. Positive favors FGSS-BO: FGSS minus LLAMBO for accuracy, reversed for MSE. Exploratory intervals are task-specific and unadjusted for multiple comparisons. LLAMBO uses the shared feasibility-preserving interface.

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 601 papers

Bayesian Optimization with Rich Auxiliary Information via LLMs
Sep 16, 2026Efe Mert Karagözlü, Rohit Sonker, Tejus Gupta +2Bayesian Optimization Large Language Model-Guided Optimization
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213
Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation
Sep 29, 2026Zhongwei Yu, Sourabh Roy, Bin Cao +3Bayesian Optimization Large Language Model-Guided Optimization
The Hong Kong University of Science and Technology (Guangzhou) · University College London (UCL) · Institute of Automation, Chinese Academy of Sciences
LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentation
May 21, 2026Zhuo Chen, Xinzhe Yuan, Jianshu Zhang +8AI for Science Bayesian Optimization
equal contribution · Shanghai Artificial Intelligence Laboratory, Shanghai, China · School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China +7

Sep 16, 2026J/K move · Enter open · S save

Efe Mert Karagözlü, Rohit Sonker, Tejus Gupta, Barnabás Póczos+1

School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213

Bayesian Optimization Large Language Model-Guided Optimization

Open paper AskPDF arXiv

Bayesian Optimization (BO) is widely used for optimizing expensive black-box functions, yet it typically reduces each expensive experiment to an input and its objective value, discarding much of the information the experiment produces. Such auxiliary information can include training curves in hyperparameter optimization, expert notes and images in scientific experimentation, or known facts about the specific optimization problem. Leveraging the ability of large language models (LLMs) to process diverse and unstructured information, we introduce two methods: AuxBO-Evolve, which uses auxiliary information to evolve beliefs over the location of the optimum, and AuxBO-Vicinity, which uses it to locally guide the acquisition function. Across synthetic tasks, hyperparameter optimization benchmarks, and a nuclear fusion task with unstructured scientist-written logs, our methods consistently improve optimization performance and outperform standard BO and existing LLM-based approaches. We further find that LLMs provide more effective probabilistic guidance when modeling likely maximizer locations rather than objective values pointwise. Overall, our results demonstrate that exploiting rich information beyond standard

(\mathbf{x}, y)

interactions can substantially improve BO performance.

iPSC overrides

iPSC EI tests

C2C12 overrides

C2C12 EI tests