HapBO: Harnessed α-π Bayesian Optimization via Expert Natural Language Priors
Authors: Jean-Sébastien Giroux, Rose Guay Hottin, Marco Bonizzato
Organizations: Department of Electrical Engineering, Polytechnique Montréal, Montréal, Quebec, Canada. · MILA – Québec AI Institute, Montréal, Quebec, Canada. · Department of Neurosciences, Centre Interdisciplinaire de Recherche sur le Cerveau et l’apprentissage (CIRCA) and Centre d’Innovation Biomédicale, Université de Montréal, Montréal, Quebec, Canada.
Bayesian Optimization (BO) can be substantially more sample-efficient when informed by prior knowledge about the problem. In practice, however, such knowledge is often available only in natural language, e.g., expert intuitions or documentation, rather than as the explicit probability distribution landscape that prior-guided BO methods may require. Leveraging it effectively demands both an expressive mapping from language to priors and robustness to information that may be misleading. We propose HapBO (Harnessed α-π Bayesian Optimization), which elicits beliefs once from a human expert, a large language model (LLM), or both, and deterministically compiles a structured specification into a reusable belief function (π:X→[0,1]), with a rich representation not restricted to a Gaussian family. Building on α-πBO, HapBO applies this belief function through a bounded positive multiplier on any nonnegative acquisition, preserving positive acquisition support and requiring no LLM calls during optimization. We prove that, for the Expected Improvement acquisition function (EI), HapBO preserves the standard simple-regret rate up to a bounded factor, even under adversarial beliefs. We evaluate HapBO on 40 HPOBench tasks, a synthetic suite in 6-15 dimensions and two real-world applications: cortical neurostimulation and constrained solar design. HapBO achieves the lowest aggregate HPOBench regret and strong real-world task performance, with zero in-loop token cost. In real-world applications it improves the optimization outcome even at budgets of 12-20 queries, and at a small fraction of the wall-clock and token costs of other LLM-based methods.
Figures & tables
Figure 1: HapBO (blue components). Left to right: an LLM reads the task card, which may include information stated by a human expert, and the PSL vocabulary and produces a document in the typed Prior Specification Language (PSL); a compiler turns the document into a prior landscape π and a trust level α . The prior is injected in the BO loop at the acquisition function level. Right: shaping re-ranks candidates toward plausible regions but cannot zero a positive base acquisition.
Figure 2: HPOBench. Average rank and normalized simple regret as a grand mean over 40 tasks for five model families. Per-family panels in App. G ; bands are ± SEM across tasks after averaging seeds.
Figure 3: Real-world tasks. A: NeuroMOSAICS cortical exploration performance (higher is better); n=40 technical replicates (muscle outputs), seeds averaged within each map. B: SOLAR_MAXEFF best feasible efficiency; zero denotes that no feasible point has yet been found.
Method
lr
rf
xgb
nn
svm
cortical
SOLAR
fail%
HapBO
0.0022
0.0015
0.0020
0.0052
0.0021
0.785
4881.2
0.0
ColaBO
0.0014
0.0031
0.0022
0.0102
0.0028
0.735
3848.9
0.0
LLAMBO
0.0017
0.0201
0.0021
0.0090
0.0041
0.819 †
4874.9 †
1.9
LGBO
0.0023
0.0088
0.0020
0.0075
0.0060
0.785
4789.6
0.0
BORA
0.0022
0.0074
0.0020
0.0081
0.0042
0.737
4858.9
0.0
vanilla BO
0.0020
0.0243
0.0020
0.0119
0.0059
0.766
4877.9
0.0
Table 1: Endpoint performance per task family. HPOBench simple regret for each of the five model families (lr, rf, xgb, nn, svm; lower is better), cortical exploration performance (higher is better — the same unit as Fig. 3 A), and SOLAR best feasible efficiency (higher); regrets are comparable across HPO families because all are validation error. fail% is the pooled fraction of task–seed runs that did not complete across every family (HPO, cortical, and SOLAR). HapBO uses matched random initialization on HPO, cortical, and SOLAR alike. Bold marks the best value in each performance column.
Figure 4: Synthetic prior-fidelity sweep. Grand-average simple regret under trustworthy ( ρ≈0.9 ), medium ( ρ≈0.5 ), and poor ( ρ≈0.1 ) priors on 4 synthetic functions. Each function is evaluated at its own budget and aligned only by fraction of budget for this summary.
Figure 5: Adversarial HPOBench. Average task-wise rank (left; over the task grid complete for every method) and grand-mean normalized simple regret over 40 tasks (right) when every method receives the same hostile task description through its native interface. Per-family panels in App. 14 ; bands are ± SEM across tasks after averaging ten seeds.
Figure 6: Prior representation ablation. The injection site is fixed at the bounded α−π BO multiplier; only how the belief becomes π varies: full PSL (ours), point-only, Gaussian-only, direct-LLM elicitation, and vanilla BO (no prior). Grand-mean normalized simple regret on A: the synthetic suite at ρ≈0.9 (four analytic functions, aligned by fraction of budget), B: HPOBench with the good prior, and C: HPOBench with the adversarial prior. Direct-LLM is absent from the synthetic suite, because its fixed anchor set degenerates in 7–15D. Ten seeds per task (synthetic prior arms: three documents × ten seeds per function); mean ± SEM across tasks; lower is better. Average ranks and per-function and per-family panels in App. Fig. 10 .
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
PSL Token
Fields
Semantics
Base layer
baseline
level [0,1] (default 0.1 ); tilt per-dim [−1,1]
default plausibility floor plus optional global slope
Marginal shapes (one dimension)
bump
( dim , mode ); ci ; skew [−1,1] ; sign {peak, trough}
single peak or trough; split-Gaussian, σ from the CI
trend
( dim , direction {increase, decrease}); knee
monotone lean, optional saturation at the knee
threshold
( dim , at , active {above, below}); sharpness {soft, sharp}
regime change at a boundary (logistic step)
Appendix
Table 2: Declaration of the PSL vocabulary. PSL v1.0: Required fields in parentheses; all shaping tokens additionally require weight ∈[0,1] and provenance ∈ { user_stated , user_confirmed , literature , llm_domain , llm_default }. CI fields are 90% intervals (half-width =1.645σ ).
Figure 7: PSL components as belief functions. Each panel is a minimal PSL document on a two-dimensional unit square ( x1 horizontal, x2 vertical), compiled by Algorithm 1 into π(x)∈[0,1] . In the categorical panel x2 has five unordered levels; in the anchors panel the anchor values override a tilted baseline at the circled points. The last panel stacks several components in one document. Meta tokens ( smoothness , noise , modality ) and the trust block describe the objective rather than the optimum’s location, so they have no panel.
Figure 8: Compiling one PSL document. The sample PSL document (top) refers to the XGBoost task on kc1 of App. B . It lives on the space of two XGBoost parameters: eta × max_depth . Step 1 Irregular document candidates (dots) become a regular grid. Step 2 Closed forms are applied. Step 3 A signed sum is performed, and clamped at zero. Step 4 Rescaling on R (dots: Halton points; crosses: declared points); u=1.5 is attained at the declared mode. Step 5 Anchors (circles: zj , labelled with vj ) will override the field within radius 0.15d (dashed: support of ψj ). Top right: α from the card to Eq. 1 .
Function
ρ(M,f)
ρ(π,M)
ρ(π,f)
π(xf⋆)
π(xM⋆)
ρ=0.9 , all functions
0.90
0.38
0.41
0.50
0.90
Hartmann-6
0.91
0.41
0.39
0.08
0.72
Styblinski–Tang-7
0.90
0.15
0.12
0.00
0.86
Levy-10
0.90
0.41
0.51
0.91
1.00
Ackley-15
0.90
0.55
0.63
1.00
1.00
ρ=0.5 , all functions
0.50
0.34
0.30
0.62
0.98
Appendix
Table 3: Prior elicitation quality in the synthetic experiment. M : the closed-form model printed on the card; f : the true objective; π : the compiled PSL prior (Qwen3.5-122B; mean over the three elicitation draws). ρ(⋅,⋅) : Pearson correlation over 20,000 uniform points of the search box. π(xf⋆) , π(xM⋆) : value of π at the global optimizer of f and of M .
Method
wall-clock/query (s)
setup tokens (once)
tokens/query
HapBO
0.20
15,001
0
ColaBO
104.30
1,770
0
LLAMBO
28.59
–
79,650
LGBO
12.44
–
566
BORA
4.08
–
266
vanilla BO
0.20
–
–
Appendix
Table 4: Per-query cost on HPOBench (Qwen3.5-122B served locally), averaged across the five families. Setup tokens is the one-off prior elicitation, compiled once per task; tokens/query is the in-loop LLM cost, zero when the model is consulted only offline.
Figure 9: Prior injection site ablation. One compiled prior (π,α) per task; every arm shares the GP surrogate, base EI, and curves are query-wise mean ± SEM over the 40 HPOBench tasks (16 queries) and the four analytic functions aligned by fraction of budget. A faithful HPO prior, B adversarial HPO prior, C synthetic ρ≈0.5 , grand mean over the four analytic functions.
Figure 10: Prior representation ablation. The injection site is fixed at the bounded α−π BO multiplier; only how the belief becomes π varies: full PSL (ours), point-only, Gaussian-only, direct-LLM elicitation, and vanilla BO (no prior). Grand means in Fig. 6 . A: average task-wise rank (1 = best) in the same three conditions (synthetic suite at ρ≈0.9 , aligned by fraction of budget; HPOBench with the good and with the adversarial prior). B–D: per synthetic function ( B ), and per HPOBench model family with the good ( C ) and with the adversarial prior ( D ). Direct-LLM is absent from the synthetic suite, because its fixed anchor set degenerates in 7–15D. Ten seeds per task (synthetic prior arms: three documents × ten seeds per function); mean ± SEM across tasks; lower is better.
Figure 11: Trust weight α of HapBO . HapBO ( α - π ) with the trust weight fixed at α=0.9 , 0.7 , 0.5 , 0.3 , or 0.1 instead of the LLM-compiled α ; 40 HPOBench tasks × 10 seeds, 16 queries, matched random start; mean ± SEM over tasks. Left: good prior. Middle: misleading prior. Right: misleading prior, final regret against α . Colors run from the highest (darkest) to the lowest trust; dashed: vanilla BO ( α=0 ); LLM-compiled α : thick line (left, middle) and outlined marker (right).
Figure 12: Prior-peak candidates alone do not transmit the prior (synthetic panel). Normalized simple regret as in Fig. 4 under trustworthy (A), medium (B), and poor (C) priors. HapBO against vanilla BO given the same prior-maximizer candidates with an unweighted acquisition, and vanilla BO.
Figure 13: Prior source and cortical optimization, complete panel. A–C: compiled belief functions for LLM-only, LLM + human, and human-only knowledge. D–F: three example ground-truth response maps from the NeuroMOSAICS dataset, each normalized independently to expose spatial structure; they are not paired with the priors above. G–I: each prior mode against matched vanilla BO over twelve queries (budget 12); mean ± SEM clustered over six rats after averaging muscles and ten seeds.
Figure 14: HPOBench, per model family. Normalized simple regret for the five families (top), average task-wise rank and grand mean over 40 tasks (bottom), using matched HapBO and every method’s native budget-16 curve. Regret is normalized per task by the full-grid span (optimum minus mean grid score); bands are ± SEM across tasks after averaging ten seeds within each task. Every method completes all 400 task–seed runs. The family panels show the aggregate gain of the main text is not driven by one model class.
Figure 15: Adversarial HPOBench, per model family. Normalized simple regret for the five families (top), average task-wise rank and grand mean over 40 tasks (bottom), using matched HapBO and the same hostile description through each method’s native interface. The rank panel uses the task grid complete for every method; regret is normalized by vanilla’s mean first-query regret; bands are ± SEM across tasks after averaging ten seeds. BORA completes 400/400, LLAMBO 400/400, and ColaBO 400/400 task–seed endpoints.
Figure 16: Bounded shaping across BO variants (matched arms; only the stated ingredient varies; details and conditions in the text of this appendix). A–C, acquisition functions: matched HapBO vs. vanilla BO under EI, PI, and UCB; top rows HPOBench (normalized simple regret, lower is better; 40 tasks), bottom rows cortical (normalized performance, higher is better; n=40 technical replicates (muscle outputs); human+LLM prior, budget 12 queries); ten seeds. D–E, batch selection: the same matched EI contrast under sequential ( q=1 ) and batch ( q=3 , Kriging Believer) selection; D reproduces the EI panels of A. F–H, GP kernel: EI-based contrast with a Matérn-5/2 (the kernel of A–E), RBF, or Matérn-1/2 kernel.
Figure 17: Downstream performance and elicitor quality. Synthetic panel, accurate cards ( ρ≈0.9 ); five elicitors and two elicitation routes (solid: one API call; hatched: agentic session). A Two-dimensional slice of the card model (Styblinski–Tang-7, x1 vs. x6 , through its optimum, white cross) and the compiled π of each elicitor’s single-call document on the same slice. B Precision (rank correlation between π and the card model) and C agreement (overlap of the top 10% of points); bars: mean over the four functions, dots: functions. D Tokens and E list-price cost per elicitation (the local model has no marginal fee). F Number of distinct PSL token types used and G share of each token type in the single-call documents. H Normalized simple regret of HapBO on the true function vs. fraction of budget, per elicitor (average of single-call and agentic curves, ± SEM); vanilla BO dashed.
Elicitor
final regret
Qwen3.5
0.048 ± 0.015
Haiku 5.5
0.048 ± 0.016
Sonnet 5.5
0.044 ± 0.012
Opus 5.5
0.045 ± 0.012
Fable 5.1
0.044 ± 0.014
vanilla BO
0.228 ± 0.160
Appendix
Table 5: Elicitor comparison: downstream regret Final normalized simple regret of HapBO on the true function at the end of the budget (lower is better), mean ± SEM of data in Fig. 17 . Bold marks the best value.
Figure 18: Baseline-implementation sensitivity. LGBO and LLAMBO, in two possible implementations, each shown with the matched HapBO curve. Left, LGBO: the paper’s mean-shift equation (solid, used in the main text) vs. the public release’s confidence operator (dashed). Right, LLAMBO: random initialization, mode 1 (dashed) vs. zero-shot warm-start, mode 2 (solid, main text). Top: standard prior, grid-span-normalized log regret. Bottom: adversarial prior, vanilla-query-1-normalized regret (log). Bands are ± SEM across tasks after averaging ten seeds.
Problem (family)
recovered %
raw % of best
recall evidence
partial recall evidence
HPOBench lr
60 ± 11
98 ± 1
no recall evidence (0/8)
0/8
HPOBench rf
48 ± 9
94 ± 1
no recall evidence (0/8)
0/8
HPOBench xgb
66 ± 9
99 ± 0
no recall evidence (0/8)
0/8
HPOBench nn
68 ± 9
96 ± 2
no recall evidence (0/8)
0/8
HPOBench svm
55 ± 16
89 ± 5
no recall evidence (0/8)
0/8
NeuroMOSAICS cortical
-12 ± 4
37 ± 6
no recall evidence (0/16)
0/16
Appendix
Table 6: Probing native recall of benchmark optima. Recovered and raw % of the blind best-guess optimum per benchmark (local elicitor, greedy decoding), with reasoning-trace recall evidence. HPO values are means over the eight tasks per family ( ± SEM); cortical over the sixteen rat–muscle maps; SOLAR is a single problem scored against the best feasible pool design.
Function
card
recovered %
dist. to true x⋆
dist. to textbook x⋆
recall evidence
Hartmann-6
ρ≈0.9
86
0.47
0.49
follows provided model, no family recognition
ρ≈0.5
7
0.23
0.27
follows provided model, no family recognition
ρ≈0.1
-3
0.31
0.37
follows provided model, no family recognition
named control
100
0.00
0.00
names Hartmann, states x⋆
Styblinski–Tang-7
ρ≈0.9
98
0.03
0.11
follows provided model, no family recognition
ρ≈0.5
41
0.44
0.50
follows provided model, no family recognition
Appendix
Table 7: Probing native recall of benchmark optima, analytic functions. De-named cards supply a perturbed model at three fidelities ρ ; the guess is scored on the true (hidden-shift) objective (recovered % and normalized RMS distance to the true and to the textbook x⋆ ; 0= exact). The named control names the function outright.
Elicitor
lr
rf
xgb
nn
svm
grand mean
recall evidence
partial recall evidence
Qwen3.5-122B
60 ± 11
48 ± 9
66 ± 9
68 ± 9
55 ± 16
60 ± 5
no recall evidence (0/40)
0/40
Claude Opus
75 ± 3
75 ± 4
74 ± 3
80 ± 4
76 ± 4
76 ± 2
no recall evidence (0/120)
2/120 †
Claude Sonnet
69 ± 5
67 ± 5
72 ± 4
80 ± 3
80 ± 5
74 ± 2
no recall evidence (0/120)
1/120 †
Claude Haiku
63 ± 6
50 ± 5
74 ± 3
74 ± 6
67 ± 6
65 ± 2
no recall evidence (0/120)
0/120
Appendix
Table 8: Probing native recall of benchmark optima across models. Recovered % of the blind best-guess per HPOBench family (Claude elicitors at temperature 1.0 , three draws per task; local elicitor greedy). No elicitation on any family recalls an optimum.
Figure 19: Leak-control benchmark. (A) the bespoke convolutional encoder optimized as an un-memorizable objective, with the four searched hyperparameters highlighted. (B) query-wise simple regret (log axis, mean ± SEM over ten seeds) on the 750-configuration grid. The optimum is a property of our network, grid, and training protocol and appears in no public corpus, so no method’s prior can reflect memorization.
Figure 20: CAKE and CALIPER vs. HapBO on HPOBench. Normalized simple regret (lower is better) per model family under a faithful prior (top) and an adversarial prior (middle), with grand means over the 40 tasks (bottom; faithful left, adversarial right). All arms share the GP surrogate and matched random initialization across ten seeds; CAKE is prior-agnostic, so the same curve appears in both conditions.
Figure 21: PriorBand and PriMO vs. HapBO on HPOBench. Normalized simple regret (lower is better) per model family under a faithful prior (top) and an adversarial prior (middle), with grand means over the 40 tasks (bottom; faithful left, adversarial right); ten seeds. PriorBand and PriMO are plotted against full-fidelity-equivalent cost (budget 16), the other arms against queries.
Figure 22: HypBO and LLM-EI vs. HapBO on HPOBench. Normalized simple regret (lower is better) per model family under a faithful prior (top) and an adversarial prior (middle), with grand means over the 40 tasks (bottom; faithful left, adversarial right). All arms run for ten seeds; HypBO and LLM-EI read the same task card as HapBO .
Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian optimisation (BO). Recently, practitioners have started to use large language models (LLMs) to generate and execute BO programs through coding harnesses. In such emerging practices, the posterior belief is shaped not only by Bayesian inference but also by LLM-generated model and data artefacts, offering a flexible route for task context to enter BO as executable code. To study whether and how LLMs can be harnessed to compile diverse contextual signals for BO, we formulate LLM-compiled BO as generalised-context decision making. We propose HarBO, a BO-specialised harness that compiles generalised context into the core artefacts of standard BO through a validated multi-stage workflow. Our theory analyses the regret under imperfect compilation and the effect of adding new context. Across synthetic functions and real-world benchmarks, we find that LLM harnesses can effectively compile context into standard BO, achieving competitive performance with specialised LLM-embedding-based and direct LLM-in-the-loop BO methods. General coding harnesses can be effective in familiar domains such as hyperparameter optimisation, but fall short in unfamiliar, context-rich domains. Together, these results establish LLM harnesses as a promising, but not automatically reliable, route for making rich task context usable in BO.
Zhongwei Yu, Sourabh Roy, Bin Cao +3
The Hong Kong University of Science and Technology (Guangzhou) · University College London (UCL) · Institute of Automation, Chinese Academy of Sciences
Bayesian Optimization (BO) is widely used for optimizing expensive black-box functions, yet it typically reduces each expensive experiment to an input and its objective value, discarding much of the information the experiment produces. Such auxiliary information can include training curves in hyperparameter optimization, expert notes and images in scientific experimentation, or known facts about the specific optimization problem. Leveraging the ability of large language models (LLMs) to process diverse and unstructured information, we introduce two methods: AuxBO-Evolve, which uses auxiliary information to evolve beliefs over the location of the optimum, and AuxBO-Vicinity, which uses it to locally guide the acquisition function. Across synthetic tasks, hyperparameter optimization benchmarks, and a nuclear fusion task with unstructured scientist-written logs, our methods consistently improve optimization performance and outperform standard BO and existing LLM-based approaches. We further find that LLMs provide more effective probabilistic guidance when modeling likely maximizer locations rather than objective values pointwise. Overall, our results demonstrate that exploiting rich information beyond standard (x,y) interactions can substantially improve BO performance.
Efe Mert Karagözlü, Rohit Sonker, Tejus Gupta +2
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213
Human-in-the-loop Bayesian optimization (HITL BO) methods utilize human expertise to improve the sample-efficiency of BO. Most HITL BO methods assume that a domain expert can quantify their knowledge, for instance by pinpointing query locations or specifying their prior beliefs about the location of the maximum as a probability distribution. However, since human expertise is often tacit and cannot be explicitly quantified, we consider a setting where domain knowledge of an expert is elicited via pairwise comparisons of designs. We interpret the expert's pairwise judgements as noisy evidence about the values of the observable objective function and develop a principled method for combining the information obtained via direct observations and pairwise queries. Specifically, we derive a cost-aware value-of-information acquisition function that balances direct observations against pairwise queries. The proposed method approaches the convex hull of the trajectories of the individual information sources: when pairwise queries are cheap it substantially improves sample-efficiency over observation-only BO, and when pairwise queries are costly or noisy, it recovers the performance of standard BO by relying on direct observations alone.
Alvar Haltia, Ville Hyvönen, Samuel Kaski
Aalto University · University of Helsinki · Aalto University, University of Manchester