STaSBO: LLM-Guided Strategy Planning and Configuration Selection for Bayesian Optimization
Organizations: Department of Electrical and Computer Engineering, University of California, Santa Barbara, California, United States.
Abstract
Bayesian optimization (BO) enables sample-efficient search for expensive black-box problems, but its performance depends strongly on surrogate-acquisition configurations whose suitability varies across problems and optimization states. We take a significant step toward LLM-driven strategic management of BO with STaSBO, a task-agnostic meta-controller that plans search behavior and jointly selects surrogate-acquisition configurations, while numerical BO retains responsibility for probabilistic modeling and candidate generation. STaSBO comprises two coordinated stages: strategy planning and strategy-guided configuration selection. A structured memory system connects prior plans, configuration selections, and candidate behaviors to optimization states, allowing the controller to revise its trajectory summary and behavioral plans during planning. To navigate the large joint configuration space, the selection stage shortlists configurations using surrogate fit evidence. Prospective BO-action diagnostics then expose the candidate behaviors, supporting final joint selection against the behavioral plans. Across 12 synthetic mathematical and 10 scientific and engineering application tasks, STaSBO achieves the best average rank against 11 baselines: 2.17 and 2.60, compared with 4.08 and 4.10 for the strongest baseline in each suite, respectively.
Figures & tables
| Source Function | Source Dim | Oracle config. | Ackley | Levy | Rastrigin | |||
|---|---|---|---|---|---|---|---|---|
| 5D | 15D | 5D | 15D | 5D | 15D | |||
| Ackley | 5D | S1–A5 | 1 | 26 | 22 | 25 | 24 | 27 |
| 15D | S5–A3 | 15 | 1 | 12 | 4 | 14 | 3 | |
| Levy | 5D | S1–A4 | 27 | 10 | 1 | 9 | 11 | 6 |
| 15D | S5–A1 | 2 | 7 | 10 | 1 | 5 | 2 | |
| Rastrigin | 5D | S5–A5 | 4 | 3 | 5 | 12 | 1 | 4 |
| Method | Ackley | Levy | Rastrigin | Rosenbrock | Avg. rank | ||||||||
| 5D | 8D | 15D | 5D | 8D | 15D | 5D | 8D | 15D | 5D | 8D | 15D | ||
| Random | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 11 | 12 | 11 | 11 | 11.75 |
| BO-EI | 4 | 4 | 3 | 8 | 4 | 7 | 9 | 4 | 9 | 8 | 4 | 7 | 5.92 |
| BO-UCB | 1 | 3 | 11 | 10 | 11 | 11 | 11 | 11 | 12 | 10 | 10 | 10 | 9.25 |
| GP-Hedge | 11 | 10 | 7 | 6 | 3 | 10 | 6 | 1 | 5 | 7 | 6 | 9 | 6.75 |
| No-PASt-BO | 9 | 9 | 9 | 5 | 5 | 9 | 1 | 10 | 8 | 4 | 7 | 5 | 6.75 |
| Method | AgNP | Concrete | Crossed Barrel | HPLC | P3HT | OER Plate | Lunar Lander | Avg. rank | |||
| 3496 | 3851 | 3860 | 4098 | ||||||||
| Random | 11 | 12 | 12 | 11 | 11 | 11 | 11 | 11 | 11 | 11 | 11.2 |
| BO-EI | 8 | 1 | 4 | 5 | 4 | 8 | 6 | 2 | 7 | 2 | 4.7 |
| BO-UCB | 9 | 3 | 5 | 4 | 1 | 6 | 1 | 1 | 1 | 10 | 4.1 |
| GP-Hedge | 7 | 7 | 7 | 10 | 9 | 9 | 5 | 4 | 4 | 5 | 6.7 |
| No-PASt-BO | 6 | 4 | 6 | 7 | 5 | 1 | 10 | 5 | 10 | 7 | 6.1 |
| Ablations | Regret |
|---|---|
| STaSBO (full) | |
| Joint Control | |
| Acquisition-only control | |
| Surrogate-only control | |
| Planning and Memory | |
| Planning from observations | |
| Decision element | Iteration 27 | Iteration 28 |
|---|---|---|
| Trajectory summary | 31 observations. Incumbent regret is 12.35. A recent intermediate improvement was followed by a miss, leaving the relative value of intermediate and near-incumbent opportunities unresolved. | 32 observations. Incumbent regret remains 12.35 after two consecutive intermediate misses. These outcomes do not establish intermediate moves as generally inferior, and refinement evidence remains limited. |
| Current focus | Compare intermediate and near-incumbent opportunities after a recent intermediate improvement followed by a miss. | Assess a distinct near-incumbent opportunity against intermediate alternatives after two consecutive intermediate misses. |
| Behavioral plans | A: Test whether intermediate movement remains useful after a recent improvement followed by a miss. Intermediate opportunity. B: Preserve a distinct local option despite sparse and mixed local evidence. Near-incumbent refinement. C: Retain a distant option given limited broad-move evidence. Substantially displaced opportunity. | A: Retain intermediate opportunities because earlier moves improved. Intermediate opportunity. B: Test a distinct near-incumbent point after two consecutive intermediate misses. Near-incumbent refinement. C: Retain a distant option given limited broad-move evidence. Substantially displaced opportunity. |
| Selected action | Matérn- isotropic + UCB, . | Matérn- isotropic + EI, . |
| Associated plan | Plan A. | Plan B. |
| Candidate Evidence | Intermediate move. . | Local refinement. . |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Profile | Kernel / acquisition | Specification |
| Surrogate profiles | ||
| S1 | Matérn- | ARD |
| S2 | Matérn- | Isotropic |
| S3 | Matérn- | ARD |
| S4 | RBF | ARD |
| S5 | RBF | Isotropic |
| Ackley | Levy | Rastrigin | ||||||
|---|---|---|---|---|---|---|---|---|
| Surrogate | Acquisition | 5D | 15D | 5D | 15D | 5D | 15D | Mean rank |
| S1 Matérn- ARD | A1 EI (0.00) | 7.312 (17) | 19.299 (13) | 4.852 (14) | 56.759 (16) | 31.130 (15) | 178.093 (18) | 15.50 |
| A2 EI (0.05) | 6.231 (12) | 20.163 (22) | 4.968 (16) | 69.683 (19) | 30.215 (12) | 176.210 (16) | 16.17 | |
| A3 EI (0.20) | 6.770 (13) | 20.565 (28) | 7.675 (25) | 84.520 (24) | 41.888 (29) | 202.348 (22) | 23.50 | |
| A4 UCB (0.5) | 12.466 (27) | 19.134 (10) | 2.296 (1) | 51.313 (9) | 30.209 (11) | 162.898 (6) | 10.67 | |
| A5 UCB (3.0) | 3.221 (1) | 20.421 (26) | 5.881 (22) | 85.357 (25) | 34.348 (24) | 229.786 (27) | 20.83 | |
| Switch after | ||
|---|---|---|
| Task | 15 iterations | 30 iterations |
| Ackley-5D | 3 | 8 |
| Ackley-15D | 13 | 12 |
| Levy-5D | 11 | 23 |
| Levy-15D | 15 | 20 |
| Rastrigin-5D | 13 | 8 |
| Decision | Stage-specific inputs | Output |
|---|---|---|
| Strategy planning | Previous strategy memory and aggregate feedback from the preceding candidate-generation step. | Updated strategy memory , comprising the trajectory summary and alternative behavioral plans. |
| Configuration shortlisting | Current strategy , fixed surrogate specifications, fit evidence , acquisition families and parameter ranges, and previous-generation feedback. | One surrogate and associated parameterized acquisitions per plan. Validation, union, and deduplication yield , retaining originating-plan associations. |
| Joint configuration selection | Current strategy , static definitions of the retained surrogates, and current candidate diagnostics with their generating configurations. | Selected configuration , an optional associated plan , and pre-evaluation feedback containing the rationale and advisory alignment assessment. |
| Profile | Kernel | Length-scale parameterization |
|---|---|---|
| Matérn- | ARD | |
| Matérn- | Isotropic | |
| Matérn- | ARD | |
| RBF | ARD | |
| RBF | Isotropic | |
| Matérn- | Isotropic |
| Acquisition family | Parameter | Allowed range |
|---|---|---|
| Expected improvement | ||
| Upper confidence bound |
| Information | Archived content | LLM-visible projection |
|---|---|---|
| Decision identity | Controller iteration and configuration, candidate, and originating-plan identifiers. | Controller iteration and an observation index. |
| Configuration | Selected surrogate profile, acquisition family, and search parameter values. | Included in all displayed records. |
| Behavioral intent | The associated historical plan, including its action, purpose, candidate preference, and revision condition, or an absent association. | Latest record only; plan content is retained without its local identifier. |
| Candidate and observation | The cached candidate and score. | Included in all displayed records. |
| Candidate geometry | Distances to the incumbent and nearest observation, together with the behavior label. | Included in all displayed records. |
| Surrounding context | Complete context snapshots before and after evaluation, including the incumbent. | Previous incumbent score in all displayed records. |
| Field | History used | Definition |
|---|---|---|
| Incumbent | All observations. | Best observed coordinates and score, with an observation index. Ties are resolved by taking the first maximizer. |
| Observation count | All observations. | Total number of evaluations, including initialization. |
| Stagnation | All observations in order. | Number of observations since the last strict incumbent improvement. |
| Recent spatial window | . | Number of observations used for the recent statistics. |
| Recent dispersion | Recent spatial window. | Mean normalized distance over all pairs within the window; zero if fewer than two observations are available. |
| Recent novelty | Recent spatial window and nearest earlier observations. | Mean distance from each eligible recent observation to its nearest observation in the full history. The first observation is excluded; the statistic is zero if no comparisons exist. |
| Field | Definition and units |
|---|---|
| Fit availability | Whether a usable fitted surrogate is available. Numerical descriptors are omitted for unavailable fits. |
| Learned length scales | One value per dimension for ARD profiles or one shared value for isotropic profiles. Values are expressed in the GP’s training-data-normalized input coordinates. |
| Likelihood-noise variance | Learned residual-noise variance of the Gaussian likelihood, expressed in squared standardized-output units. |
| Kernel output scale | Learned multiplicative scale of the base-kernel covariance, expressed in squared standardized-output units. |
| Noise-to-kernel variance ratio | Likelihood-noise variance divided by kernel output scale. This derived quantity is dimensionless and is included when the ratio is finite and defined. |
| Field | Content |
|---|---|
| Candidate identifier | An iteration-local identifier for the exact-point group. |
| Candidate coordinates | The validated point in the task’s original parameter units. |
| Incumbent distance | Normalized distance to the current best observed point, . |
| Nearest-observation distance | Minimum normalized distance to any previously evaluated point, . |
| Behavior label | Local refinement, intermediate move, or broad move, determined by incumbent distance. |
| Generating configurations | Configuration identifiers, surrogate profiles, acquisition families and parameter values, and originating-plan identifiers. |
| Function | Objective | Search domain | Shift |
|---|---|---|---|
| Ackley | |||
| Levy | |||
| Rastrigin | |||
| Rosenbrock |
| Task | Domain | Objective | ||
|---|---|---|---|---|
| Concrete | Civil engineering | 7 | 427 | Compressive strength |
| Crossed Barrel | Mechanical design | 4 | 600 | Toughness |
| AgNP | Nanomaterials synthesis | 5 | 164 | Archived spectrum score |
| P3HT | Functional materials | 5 | 178 | Electrical conductivity |
| HPLC | Analytical chemistry | 6 | 1,007 | Recorded peak area |
| OER 3496 | Electrocatalysis | 6 | 2,121 | Overpotential |
| Task | Variables, bounds, and constraints |
|---|---|
| Concrete | Cement , blast-furnace slag , fly ash , water , superplasticizer , coarse aggregate , and fine aggregate , in . No fixed-total constraint is imposed. |
| Crossed Barrel | Column count , twist angle degrees, outer radius mm, and wall thickness mm. Column count is continuously relaxed. |
| AgNP | Feed ratios for silver nitrate , polyvinyl alcohol , trisodium citrate , and silver seeds , on the recorded percentage scale; total flow . No additional coupled flow constraint is imposed. |
| P3HT | P3HT , long single-walled nanotubes , short single-walled nanotubes , multi-walled nanotubes , and double-walled nanotubes , on the recorded percentage scale. No fixed-sum constraint is imposed. |
| HPLC | Sample-loop volume mL, additional volume mL, tubing volume mL, sample flow mL/min, push speed Hz, and wait time s. |
| OER 3496 | Relative loadings of Ni, Fe, Co, Mn, Ce, and La. |
| Method | Configuration and settings |
|---|---|
| Random | Latin hypercube sampling for boxes; sampling for OER simplices. |
| BO-EI | Matérn- ARD GP; EI with . |
| BO-UCB | Matérn- ARD GP; UCB with . |
| GP-Hedge | EI/PI/UCB portfolio; softmax parameter . |
| No-PASt-BO | EI/PI/UCB portfolio; memory coefficient and . |
| SETUP-BO | EI/PI/UCB portfolio; initial memory and softmax-parameter distributions and . |
| Parameter | Setting |
|---|---|
| Surrogate profiles | 7 |
| Behavioral plans | 3 |
| Acquisition specifications per plan | 3 |
| EI improvement-margin fraction | |
| UCB coefficient | |
| Recent-statistics window | 5 |
| Variant | Modification relative to full STaSBO |
|---|---|
| Joint Control | |
| Acquisition-only control | Fix the surrogate profile to Matérn- ARD while continuing to fit its GP hyperparameters and expose its fit evidence. Each plan nominates three acquisition specifications, including their parameter values. |
| Surrogate-only control | Fix the acquisition to EI with . Each plan nominates three distinct surrogate profiles, all paired with this fixed acquisition. Fit evidence and candidate diagnostics remain available. |
| Planning and Memory | |
| Planning from observations | Generate a new trajectory summary and behavioral plans from the complete observation history and numerical state at each iteration. Exclude previous strategy memory, decision records, and previous-iteration proposal feedback from LLM inputs. |
| w/o trajectory summary | Neither generate nor supply the trajectory summary. Retain behavioral plans, including the previous plans used during planning, together with recent decision records and previous-iteration proposal feedback. |
| Category | EI | UCB |
|---|---|---|
| Exploitation-oriented | ||
| Balanced | ||
| Exploration-oriented |
| Method | Ackley | Levy | Rastrigin | Rosenbrock | Avg. norm. regret | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5D | 8D | 15D | 5D | 8D | 15D | 5D | 8D | 15D | 5D | 8D | 15D | ||
| Random | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.000 |
| BO-EI | 0.40 | 0.74 | 0.93 | 0.53 | 0.31 | 0.56 | 0.74 | 0.69 | 0.79 | 0.22 | 0.18 | 0.38 | 0.539 |
| BO-UCB | 0.18 | 0.72 | 0.99 | 0.64 | 0.50 | 0.85 | 0.82 | 0.85 | 1.02 | 0.51 | 0.44 | 0.92 | 0.702 |
| GP-Hedge | 0.80 | 0.90 | 0.97 | 0.40 | 0.30 | 0.64 | 0.62 | 0.58 | 0.74 | 0.18 | 0.20 | 0.40 | 0.561 |
| No-PASt-BO | 0.78 | 0.89 | 0.97 | 0.34 | 0.36 | 0.59 | 0.46 | 0.79 | 0.78 | 0.14 | 0.22 | 0.34 | 0.555 |
| Method | AgNP | Concrete | Crossed Barrel | HPLC | P3HT | OER Plate | Lunar Lander | Avg. norm. regret | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 3496 | 3851 | 3860 | 4098 | ||||||||
| Random | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.000 |
| BO-EI | 0.66 | 0.77 | 0.41 | 0.75 | 0.52 | 0.56 | 0.59 | 0.57 | 0.30 | 0.60 | 0.572 |
| BO-UCB | 0.67 | 0.78 | 0.47 | 0.72 | 0.36 | 0.50 | 0.33 | 0.49 | 0.18 | 0.93 | 0.544 |
| GP-Hedge | 0.62 | 0.81 | 0.51 | 0.88 | 0.62 | 0.57 | 0.52 | 0.61 | 0.25 | 0.66 | 0.606 |
| No-PASt-BO | 0.62 | 0.79 | 0.50 | 0.80 | 0.54 | 0.33 | 0.69 | 0.62 | 0.36 | 0.68 | 0.594 |
Bayesian Optimization
Bayesian optimization uses probabilistic models and sequential experiments to find strong solutions when evaluating a black-box objective is costly.
See all 276 papers