We introduce LAGO, a LocAl-Global Optimization framework coupling Bayesian Optimization (BO) and gradient-based trust region local refinement through an adaptive competition mechanism for smooth expensive-to-evaluate objective functions with available gradients. At each iteration, global and local optimization strategies independently propose candidate points, and the next evaluation is selected based on predicted improvement. LAGO separates global exploration from local refinement at the proposal level: the BO acquisition function is optimized outside the active trust region, while local candidates are proposed within the trust region. Points in the vicinity of the accepted local step are incorporated in the global GP dataset only when satisfying a lengthscale-based minimum-distance criterion, hence reducing the risk of numerical instability during local exploitation. LAGO enhances BO with efficient local refinement when reaching promising regions, and reverts to exploratory behavior when local steps are not competitive.
Figures & tables
Figure 1: One LAGO iteration. The TR is centered at the current best minimizer. A local candidate is proposed inside the region, while a global candidate is proposed outside. The shaded contours are the GP posterior mean outside of the TR, and the quadratic surrogate inside it.
xG←argmaxx∈X∖B(xc,Δ)EI(x).
Algorithm 1 LAGO: Local-Global Framework Combining BO and SR1-TR.
Figure 2: Convergence comparison. Convergence of optimization routines on a PDE-constrained optimization problem and synthetic benchmarks (Median ± IQR). LAGO performs consistently well across benchmarks, both convex and multimodal ones. It also performs strongly in higher dimensional settings, where other hybrid methods start struggling. Gradient evaluations cost d function evaluations, except for the PDE example where gradients have unit cost (leveraging adjoints).
Figure 3: RQ1. Performance comparison of LAGO and phase-switching methods on synthetic benchmarks (Median ± IQR error). LAGO exhibits more robust performance, particularly on benchmarks where phase-based switching fails to identify the correct basin, leading to high variance.
Figure 4: RQ2. Convergence of LAGO (Median ± IQR) for different values of ν on Lévy (left), and increase in the kernel matrix condition number ratio (Median ± IQR) when ν=0 , shown over a window of ±5 iterations around the iteration where the first trigger of ( 7 ) happens (right). Setting ν too high delays convergence, while setting it to 0 increases the ill-conditioning of the GP.
Figure 5: RQ3. Comparison of LAGO and LAGO-grad, together with their BO and gradBO counterparts (median ± IQR, shaded). LAGO-grad improves over LAGO primarily on ill-conditioned problems such as Rosenbrock but can underperform in multimodal settings such as Lévy.
Figure 6: RQ4. Median optimization error ( ± IQR) on Rastrigin and Griewank (2D), and Styblinski-Tang (10D). LAGO-BO closely matches BO, indicating negligible overhead in highly multimodal regimes, and outperforms BO in 10D on this example. This confirms that LAGO adapts to the problem structure and reverts to BO-like behavior when exploration dominates.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Objective
Definition f:Rd→R
Domain
Global minimum
Branin
(x2−4π25.1x12+π5x1−6)2+10(1−8π1)cos(x1)+10
[−5,10]×[0,15]
0.39789
Styblinski-Tang
21∑i=1d(xi4−16xi2+5xi)
[−5,5]d
−39.16599d
Rosenbrock
(1−x1)2+100(x2−x12)2
[−5,10]2
0.0
Lévy
sin2(πw1)+(w1−1)2[1+10sin2(πw1+1)]
[−10,10]2
0.0
+(w2−1)2[1+sin2(2πw2)],wi=1+4xi−1
Perturbed Branin
Branin(x)+10−6∥x−[−π,12.275]∥2
[−5,10]×[0,15]
0.39789
Appendix
Table 1: Information about the synthetic test functions used in Section 4 .
Figure 7: Contour and surface plots of the 2D synthetic test functions. The red stars show the global minima. Surface plots are used for highly multimodal functions for clarity purposes, and the Rastrigin one is zoomed in a smaller domain [−5,5]2 .
Figure 8: Landscape of the reduced cost function J for the PDE-constrained optimization problem. The objective exhibits multiple local minima and a distinct global minimizer, highlighting the need for global exploration combined with local refinement.
Figure 9: Diffusion coefficient κ , desired state yd , optimal forcing term u , and obtained state variable y . The forcing term is approximately placed at the highest point of the desired state yd , as the resulting state y will minimize the L2 distance with yd .
Figure 10: γ -ablation. Performance of LAGO (Median ± IQR) on the Lévy function for different values of γ in ( 6 ) (default γ=1 ). Larger values of γ , which place greater emphasis on local refinement, lead to lower error on this synthetic benchmark.
Figure 13
Figure 12: Local-global interleaving. LAGO trajectories across benchmarks with local (green) and global (blue) steps. LAGO predominantly selects global steps in moderate and highly multimodal settings, while increasing the use of local refinement in convex or weakly multimodal regimes.
Objective
Local steps (%)
Mostly global behavior: local steps <5%
Styblinski-Tang (10D)
1.0 ± 0.2
Styblinski-Tang (5D)
1.5 ± 0.3
Griewank (2D)
1.9 ± 0.9
Lévy (2D)
2.1 ± 0.5
Rastrigin (2D)
3.9 ± 2.0
Appendix
Table 3: Percentage of local steps versus global steps across benchmark problems (reported as mean ± standard deviation).
Bayesian optimization (BO) is a popular and effective approach for tuning expensive, noisy experiments, but requires the formulation of an explicit objective function. Preferential BO (PBO) removes this requirement by learning from pairwise human feedback, yet existing methods struggle to efficiently optimize beyond low- and medium-dimensional problems due to their global search approaches. We address this limitation by developing a family of local PBO methods that transfer key ideas from high-dimensional BO to the preferential setting. In particular, we introduce local PBO methods which adapt trust-region and derivative-informed local search to pairwise preference feedback, where the latter exploits first- and second-order derivatives of the Laplace-approximated GP posterior. Our benchmark on GP sample paths, standard optimization benchmark functions, and policy-search tasks shows that local PBO methods are especially effective in high-dimensional and complex landscapes with steep optima. Compared with global preference-based baselines, they can substantially reduce cumulative regret, making them particularly useful for real-world preference-based optimization tasks such as policy search.
Johanna Menn, Miriam Kober, Paul Brunzema +2
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1
TREGO, a trust-region Bayesian optimization (BO) method for expensive blackbox problems, has convergence guarantees, but only for exact function values. In high dimension, its performance tends to deteriorate. This paper asks whether a trust-region BO method can be extended to handle noisy evaluations while being competitive in high dimension. We first generalize TREGO into a trust-region framework for noisy BO and extend its convergence analysis to noisy function estimates that are sufficiently accurate with sufficiently high probability. We then propose STREGO, a scalable implementation of this framework. Its global phase selects a batch of candidates by solving a bi-objective problem on the posterior mean of a Gaussian process and the integrated variance reduction. Unlike the predictive variance, this second objective does not favor boundary points merely because they are far from the data. Its local phase minimizes the posterior mean of a second Gaussian process, trained only on points close to the incumbent. STREGO is compared with TREGO, TURBO, BAXUS and NSMA-BO on synthetic and real-world problems of dimension 4 to 124, with and without noise. It gives the best average results in every synthetic setting tested and the best or tied-best solution on four of the six real-world problems. An implementation of STREGO is available at https://github.com/Mayousk/STREGO.
Youssef Diouane, Issmail El Hallaoui, Meisseme Kadri +1
GERAD & Department of Mathematics and Industrial Engineering, Polytechnique Montr´eal. · GERAD & Department of Management, Universit´e du Qu´ebec `a Trois-Rivi`eres.
Recently, Morris et al. introduced TinyLoRA, a strategy for LLM fine-tuning which introduces arbitrarily few parameters, and demonstrated accuracy improvements using less than 10 parameters rather than the ≳10,000 typically demanded by LoRA-based methods. However, that study continued to use gradient-based local search methods of the kind deployed for training multi-billion parameter LLMs in the first place. This article is predicated on the observation that, in the low dimensional setting, global optimization methods become tractable, unlocking Large language Model Bayesian Optimization (LlMBO). LlMBO is sample-efficient, gradient-free, memory-light, parallelizes across devices without synchronization, and, unlike policy-gradient methods, can optimize greedy-decoding accuracy directly. We describe numerical experiments comparing LlMBO against GRPO on fine-tuning tasks for a variety of models on a math benchmark suite which show that LlMBO can be a suitable alternative in this regime, matching or exceeding GRPO's fine tuning gains, while using a fraction of the compute and memory.
Aditya Kulkarni, Davide Sartor, Carlos Soto +1
Department of Physics, University of Massachusetts Amherst · Department of Mathematics and Statistics, University of Massachusetts Amherst