Black-Box Optimization

Follow this topic Ask about this topic

Black-box optimization studies how to choose inputs that maximize or minimize an objective when its internal workings or gradients are unavailable, often because each evaluation is costly. Common approaches include Gaussian-process Bayesian optimization, evolutionary strategies such as CMA-ES and differential evolution, and bandit or partitioning methods that balance exploration with refinement. Surrogate models, multi-fidelity evaluations, and learned or generative proposals extend these methods to noisy, constrained, discrete, and high-dimensional tasks, including scientific design and system tuning. Benchmarks such as COCO/BBOB and CEC help compare methods, but performance often depends on the problem and evaluation budget, leaving robust transfer and fair comparisons across settings as central challenges.

Latest papers

177 papers25 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Mar 25, 2026

Trust Region Constrained Bayesian Optimization with Penalized Constraint Handling

Constrained optimization in high-dimensional black-box settings is difficult due to expensive evaluations, the lack of gradient information, and complex feasibility regions. In this work, we propose a Bayesian optimization method that combines a penalty formulation, a surrogate model, and a trust region strategy. The constrained problem is converted to an unconstrained form by penalizing constraint violations, which provides a unified modeling framework. A trust region restricts the search to a local region around the current best solution, which improves stability and efficiency in high dimensions. Within this region, we use the Expected Improvement acquisition function to select evaluation points by balancing improvement and uncertainty. The proposed Trust Region method integrates penalty-based constraint handling with local surrogate modeling. This combination enables efficient exploration of feasible regions while maintaining sample efficiency. We compare the proposed method with state-of-the-art methods on synthetic and real-world high-dimensional constrained optimization problems. The results show that the method identifies high-quality feasible solutions with fewer evaluations and maintains stable performance across different settings.

Raju Chowdhury, Tanmay Sen, Biswabrata Pradhan

Statistical Quality Control and Operations Research Unit, Indian Statistical Institute, Kolkata, India.

Bayesian Optimization Black-Box Optimization Surrogate-Assisted Optimization Trust-Region Optimization

Mar 24, 2026

AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization

Optimizing AscendC (Ascend C) operators for Ascend NPUs is difficult for two reasons. First, unlike CUDA, the ecosystem offers few public kernels to learn from. Second, performance depends on a coupled two-part implementation: a host-side tiling program that controls data movement and a kernel program that schedules and pipelines computation. We present AscendOptimizer, an episodic agent that builds missing optimization knowledge from execution itself. For kernel optimization, AscendOptimizer rewinds strong implementations by removing optimizations in a controlled way, then keeps the changes whose removal measurably hurts performance as reusable experience for later rewriting. For host-side optimization, it runs profiling-in-the-loop evolutionary search to find valid, fast tiling and data-movement configurations directly from hardware feedback. This combination lets the agent improve kernel structure and host-side scheduling together. On a benchmark of 101 real AscendC operators, AscendOptimizer achieves a 1.21x geometric-mean speedup over the open-source baseline, and 53.47% of operators run faster than their references. Given a same budget of evaluations per operator, AscendOptimizer consistently outperforms Best-of-N sampling and OpenEvolve in terms of geometric mean speedup, fast_p tail speedup ratios, and overall optimization progress across varying budgets.

Jiehao Wu, Zixiao Huang, Wenhao Li +3

School of Computer Science and Technology, East China Normal University · School of Computer Science and Technology, Tongji University · Shanghai University of International Business and Economics +3

GPU Kernel Optimization Black-Box Optimization

Mar 18, 2026

Constrained Hybrid Metaheuristic: A Universal Framework for Continuous Optimisation

This paper presents the constrained Hybrid Metaheuristic (cHM) algorithm as a general framework for continuous optimisation. Unlike many existing metaheuristics that are tailored to specific function classes or problem domains, cHM is designed to operate across a broad spectrum of objective functions, including those with unknown, heterogeneous, or complex properties such as non-convexity, non-separability, and varying smoothness. We provide a formal description of the algorithm, highlighting its modular structure and two-phase operation, which facilitates dynamic adaptation to the problem's characteristics. A key feature of cHM is its ability to harness synergy between both candidate solutions and component metaheuristic strategies. This property allows the algorithm to apply the most appropriate search behaviour at each stage of the optimisation process, thereby improving convergence and robustness. Our extensive experimental evaluation on 28 benchmark functions demonstrates that cHM consistently matches or outperforms traditional metaheuristics in terms of solution quality and convergence speed. In addition, a practical application of the algorithm is demonstrated for a feature selection problem in the context of data classification. The results underscore its potential as a versatile and effective black-box optimiser suitable for both theoretical research and practical applications.

Piotr A. Kowalski, Szymon Kucharczyk, Jacek Mańdziuk

Faculty of Physics and Applied Computer Science, AGH University of Krakow, al. A. Mickiewicza 30, 30-059 Cracow, Poland. · Systems Research Institute, Polish Academy of Sciences, ul. Newelska 6, 01-447 Warsaw, Poland. · AGH Doctoral School, AGH University of Krakow, al. A. Mickiewicza 30, 30-059 Cracow, Poland. +2

Metaheuristic Optimization Black-Box Optimization Constrained Optimization

Mar 17, 2026

Quadratic Surrogate Attractor for Particle Swarm Optimization

This paper presents a particle swarm optimization algorithm that leverages surrogate modeling to replace the conventional global best solution with the minimum of an n-dimensional quadratic form, providing a better-conditioned dynamic attractor for the swarm. This refined convergence target, informed by the local landscape, enhances global convergence behavior and increases robustness against premature convergence and noise, while incurring only minimal computational overhead. The surrogate-augmented approach is evaluated against the standard algorithm through a numerical study on a set of benchmark optimization functions that exhibit diverse landscapes. To ensure statistical significance, 400 independent runs are conducted for each function and algorithm, and the results are analyzed based on their statistical characteristics and corresponding distributions. The quadratic surrogate attractor consistently outperforms the conventional algorithm across all tested functions. The improvement is particularly pronounced for quasi-convex functions, where the surrogate model can exploit the underlying convex-like structure of the landscape.

Maurizio Clemente, Marcello Canova

Center for Automotive Research, The Ohio State University, Columbus, OH 43212 USA

Particle Swarm Optimization Surrogate-Assisted Optimization Black-Box Optimization Surrogate Modeling

Mar 17, 2026

DISCOVER: A Solver for Distributional Counterfactual Explanations

Counterfactual explanations (CE) explain model decisions by identifying input modifications that lead to different predictions. Most existing methods operate at the instance level. Distributional Counterfactual Explanations (DCE) extend this setting by optimizing an optimal transport objective that balances proximity to a factual input distribution and alignment to a target output distribution, with statistical certification via chance constrained bounds. However, DCE relies on gradient based optimization, while many real-world tabular pipelines are dominated by non-differentiable models. We propose DISCOVER, a model-agnostic solver for distributional counterfactual explanations. DISCOVER preserves the original DCE objective and certification while replacing gradient descent with a budgeted propose-and-select search paradigm. It exploits a sample-wise decomposition of the transport objective to compute per-row impact scores and enforce a top-k intervention budget, focusing edits on the most influential samples. To guide candidate generation without predictor gradients, DISCOVER introduces an OT-guided cone sampling primitive driven by input-side transport geometry. Experiments on multiple tabular datasets demonstrate strong joint alignment of input and output distributions, extending distributional counterfactual reasoning to modern black box learning pipelines. A code repository is available at: https://github.com/VALHALLA9/Discover.

Yikai Gu, Lele Cao, Bo Zhao +2

Technical University of Denmark · Scholar7 · Aalto University +1

Counterfactual Explanations Black-Box Optimization

Mar 15, 2026

Trust-Region Noise Search for Black-Box Alignment of Diffusion and Flow Models

Optimizing the noise samples of diffusion and flow models is an increasingly popular approach to align these models to target rewards at inference time. However, we observe that these approaches are usually restricted to differentiable or cheap reward models, the formulation of the underlying pretrained generative model, or are memory/compute inefficient. We instead propose a simple trust-region based search algorithm (TRS) which treats the pre-trained generative and reward models as a black-box and only optimizes the source noise. Our approach achieves a good balance between global exploration and local exploitation, and is versatile and easily adaptable to various generative settings and reward models with minimal hyperparameter tuning. We evaluate TRS across text-to-image, molecule and protein design tasks, and obtain significantly improved output samples over the base generative models and other inference-time alignment approaches which optimize the source noise sample, or even the entire reverse-time sampling noise trajectories in the case of diffusion models. Our source code is publicly available.

Niklas Schweiger, Daniel Cremers, Karnik Ram

Technical University of Munich, Germany · Munich Center for Machine Learning, Germany

Diffusion Model Alignment Black-Box Optimization Diffusion Model Sampling Molecule Design Protein Design T2I Diffusion Models

Mar 13, 2026

Batched Kernelized Bandits: Refinements and Extensions

In this paper, we consider the problem of black-box optimization with noisy feedback revealed in batches, where the unknown function to optimize has a bounded norm in some Reproducing Kernel Hilbert Space (RKHS). We refer to this as the Batched Kernelized Bandits problem, and refine and extend existing results on regret bounds. For algorithmic upper bounds, (Li and Scarlett, 2022) shows that

B=O(\log\log T)

batches suffice to attain near-optimal regret, where

T

is the time horizon and

B

is the number of batches. We further refine this by (i) finding the optimal number of batches including constant factors (to within

1+o(1)

), and (ii) removing a factor of

B

in the regret bound. For algorithm-independent lower bounds, noticing that existing results only apply when the batch sizes are fixed in advance, we present novel lower bounds when the batch sizes are chosen adaptively, and show that adaptive batches have essentially same minimax regret scaling as fixed batches. Furthermore, we consider a robust setting where the goal is to choose points for which the function value remains high even after an adversarial perturbation. We present the robust-BPE algorithm, and show that a suitably-defined cumulative regret notion incurs the same bound as the non-robust setting, and derive a simple regret bound significantly below that of previous work.

Chenkai Ma, Keqin Chen, Jonathan Scarlett

National University of Singapore

Black-Box Optimization Multi-Armed Bandits

Mar 11, 2026

Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI

In this paper, we present a novel black-box online controller that uses only end-to-end measurements over short segments, without internal instrumentation, and hill climbing to maximize goodput, defined as the throughput of requests that satisfy the service-level objective. We provide empirical evidence that this design is well-founded. Using this advance in LLM serving as a concrete example, we then discuss the importance of integrating system performance and sustainability metrics into Factsheets for organizations adopting AI systems.

Yonas Atinafu, Henry Lin, Robin Cohen

David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

LLM Inference Serving LLM Serving Black-Box Optimization

Mar 4, 2026

Empirical Evaluation of No Free Lunch Violations in Permutation-Based Optimization

The No Free Lunch (NFL) theorem guarantees equal average performance only under uniform sampling of a function space closed under permutation (c.u.p.). We ask when this averaging ceases to reflect what benchmarking actually reports. We study an iterative-search setting with sampling without replacement, where algorithms differ only in evaluation order. Binary objectives allow exhaustive evaluation in the fully enumerable case, and efficiency is defined by the first time the global minimum is reached. We then construct two additional benchmarks by algebraically recombining the same baseline functions through sums and differences. Function-algorithm relations are examined via correlation structure, hierarchical clustering, delta heatmaps, and PCA. A one-way ANOVA with Tukey contrasts confirms that algebraic reformulations induce statistically meaningful shifts in performance patterns. The uniformly sampled baseline remains consistent with the global NFL symmetry. In contrast, the algebraically modified benchmarks yield stable re-rankings and coherent clusters of functions and sampling policies. Composite objectives can also exhibit non-additive search effort despite being built from simpler components. Monte Carlo experiments indicate that order effects persist in larger spaces and depend on function class. Taken together, the results show how objective reformulation and benchmark design can generate structured local departures from NFL intuition. They motivate algorithm choice that is aware of both the problem class and the objective representation. This message applies to evolutionary computation as well as to statistical procedures based on relabeling, resampling, and permutation tests.

Grzegorz Sroka

Department of Nonlinear Analysis, Rzeszów University of Technology, Powstańców Warszawy 12, 35-959 Rzeszów, Poland

Black-Box Optimization Benchmark Design Evolutionary Optimization

Mar 3, 2026

LAGO: A Local-Global Optimization Framework Combining Trust Region Methods and Bayesian Optimization

We introduce LAGO, a LocAl-Global Optimization framework coupling Bayesian Optimization (BO) and gradient-based trust region local refinement through an adaptive competition mechanism for smooth expensive-to-evaluate objective functions with available gradients. At each iteration, global and local optimization strategies independently propose candidate points, and the next evaluation is selected based on predicted improvement. LAGO separates global exploration from local refinement at the proposal level: the BO acquisition function is optimized outside the active trust region, while local candidates are proposed within the trust region. Points in the vicinity of the accepted local step are incorporated in the global GP dataset only when satisfying a lengthscale-based minimum-distance criterion, hence reducing the risk of numerical instability during local exploitation. LAGO enhances BO with efficient local refinement when reaching promising regions, and reverts to exploratory behavior when local steps are not competitive.

Eliott Van Dieren, Tommaso Vanzan, Fabio Nobile

Institute of Mathematics, EPFL, Lausanne, Switzerland · Dipartimento di Scienze Matematiche, Politecnico di Torino, Turin, Italy

Black-Box Optimization Bayesian Optimization Trust-Region Optimization

Mar 3, 2026

From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors

Large Language Models (LLMs) have already been widely adopted for automated algorithm design, demonstrating strong abilities in generating and evolving algorithms across various fields. Existing work has largely focused on examining their effectiveness in solving specific problems, with search strategies primarily guided by adaptive prompt designs. In this paper, through investigating the token-wise attribution of the prompts to LLM-generated algorithmic codes, we show that providing high-quality algorithmic code examples can substantially improve the performance of the LLM-driven optimization. Building upon this insight, we propose leveraging prior benchmark algorithms to guide LLM-driven optimization and demonstrate superior performance on two black-box optimization benchmarks: the pseudo-Boolean optimization suite (pbo) and the black-box optimization suite (bbob). Our findings highlight the value of integrating benchmarking studies to enhance both efficiency and robustness of the LLM-driven black-box optimization methods.

Qi Huang, Furong Ye, Ananta Shahane +2

Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2333 CC Leiden, The Netherlands

Large Language Model-Guided Optimization Automated Algorithm Discovery Automated Heuristic Design Black-Box Optimization

Mar 1, 2026

Active Flow Matching

Discrete diffusion and flow matching models capture complex, non-additive and non-autoregressive structure in high-dimensional objective landscapes through parallel, iterative refinement. However, their implicit generative nature precludes direct integration with principled variational frameworks for online black-box optimisation, such as variational search distributions (VSD) and conditioning by adaptive sampling (CbAS). We introduce Active Flow Matching (AFM), which reformulates variational objectives to operate on conditional endpoint distributions along the flow, enabling gradient-based steering of flow models toward high-fitness regions while preserving the rigour of VSD and CbAS. We derive forward and reverse Kullback-Leibler (KL) variants using self-normalised importance sampling. Across a suite of online protein and small molecule design tasks, forward-KL AFM consistently performs competitively compared to state-of-the-art baselines, demonstrating effective exploration-exploitation under tight experimental budgets.

Yashvir S. Grewal, Daniel M. Steinberg, Thang D. Bui +2

School of Computing, College of Systems and Society, Australian National University, Australia · Data61, CSIRO, Australia

Molecule Design Flow-Based Generative Modeling Black-Box Optimization Variational Inference

Feb 23, 2026

Goal-Oriented Influence-Maximizing Data Acquisition for Learning and Optimization

Active data acquisition is central to many learning and optimization tasks in deep neural networks, yet remains challenging because most approaches rely on predictive uncertainty estimates that are difficult to obtain reliably. To this end, we propose Goal-Oriented Influence- Maximizing Data Acquisition (GOIMDA), an active acquisition algorithm that avoids explicit posterior inference while remaining uncertainty-aware through inverse curvature. GOIMDA selects inputs by maximizing their expected influence on a user-specified goal functional, such as test loss, predictive entropy, or the value of an optimizer-recommended design. Leveraging first-order influence functions, we derive a tractable acquisition rule that combines the goal gradient, training-loss curvature, and candidate sensitivity to model parameters. We show theoretically that, for generalized linear models, GOIMDA approximates predictive-entropy minimization up to a correction term accounting for goal alignment and prediction bias, thereby, yielding uncertainty-aware behavior without maintaining a Bayesian posterior. Empirically, across learning tasks (including image and text classification) and optimization tasks (including noisy global optimization benchmarks and neural-network hyperparameter tuning), GOIMDA consistently reaches target performance with substantially fewer labeled samples or function evaluations than uncertainty-based active learning and Gaussian-process Bayesian optimization baselines.

Weichi Yao, Bianca Dumitrascu, Bryan R. Goldsmith +1

University of Michigan · Columbia University

Active Information Gathering Active Learning Black-Box Optimization

Feb 18, 2026

Factorization Machine with Quadratic-Optimization Annealing for RNA Inverse Folding and Evaluation of Binary-Integer Encoding and Nucleotide Assignment

The RNA inverse folding problem aims to identify nucleotide sequences that preferentially adopt a given target secondary structure. While various heuristic and machine learning-based approaches have been proposed, many require a large number of sequence evaluations, which limits their applicability when experimental validation is costly. We propose a method to solve the problem using a factorization machine with quadratic-optimization annealing (FMQA). FMQA is a discrete black-box optimization method reported to obtain high-quality solutions with a limited number of evaluations. Applying FMQA to the problem requires converting nucleotides into binary variables. However, the influence of integer-to-nucleotide assignments and binary-integer encoding on the performance of FMQA has not been thoroughly investigated, even though such choices determine the structure of the surrogate model and the search landscape, and thus can directly affect solution quality. Therefore, this study aims both to establish a novel FMQA framework for RNA inverse folding and to analyze the effects of these assignments and encoding methods. We evaluated all 24 possible assignments of the four nucleotides to the ordered integers (0-3), in combination with four binary-integer encoding methods. Our results demonstrated that one-hot and domain-wall encodings outperform binary and unary encodings in terms of the normalized ensemble defect value. In domain-wall encoding, nucleotides assigned to the boundary integers (0 and 3) appeared with higher frequency. In the RNA inverse folding problem, assigning guanine and cytosine to these boundary integers promoted their enrichment in stem regions, which led to more thermodynamically stable secondary structures than those obtained with one-hot encoding.

Shuta Kikuchi, Shu Tanaka

Graduate School of Science and Technology, Keio University, Yokohama, Kanagawa 223-8522, Japan · Keio University Sustainable Quantum Artificial Intelligence Center (KSQAIC), Keio University, Minato-ku, Tokyo 108-8345, Japan · Department of Applied Physics and Physico-Informatics, Keio University, Yokohama, Kanagawa 223-8522, Japan +1

Inverse Design Black-Box Optimization

Feb 16, 2026

Extending Multi-Source Bayesian Optimization With Causality Principles

Multi-Source Bayesian Optimization (MSBO) serves as a variant of the traditional Bayesian Optimization (BO) framework applicable to situations involving optimization of an objective black-box function over multiple information sources such as simulations, surrogate models, or real-world experiments. However, traditional MSBO assumes the input variables of the objective function to be independent and identically distributed, limiting its effectiveness in scenarios where causal information is available and interventions can be performed, such as clinical trials or policy-making. In the single-source domain, Causal Bayesian Optimization (CBO) extends standard BO with the principles of causality, enabling better modeling of variable dependencies. This leads to more accurate optimization, improved decision-making, and more efficient use of low-cost information sources. In this article, we propose a principled integration of the MSBO and CBO methodologies in the multi-source domain, leveraging the strengths of both to enhance optimization efficiency and reduce computational complexity in higher-dimensional problems. We present the theoretical foundations of both Causal and Multi-Source Bayesian Optimization, and demonstrate how their synergy informs our Multi-Source Causal Bayesian Optimization (MSCBO) algorithm. We compare the performance of MSCBO against its foundational counterparts for both synthetic and real-world datasets with varying levels of noise, highlighting the robustness and applicability of MSCBO. Based on our findings, we conclude that integrating MSBO with the causality principles of CBO facilitates dimensionality reduction and lowers operational costs, ultimately improving convergence speed, performance, and scalability.

Luuk Jacobs, Mohammad Ali Javidian

Radboud University Nijmegen, The Netherlands · Appalachian State University Boone, United States

Bayesian Optimization Black-Box Optimization

Feb 6, 2026

BONSAI: Bayesian Optimization with Natural Simplicity and Interpretability

Bayesian optimization (BO) is a popular technique for sample-efficient optimization of black-box functions. In many applications, the parameters being tuned come with a carefully engineered default configuration, and practitioners only want to deviate from this default when necessary. Standard BO, however, does not aim to minimize deviation from the default and, in practice, often pushes weakly relevant parameters to the boundary of the search space. This makes it difficult to distinguish between important and spurious changes and increases the burden of vetting recommendations when the optimization objective omits relevant operational considerations. We introduce BONSAI, a default-aware BO policy that prunes low-impact deviations from a default configuration while explicitly controlling the loss in acquisition value. BONSAI is compatible with a variety of acquisition functions, including expected improvement and upper confidence bound (GP-UCB). We theoretically bound the regret incurred by BONSAI, showing that, under appropriate conditions, it retains the no-regret property of vanilla GP-UCB and removes irrelevant changes. Across many real-world applications, we empirically find that BONSAI substantially reduces the number of non-default parameters in recommended configurations while maintaining competitive optimization performance with little effect on wall time. Its candidate-generation cost averages only

1.5\times

that of standard BO, compared with

7

-

34\times

for prior sparse-BO methods.

Samuel Daulton, David Eriksson, Maximilian Balandat +1

Meta

Bayesian Optimization Black-Box Optimization

Jan 31, 2026

Meta-Learning-Assisted Constraint Relaxation for Constrained Black-Box Optimization

Constraint handling is central to constrained black-box optimization (BBO), where objective improvement and feasibility restoration often provide conflicting search signals. Existing

ε

-relaxation methods are simple and effective, but their relaxation schedules are usually fixed or manually designed for a limited range of problems. To address this limitation, this letter proposes MeCO, a meta-learning-assisted optimizer that learns an adaptive

ε

-relaxation policy for constrained BBO. MeCO couples a SHADE optimizer with a Double Deep Q-Network controller. At each optimization step, the controller observes compact population and constraint features and selects a scalar action, which is decoded into a relaxation vector for the candidate comparison rule. The policy is trained across constrained BBO instances and then deployed on held-out problems without problem-specific tuning. Experiments on the CEC2017 constrained benchmark, 16 UAV path-planning tasks and eight real-world engineering problems provide evidence that MeCO transfers across held-out benchmark functions, higher dimensions, and an application-domain setting. Ablation and behavior analyses further clarify the roles of constraint-related state features, action scaling, reward shaping, and meta-training.

Sijie Ma, Zeyuan Ma, Yue-Jiao Gong +1

Hong Kong Polytechnic University · South China Normal University · South China University of Technology

Black-Box Optimization Learning to Optimize Constrained Optimization Derivative-Free Optimization

Jan 27, 2026

Online Regime-aware Calibration for Black-box Social Simulators via Posterior-assisted Evolutionary Dynamic Optimization

Evolutionary dynamic optimization (EDO) commonly assumes that environmental changes can be detected from fitness variations and handled through random re-initialization, historical solutions, or learned transition patterns. Online calibration of black-box simulators introduces a different setting, where the dynamic objective is induced by sequential observations and a changing calibration window, rather than being controlled by explicit variables. Fitness variations therefore cannot be directly attributed to regime changes, while the unknown relationship between successive regimes limits conventional adaptation. We formulate this setting as an observation-driven dynamic optimization problem and propose PosEDO, which augments fitness-based EDO with an observation-conditioned parameter-space signal. PosEDO learns this signal online as a posterior distribution over simulator parameters from parameter-trajectory pairs generated during evolutionary evaluation, using posterior shifts for change detection and posterior samples for population adaptation. The new evaluation records are further utilized for online posterior updating without additional simulator calls. Experiments on nonstationary economic and financial simulators show that PosEDO improves calibration accuracy, optimization performance, and change-detection quality over representative EDO baselines.

Peng Yang, Zhenhua Yang, Boquan Jiang +3

Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation, Southern University of Science and Technology, Shenzhen 518055, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen 518055, China +2

Evolutionary Optimization Black-Box Optimization Model Calibration

Jan 23, 2026

How Sequential Algorithm Portfolios can benefit Black Box Optimization

In typical black-box optimization applications, the available computational budget is often allocated to a single algorithm, typically chosen based on user preference with limited knowledge about the problem at hand or according to some expert knowledge. However, we show that splitting the budget across several algorithms yield significantly better results. This approach benefits from both algorithm complementarity across diverse problems and variance reduction within individual functions, and shows that algorithm portfolios do NOT require parallel evaluation capabilities. To demonstrate the advantage of sequential algorithm portfolios, we apply it to the COCO data archive, using over 200 algorithms evaluated on the BBOB test suite. The proposed sequential portfolios consistently outperform single-algorithm baselines, achieving relative performance gains of over 14%, and offering new insights into restart mechanisms and potential for warm-started execution strategies.

Catalin-Viorel Dinu, Diederick Vermetten, Carola Doerr

Sorbonne Universit´e, CNRS, LIP6 in Paris, France

Algorithm Selection Black-Box Optimization

Jan 13, 2026

Amortized Acquisition Optimization for Bayesian Optimization with Variational Mutual Information

Bayesian optimization (BO) of expensive black-box functions is traditionally addressed with Gaussian processes (GPs), which scale cubically with observations, or Bayesian neural networks (BNNs), which incur costly posterior sampling and inner-loop acquisition optimization. We propose VBO-MI (Variational Bayesian Optimization with Mutual Information), a fully gradient-based BO framework that requires no explicit GP prior or fixed parametric posterior family over objective function and treats it as a strict black box. An actor-critic architecture pairs an action-net with a variational critic that estimates information gain, eliminating the acquisition optimization bottleneck and achieving up to

10^{2}\times

fewer FLOPs than BNN-BO baselines. A lightweight surrogate network further reduces real function queries to one batch per iteration. We establish consistency guarantees and evaluate VBO-MI on synthetic benchmarks (Ackley, Levy, Griewank) and real-world tasks (Rover Trajectory, Lunar Lander, Pest Control), demonstrating competitive or superior performance over the baselines.

Farhad Mirkarimi

Bayesian Optimization Black-Box Optimization

Nov 11, 2025

Hyperellipsoid Density Sampling: Exploitative Sequences to Accelerate High-Dimensional Numerical Optimization

The curse of dimensionality remains a persistent challenge in modern optimization problems. Expanding the search space into higher dimensions exponentiates the difficulty of finding optimal solutions, rendering traditional algorithms inefficient. An efficient sampling strategy is presented to accelerate high-dimensional optimization as an alternative to uniform quasi-Monte Carlo (QMC) methods. This method, referred to as Hyperellipsoid Density Sampling (HDS), generates sequences by defining multiple hyperellipsoids throughout the search space. HDS utilizes three types of unsupervised learning algorithms to bypass high-dimensional geometric calculations, producing a non-uniform sample sequence that exploits statistically promising regions of the parameter space. The ability to influence its distribution towards regions of interest makes HDS versatile for applications beyond global optimization, where models benefit from samples focused in specific regions. HDS was evaluated against Sobol, a highly uniform QMC sampling method, using differential evolution (DE) on the challenging set of 29 CEC2017 benchmark test functions. The results show statistically significant improvements in final solution geometric mean error (p<0.05), with average performance gains ranging from 37% in 10D to 11% in 100D. This paper demonstrates the efficacy of HDS as a robust alternative to uniform QMC sampling in high-dimensional optimization.

Julian G. Soltes

Regis University, Denver, Colorado, USA

Black-Box Optimization

Jul 2, 2025

Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training

Gradient-based optimization is the workhorse of deep learning, offering efficient and scalable training via backpropagation. However, exposing gradients during training can leak sensitive information about the underlying data, raising privacy and security concerns such as susceptibility to data poisoning attacks. In contrast, black-box optimization methods, which treat the model as an opaque function, relying solely on function evaluations to guide optimization, offer a promising alternative in scenarios where data access is restricted, adversarial risks are high, or overfitting is a concern. This paper introduces BBoxER, an evolutionary black-box method for LLM post-training that induces an information bottleneck via implicit compression of the training data. Leveraging the tractability of information flow, we provide non-vacuous generalization bounds and strong theoretical guarantees for robustness to data poisoning attacks and extraction attacks, while ensuring privacy by design. In experiments with LLMs, we demonstrate empirically that black-box optimization methods-despite the scalability and computational challenges inherent to black-box approaches-are able to learn, showing how a few iterations of BBoxER improve performance, generalize well on a benchmark of reasoning datasets, and are robust to membership inference attacks. This positions BBoxER as an attractive add-on on top of gradient-based optimization, offering suitability for deployment in restricted environments while also providing non-vacuous generalization guarantees.

Ismail Labiad, Mathurin Videau, Matthieu Kowalski +4

Meta FAIR · Université Paris-Saclay · LISN, Inria, CNRS +2

Privacy-Preserving Language Models Evolutionary Optimization Black-Box Optimization Information-Theoretic Generalization Bounds LLM Post-Training Data Poisoning Attacks+1

Jun 9, 2024

Probabilistic Approach to Black-Box Binary Optimization with Budget Constraints: Application to Sensor Placement

This paper presents a fully probabilistic approach for solving optimal experimental design problems under budget constraints. The experimental design is viewed as a random variable and is associated with a parametric conditional distribution that inherently models the budget constraints. The original optimization problem is replaced with an optimization over the expected value of the original objective, which is then optimized over the distribution parameters. The resulting optimal parameter (policy) is used to sample the feasible region of binary space to produce estimates of the optimal solution(s) of the original optimization problem. In this work we extend the family of conditional Bernoulli models to model the random variable conditioned by the total number of nonzero entries, that is, the budget constraint. This approach (a) is generally applicable to binary optimization problems with nonstochastic black-box objective functions and budget constraints; (b) employs conditional probabilities to model and sample only the feasible region and thus considerably reduces the computational cost compared with employing soft constraints; and (c) does not employ soft constraints and thus does not require tuning of a regularization parameter, for example to promote sparsity, which is generally challenging. The proposed approach is verified numerically using an optimal sensor placement experiment based on an advection-diffusion forward model in a parameter identification setup.

Ahmed Attia

Mathematics and Computer Science Division, Argonne National Laboratory, Lemont, IL, USA

Black-Box Optimization Combinatorial Optimization Bayesian Experimental Design

Jun 6, 2024

Simulating, Fast and Slow: Learning Policies for Black-Box Optimization

Simulators are vital in science and engineering, as they faithfully model the influence of design parameters on real-world observations. Optimizing design parameters to minimize a desired objective function is notoriously challenging since simulators are often non-differentiable black boxes. Moreover, each simulation incurs significant compute time, thus doing such optimizations naively may be prohibitively expensive. Furthermore, in experiment design settings, practitioners are typically required to solve the same optimization problem for various input distributions before proceeding to real-world experiments. Starting these related optimizations from scratch each time can be inefficient if the forward simulation model is expensive to evaluate. To address these challenges, this paper introduces a novel method for solving black-box optimization problems on various input distributions by learning a policy that guides the training of a differentiable surrogate. The surrogate's gradients are then used to optimize the simulation parameters with SGD. After training the policy, the cost for downstream optimization of problems involving black-box simulators is amortized, and we require up to

10\times

fewer expensive simulator calls compared to baselines such as previous local surrogate-based approaches, numerical optimization, and Bayesian optimization.

Tim Bakker, Fabio Valerio Massoli, Thomas Hehn +2

University of Amsterdam § · Qualcomm AI Research‡

Black-Box Optimization Learning to Optimize Surrogate-Assisted Optimization

Jun 5, 2024

BEACON: A Bayesian Optimization Inspired Strategy for Efficient Novelty Search

Novelty search (NS) aims to uncover diverse system behaviors through simulation or experiment without requiring a pre-specified scalar objective. This capability is especially relevant to modern discovery problems in chemistry, materials science, and molecular design, where researchers often seek broad coverage of attainable property space rather than a single optimum and where each evaluation may require a costly computation or experiment. For such expensive black-box settings, we propose BEACON, a sample-efficient NS strategy inspired by Bayesian optimization principles. BEACON models the input-to-outcome mapping using multi-output Gaussian processes and selects new inputs by scoring how far plausible posterior outcomes lie from a denoised archive of previously observed outcomes. This gives a distance-based novelty acquisition that accounts for predictive uncertainty and observational noise while operating directly in continuous outcome space, rather than requiring direct optimization over a discretized partition of behaviors. By leveraging efficient posterior sampling together with scalable high-dimensional Gaussian process models, the proposed framework can be extended to settings with large data sets and high-dimensional design variables. We demonstrate BEACON on established benchmark problems together with real-world case studies in materials and molecular discovery. Across these settings, BEACON consistently discovers broader sets of distinct behaviors than several competing baselines under limited evaluation budgets.

Wei-Ting Tang, Ankush Chakrabarty, Joel A. Paulson

University of Wisconsin-Madison, Department of Chemical and Biological Engineering, Madison, WI, USA. · The Ohio State University, Department of Chemical and Biomolecular Engineering, Columbus, OH, USA. · Insulet Corporation, Acton, MA, USA.

Novelty Search Black-Box Optimization Materials Discovery

May 29, 2024

MEGO: Learning Mixture-of-Experts for General-Purpose Binary Optimization

Discrete optimization is ubiquitous in science and engineering. The vast array of existing discrete optimization problems, coupled with the continuous emergence of new ones, necessitates off-the-shelf optimizers capable of generating high-quality solutions for a large variety of optimization problems. This article introduces MEGO, a novel general-purpose neural optimizer for binary optimization under the black-box setting, intended for broad applicability across diverse binary optimization problem classes with minimal problem-specific customization. MEGO comprises a mixture-of-experts trained without domain knowledge. When presented with a new problem instance to solve, it employs a routing policy to dynamically activate the most relevant expert models to generate high-quality solutions. The strong generalization capability of MEGO is demonstrated on six problem classes from different disciplines, including classic problems and real-world applications. Trained solely on classic problems, MEGO effectively generalizes to unseen and complex real-world problem classes, significantly outperforming widely-used general-purpose optimizers in both solution quality and efficiency. Furthermore, MEGO provides a computational approach for quantifying similarity between optimization problems and classifying them, which is fundamentally different from the conventional analysis-based problem classification.

Shengcai Liu, Zhiyuan Wang, Yew-Soon Ong +2

Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · College of Computing & Data Science, Nanyang Technological University, Singapore 639798 · School of Data Science, Lingnan University, Hong Kong, China

Black-Box Optimization Mixture of Experts Learning to Optimize Neural Combinatorial Optimization

May 19, 2024

Gradient Testing and Estimation by Comparisons

We study gradient testing and gradient estimation of smooth functions using only a comparison oracle that, given two points, indicates which one has the larger function value. For any smooth

f\colon\mathbb R^n\to\mathbb R

,

\mathbf{x}\in\mathbb R^n

, and

\varepsilon>0

, we design a gradient testing algorithm that determines whether the normalized gradient

\nabla f(\mathbf{x})/\|\nabla f(\mathbf{x})\|

is

\varepsilon

-close or

2\varepsilon

-far from a given unit vector

\mathbf{v}

using

O(1)

queries, as well as a gradient estimation algorithm that outputs an

\varepsilon

-estimate of

\nabla f(\mathbf{x})/\|\nabla f(\mathbf{x})\|

using

O(n\log(1/\varepsilon))

queries which we prove to be optimal. Furthermore, we study gradient estimation in the quantum comparison oracle model where queries can be made in superpositions, and develop a quantum algorithm using

O(\log (n/\varepsilon))

queries.

Xiwen Tao, Chenyi Zhang, Helin Wang +2

School of Electronics Engineering and Computer Science, Peking University · Computer Science Department, Stanford University · Center on Frontiers of Computing Studies, Peking University +1

Black-Box Optimization Zeroth-Order Optimization