Black-Box Optimization

Ask about this topic

Overview

Black-box optimization studies how to choose inputs that maximize or minimize an objective when its internal workings or gradients are unavailable, often because each evaluation is costly. Common approaches include Gaussian-process Bayesian optimization, evolutionary strategies such as CMA-ES and differential evolution, and bandit or partitioning methods that balance exploration with refinement. Surrogate models, multi-fidelity evaluations, and learned or generative proposals extend these methods to noisy, constrained, discrete, and high-dimensional tasks, including scientific design and system tuning. Benchmarks such as COCO/BBOB and CEC help compare methods, but performance often depends on the problem and evaluation budget, leaving robust transfer and fair comparisons across settings as central challenges.

Latest papers

CardsList

177 papers25 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Jun 1, 2026

Don't Gamble, GAMBLe: An Analytical Framework for AI-Driven Research Systems

AI-Driven Research Systems (ADRS) -- systems coupling LLMs with automated evaluation to discover algorithms, proofs, and designs -- are being optimized and adopted across domains, but the tools to analyze them have not kept pace. ADRS performance depends on component interactions that are poorly understood, expensive to explore, and (as we show) not well captured by standard convergence guarantees. These guarantees rely on structural assumptions that do not hold under the ADRS process we formalize. We introduce GAMBLe, a framework that decomposes ADRS behavior into four parameters (generator GG, assessor A\mathcal{A}, discovery mechanism M\mathcal{M}, budget BB) and one compositional object, the effective landscape Leff=A∘GL_{\text{eff}} = \mathcal{A} \circ G, which reveals that distinct generator-assessor pairs induce structurally different per-problem optimization landscapes. We exercise the framework on 760+ replicated runs (>46,000 iterations) spanning generators from single LLMs to dynamically-adaptive ensembles, mechanisms from greedy selection to co-evolutionary meta-search, and three NP-hard problems whose assessors range from continuous scoring to cliff functions. The experiments reveal no total ordering of generators or mechanisms: frontier models can underperform open-source alternatives and the simplest mechanism sometimes outperforms state-of-the-art meta-search. Results show that even under limited budgets (60 iterations per run), the right component choices can improve performance by 13-67% and search efficiency by 6-39x.

Marquita Ellis, Paul Castro

IBM Research

Automated Algorithm DiscoveryBlack-Box OptimizationLarge Language Model-Guided Optimization
AskPDF
May 29, 2026

Trajectory-Aware Best-Arm Identification for Local Search Allocation in Bayesian Optimization and Beyond

Bayesian optimization (BO) is effective for expensive black-box optimization, but its performance often degrades on multimodal or high-dimensional problems. A promising remedy is to run multiple local search processes, such as trust region--based BO, and allocate evaluations to those with high potential. However, current best values do not always reflect the final performance of each process. We propose a trajectory-aware best-arm identification (BAI) framework that uses optimization history to guide this allocation. The proposed method extrapolates improvement trajectories, estimates final performance, and progressively eliminates low-potential candidates. For trust region--based BO, we theoretically show that the proposed BAI-guided allocation accelerates convergence to the global optimum under mild assumptions. Experiments on synthetic and real-world benchmarks demonstrate that our method improves BO on multimodal problems. We also show that the same mechanism can enhance other local or population-based optimizers, suggesting its potential as a general extension strategy for multimodal black-box optimization.

Nobuo Namura, Sho Takemori

Fujitsu Limited, Kawasaki, Japan.

Multi-Armed BanditsBayesian OptimizationBlack-Box OptimizationBest-Arm Identification
AskPDF
May 28, 2026

Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability

Large language models (LLMs) are increasingly used as generators in iterative neural architecture search (NAS), yet no formal convergence theory exists for this class of algorithms. We model iterative LLM-NAS as a parametric Cross-Entropy (CE) method over executable programs and prove six results: (1) iterative LLM fine-tuning on elite architectures is equivalent to the CE update restricted to the LLM parametric family; (2) expected architecture quality is monotonically non-decreasing across cycles; (3) elite-set probability converges to a fixed point at a geometric rate C_t >= 1-(1-rho_0)^t; (4) delta-based generation achieves a strictly higher valid-generation rate than full-code generation under a first-order Markov token-error model; (5) the MinHash-Jaccard novelty filter prevents mode collapse; (6) proxy reliability admits the closed-form rho_S = (6/pi) arcsin(rho_P(SNR)/2), yielding the practical diagnostic sigma^2_arch >> sigma^2_noise as a necessary condition for trustworthy proxy-based rankings. Testing against a 22-cycle, three-LLM, six-dataset experiment with 3,300 generated architectures confirms two predictions quantitatively, two at direction-of-effect level, and explains the proxy-reliability ceiling effect previously reported empirically but left unexplained.

Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov

Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

Black-Box OptimizationLarge Language Model-Guided OptimizationNeural Architecture Search
AskPDF
May 27, 2026

Information-Geometric Optimization on Spheres

We consider the black-box optimization problem on a sphere. Two information-geometric optimization flows (IGO flows) are designed with rigorous calculation of natural search gradients based on hyperbolic (information) geometry of Poincar' e and Bergman balls. We demonstrate that ensembles of generalized Kuramoto oscillators on spheres compute natural search gradients and realize IGO algorithms on both manifolds. The relationship between natural gradient policies in Bergman balls and quantum decision making is pointed out.

Vladimir Ja\' cimović

Faculty of Natural Sciences and Mathematics University of Montenegro Cetinjski put bb., 81000 Podgorica Montenegro

Information GeometryKuramoto ModelRiemannian OptimizationBlack-Box Optimization
AskPDF
May 27, 2026

Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization

In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring unnecessary computational cost. We study a learning-based extension of a recent empirical online heuristic that adaptively estimates the required number of runs using outlier handling and skewness-based symmetry checks. Using annotated outcomes from 132{,}000 Nevergrad runs on COCO (24 problems in 20 dimensions, 10 instances each, 11 optimizers), we train classifiers on 23 statistical, energy-free, and shape and stability features to predict whether a run-number estimate is reliable, prioritizing detection of incorrect estimates via minority-class recall. We evaluate reliability prediction using a within-configuration learning setup, where models are trained and tested on data sharing the same optimizer. The results show that run-number reliability can be learned in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall, although performance remains limited by the restricted data diversity within fixed configurations.

Sara Gjorgjieva, Eva Tuba, Tome Eftimov

Jožef Stefan Institute, Ljubljana, Slovenia

Stochastic OptimizationBlack-Box Optimization
AskPDF
May 27, 2026

On the Structural (Dis)Agreement of Landscape Representations in Black-Box Optimization

Landscape feature representations play a central role in automated algorithm selection and meta-learning for black-box optimization, yet little is known about how different representations agree (or disagree) in the structures they impose on problem spaces. This paper presents a systematic unsupervised evaluation of four state-of-the-art representations (ELA, DeepELA, TransOptAS, and DoE2Vec) using a diverse set of affine combinations of BBOB functions (MA-BBOB). By applying extensive clustering analyses, coverage-based stability measures, and cross-representation similarity assessments, we show that each representation organizes the same problems in markedly different ways: ELA and TransOptAS form compact geometric structures, DeepELA provides a balanced intermediate view, and DoE2Vec achieves strong semantic alignment but with substantial fragmentation. Our results reveal that no single representation dominates; rather, they capture complementary aspects of the underlying landscapes. These findings highlight the importance of multi-view analyses for understanding representation behavior and offer guidance on selecting or combining representations in downstream meta-learning and algorithm selection tasks. In addition, across two different algorithm families (Differential Evolution and Particle Swarm Optimization), we show that landscape representations face an inherent trade-off in how well they align structural landscape descriptions with observed performance, indicating that no single representation can fully capture algorithm performance.

Sara Gjorgjieva, Eva Tuba, Barbara Koroušić Seljak +2

Jožef Stefan Institute, Ljubljana, Slovenia · Sorbonne Université, CNRS, LIP6, Paris, France

Fitness Landscape AnalysisRepresentation LearningBlack-Box Optimization
AskPDF
May 26, 2026

Probabilistic Smoothing with Ratio-Monotone Transforms for Global Optimization

Probabilistic smoothing is a standard tool for global optimization, but existing methods rely on Gaussian kernels and specific transforms, often resulting in strong hyperparameter sensitivity and limited robustness. We propose a general smoothing framework that combines flexible symmetric unimodal kernels with monotonic ratio-based transformations. Under mild conditions, we show that the smoothed objective preserves the global maximizer and that all stationary points concentrate near the true optimum for sufficiently large amplification, without requiring a decreasing smoothing schedule. We further provide explicit complexity bounds for stochastic gradient ascent and show that a leave-one-out baseline provably reduces variance. Experiments on high-dimensional benchmarks and black-box adversarial attacks demonstrate improved robustness and competitive performance.

Kukyoung Jang, Taehyun Cho, Junrui Zhang +2

Department of Statistics, Seoul, Korea University · Department of Electrical and Computer Engineering, Seoul, Seoul National University · Shandong University at Weihai, Weihai, China.

Black-Box OptimizationGlobal OptimizationStochastic Optimization Convergence
AskPDF
May 26, 2026

MatFormBench: A Benchmarking Evaluation Framework for Target-Driven Materials Formulation

Inverse design of materials has significantly advanced target-driven formulation optimization, yet existing materials machine learning benchmarks remain limited to forward property prediction, failing to systematically evaluate inverse optimization and generation algorithms, a critical gap that hinders the progress of target-driven materials design. To address this limitation, we propose MatFormBench, a novel benchmarking ecosystem tailored to evaluate and guide generative strategies for target-driven formulation. MatFormBench integrates a physics-driven formulation generation scheme to generate synthetic samples that faithfully emulate realistic materials structure-property response relationships, complemented by five escalating difficulty levels to quantify the complexity of these relationships. To rigorously assess algorithm performance, we further propose MatFormScore, a multi-dimensional metric that comprehensively quantifies performance across five critical axes: target success, search efficiency, exploratory capacity, robustness, and stability. We validate MatFormBench by evaluating 39 diverse inverse design algorithms, covering classical surrogate-assisted black-box search, state-of-the-art deep generative models, and increasingly popular Large Language Model (LLM)-based recommendation strategies. Across 1170 standardized algorithm-task evaluations, diffusion-based models demonstrate the strongest overall performance, while Variational Autoencoder (VAE)-based and Genetic Algorithm (GA)-based methods exhibit distinct advantages in specific scenarios. By establishing a unified evaluation standard for target-driven materials formulation, MatFormBench enables reproducible benchmarking, principled algorithm comparison, and diagnostic analysis of inverse design strategies, providing a foundational tool for advancing materials inverse design.

Linhan Wu, Chenxi Wang, Chuhan Yang +2

DeepVerse

Benchmark DesignBlack-Box OptimizationMaterials ScienceSynthetic Benchmark GenerationInverse Design
AskPDF
May 26, 2026

Self-Improvement Imitation with Biologically Guided Search for Protein Design Under Oracle Budgets

Protein sequence optimization under tight oracle budgets requires methods that explore vast combinatorial spaces while making each evaluation informative. Existing reinforcement learning and off-policy generative approaches often degrade under surrogate noise, and position-agnostic mutation proposals risk disrupting functionally critical residues. We introduce SILO, a trajectory-level self-improvement imitation framework for oracle-budgeted protein design. SILO uses a hierarchical edit policy that decomposes each mutation into a position choice followed by a residue choice. In each active-learning round, the policy samples candidate trajectories via incremental stochastic beam search without replacement (SBS), and a UCB-based proxy ensemble, combined with an alanine-scan fitness score (AFS), selects candidates with functionally relevant edits for in silico oracle evaluation. The policy is then updated by next-action cross-entropy imitation on the round's best oracle-labeled trajectories, avoiding value-function estimation. Across eight reproduced protein fitness landscapes and five strong baselines from prior work, SILO achieves the highest maximum and top-100 mean fitness on 8 of 8 landscapes within our evaluations, often exhibiting faster early-stage improvement. In low-data and noisy-proxy stress tests on two landscapes per setting, SILO remains competitive or best when several baselines degrade. Ablations show that SBS with AFS account for much of the gains, with iterative imitation providing additional improvement. Code is available at: https://github.com/grimmlab/SILO.git

Ashima Khanna, Dominik Grimm

Bioinformatics, Technical University of Munich, TUM Campus Straubing, 94315 Straubing, Germany · University of Applied Sciences Weihenstephan-Triesdorf, 94315 Straubing, Germany

Surrogate-Assisted OptimizationBlack-Box OptimizationImitation LearningMolecular GenerationProtein Design
AskPDF
May 22, 2026

An Open-Source Training Dataset for Foundation Models for Black-box Optimization

Most black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to generalize across different optimization domains. Foundation models for black-box optimization that learn optimization principles from a large collection of optimization trajectories offer a promising alternative, with the potential to outperform manually designed methods across diverse problem classes. However, prior work has either relied on non-public datasets or on purely synthetic data, limiting reproducibility and generalization to real-world problems. As a result, progress in this area has been constrained by the lack of large-scale, real-world, publicly available pre-training data. We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories evaluated across 3095 different black-boxes for different optimizers, which represents by far the largest public dataset for this task. Using this dataset, we train a family of foundation models at multiple scales, ranging from 2M to 80M parameters and from 200M to 2B training tokens, and study their scaling behavior with respect to compute. Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate black-box optimization methods, paving the way for future research in this direction.

Aaron Klein, Herilalaina Rakotoarison, Luca Thale-Bombien +1

University of Helsinki · ELLIS Institute Tübingen, Prior Labs

Black-Box OptimizationLearning to OptimizeScaling Laws
AskPDF
May 20, 2026

Beyond Numerical Features: CNN-Driven Algorithm Selection via Contour Plots for Continuous Black-Box Optimization

The present paper introduces a new representation-driven approach to per-instance algorithm selection, applied to black-box optimization, for automatically choosing the most promising solver from a fixed portfolio. Prior work in continuous optimization largely relies on numerical descriptors, including Exploratory Landscape Analysis features and learned embeddings such as Deep-ELA. This work studies a complementary representation: contour-map visualizations of probed landscapes. A CNN regressor takes multiple instance-specific contour views (stacked or encoded per view and aggregated) and predicts per-solver performance, enabling selection by the predicted best value. On the standard BBOB 2009 single-objective protocol, the resulting selectors significantly outperform the single best solver (SBS) and are competitive with feature-based baselines. A subsequent bi-objective evaluation under the DeepELA setting further indicates that the same image-based principle can be competitive when using windowed contour views. Overall, the results suggest that simple vision models can exploit spatial structure in probed landscapes for algorithm selection without handcrafted ELA features.

Yiliang Yuan, Xiang Shi, Mustafa Misir

Mohamed bin Zayed University of Artificial Intelligence, Masdar City, United Arab Emirates · Duke Kunshan University, Suzhou, China

Algorithm SelectionBlack-Box Optimization
AskPDF
May 19, 2026

optimize_anything: A Universal API for Optimizing any Text Parameter

Can a single LLM-based optimization system match specialized tools across fundamentally different domains? We show that when optimization problems are formulated as improving a text artifact evaluated by a scoring function, a single AI-based optimization system-supporting single-task search, multi-task search with cross-problem transfer, and generalization to unseen inputs-achieves state-of-the-art results across six diverse tasks. Our system discovers agent architectures that nearly triple Gemini Flash's ARC-AGI accuracy (32.5% to 89.5%), finds scheduling algorithms that cut cloud costs by 40%, generates CUDA kernels where 87% match or beat PyTorch, and outperforms AlphaEvolve's reported circle packing solution (n=26). Ablations across three domains reveal that actionable side information yields faster convergence and substantially higher final scores than score-only feedback, and that multi-task search outperforms independent optimization given equivalent per-problem budget through cross-task transfer, with benefits scaling with the number of related tasks. Together, we show for the first time that text optimization with LLM-based search is a general-purpose problem-solving paradigm, unifying tasks traditionally requiring domain-specific algorithms under a single framework. We open-source optimize_anything with support for multiple backends as part of the GEPA project at https://github.com/gepa-ai/gepa .

Lakshya A Agrawal, Donghyun Lee, Shangyin Tan +11

UC Berkeley USA · MIT USA

Black-Box OptimizationLarge Language Model-Guided OptimizationLLM-Based Program SynthesisMulti-Task Optimization
AskPDF
May 18, 2026

Adaptive Stochastic Natural Gradient Method for Safe Optimization on Binary Space

Optimization problems in real-world applications across the medical and engineering domains often involve potential risks when evaluating candidate solutions. Safe optimization aims to perform optimization while suppressing unsafe solution evaluations in such situations. For continuous search spaces, there exist safe optimization methods based on evolutionary computation. However, the algorithm development of safe optimization methods for binary search spaces has not been adequately addressed. In this study, we incorporate additional mechanisms for safe optimization into a binary optimization method, the adaptive stochastic natural gradient method (ASNG) with a family of Bernoulli distributions. For safety functions that must be kept non-negative during optimization, the proposed method, safe ASNG, estimates the Lipschitz constants with respect to the Hamming distance by constructing surrogate models of safety functions based on discrete Walsh functions. Then, safe ASNG computes a safe region that consists of safe solutions around the previously evaluated safe solutions. By projecting newly generated solutions to their nearest neighbors within the safe region, safe ASNG suppresses unsafe solution evaluations. Experimental results on benchmark problems on binary domains confirm that, while the comparative methods fail to suppress unsafe solution evaluations, safe ASNG achieves efficient optimization while effectively suppressing unsafe solution evaluations.

Kento Uchida, Ryoki Hamano, Masahiro Nomura +1

Yokohama National University Yokohama, Kanagawa, Japan · CyberAgent Shibuya, Tokyo, Japan · Institute of Science Tokyo Meguro, Tokyo, Japan

Black-Box OptimizationNatural Gradient MethodsConstrained Optimization
AskPDF
May 16, 2026

BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at https://github.com/chewwt/bolt. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.

Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1

National University of Singapore

Bayesian OptimizationBenchmark DesignBlack-Box OptimizationHyperparameter Optimization
AskPDF
May 15, 2026

Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing

Sampling-based optimization (SBO), like cross-entropy method and evolutionary algorithms, has achieved many successes in solving non-convex problems without gradients, yet its convergence is poorly understood. In this paper, we establish a non-asymptotic convergence analysis for SBO through the lens of smoothing. Specifically, we recast SBO as gradient descent on a smoothed objective, mirroring noise-conditioned score ascent in diffusion models. Our first contribution is a landscape analysis of the smoothed objective, demonstrating how smoothing helps escape local minima and uncovering a fundamental coverage-optimality trade-off: smoothing renders the landscape more benign by enlarging the locally convex region around the global minimizer, but at the cost of introducing an optimality gap. Building on this insight, we establish non-asymptotic convergence guarantees for SBO algorithms to a neighborhood of the global minimizer. Furthermore, we propose an annealed SBO algorithm, Diffusion-Inspired Dual-Annealing (DIDA), which is provably convergent to the global optimum. We conduct extensive numerical experiments to verify our landscape results and also demonstrate the compelling performance of DIDA compared to other gradient-free optimization methods. Lastly, we discuss implications of our results for diffusion models.

Zeji Yi, Chaoyi Pan, Guanya Shi +1

Carnegie Mellon University

Optimization Convergence AnalysisBlack-Box OptimizationGlobal OptimizationNonconvex OptimizationStochastic Optimization ConvergenceDerivative-Free Optimization
AskPDF
May 13, 2026

Stochastic global optimization of continuous functions via random walks on Grassmannians

We introduce a stochastic global optimization method based on random walks on Grassmannian manifolds. To minimize a continuous objective ℓ:Rd→R\ell:\mathbb{R}^d\rightarrow\mathbb{R}, the method repeatedly samples random kk-dimensional linear subspaces (with k≪dk\ll d), solves the resulting low-dimensional restrictions of these problems to these subspaces using an arbitrary black-box optimizer, and updates the iterate (which monotonically improves upon the previous iterate). Unlike classical optimization analyses that rely on convexity, smoothness, Lipschitz bounds, or Polyak-Lojasiewicz-type conditions, our convergence guarantees depend only on the geometric distribution of restricted minima across the kk-dimensional subspaces passing through a given point in Rd\mathbb{R}^d. We identify a gap parameter -- an analogue of a spectral gap for random walks -- that controls the rate at which the iterates approach the global minimum value. Finally, we argue that the same analysis yields a blind-spot robustness property: sufficiently narrow, deep dips of the loss function (small-measure regions where ℓ\ell spikes downward) have limited influence on the algorithm's trajectory, since they are unlikely to be encountered by random subspace sampling.

Kartik Gupta, Stephen D. Miller, Pradeep Ravikumar +1

Stochastic OptimizationSubspace OptimizationBlack-Box OptimizationGlobal OptimizationStochastic Optimization Convergence
AskPDF
May 13, 2026

WASHH: An Anchor-Aware Whale-Guided Selection Hyper-Heuristic for Continuous Optimization and SVC Configuration

Learning-assisted algorithm design often has to make reliable search decisions under small evaluation budgets, where committing to a single metaheuristic can be unreliable. We propose WASHH, a Whale-guided Adaptive Selection Hyper-Heuristic for continuous black-box optimization. WASHH uses WOA as the main exploitation backbone, but treats PSO-style memory, GWO-style leader averaging, DE-style variation, local coordinate search, and anchor-guided refinement as selectable search behaviors. An online reward controller allocates evaluations according to observed improvements, while anchor refinement exploits inexpensive reference configurations such as box centers or default model settings without bypassing black-box evaluation. On ten 30-dimensional benchmark functions with 10 independent runs and 12,000 evaluations, WASHH achieves the best average rank, 1.10, and is best or tied best on all ten functions. It strictly improves over WOA on eight functions and ties WOA at the numerical optimum on Rastrigin and Griewank. We further study SVC hyperparameter configuration for breast cancer diagnosis under a 300-evaluation budget. WASHH obtains the lowest mean validation log loss among the compared optimizers, suggesting that anchor-aware selection hyper-heuristics are a practical lightweight direction for LEAD systems.

Yifu Zhao, Xiaofan Zou, Junhao Wei +9

Faculty of Applied Sciences, Macao Polytechnic University, Macao, 999078, China · School of Mechanical and Electrical Engineering and Automation, Shanghai University, Shanghai, 200444, China · Pazhou Lab (Huangpu), Guangzhou, 510555, China +2

Black-Box OptimizationMetaheuristic OptimizationIsing OptimizationHyperparameter Optimization
AskPDF
May 13, 2026

The Geno-Synthetic Algorithm: Type-Factored Coevolutionary Optimization for Heterogeneous Genotypes and Assembled Phenotypes

Many real-world optimization problems are not naturally homogeneous vectors but composite design objects with heterogeneous parameters: integers, real values, Booleans, categoricals, complex-valued descriptors, and embedding vectors. Standard evolutionary algorithms flatten these into a single chromosome and apply generic operators with rounding and repair, sacrificing representational fidelity. We introduce the Geno-Synthetic Algorithm (GSA), a type-factored coevolutionary framework in which gene families are partitioned by representational type, evolved in parallel with type-native operators, and assembled into executable phenotypes for joint fitness evaluation. GSA is formalized as a typed product-space search procedure with an explicit assembly operator. An open-source reference implementation (gsa-experiments, MIT-licensed) is released. A focused empirical study compares eight GSA variants against five baselines across seven benchmark problems (six synthetic plus the external COCO BBOB-MixInt suite) at budgets from 5,000 to 100,000 evaluations. The headline finding is architectural: GSA is the only method that operates when gene families include complex-valued descriptors or embedding vectors. On smooth synthetic multi-family problems, well-tuned flattened differential evolution remains the strongest baseline; on BBOB-MixInt at 100,000 evaluations, GSA_DIRECT becomes statistically indistinguishable from FLATTENED_DE while FLATTENED_EA drops from second to fifth rank, an asymptotic crossover. Ablations confirm that type-native operators are essential, elite credit dominates ensemble credit, and active assembly outperforms passive concatenation on gated benchmarks. The framework extends naturally to prompt and embedding optimization for large language model systems.

Alex Bogdan

Evolutionairy AI Toronto, Canada

Evolutionary OptimizationBlack-Box OptimizationGenetic Algorithms
AskPDF
May 12, 2026

A Family of Quaternion-Valued Differential Evolution Algorithms for Numerical Function Optimization

The numerical optimization of continuous functions is a fundamental task in many scientific and engineering domains, ranging from mechanical design to training of artificial intelligence models. Among the most effective and widely used algorithms for this purpose is Differential Evolution (DE), known for its simplicity and strong performance. Recent research has shown that adapting AI models to operate over alternative number systems-such as complex numbers, quaternions, and geometric algebras-can improve model compactness and accuracy. However, such extensions remain underexplored in bio-inspired optimization algorithms. In particular, the use of quaternion algebra represents an emerging area in computational intelligence. This paper introduces a family of novel Quaternion-Valued Differential Evolution (QDE) algorithms that operate directly in the quaternion space. We propose several mutation strategies specifically designed to exploit the algebraic and geometric properties of quaternions. Results show that our QDE variants achieve faster convergence and superior performance on several function classes in the BBOB benchmark compared to the traditional real-valued DE algorithm.

Gerardo Altamirano-Gomez, Álvaro Gallardo, Carlos Ignacio Hernández Castellanos

Instituto de Investigaciones en Matem´aticas Aplicadas y Sistemas, Universidad Nacional Aut´onoma de M´exico, M´exico · Universidad Iberoamericana, M´exico

Differential EvolutionEvolutionary OptimizationBlack-Box Optimization
AskPDF
May 12, 2026

Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging

Model merging has emerged as a cost-effective paradigm for combining capabilities of multiple large language models (LLMs) without additional training. Among existing approaches, Data Flow Space (DFS) merging operates by selectively activating parameter groups across model layers, offering greater flexibility than naive parameter averaging. However, the optimization challenges underlying DFS merging remain poorly understood. In this paper, we formally characterize the DFS merging problem as a mixed binary-continuous black-box optimization problem, identifying three fundamental challenges: (1) mixed binary-continuous variable types, (2) high-dimensional search spaces with up to 291 parameter groups, and (3) conditional variable dependencies corresponding to Type-I interaction as characterized by Akimoto et al. [1]. We empirically evaluate seven merging methods on Mistral-7B models across official GSM8K (1,319 questions) and MMLU (14,042 questions, 57 subjects) benchmarks. Our structured approach achieves a 49.5% reduction in effective search space. Results reveal that all tested merging methods fail to surpass the stronger source model on official benchmarks, with TIES (13.72%) and DELLA (4.12%) performing catastrophically on GSM8K. Notably, DELLA scores below even the base model (6.67%), demonstrating that standard parameter space methods fail when merging base and instruction-tuned models. These findings directly validate the difficulty of the Type-I interaction structure we identify, and motivate the application of algorithms specifically designed for this challenge such as the concurrent IGBD work of Ong et al. [2].

Md. Robiul Islam Niloy, Youhei Akimoto

Department of Computer Science and Engineering, BRAC University, Dhaka, Bangladesh · Institute of Systems and Information Engineering / Center for Artificial Intelligence Research, University of Tsukuba, Tsukuba, Japan

Evolutionary OptimizationBlack-Box OptimizationLanguage Model Merging
AskPDF
May 11, 2026

Support-Proximity Augmented Diffusion Estimation for Offline Black-Box Optimization

Offline black-box optimization aims to discover novel designs with high property scores using only a static dataset, a task fundamentally challenged by the out-of-distribution (OOD) extrapolation problem. Existing approaches typically bifurcate into inverse methods, which struggle with the ill-posed nature of mapping scores to designs, and forward methods, which often lack the distributional expressivity to quantify uncertainty effectively. In this work, we propose SPADE (Support-Proximity Augmented Diffusion Estimation), a novel framework that reimagines forward surrogate modeling through the lens of conditional generative modeling. SPADE models the forward likelihood p(y|x) using a diffusion model, but with two critical enhancements to tailor it for optimization: (1) a Calibrated Diffusion Estimation module that enforces global consistency in statistical moments and pairwise rankings, and (2) a Support-Proximity Regularization mechanism that implicitly internalizes the data manifold constraint p(x) via kNN-based density estimation. Theoretically, we prove that our regularization is first-order equivalent to maximizing a Bayesian posterior with a valid design prior. Empirically, SPADE achieves state-of-the-art performance across Design-Bench tasks and an LLM data mixture optimization benchmark.

Yonghan Yang, Ye Yuan, Zipeng Sun +5

MBZUAI - Mohamed bin Zayed University of Artificial Intelligence · McGill University · Mila - Quebec AI Institute +1

Surrogate ModelingSurrogate-Assisted OptimizationBlack-Box OptimizationOOD Generalization
AskPDF
May 11, 2026

Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputs

Guided-diffusion black-box optimization (BO) has shown strong empirical performance on structured design problems such as molecules and crystals, but its regret behavior remains poorly understood. Existing BO regret analyses typically rely on maximum information gain, non-pretrained surrogate models, or exact acquisition maximization -- assumptions that break down in modern diffusion -- BO pipelines, where pretrained diffusion models serve as powerful priors over valid structures and acquisition maximization is replaced by approximate sampling over astronomically large discrete spaces. We develop a first certificate-based expected simple-regret framework for guided-diffusion BO that avoids maximum-information-gain bounds, RKHS assumptions, and exact acquisition maximization. The central quantity in our analysis is mass lift: the increase in probability mass assigned to near-optimal designs relative to the pretrained generator. This view explains how exponential-looking finite-budget convergence and polynomial acceleration can all arise from the same mechanism. We also give practical diagnostics for estimating search exponents from finite candidate pools and a proposal-corrected resampling construction that provides a fully certified sampler instance.

Masaki Adachi, Anita Yang, Yakun Wang +1

Toyota Motor Corporation · Lattice Lab · University of Bristol

Bayesian OptimizationBlack-Box OptimizationDiffusion Guidance
AskPDF
May 9, 2026

ARES-LSHADE: Autoresearch-Enhanced LSHADE with Memetic Polish for the GNBG Benchmark

We present ARES-LSHADE, a memetic differential-evolution variant submitted to the GECCO 2026 competition on LLM-designed evolutionary algorithms for the Generalized Numerical Benchmark Generator (GNBG). The algorithm builds on the LLM-LSHADE 2025 winner, contributing two new components: (a) a scout-augmented mutation operator with adaptive CMA-ES integration, produced by an autonomous research loop across approximately thirty LLM-driven design experiments, and (b) a multi-start L-BFGS-B polish phase that respects strict blackbox treatment of the benchmark. On the official 31-run-per-function evaluation with the competition-specified function-evaluation budgets, ARES-LSHADE obtains 510 of 744 wins (per-function gap below 1e-8), reaching machine precision on 18 of 24 functions. The remaining six functions exhibit characteristic plateau signatures consistent with GNBG's compositional structure, and were independently identified by the autoresearch loop as the hardest of the suite. Beyond the result itself, this report documents two methodological observations: (i) an LLM-driven research loop with operator-only edit surface and fitness-only observation space converges to a characteristic plateau on this benchmark; (ii) when we initially widened the observation space to include the benchmark's compositional metadata, the resulting algorithm trivially solved all 24 functions but violated the competition's blackbox rule, which we identified before submission. We discuss this tension between LLM capability and benchmark integrity as a design consideration for future LLM-driven optimization-algorithm research. Code and reproducibility artifacts are available at https://github.com/anaeem1/ARES-LSHADE.

Abdullah Naeem, Md Wasi Ul kabir, Manish Bhatt +3

University of New Orleans · Amazon†

Differential EvolutionEvolutionary OptimizationBlack-Box OptimizationLarge Language Model-Guided Optimization
AskPDF
May 9, 2026

Drain-Vortex Optimization: A Population-Based Metaheuristic Inspired by Multi-Drain Free-Vortex Flow

This paper proposes Drain-Vortex Optimization (DVO), a population-based metaheuristic for continuous optimization. DVO models each candidate solution as a particle moving in a multi-drain vortex field. Its update rule decomposes motion into radial attraction toward selected drain centres and tangential rotation governed by a regularized free-vortex law. A three-phase mechanism switches between far-field exploration, spiral inward motion, and localized core exploitation according to the normalized distance to the assigned drain. The method also uses adaptive spiral exploitation, population-level vortex basin assignment, and optional stochastic basin switching to support structured diversity. DVO is evaluated against PSO, GWO, WOA, SCA, AOA, EO, and SVOA using a calibration--validation protocol. CEC 2022 is used only to select the final DVO configuration, while CEC 2017, classical functions, and five constrained engineering design problems are used for out-of-sample validation. On CEC 2017, DVO achieves the best mean log⁡10\log_{10} error on 34 of 58 cases and the best Friedman average rank (1.67), and is significantly better than every baseline under Holm-corrected Wilcoxon tests. On CEC 2022, DVO obtains the best Friedman rank (2.13) and is significantly better than five of the seven baselines; the differences against PSO and SVOA are not significant. DVO is less competitive on simple scalable classical functions and on small constrained engineering designs, which clarifies its operating regime. The algorithm is implemented in a vectorized GPU form that executes independent runs in parallel.

Mohsen Omidi, Brian Vaughan

Technological University Dublin, Grangegorman Lower, Dublin, D07 C972, Ireland

Black-Box OptimizationMetaheuristic OptimizationIsing Optimization
AskPDF
May 8, 2026

ADKO: Agentic Decentralized Knowledge Optimization

We present Agentic Decentralized Knowledge Optimization (ADKO), a framework for collaborative black-box optimization across autonomous agents that achieves sample efficiency, privacy preservation, heterogeneous-objective handling, and communication efficiency. Each agent maintains a private Gaussian Process (GP) surrogate trained on local data and communicates only through knowledge tokens-compact, lossy summaries containing directional signals, advantage scores, and optional language-model (LM) insights-without sharing raw data or model parameters. ADKO unifies GP-Upper Confidence Bound (GP-UCB), parallel Bayesian optimization, decentralized learning, and LM-guided discovery. We provide the first formal analysis of dual information loss: token compression, quantified via mutual-information-based fidelity, and LM approximation error, decomposed into bias and stochastic noise. Our main result shows cumulative regret decomposes into GP error, LM bias, LM noise, and compression loss, with necessary and sufficient conditions for sublinear regret. We also propose fidelity-aware token pruning to preserve high-information tokens under memory budget. Experiments on neural architecture search and scientific discovery validate the theory and show consistent improvements over strong baselines.

Lucas Nerone Rillo, Zhanhong Jiang, Nastaran Saadati +4

Iowa State University · New York University

Multi-Agent System OptimizationAI Agents for Scientific DiscoveryBayesian OptimizationBlack-Box OptimizationLarge Language Model-Guided OptimizationDecentralized Optimization
AskPDF
May 8, 2026

POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles

Balancing exploration with exploitation is a core challenge in black-box optimization and reinforcement learning. We introduce POETS (Po\textbf{Po}licy E\textbf{E}nsembles for T\textbf{T}hompson S\textbf{S}ampling), a novel framework that bridges uncertainty quantification and policy optimization. Our approach is grounded in the insight that policies trained with Kullback-Leibler (KL) regularization implicitly encode an underlying reward function. Building on this, POETS bypasses the complex, nested process of training an uncertainty-aware reward model and separately fitting a policy to this model. Instead, we directly train a policy ensemble to capture epistemic uncertainty by matching implicitly encoded reward functions to online, bootstrapped data. To overcome the prohibitive compute and memory constraints of ensembling Large Language Models (LLMs), POETS utilizes an efficient architecture: the ensemble shares a pre-trained backbone while maintaining diversity through independent Low-Rank Adaptation (LoRA) branches. Theoretically, we show that POETS implicitly approximates KL-regularized Thompson sampling, a principled optimization strategy with optimal cumulative regret bounds of O(TγT)O(\sqrt{T γ_T}). Empirically, we demonstrate that POETS achieves state-of-the-art sample efficiency across diverse scientific discovery domains, including protein search, therapeutic molecular design, and quantum circuit design. POETS also improves the optimization trajectory and out-of-distribution transfer of reinforcement learning across mathematical reasoning and code generation tasks, proving particularly robust in off-policy settings with experience replay or in small dataset regimes.

Nicolas Menet, Andreas Krause, Abbas Rahimi

IBM Research – Zurich · Department of Computer Science, ETH Zürich

Thompson SamplingBlack-Box OptimizationLarge Language Model-Guided OptimizationLLM Uncertainty EstimationLanguage Model EnsemblesLarge Language Model-Guided Scientific Discovery
AskPDF
May 8, 2026

Flow-Direct: Feedback-Efficient and Reusable Guidance for Flow Models via Non-Parametric Guidance Field

Training-free guidance enables pre-trained diffusion and flow models to optimize application-specific objectives using feedback from external black-box reward functions. However, existing methods are feedback-inefficient because reward feedback is used only transiently to inform a localized gradient approximation or a discrete search decision, and is subsequently discarded. To address this limitation, we propose Flow-Direct, a framework that guides the generation process via a persistent guidance field. Theoretically, this guidance field is analytically derived from the log-density ratio between the base and reward-weighted target distributions; it transports the pre-trained distribution to the target distribution. In practice, the field is implemented as a non-parametric estimator constructed from all accumulated reward-evaluated samples. As more samples are collected during optimization, this empirical guidance field becomes increasingly accurate. This persistent formulation yields two major advantages. First, Flow-Direct is highly feedback-efficient: because every evaluated sample is used to refine the global guidance field, no reward information is wasted. Second, the framework is naturally reusable: once optimization is complete, the collected dataset defines a reusable guidance field for generating novel target samples without additional reward evaluations, and distinct guidance fields can be combined to generate samples that simultaneously satisfy multiple objectives.

Kim Yong Tan, Yueming Lyu, Ivor Tsang +1

College of Computing and Data Science, Nanyang Technological University, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore

Diffusion Model GuidanceBlack-Box OptimizationFlow-Based Generative ModelingInference-Time Guidance
AskPDF
May 7, 2026

In-Context Black-Box Optimization with Unreliable Feedback

Black-box optimization in science and engineering often comes with side information: experts, simulators, pretrained predictors, or heuristics can suggest which candidates look promising. This information can accelerate search, but it can also be biased, input-dependent, or misleading. Feedback-aware BO methods typically handle one task at a time, limiting their ability to generalize over multiple sources of feedback. In-context optimizers address cross-task adaptation, but usually assume that optimization history is the only available signal at test time. We study feedback-informed in-context black-box optimization (FICBO), where a pretrained optimizer conditions on both the observed history and cheap auxiliary feedback for the current candidate set. We introduce a structured feedback prior that models how feedback sources vary in their access, relevance, and distortion relative to the true objective, and use it to pretrain a feedback-aware transformer. At test time, the model estimates source reliability in context by comparing observed objective values with auxiliary signals, improving query selection. On synthetic and real-world tasks, FICBO effectively exploits informative feedback while remaining robust to weak or misleading sources, improving over other baselines. Empirical investigations further illustrate how the model perceives test-time sources, offering insights into its interpretability and decision-making process.

Nicolas Samuel Blumer, Julien Martinelli, Samuel Kaski

ELLIS Institute Finland · Aalto University · University of Manchester

Bayesian OptimizationBlack-Box OptimizationIn-Context Learning
AskPDF
May 6, 2026

S-LCG: Structured Linear Congruential Generator-Based Deterministic Algorithm for Search and Optimization

This study presents a novel deterministic optimization algorithm based on a special variant of the Linear Congruential Generator (LCG). While conventional algorithms generally operate within the search space, the introduced technique follows a two-level architecture. In particular, an external loop that adaptively balances between exploration and exploitation, while the internal loop evaluates solutions. It is motivated by the intrinsic structure of the generator, the reason behind naming it the Structured Linear Congruential Generator (S- LCG). which enjoys a number of unique characteristics as follows: 1) a memoryless scheme, which ensures non-overlapping sequences based on distinct seeds, thus ensuring no evaluation redundancy; 2) bit splitting representation, which converts LCG states into multi-dimensional points to overcome the Marsaglia lattice effect; 3) adaptive exploration-exploitation of the generator space, which leads to implicit optimization of the surrogate smooth objective function; and 4) constant information gathering speed to avoid the problem of premature convergence. Extensive testing on 26 benchmark functions across dimensions d = 2 to 30 demonstrates that S-LCG comes within 1% of the global optimum in 83.3% of 138 cases (100% at d = 2, 81.2% at d = 30) while the nearest competitor GA achieved 75.4%. Statistical validation shows that S-LCG outperforms eight cutting-edge binary algorithms. Furthermore, its practical value is confirmed by validation on three constrained engineering design problems. In the end, S-LCG offers an optimization framework that is strictly reproducible and requires only one sensitive parameter to be tuned.

Ahmed Qasim Mohammed, Haider Banka, Anamika Singh

Department of Computer Science and Engineering, IIT Dhanbad, India · Department of Computer Science and Engineering, Indian Institute of Technology Dhanbad, India · Department of Mathematics and Computing, Indian Institute of Technology Dhanbad, India

Black-Box OptimizationMetaheuristic Optimization
AskPDF
May 6, 2026

On the Influence of the Feature Computation Budget on Per-Instance Algorithm Selection for Black-Box Optimization

Per-instance algorithm selection (PIAS) takes advantage of complementarity between a set of algorithms by deciding which algorithm to run on a given instance. This decision is based on features of the instances, which, in the context of black-box optimization (BBO), require a part of the optimization budget to be computed. This raises two questions: (a) from which fraction of the budget spent on feature computation does PIAS become worth it for BBO, and (b) which fraction of the budget optimizes the tradeoff between feature accuracy and PIAS performance. To this end, we perform a broad study where PIAS with varying sampling budgets for feature computation is compared to the single best algorithm on a broad range of algorithm selection scenarios. These scenarios consist of two portfolio sizes, three problem sets, 4 dimensionalities, and 10 target budgets. We find that PIAS is viable for the majority of tested scenarios, even when as much as a quarter of the total budget is spent on feature computation. The tradeoff for the fraction of the budget spent on feature computation to maximize the benefit of PIAS is highly dependent on the specific AS scenario. Further, on average 20 percent of PIAS loss to the virtual best solver is explained by the budget spent on feature computation, highlighting the importance of properly accounting for the feature budget.

Koen van der Blom, Diederick Vermetten

Centrum Wiskunde & Informatica, Amsterdam, The Netherlands · Sorbonne Universit´e, CNRS, LIP6, Paris, France

Algorithm SelectionBlack-Box Optimization
AskPDF