Simulating, Fast and Slow: Learning Policies for Black-Box Optimization

Tim Bakker; Fabio Valerio Massoli; Thomas Hehn; Tribhuvanesh Orekondy; Arash Behboodi

cs.LGJun 6, 2024

Simulating, Fast and Slow: Learning Policies for Black-Box Optimization

Authors: Tim Bakker, Fabio Valerio Massoli, Thomas Hehn, Tribhuvanesh Orekondy, Arash Behboodi

Organizations: University of Amsterdam § · Qualcomm AI Research‡

Abstract

Simulators are vital in science and engineering, as they faithfully model the influence of design parameters on real-world observations. Optimizing design parameters to minimize a desired objective function is notoriously challenging since simulators are often non-differentiable black boxes. Moreover, each simulation incurs significant compute time, thus doing such optimizations naively may be prohibitively expensive. Furthermore, in experiment design settings, practitioners are typically required to solve the same optimization problem for various input distributions before proceeding to real-world experiments. Starting these related optimizations from scratch each time can be inefficient if the forward simulation model is expensive to evaluate. To address these challenges, this paper introduces a novel method for solving black-box optimization problems on various input distributions by learning a policy that guides the training of a differentiable surrogate. The surrogate's gradients are then used to optimize the simulation parameters with SGD. After training the policy, the cost for downstream optimization of problems involving black-box simulators is amortized, and we require up to $10\times$ fewer expensive simulator calls compared to baselines such as previous local surrogate-based approaches, numerical optimization, and Bayesian optimization.

Figures & tables

Figure 1: Schematic view of our approach . (a) We study black-box optimization problem (over parameters $\bm{\psi}$ ), with an emphasis on using gradient information from a fast differentiable surrogate $f_{\phi}$ (b) To optimize $\bm{\psi}$ sample-efficiently, we employ a policy $\pi_{\theta}$ to actively determine whether retraining the surrogate is necessary before using the gradient information.

Panel of Figure 4 — Figure 4: Benchmark function results. Top row: Probabilistic Three Hump problem. Middle row: Rosenbrock problem. Bottom row: Nonlinear Submanifold Hump Problem.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Schematic for the Actor (left) and Critic (right) MLPs, which together form the policy. Both models take $(\bm{\psi},t,l,\sigma)$ as input.

Figure 9: Schematic for the surrogate architecture. The surrogate is an MLP trained to mimic the simulator. It takes

(\bm{\psi},\bm{x},\bm{z})

as input and outputs

\bm{y}

Figure 10: Loss landscape for the Probabilistic Three Hump Problem for different bounds on the $x$ -distribution. Shown is $\frac{1}{n}\,\sum_{i=0}^{n-1}\mathcal{L}(f_{s}(\bm{\psi},\bm{x}_{i}))$ for a grid of $\bm{\psi}$ values ( $n=100$ ). The yellow region denotes $\bm{\psi}$ values that lead to termination.

Episodes
Maximum number of timesteps $T$	1000
Maximum number of simulator calls $L$ (High Energy Physics)	30
Maximum number of simulator calls $L$ (rest)	50
Episodes per PPO iteration (High Energy Physics)	5
Episodes per PPO iteration (Submanifold Hump & Wireless)	10
Episodes per PPO iteration (ThreeHump & Rosenbrock)	16

Appendix

Table 1: Hyperparameters used for RL training

Figure 11: Loss landscape and learned optimization trajectory for the 2D Three Hump problem . The yellow region denotes $\bm{\psi}$ values that lead to termination. The $\epsilon=0.5$ region around $\bm{\psi}_{0}$ (black cross) is visualized as the red box. Light green and blue arrows respectively represent gradients from a previously trained surrogate or after a simulator call.

Figure 12: Top : Average number of simulator calls (purple curve) and average sum of rewards (blue curve) after each PPO iteration. Values are averaged across evaluation episodes. The upper and lower bound for the shadowed areas represent the max. and min. for each of the two mentioned metrics, respectively. Bottom : Average value of the probability of calling the black-box simulator (orange curve) and average critic loss (green curve). The shaded region represents the standard deviation. Values are averages across evaluation episodes and time steps within an episode for each PPO iteration. Note that the episodic return stabilizes before the 20th PPO iteration.

Black-Box Optimization

Black-box optimization finds high-quality solutions when objectives can only be queried, not directly analyzed or differentiated.

See all 176 papers

Exploiting Gradients in Bayesian Inference of Expensive Simulators
Oct 8, 2026Šimon Soldát, Václav ŠmídlSimulation-Based Inference Bayesian Optimization
Department of Computer Science, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic · Institute of Plasma Physics of the Czech Academy of Sciences, Prague, Czech Republic
D-IMPL: A Diffusion-based Solver for Parameterized BBOs
Sep 19, 2026Yang Hu, Na LiBlack-Box Optimization Learning to Optimize
School of Engineering and Applied Sciences, Harvard University, Boston, MA
Exact Discrete Stochastic Simulation with Deep-Learning-Scale Gradient Optimization
Feb 23, 2026Jose M. G. Vilar, Leonor SaizStraight-Through Estimator Markov Chains
Biofisika Institute (CSIC, UPV/EHU), University of the Basque Country (UPV/EHU), P.O. Box 644, 48080 Bilbao, Spain · IKERBASQUE, Basque Foundation for Science, 48011 Bilbao, Spain · Department of Biomedical Engineering, University of California, 451 E. Health Sciences Drive, Davis, CA 95616, USA

Oct 8, 2026J/K move · Enter open · S save

Exploiting Gradients in Bayesian Inference of Expensive Simulators

Šimon Soldát, Václav Šmídl

Department of Computer Science, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic · Institute of Plasma Physics of the Czech Academy of Sciences, Prague, Czech Republic

Simulation-Based Inference Bayesian Optimization Surrogate-Assisted Optimization Active Learning Gaussian Process Regression

Open paper AskPDF arXiv

Simulators based on differential equations are ubiquitous in science and engineering. They are often used in simulation-based inference to evaluate the posterior distribution of the input parameters based on real-world observations of the simulator outputs. However, inference becomes challenging when individual simulator evaluations are computationally expensive. In such cases, a Bayesian optimization-based active learning approach with Gaussian process surrogate models has been used to maximize the information obtained from a limited simulation budget. Recently, gradients of simulator outputs with respect to input parameters have become increasingly available, yet they are rarely exploited for inference. Even though we only need to learn the simulator input-output relationship, gradient information can provide an additional valuable signal to guide the active learning procedure. This is of particular interest in the case of expensive simulators, when sample efficiency is crucial. In this paper, we demonstrate how incorporating gradient information into the Gaussian process surrogate accelerates Bayesian optimization-based inference under a limited simulation budget. Our results show significant improvement in convergence speed from using gradient information. For reverse-mode differentiation, the inference efficiency gains are maintained when accounting for the additional computational cost. In contrast, for forward-mode differentiation, the inference speed-up does not outweigh the computational costs. These results indicate that gradient-enhanced surrogates are beneficial primarily in problems where the number of parameters exceeds the output dimensionality, where reverse-mode differentiation is efficient.

Abstract

Black-Box Optimization

Explore similar work

Exploiting Gradients in Bayesian Inference of Expensive Simulators

D-IMPL: A Diffusion-based Solver for Parameterized BBOs

Exact Discrete Stochastic Simulation with Deep-Learning-Scale Gradient Optimization