Organizations: Robotics Institute, Carnegie Mellon University · Princeton University · Data and Systems Engineering, University of Hong Kong · Princeton Plasma Physics Laboratory
Many real-world optimization problems involve optimizing black-box objectives for which only a limited number of evaluations are available. Bayesian optimization (BO) is a powerful framework for such problems, but its effectiveness often deteriorates in high-dimensional spaces. Recent work leverages diffusion models for black-box optimization, often through pre-training a reward-conditioned score network to steer generation towards high-reward samples. Alternatively, gradient-based guidance methods assume access to a differentiable reward function from which guidance gradients are queried. However, these approaches either require large amounts of labeled data for offline training or access to exact reward gradients, limiting their applicability in black-box settings. We propose BReaD (Black-box Reward-guided Sampling of Diffusion Model), a guidance sampling framework for optimizing black-box rewards without assuming access to a labeled dataset or reward gradients. The proposed method uses a Gaussian process (GP) as a differentiable surrogate for the unknown objective and actively queries labels for the generated samples. We provide a theoretical analysis of the total-variation distance between the generated distribution and the target distribution. Using Upper Confidence Bound (UCB) as guidance signals, our method balances exploration and exploitation during generation and enables the discovery of high-reward regions in the data space. Numerical experiments on scientific discovery and image generation tasks demonstrate that our framework can effectively optimize black-box rewards in low-data scenarios.
Figures & tables
Figure 1: Black-box Reward-guided Sampling of Diffusion Models ( BReaD ). By actively exploring and querying reward labels in promising regions of the (embedded) sample space, BReaD can effectively optimize black-box rewards in high-dimensional but low-data regimes.
Algorithm 1 ForwardUniversalGuidance ( Bansal et al., 2024 )
Figure 2: Human preference-guided generation (Task 2, Section 4.1 ). A painting of a girl walking in a hallway and suddenly finds a giant sunflower on the floor blocking her way . BReaD generates high-score samples with only a few evaluations without accessing reward gradients. Images are generated with the same seed.
Figure 3: Image generation guided by black-box rewards (Section 4.1 ). Optimization for style loss (Left) and ImageReward score (Right). Results are averaged over 10 trials.
Figure 4: Ligand optimization for Vina scores. DPP4 and p38-MAPK. Results are averaged over 10 trials.
Figure 5: Accessing the Wide-Pedesatal QH in a simulated tokamak environment. Results are averaged over 10 trials.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Definition
f⋆
(Unknown) Reward function
N(0,τ)
Noise in the observed rewards
y⋆
Target score
qdata
Data distribution
qt
Marginal distribution of Xˉt in process ( 17 )
QT←
Path distribution of the backward SDE ( 18 )
Appendix
Table 1: Notation and definitions in the proof
Variable
Ensemble EV
Variable
Ensemble EV
βN
0.7983
denv1
0.4912
βp
0.8137
temp_pca1
0.6892
WMHD
0.8012
temp_pca2
0.5145
prmtan_tewid
0.3132
temp_pca3
0.3936
prmtan_newid
0.2460
temp_pca4
0.4799
prmtan_teped
0.3803
dens_pca1
0.6918
Appendix
Table 2: Per-variable explained variance (EV) of the RPNN ensemble trained on historical DIII-D experimental data. Higher values indicate better agreement with held-out real trajectories.
Signal Group
Signals
Actuator
Controllable
State
Scalar States
βN (Normalized Plasma Pressure)
✗
✗
✓
βp (Poloidal Beta)
✗
✗
✓
MHD Stored Energy ( WMHD )
✗
✗
✓
Electron Temperature Pedestal Width (prmtan_tewid)
✗
✗
✓
Electron Density Pedestal Width (prmtan_newid)
✗
✗
✓
Electron Temperature Pedestal Height (prmtan_teped)
✗
✗
✓
Appendix
Table 3: Plasma state and actuator signals used for modeling the DIII-D tokamak dynamics in the WPQH experiments. The controllable column indicates actuator signals that can be optimized when searching for WPQH access strategies.
Figure 6: Style guidance on Stable Diffusion v1.5 to match the text prompt, A portrait of a woman , and the style of Pablo Picasso’s painting, Brick Factory at Tortosa . Results are averaged over 10 trials.
Figure 7: Style guidance on Stable Diffusion v1.5 to match the text prompt, A colorful photo of an Eiffel Tower , and the style of Vincent van Gogh’s painting, The Starry Night . Results are averaged over 10 trials.
Figure 8: Style guidance on Stable Diffusion v1.5 to match the text prompt, A fantasy photo of an volcanoes , and the style of Leonid Afremov’s painting, Melody of The Night . Results are averaged over 10 trials.
Figure 9: Guidance sampling of Stable Diffusion v1.5 to match the text prompt, Landscape photography by Marc Adams, mountains with some forests, small lake in the center, fog in the background, sunrays, golden hour, high quality , where the rewards are given by the ImageReward model. The images demonstrated use the same seed. Results are averaged over 10 trials.
Figure 10: Image generation for optimizing the ImageReward score. A painting of a girl walking in a hallway and suddenly finds a giant sunflower on the floor blocking her way.
Figure 11: Ligand optimization. VEGFR2 and ER α . Results are averaged over 10 trials.
Guided-diffusion black-box optimization (BO) has shown strong empirical performance on structured design problems such as molecules and crystals, but its regret behavior remains poorly understood. Existing BO regret analyses typically rely on maximum information gain, non-pretrained surrogate models, or exact acquisition maximization -- assumptions that break down in modern diffusion -- BO pipelines, where pretrained diffusion models serve as powerful priors over valid structures and acquisition maximization is replaced by approximate sampling over astronomically large discrete spaces. We develop a first certificate-based expected simple-regret framework for guided-diffusion BO that avoids maximum-information-gain bounds, RKHS assumptions, and exact acquisition maximization. The central quantity in our analysis is mass lift: the increase in probability mass assigned to near-optimal designs relative to the pretrained generator. This view explains how exponential-looking finite-budget convergence and polynomial acceleration can all arise from the same mechanism. We also give practical diagnostics for estimating search exponents from finite candidate pools and a proposal-corrected resampling construction that provides a fully certified sampler instance.
Masaki Adachi, Anita Yang, Yakun Wang +1
Toyota Motor Corporation · Lattice Lab · University of Bristol
Bayesian optimization (BO) is a widely used approach for black-box optimization that uses a Gaussian process (GP) as a surrogate and guides sequential evaluations via an acquisition function, with the ultimate goal of locating the global optimum x⋆. To align with this goal, information-based acquisition functions such as Predictive Entropy Search (PES) model x⋆ as a random variable and reduce the entropy of its distribution, but approximating this distribution via traditional GP posterior sampling is computationally expensive. To address this limitation, we leverage Conditional Diffusion Models (CDMs) to efficiently approximate the distribution of x⋆ and develop BO-inherent training strategies for CDMs. Motivated by the structural properties of the CDM-learned distribution, we further develop an acquisition strategy termed Diffusion-based Mode Seeking (DMS) to guide the sequential evaluation. We establish a sub-optimality guarantee for the CDM-learned distribution and demonstrate through extensive experiments that DMS outperforms standard BO baselines.
Yilin Zheng, Haowei Wang, Szu Hui Ng +1
National University of Singapore · Singapore · Georgia Institute of Technology +1
Black-box optimization is a fundamental tool in science and engineering for optimizing objectives when gradient information is unavailable. It becomes especially difficult when the objective function is expensive to evaluate, limiting the evaluation budget to a few tens or hundreds of queries, and when good solutions occupy complex, low-measure regions of the search space. Generative models can supply useful structural priors in such settings, but existing generative BBO approaches bring significant evaluation cost. We identify three design principles for generative optimization under such low-budget conditions: avoid objective learning, optimize in candidate space, and make every evaluation count. Together, these principles motivate separating structural modeling from objective-driven search. We instantiate them in SPARROW, a simple sequential optimizer that maintains a persistent, ranked archive of evaluated candidates and uses a fixed, unconditional generative sampler solely as a corruption-refinement operator. SPARROW requires only access to the sampler's corruption and refinement processes, and never needs to evaluate the objective to train or guide it. Across three complementary settings, probing thin feasible geometry, disconnected high-performing regions, and failure-prone evaluations, SPARROW outperforms classical and generative baselines under strict evaluation budgets. These results demonstrate that separating structural priors from objective-driven search can be effective when evaluations are scarce and the search geometry is challenging.