cs.LGSep 29, 2026
SaveDisCoMBO: Steering Expert-in-the-Loop Black Box Optimization via Distributional Conformance
Organizations: Computer Science Department TU Darmstadt · TU Darmstadt, Hessian.AI, DFKI
Abstract
Sequential Model-Based Optimization (SMBO) traditionally relies on Bayesian or ensembling surrogates for uncertainty quantification. While historically treated as fully data-driven, SMBO increasingly integrates external domain expertise to accelerate discovery. To overcome the opaque guidance and diminished integration fidelity of standard acquisition re-weighting, Probabilistic Circuits (PCs) have emerged as a generative surrogate alternative, enabling direct knowledge injection via conditional sampling. However, these generative routines lack the formal exploration-exploitation semantics required for rigorous optimization. We introduce the Distributional Conformance Score (DisCo), a novel metric that unifies the flexibility and efficiency of PCs with a formal uncertainty framework. DisCo provides a bounded, -normalized measure of model "surprise" that (1) recovers properties comparable to kernel-based uncertainty known from, e.g., Gaussian Processes, while maintaining linear-time inference, and (2) enables accurate assessment of conformance of external knowledge w.r.t. model evidence. We then present DisCoMBO, a framework leveraging these properties for robust, knowledge-aware optimization. We prove that DisCoMBO is a zero-regret algorithm and demonstrate its effectiveness across diverse benchmarks from AutoML, material optimization, and wind park optimization.
Figures & tables
Figure 1: DisCO provides uncertainty estimates for PC-based SMBO and allows for accurate conformance assessment of external knowledge – both with linear inference time. (a) Given a learned PC from observations, DisCo ( ) provides uncertainty estimates with linear inference cost based on how well a given point conforms with the distribution represented by the PC, resulting in uncertainty quantification akin to Gaussian Processes (GPs; ). (b) DisCo can assess whether externally provided knowledge conforms to the PC’s expectation for improving upon , serving as a scale-independent basis for acceptance or rejection of given knowledge.
Figure 2: DisCoMBO performs DisCo-guided candidate generation. To generate new candidate configurations, DisCoMBO samples a large set of configurations from the conditional (Left) along with random candidates from the search space. These configurations are considered potentially promising and are scored using DisCo in a single bottom-up pass (Right) . Resulting DisCo scores are used to guide exploration and exploitation as well as to rate conformance of potentially given priors.
Figure 3: DisCoMBO is competitive to top-tier SMBO methods. DisCoMBO ranks first across 9 tasks when no prior is given, being statistically indistinguishable to strong baselines like HEBO. DisCoMBO significantly outperforms IBO-HPC, its PC-based counterpart, lacking a well-defined exploitation-exploration trade-off. Assessment is based on Wilcoxon signed-rank test ( ).
Figure 4: DisCoMBO is effective on diverse tasks. DisCoMBO ( ) effectively optimizes black-box functions across tasks with diverse search spaces and optimization landscapes. In 8/9 tasks, DisCoMBO is competitive to top-tier SMBO algorithms (see App. C.4 for remaining tasks). We achieve significant improvement upon IBO-HPC ( ) across 9/9 tasks.
Figure 5: DisCoMBO leverages beneficial priors effectively. We find that DisCoMBO ( ) retains IBO-HPC’s ( ) efficacy in incorporating priors. We outperform baselines or perform competitively on 7/9 tasks. See App. C.4 for additional results and discussion.
Figure 6: DisCoMBO successfully recovers from and detects misleading priors. ( Left ) DisCoMBO effectively recovers ( ) from misleading priors, even in worst-case scenarios in which misleading priors are always accepted. Also, it effectively handles sequences of priors ( ) with contradicting information. ( Right ) We find that our DisCo-based detection mechanism’s accuracy ( ) is significantly higher compared to DynaBO’s heuristic-based detection mechanism ( ), indicating DisCoMBO is able to detect misleading priors.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| S1 | S2 | S3 | |
|---|---|---|---|
| Activation | [Mish, ReLU, Hardswish] | [Mish, ReLU, Hardswish] | [Mish, ReLU, Hardswish] |
| Learning Rate | [1e-3, 1e0] | [1e-3, 1e0] | [1e-3, 1e0] |
| Weight Decay | [1e-5, 1e-2] | [1e-5, 1e-2] | [1e-5, 1e-2] |
| Trivial Argument | [True, False] | [True, False] | [True, False] |
| Op1 | 0-6 | 0-6 | 0-6 |
| Op2 | 0-6 | 0-6 | 0-6 |
Table 1: JAHS Search Space. We define three versions of the JAHS search space, ranging from simpler to harder spaces.
Figure 7: DisCoMBO achieves top-tier performance without external priors. Without guidance with external priors, DisCoMBO ( ) achieves top-tier performance on 8/9 tasks. On Floris, we still significantly improve upon IBO-HPC ( ), however, DisCoMBO cannot fully catch up with top-tier baselines (see App. C.4 ) for further discussion.
Figure 8: DisCoMBO is competitive to top-tier SMBO methods without priors. DisCoMBO ranks first across 9 tasks when no prior is given, being statistically indistinguishable from strong baselines like HEBO. DisCoMBO significantly outperforms IBO-HPC, its PC-based counterpart, lacking a principled exploitation-exploration trade-off. Assessment is based on Wilcoxon signed-rank test ( ), corrected for multiple tests.
Figure 9: DisCoMBO retains knowledge integration efficacy of generative PC-based SMBO. When provided with external priors early in optimization, DisCoMBO ( ) retains the efficacy of IBO-HPC ( ) in leveraging that provided knowledge to converge faster and to better solutions. We outperform baselines on 7/9 tasks.
Figure 10: DisCoMBO effectively incorporates priors into optimization. DisCoMBO ranks first across 9 tasks when a prior is given, being statistically indistinguishable to IBO-HPC and BO. This also demonstrates that DisCoMBO retains the efficacy of IBO-HPC in integrating external priors into the optimization process. Assessment is based on Wilcoxon signed-rank test ( ) and corrected for multiple tests.
Figure 11: DisCoMBO effectively recovers from misleading priors. DisCoMBO ( ) successfully recovers from misleading priors on all tasks. In 4/6 tasks, we recover faster than our baselines, demonstrating robustness. We evaluate recovery on JAHS, NAS-Bench-201, Floris, and PV as a representative subset of tasks. On Floris, BO was not able to recover from the misleading prior.
Figure 12: DisCoMBO effectively recovers from misleading priors. DisCoMBO ranks first across 9 tasks when a misleading prior is given. While DisCoMBO is statistically indistinguishable to IBO-HPC and DynaBO, the results show that DisCoMBO retains or even improves upon the robustness of all competing methods. Assessment is based on Wilcoxon signed-rank test ( ) and corrected for multiple tests.
Figure 13: DisCoMBO effectively handles sequences of priors. We demonstrate that DisCoMBO ( ) effectively handles sequences of priors, even if they contain contradicting information. We first provide a misleading prior, followed by a beneficial prior, followed by a misleading prior, and finally a beneficial prior again. It can be observed that DisCoMBO leverages beneficial priors effectively while not suffering from misleading ones. BO does not allow for multiple priors.
Figure 14: DisCoMBO effectively handles sequences of priors. DisCoMBO ranks first across 9 tasks when a sequence of priors is given. This demonstrates that DisCoMBO successfully handles multiple, even contradicting priors. Assessment is based on Wilcoxon signed-rank tests ( ) and corrected for multiple tests.
Figure 15: Average algorithmic rank over time across 50 independent seeds. DisCoMBO ( ) achieves the top final rank on 6/9 tasks while remaining highly competitive on the remaining landscapes, with the exception of Floris (see App. C.4 ). On the AutoML suite, DisCoMBO quickly identifies high-quality solutions early in the optimization trajectory. On PV (crossed-barrel), it benefits from structured exploration to effectively exploit the search space in later iterations.
Figure 16: Average algorithmic rank over time given a beneficial prior at iteration 5 (averaged across 50 seeds). DisCoMBO ( ) achieves the top final rank on 7/9 tasks. Across all benchmarks except FLORIS, the framework successfully capitalizes on the injected prior, demonstrating that DisCoMBO retains or enhances the targeted knowledge integration capabilities of purely generative approaches like IBO-HPC.
Figure 17: Average algorithmic rank over time given a misleading prior at iteration 5 (averaged across 50 seeds). DisCoMBO ( ) achieves the top final rank on 4/6 tasks. Across all benchmarks except Floris and PV (crossed-barrel), the framework successfully recovers from a misleading prior given after 5 iterations, demonstrating that DisCoMBO retains or enhances the targeted knowledge integration capabilities of purely generative approaches like IBO-HPC.
Figure 18: Average algorithmic rank over time given a sequence of priors (averaged across 50 seeds). DisCoMBO ( ) achieves the top final rank on 5/6 tasks. Across all benchmarks except PV (crossed-barrel), the framework successfully recovers from a misleading prior given after 5 iterations, demonstrating that DisCoMBO retains or enhances the targeted knowledge integration capabilities of purely generative approaches like IBO-HPC.
Figure 19: Relative improvement of DisCoMBO against IBO-HPC over time without priors. We observe that DisCoMBO ( ) consistently achieves a significant improvement over its fully generative counterpart IBO-HPC on 8/9 tasks, demonstrating the effectiveness of our exploration-exploitation strategy driven by DisCo.
Figure 20: Relative improvement of DisCoMBO against IBO-HPC over time with beneficial prior at 5 iterations. We observe that DisCoMBO ( ) consistently achieves a significant improvement over its fully generative counterpart IBO-HPC or matches its performance on 7/9 tasks when beneficial priors are provided early during optimization, demonstrating that DisCoMBO effectively incorporates priors.
Figure 21: Relative improvement of DisCoMBO against IBO-HPC over time with misleading prior at 5 iterations. We observe that DisCoMBO ( ) consistently achieves a significant improvement over its fully generative counterpart IBO-HPC or matches its performance on 4/6 tasks when misleading priors are provided early during optimization, demonstrating that DisCoMBO effectively recovers from misleading priors.
Figure 22: Relative improvement of DisCoMBO against IBO-HPC over time with a sequence of priors given. We observe that DisCoMBO ( ) consistently achieves a significant improvement over its fully generative counterpart IBO-HPC or matches its performance on 4/6 tasks when a sequence of priors with contradicting information is provided during optimization, demonstrating that DisCoMBO effectively handles multiple priors.
Figure 23: DisCoMBO is robust across different hyperparameter settings. It can be seen that DisCoMBO achieves similar performances regardless of how hyperparameters , , , and are set. This demonstrates that no careful tuning of DisCoMBO’s hyperparameters is necessary to achieve strong performance.
Figure 24: DisCoMBO can discriminate between beneficial and adversarial priors. On 4/5 datasets, our method ( ) is able to identify most of the 10 good and 10 bad interventions provided to the model correctly as such. On 5/5 , DisCoMBO directly outperforms DynaBO ( ), indicating it is well-suited for human-in-the-loop optimization scenarios even without extensive domain knowledge.
| Benchmark/task | DisCoMBO (no interaction) | DisCoMBO (broad prior) |
|---|---|---|
| JAHS/cifar10 | 1.0000 | 2.1603 |
| JAHS/colorectal_histology | 1.0000 | 1.9342 |
| JAHS/fashion_mnist | 1.0000 | 1.7872 |
| HPO-B/6767:31 | 1.0000 | 2.7492 |
| HPO-B/6767:9914 | 1.0000 | 2.1132 |
| HPO-B/6794:3 | 1.0000 | 1.7865 |
Table 2: DisCoMBO is effective with broader priors. When priors are not as specific as in our main results, DisCoMBO still clearly benefits from provided priors in terms of average speed-up. A value means, the algorithm was % faster compared to DisCoMBO with no prior provided.
| input | multiplication | arithmetic mean | Fisher | |
|---|---|---|---|---|
| a) | (1e-10, 1e-10) | |||
| b) | (1e-10, 0.01) | |||
| c) | (1e-10, 1) | |||
| d) | (0.01, 1) | |||
| e) | (1, 1) | |||
| f) | 1 1e-10, 9 1 |
Table 3: Fisher’s method ranks cases similarly to raw multiplication (a–c, h, i), but accounts for the number of observations. In g), nine ones and one give the same product as d), yet Fisher’s reports instead of — a single weak signal among nine nulls is correctly judged uninformative. The arithmetic mean stays above even when nine of ten observations are tiny (h, i), washing out localized signal.
More in this topic
Black-Box Optimization
Black-box optimization finds high-quality solutions when objectives can only be queried, not directly analyzed or differentiated.
See all 176 papers