Preferential Bayesian Optimization with Crash Feedback

Johanna Menn; David Stenger; Sebastian Trimpe

cs.ROApr 2, 2026

Preferential Bayesian Optimization with Crash Feedback

Authors: Johanna Menn, David Stenger, Sebastian Trimpe

Organizations: Institute for Data Science in Mechanical Engineering (DSME), RWTH Aachen University, Aachen, Germany · aiXopt GmbH, Aachen, Germany

Abstract

Bayesian optimization is a popular black-box optimization method for parameter learning in control and robotics. It typically requires an objective function that reflects the user's optimization goal. However, in practical applications, this objective function is often inaccessible due to complex or unmeasurable performance metrics. Preferential Bayesian optimization (PBO) overcomes this limitation by leveraging human feedback through pairwise comparisons, eliminating the need for explicit performance quantification. When applying PBO to hardware systems, such as in quadcopter control, crashes can cause time-consuming experimental resets, wear and tear, or otherwise undesired outcomes. Standard PBO methods cannot incorporate feedback from such crashed experiments, resulting in the exploration of parameters that frequently lead to experimental crashes. We thus introduce CrashPBO, a user-friendly mechanism that enables users to both express preferences and report crashes during the optimization process. Benchmarking on synthetic functions shows that this mechanism reduces crashes by 63% and increases data efficiency. Through experiments on three robotics platforms, we demonstrate the wide applicability and transferability of CrashPBO, highlighting that it provides a flexible, user-friendly framework for parameter learning with human feedback on preferences and crashes.

Figures & tables

Fig. 1 : CrashPBO enables optimization of robotic tasks directly from human preferences. The human provides feedback on preferences (“Experiment A was better than B”) and crashes (i.e., totally undesirable experiments). A crashed experiment is ranked worse than all successful ones, preventing the exploration of unsafe or undesirable regions.

Panel of Figure 2 — Fig. 2 : Applications of CrashPBO

Fig. 3 : Crash mechanism in PBO: By adding virtual comparisons of the crashed experiment (red) with all successful experiments (black), we ensure that the posterior is worse in the crashed regions. This reduces the likelihood of exploration in the crashed region. Comparisons between data points are indicated by

\mathbf{x}_{\mathrm{A}}\succ\mathbf{x}_{\mathrm{B}}

, meaning that

\mathbf{x}_{\mathrm{A}}

is preferred over

\mathbf{x}_{\mathrm{B}}

, a comparison is indicated with a line in the plot.

Fig. 4 : Synthetic results showing average normalized performance and average crashes for within-model comparison and synthetic functions. We compare CrashPBO to the PBO method EUBO, the standard BO method MES, the safe BO method SafeOpt, and random search. CrashPBO performs similarly to MES and EUBO and outperforms safe BO and random search while reducing crashes compared to EUBO in both settings.

Algorithm	Metric	best	last	two new
EUBO	Perf. $\uparrow$	$0.85\pm 0.11$	$0.83\pm 0.12$	$0.83\pm 0.13$
CrashPBO	Perf. $\uparrow$	$\color[rgb]{0,0,1}{0.87\pm 0.11}$	$0.84\pm 0.13$	$0.85\pm 0.11$
EUBO	Crashes $\downarrow$	$0.30\pm 0.10$	$0.34\pm 0.14$	$0.18\pm 0.07$
CrashPBO	Crashes $\downarrow$	$\color[rgb]{0,0,1}{0.11\pm 0.04}$	$0.17\pm 0.08$	$0.12\pm 0.05$

TABLE I : Comparison on average performance (“higher is better”,

\uparrow

) and crashes (“fewer is better”,

\downarrow

) over four test functions and within-model comparison for one- to eight-dimensional GPs after

10d

evaluations, where

d

is the dimension of the test function.

Fig. 5 : Experimental outcomes of three DMs tuning the backflip maneuver with subjective preferences. The top row shows the mean of the learned pairwise GP after the optimization, and the bottom row shows the trajectories of all evaluated flips. The black points indicate the shared initial parameters, which correspond to the black trajectories below. The final preferred parameters are marked with a star and appear as the bold trajectory. All other evaluated parameters are shown as points when the flip succeeded and as crosses when the experiment crashed.

Panel of Figure 7 — Fig. 6 : (a) Furuta pendulum. (b)–(c) State distributions during learning with CrashPBO, EUBO, and random search, estimated using kernel density estimation (KDE). Successful swing-ups yield $\alpha,\theta\approx 0$ , while large $\theta$ values indicate aggressive motion or cable protection impacts. CrashPBO’s lighter tails reflect more stable and less aggressive behavior.

	CrashPBO	EUBO	Random
no weights	4/20	8/20	9/20
weights	6/20	8/20	12/20

TABLE II : Failed pendulum swing-ups

Experiment	Decisions	Crashes	Final Parameters
Figure 8	15/20	4/20	0.79, 0.64, 0.0081, 0.82
Ramp	15/20	3/20	0.55, 0.55, 0.000, 2.61

TABLE III : Summary of Mini Wheelbot results, as in this setting it is hard to recall the best observed parameters, the best parameters can be repeated, so that not every experiment results in a decision.

Bayesian Optimization

Bayesian optimization uses probabilistic models and sequential experiments to find strong solutions when evaluating a black-box objective is costly.

See all 243 papers

Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models
Mar 25, 2026Lukas Theiner, Maik Pfefferkorn, Yongpeng Zhao +2Bayesian Optimization Multi-Fidelity Optimization
Control and Cyber-Physical Systems Laboratory, Technical University of Darmstadt, Darmstadt, Germany · Volkswagen AG, Wolfsburg, Germany
A Decade of Bayesian Optimization for Controller Tuning and Robot Learning: Tutorial, Review, and Future Prospects
Sep 8, 2026David Stenger, Paul Brunzema, Johanna Menn +3Robotic Control Bayesian Optimization
RWTH Aachen University, Aachen, Germany · University of Technology Nuremberg, Nuremberg, Germany · Technical University of Munich, Munich, Germany
Local Preferential Bayesian Optimization
Jun 1, 2026Johanna Menn, Miriam Kober, Paul Brunzema +2Pairwise Preference Learning Pairwise Preference Evaluation
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1

Mar 25, 2026J/K move · Enter open · S save

Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models

Lukas Theiner, Maik Pfefferkorn, Yongpeng Zhao, Sebastian Hirt+1

Control and Cyber-Physical Systems Laboratory, Technical University of Darmstadt, Darmstadt, Germany · Volkswagen AG, Wolfsburg, Germany

Bayesian Optimization Multi-Fidelity Optimization Surrogate Modeling

Open paper AskPDF arXiv

Tuning control policies manually to meet high-level objectives is often time-consuming. Bayesian optimization provides a data-efficient framework for automating this process using numerical evaluations of an objective function. However, many systems, particularly those involving humans, require optimization based on subjective criteria. Preferential Bayesian optimization addresses this by learning from pairwise comparisons instead of quantitative measurements, but relying solely on preference data can be inefficient. We propose a multi-fidelity, multi-modal Bayesian optimization framework that integrates low-fidelity numerical data with high-fidelity human preferences. Our approach employs Gaussian process surrogate models with both hierarchical, autoregressive and non-hierarchical, coregionalization-based structures, enabling efficient learning from mixed-modality data. We illustrate the framework by tuning an autonomous vehicle's trajectory planner, showing that combining numerical and preference data significantly reduces the need for experiments involving the human decision maker while effectively adapting driving style to individual preferences.

Abstract

Bayesian Optimization

Explore similar work

Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models

A Decade of Bayesian Optimization for Controller Tuning and Robot Learning: Tutorial, Review, and Future Prospects

Local Preferential Bayesian Optimization