Uncertainty-Aware Selection of Online Algorithms with Simulator Ensembles

Yongyi Guo; Zifan Xu; Ziping Xu; Kelly W. Zhang

cs.LGSep 26, 2026

Uncertainty-Aware Selection of Online Algorithms with Simulator Ensembles

Authors: Yongyi Guo, Zifan Xu, Ziping Xu, Kelly W. Zhang

Organizations: Department of Statistics, University of Wisconsin–Madison · Department of Computer Science, The University of Texas at Austin · School of Data and Information Sciences, University of North Carolina at Chapel Hill · Department of Mathematics, Imperial College London

Abstract

The performance of online reinforcement learning depends critically on design choices, especially those that affect exploration. These choices are often selected by fitting a simulator to offline data, evaluating candidate algorithms in that simulator, and deploying the best-performing one. The simplest Plug-In selection rule simply selects the best performing algorithm on the fitted simulator, making evaluations unreliable when the offline data used to fit the simulator are limited. We investigate Uncertainty-Aware selection, which forms an ensemble of simulators---for example, obtained by bootstrap resampling---and selects the online algorithm with the best average performance across the ensemble. While ensemble-based approaches have been used to mitigate distribution shift and facilitate sim-to-real transfer, we formally show that this approach can mitigate the effects of limited data when fitting the simulator and theoretically has significant regret gains compared to Plug-In selection in multi-armed bandits. We also empirically investigate the Uncertainty-Aware selection approach in deep RL experiments on robotic control tasks that involve selecting reward-shaping hyperparameters, and show that it leads to more reliable selection and improved online performance.

Figures & tables

Figure 1: Selecting an online RL algorithm using limited offline data. Both rules fit simulators to $\mathcal{D}_{\textnormal{off}}$ and then deploy the selected algorithm in the true environment $M^{*}$ . The Plug-In rule commits to a single fitted simulator $\widehat{M}$ ; the Uncertainty-Aware rule averages the objective over $\widehat{M}^{(1)},\dots,\widehat{M}^{(n)}$ , favoring algorithms that perform consistently across plausible realizations.

Figure 2: Asymmetric costs of exploration and the benefit of Uncertainty-Aware selection. Simulations are in a two-armed Gaussian bandit with $T=5000$ , $T_{\textnormal{off}}=25$ , margin $\mu_{1}-\mu_{2}=1$ , and noise standard deviation $\sigma=3$ . Top: Regret of UCB (left) and Thompson Sampling (right) as a function of the exploration level $\theta$ (see ( 4 ) and ( 5 )). Regret rises far more steeply below the optimal value $\theta_{T}^{*}$ (dotted vertical line) than above it, i.e., underexploration is more costly. Bottom: Using multiple samples of $\mathcal{D}_{\textnormal{off}}$ , Uncertainty-Aware selection (ensemble models constructed via bootstrap) more often selects higher values of $\theta$ compared to Plug-In, which leads to lower regret on average. Additionally, Uncertainty-Aware selection has lower variance in regret across trials.

Figure 3: Uncertainty-Aware (UA) selection leads to greater exploration than Plug-In under UCB. We independently sample $10^{6}$ $\mathcal{D}_{\textnormal{off}}$ datasets and compute an estimate of the margin $\Delta=\mu_{1}-\mu_{2}$ and the reward standard deviation $\sigma$ . The star depicts the true value of $\Delta=1$ and $\sigma=3$ in the environment. The contour lines at 50% depicts where 50% of the $10^{6}$ estimated $\widehat{\Delta},\widehat{\sigma}$ are; similarly for the 90%, and 99% contour lines. We depict the selected value of $\theta/\sigma^{2}$ for each combination of $(\Delta,\sigma)$ under the Plug-In selection rule (left) and that for the UA selection rule (center). In the right most plot, we depict the difference in chosen $\theta/\sigma^{2}$ for the Plug-In and UA rules. The plot shows that the UA selection rule leads to similar or greater exploration than Plug-In for most values of $(\Delta,\sigma)$ . Combined with the asymmetric cost of underexploration (Section 3.1 ), this helps explain how UA decreases regret compared to Plug-In.

	Hopper , $J_{T}(\pi_{\theta_{T}^{}},M^{})=0.991$				Walker2d , $J_{T}(\pi_{\theta_{T}^{}},M^{})=0.689$
$T_{\textnormal{off}}$	Plug-In	UA	$q_{10}$ Plug-In	$q_{10}$ UA	Plug-In	UA	$q_{10}$ Plug-In	$q_{10}$ UA
$20$	$0.933\pm 0.007$	$\mathbf{0.956\pm 0.005}^{*}$	$0.764$	$\mathbf{0.930}$	$0.585\pm 0.006$	$\mathbf{0.594\pm 0.006}~~$	$0.456$	$0.456$
$80$	$0.965\pm 0.003$	$\mathbf{0.971\pm 0.003}~~$	$0.930$	$0.930$	$0.594\pm 0.005$	$\mathbf{0.616\pm 0.005}^{*}$	$0.456$	$\mathbf{0.517}$
$320$	$0.976\pm 0.002$	$\mathbf{0.981\pm 0.002}~~$	$0.930$	$\mathbf{0.936}$	$0.590\pm 0.005$	$\mathbf{0.610\pm 0.005}^{*}$	$0.456$	$\mathbf{0.527}$

Table 1: Success score of the weight vector selected by the Plug-In and UA rules on the true environment, mean

\pm

standard error and

10

th percentile over

300

paired replications; higher is better. The better of the two rules is in bold, and ∗ marks a significant difference.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The cost of misspecified exploration is asymmetric in every environment tested, under UCB. Excess regret ( 21 ) of UCB

(\theta)

against

\theta/\sigma^{2}

, for the environments ( 20 ), at

T=5000

. The triangle and the dashed line mark the best candidate. The vertical scale is per panel, since the regret at the best candidate varies by three orders of magnitude across these environments.

Figure 5: The cost of misspecified exploration is asymmetric in every environment tested, under Thompson Sampling. The counterpart of Figure 4 for the algorithm of ( 5 ).

Figure 6: Uncertainty-Aware (UA) selection leads to greater exploration than Plug-In under Thompson Sampling. The counterpart of Figure 3 for the algorithm of ( 5 ), on the candidate grid $\Theta_{\textnormal{TS}}$ of ( 12 ). Star, contours and panels are as in Figure 3 .

task	$x_{\textnormal{lo}}$	$x_{\textnormal{hi}}$	$c_{\max}$
Hopper	$1.523$	$3.047$	$1.548$
Walker2d	$1.669$	$3.337$	$3.346$

Appendix

Table 2: Thresholds of the success criterion ( 24 ), in meters and in units of

\lVert A\rVert_{2}^{2}

Hopper	$\theta_{\textnormal{ctrl}}$
$\theta_{\textnormal{fwd}}$	$0.002$	$0.01$	$0.05$	$0.25$	$1$
$0.3$	$0.190$	$0.067$	$0.062$	$0.218$	$0.000$
$1$	$0.381$	$0.555$	$0.872$	$\mathbf{0.991}$	$0.553$
$3$	$0.365$	$0.456$	$0.783$	$0.930$	$0.936$
$10$	$0.378$	$0.360$	$0.552$	$0.764$	$0.764$
$30$	$0.255$	$0.368$	$0.438$	$0.614$	$0.637$

Appendix

Table 3: Success score

J_{T}(\pi_{\theta},M^{*})

of every weight vector

\theta\in\Theta

on the true environment,

\theta_{\textnormal{surv}}=1

, under the sensor at

\sigma_{\mathrm{rel}}=0.05

;

30

PPO seeds per cell on Hopper and

32

on Walker2d ,

50

evaluation episodes per seed. The optimum

\theta_{T}^{*}

of each task is in bold; the success score of a selected

\theta

is read from it.

Hopper
		mass		friction		damping
$T_{\textnormal{off}}$	fit	bias	sd	bias	sd	bias	sd	boot/true
$20$	naive	$+0.008$	$0.042$	$+0.069$	$0.137$	$+0.932$	$0.402$	$0.71$
$20$	SIMEX	$+0.011$	$0.200$	$+0.162$	$0.470$	$+0.463$	$0.914$	$0.91$
$80$	naive	$+0.024$	$0.021$	$+0.015$	$0.006$	$+0.814$	$0.122$	$1.15$
$80$	SIMEX	$+0.022$	$0.133$	$+0.007$	$0.194$	$+0.238$	$0.467$	$1.01$

Appendix

Table 4: Physics identification under the sensor (

\sigma_{\mathrm{rel}}=0.05

). The truth is

\lambda^{*}=(1.5,0.7,1.4)

on both robots. Bias and standard deviation of the estimate across independent offline datasets (

12

datasets for the naive fit,

24

for SIMEX); the last column is the bootstrap standard deviation of the damping estimate divided by its across-dataset standard deviation, which is

1

for a calibrated ensemble of members.

task	$T_{\textnormal{off}}$	friction axis	damping axis	mass
Hopper	$20$	$\{0.24,0.72,2.03\}$	$\{0.42,1.28,1.77,2.32,3.77\}$	$1.511$
Hopper	$80$	$\{0.34,0.71,1.06\}$	$\{0.79,1.37,1.63,2.05,2.31\}$	$1.522$
Hopper	$320$	$\{0.60,0.74,0.88\}$	$\{1.25,1.40,1.52,1.64,1.78\}$	$1.544$
Walker2d	$20$	$\{0.10,0.58,2.03\}$	$\{0.14,1.03,1.68,2.34,2.95\}$	$1.504$
Walker2d	$80$	$\{0.44,0.64,1.34\}$	$\{1.18,1.43,1.60,1.84,1.96\}$	$1.508$
Walker2d	$320$	$\{0.42,0.73,0.95\}$	$\{1.24,1.49,1.62,1.68,1.81\}$	$1.514$

Appendix

Table 5: The lattice of identified physics parameters

\lambda

on which the simulated success scores are tabulated, one per cell. The two axes are the

2

nd to

98

th percentiles of the SIMEX estimates over

24

independent offline datasets; the third parameter is held at their mean.

Hopper , $J_{T}(\pi_{\theta_{T}^{}},M^{})=0.991$
		mean		$10$ th percentile		Borda		share at optimum
$T_{\textnormal{off}}$	Plug-In	score	$t$	score	$t$	score	$t$	Plug-In	UA
$20$	$0.933$	$0.956$	$\mathbf{+3.80}$	$0.929$	$-0.35$	$0.926$	$-0.89$	$62\%$	$69\%$
$80$	$0.965$	$0.971$	$+1.57$	$0.968$	$+0.69$	$0.969$	$+0.87$	$72\%$	$81\%$
$320$	$0.976$	$0.981$	$+1.91$	$0.978$	$+0.73$	$0.969$	$-1.91$	$84\%$	$88\%$
Walker2d , $J_{T}(\pi_{\theta_{T}^{}},M^{})=0.689$

Appendix

Table 6: Success score

J_{T}(\pi_{\widehat{\theta}},M^{*})

of the weight vector selected by the Plug-In and by the UA rule under three ways of combining the members, in the six cells of Table 1 . Mean score and the paired

t

statistic of the difference from the Plug-In (positive favors UA; bold where

|t|>1.96

300

replications. “share at optimum” is the fraction of replications that select

\theta_{T}^{*}

Figure 7: The UA selection

\widehat{\theta}_{\textnormal{UA},T}

given the Plug-In selection

\widehat{\theta}_{\textnormal{plug},T}

, in the six cells of Table 1 ,

300

paired offline datasets each. The weight vectors on both axes are sorted by the ratio

\theta_{\textnormal{fwd}}/\theta_{\textnormal{ctrl}}

; a larger ratio pays more for moving forward and charges less for actuation, so the trained policy runs harder. Only weight vectors selected at least once are shown. Each column is one Plug-In selection, with the number of datasets in parentheses; the bars in a column give the conditional distribution of the UA selection at its position on the vertical axis. Bars on the dashed diagonal mean the two rules agree; red bars above it mean UA selected a larger ratio than the Plug-In did, blue bars below it a smaller one.

\Xi_{i}(t;\theta)=\widehat{\mu}_{i}(t-1;\theta)+\sqrt{\frac{\theta}{N_{i}(t-1;\theta)}}Z_{i,t}.

Appendix

Algorithm 2 TS

(\theta)

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Bayesian Online Model Selection
Feb 20, 2026Aida Afshar, Yuke Zhang, Aldo PacchianoMarkovian Bandits Model Selection
Boston University · Broad Institute of MIT and Harvard
Active Exploration via Autoregressive Generation of Missing Data
May 29, 2024Tiffany Tianhui Cai, Hongseok Namkoong, Daniel Russo +1RL Exploration Autoregressive Generation
Department of Statistics Columbia University New York, NY 10025, USA · Decision, Risk, and Operations Division Columbia Business School New York, NY 10025, USA · Department of Mathematics Imperial College London London SW7 2AZ, United Kingdom
What Matters for Simulation to Online Reinforcement Learning on Real Robots
Feb 23, 2026Yarden As, Dhruva Tirumala, René Zurbrügg +4RL for Robotics Robotic RL
ETH Zurich · Google DeepMind

Feb 20, 2026J/K move · Enter open · S save

Bayesian Online Model Selection

Aida Afshar, Yuke Zhang, Aldo Pacchiano

Boston University · Broad Institute of MIT and Harvard

Markovian Bandits Model Selection Multi-Armed Bandits

Open paper AskPDF arXiv

Online model selection in Bayesian bandits raises a fundamental exploration challenge: When an environment instance is sampled from a prior distribution, how can we design an adaptive strategy that explores multiple bandit learners and competes with the best one in hindsight? We address this problem by introducing a new Bayesian algorithm for online model selection in stochastic bandits. We prove an oracle-style guarantee of

O\left( d^* M \sqrt{T} + \sqrt{(MT)} \right)

on the Bayesian regret, where

M

is the number of base learners,

d^*

is the regret coefficient of the optimal base learner, and

T

is the time horizon. We also validate our method empirically across a range of stochastic bandit settings, demonstrating performance that is competitive with the best base learner. Additionally, we study the effect of sharing data among base learners and its role in mitigating prior mis-specification.

Abstract

Reinforcement Learning

Explore similar work

Bayesian Online Model Selection

Active Exploration via Autoregressive Generation of Missing Data

What Matters for Simulation to Online Reinforcement Learning on Real Robots