Betting for Sim-to-Real Performance Certificates

Yujia Chen; Bowen Weng

cs.ROAug 21, 2026

Betting for Sim-to-Real Performance Certificates

Authors: Yujia Chen, Bowen Weng

Organizations: Department of Computer Science, Iowa State University, Ames, IA 50011, USA

Abstract

Consider a typical test of a robot system: one observes a sequence of outcomes concerning some aspect of interest (crash or no crash, tracking error, time to completion), and reports a mean (crash risk, average error, mean time to completion) and, more importantly, an interval guaranteed to contain that mean at a prescribed confidence, referred to as a performance certificate. Given expensive real-world trials, the sample size is therefore small, and the certificate is often loose. Now consider the same procedure, except that before each real outcome is revealed, the operator peeks'' at a large bank of simulated results, and places a bet on where the real outcome will land. As the real outcomes settle the bets, the operator gains or loses wealth. One's trust'' over simulators also shifts within the portfolio. This paper develops that idea into a sim-to-real betting certificate framework with three contributions: (i) An algorithm that links a scalable bank of simulators to effective bets, and the accumulated betting wealth to the certificate. (ii) A proof that the returned certificate is anytime valid, covering the true mean with the prescribed probability, using any simulator bank. (iii) The guaranteed wealth-regret bounds yield configuration principles for the proposed algorithm and simulator bank design to deliver tight certificates. Experiments across synthetic distributions and real-world robot tests, covering both replayed standardized testing outcomes and online runtime evaluation, show the proposed method narrows the certificate by $51.6\%\pm16\%$ against classic and state-of-the-art baselines, and by $32.26\%\pm8\%$ in the extremely limited-sample regime ( $\leq30$ samples).

Figures & tables

Fig. 1 : A sequence of $8$ sampled measures $\{0,0,0,0,0,1,0,0\}$ are generated from $x\sim\mathrm{Bernoulli}(0.55)$ (a toy example emulating a success rate certificate task of a peg-in-hole operation [ 1 ] ). The lower bounds of all certificates remain at zero across all sample sizes. The upper bound of each certificate is shown. Note p-value based methods occasionally have invalid coverage. All calculations are based on Section I-A .

\pi_{t}^{k}\!=\!\frac{\exp\big(\eta L_{t-1}^{k}\big)}{\sum_{j=1}^{K}\!\exp\big(\eta L_{t-1}^{j}\big)},\!m_{t}\!=\!\sum_{k=1}^{K}\!\pi_{t}^{k}\mu_{k},\!v_{t}\!=\!\sum_{k=1}^{K}\!\pi_{t}^{k}\sigma_{k}^{2}

(10)

Algorithm 1 Sim-to-Real Betting for E-Process Certificate

Fig. 2 : Illustration of Theorem 1 and 2 on the same synthetic real distribution: (a) The expected average wealth regret, $\mathbb{E}[\overline{\mathcal{R}}_{t}]/t$ , for dense and separated simulators, together with the corresponding theorem bounds. (b) The mean and variance of the real distribution and each simulator. The two simulator sets yield similar regret while satisfying the different structures illustrated by the two theorems.

Panel of Figure 4 — Fig. 3 : An overview of the three experimental categories and four testing tasks is provided below. For the synthetic evaluation , Fig. 3(a) illustrates five simulator banks relative to three sets of target, or “real” distributions. Both the simulators and the target distributions are synthetic and are constructed from parameterized distribution families, including Bernoulli, beta, truncated-normal, bimodal-mixture, uniform-spike, and Gaussian-mixture distributions (adapted and extended from [ 26 , 5 ] ). Each individual distribution is represented as a point in the mean-variance plane, with different markers distinguishing the target distributions from the simulators. Each simulator-bank name begins with Sim followed by the number of distinct parameterized distributions contained in that bank. For example, Sim_756 contains 756 simulator distributions. An additional suffix describes a special bank construction when needed, as in Sim_7_biased . A similar naming strategy applies to the group of Real distributions as well. Fig. 3(c) presents the open-source NIST Continuous Mobile Manipulator Performance Measurement Dataset [ 1 ] . The dataset records a wheeled mobile manipulator performing a peg-in-hole task according to standardized testing procedures. Fig. 3(d) presents a test conducted under ASTM Work Item WK86916 [ 3 ] developed by ASTM Subcommittee F45.06 to standardize the evaluation of legged robots under push-over disturbances. In this test, a Unitree Go2 robot operating with its manufacturer-provided control modules is subjected to controlled impacts delivered by a swinging-pendulum impactor. Fig. 3(b) showcases (a simulator and a real views of) a command tracking accuracy testing task of a Unitree G1 humanoid running the GR00T [ 16 ] locomotion controller under joystick commands.

Panel of Figure 5 — Fig. 4 : Empirical results of C1 . At every reported sample size ( $t$ ), Fig. 4(a) ’s certificate widths are averaged over all distributions in the corresponding Real group and over 100 independently seeded trials for each distribution. Fig. 4(b) ’s empirical coverage is shown as the observed probability for the obtained $\mathcal{C}_{t}$ covering the analytical $\mu^{*}$ (note the theoretical coverage is 95% given $\alpha=0.05$ .). Fig. 4(c) ’s width is calculated similar to that in Fig. 4(a) .

Fig. 5 : The evolution of the average certificate width for each method for the C2 task. Given the command $x^{c}=[v_{x}^{c},v_{y}^{c},\omega^{c}]$ (i.e., desired linear velocities and yaw rate) and the measured robot state $x^{r}=[v_{x}^{r},v_{y}^{r},\omega^{r}]$ , the weighted command tracking error takes the form of $0.25\sqrt{(v_{x}^{c}-v_{x}^{r})^{2}+(v_{y}^{c}-v_{y}^{r})^{2}}+0.5|(\omega^{c}-\omega^{r}|$ .

Panel of Figure 7 — Fig. 6 : The evolution of the average certificate width for various methods in two different C3 tasks.

Fig. 7 : Overview of the experimental setup. The left side shows the Type B CMMA, while the right side shows the mobile manipulator mounted on the AMR-CT. The left corner highlights the Retro-reflective Laser Sensor and Emitter (RLS), and the right corner shows the optical tracking system (OTS) markers mounted on the end effector of the manipulator. Note: the figure is adapted from [ 1 ]

Explore similar work

Sim-to-Real Betting on the E-Process: Bringing "simulators" to anytime-valid confidence sequences
Jun 23, 2026Yujia Chen, Bowen WengAnytime-Valid Inference
Department of Computer Science, Iowa State University
Betting for Sim-to-Real Performance Evaluation
Apr 27, 2026Zaid Mahboob, Yujia Chen, Bowen WengSim-to-Real Transfer
Department of Computer Science Iowa State University, Ames, Iowa 50011
How Should a Simulation-to-Reality Transfer Budget Be Spent?
Jun 20, 2026Syed Hamzah Rizvi, Yash Vardhan TomarRobot Dynamics Identification Domain Randomization
Purdue University West Lafayette, USA

Jun 23, 2026, cs.ROJ/K move · Enter open · S save

Sim-to-Real Betting on the E-Process: Bringing "simulators" to anytime-valid confidence sequences

Yujia Chen, Bowen Weng

Department of Computer Science, Iowa State University

Anytime-Valid Inference

Open paper AskPDF arXiv

This note describes an integration of the sim-to-real performance estimate with betting (from Chen et al.) and the safe anytime-valid inference (from Ramdas et al.). Using the scaled simulators. The method produces efficient, reliable certificates for the mean estimate, an approach that is especially valuable in robot performance testing. This note gives a primary, self-contained account of the construction; preliminaries of the respective methods are kept at a minimum, and one shall refer to the original works for full detail. Some synthetic examples demonstrating the proposed algorithm can be found at https://github.com/ISUSAIL/Bet4Sim2Real-EProcess.

Abstract

Explore similar work

Sim-to-Real Betting on the E-Process: Bringing "simulators" to anytime-valid confidence sequences

Betting for Sim-to-Real Performance Evaluation

How Should a Simulation-to-Reality Transfer Budget Be Spent?