We present Dejavu, a method that makes sim-to-real transfer of reinforcement learning policies in constrained Markov decision processes probably safe over an unbounded sequence of policy updates. An anytime valid test using per-decision importance sampling and a betting confidence sequence certifies each candidate trained in simulation before deployment using only trajectories collected by a safe baseline policy operating in production. Simulation quality does not affect the safety guarantee, only the production sample efficiency of the method. Finally, a summable per-test failure probability yields the lifetime valid property. Empirical results across three sim-to-real benchmarks support the theoretical work.
Figures & tables
Figure 1: Deployment benchmark, one column per environment: the baseline policy’s true production-environment reward (top) and cost return with the limit (middle), and the share of seeds whose current baseline policy exceeds the cost constraint (bottom). Lines are means over 20 seeds, shaded regions are ±1 standard deviation. The normalized deployment lifecycle describes the fraction of the total production data budget consumed for training by each method. SPiDR uses 0 production data; the others have a budget of 10 M steps on the CRAX environments and 1 M steps on IcyLake.
Table 2: Environment settings, identical across methods. Production parameters are drawn once per seed, uniformly over the range shown. ∗ Selected among ℓ~∈{1,0.5} (HalfCheetahVelocity, warm-started from an unconstrained policy) or {2,1} (PointGoal) by a 64 -episode evaluation at the most adverse corner of the production range.
Method
Setting
HalfCheetahVelocity
PointGoal
IcyLake
Dejavu
Tests (maximum)
50
50
40
Simulation steps per test †
4.1 M
4.1 M
4.1 M
Parallel simulation environments †
256
256
256
Rollout length, epochs, minibatch †
128 , 4 , 512
128 , 4 , 512
128 , 4 , 512
Ratio clip, reward and cost †
0.2 , 0.05
0.2 , 0.05
0.2 , 0.05
Value-loss coefficient †
0.5
0.5
0.5
Appendix
Table 3: Per-method settings. Simulation training (rows marked † ) is shared by the Dejavu proposer and SPiDR.
Method
HalfCheetahVelocity
PointGoal
IcyLake
Dejavu
9.57 M ( 9.98 M)
8.65 M ( 9.98 M)
0.80 M ( 1.00 M)
SPiDR
0
0
0
CPO
10 M
10 M
1 M
HCPI
9.86 M ( 9.86 M)
9.86 M ( 9.86 M)
0.98 M ( 1.00 M)
Appendix
Table 4: Production steps consumed per seed, mean (maximum) over seeds. Dejavu includes the initial policy’s certification. Tuning trials and truth evaluations are not counted for any method.
We address safe sim-to-real transfer, in which an agent leverages an imperfect simulator and limited real-world interaction while ensuring safety throughout data collection in the real system. This problem arises in applications such as robotics and healthcare: simulators provide cheap data, but sim-to-real mismatch makes direct transfer unreliable, and collecting real-world data to correct this mismatch must itself be safe. Moreover, deployment objectives may vary across tasks, making it costly to collect new data for each reward function. We therefore formulate safe sim-to-real transfer as a reward-free safe reinforcement learning (RL) problem, in which data are collected once and reused to plan for arbitrary reward functions. We develop a computationally efficient algorithm that identifies where the simulator and real dynamics differ, uses certified simulator transitions where they are reliable, and estimates mismatched transitions from safely collected data. With high probability, every policy deployed during learning is feasible, and the collected data support the computation of a feasible and near-optimal policy for any reward function. When the simulator is uninformative, our algorithm recovers online reward-free safe RL while improving the best-known sample complexity by a factor of Θ(H/ξ2), where ξ is the safety margin of a baseline policy. When the simulator is accurate on most transitions, this improvement grows to Θ(H2∣\mcS∣∣\mcA∣/(ξ2∣\mcB∣)), where ∣\mcB∣ denotes the size of the sim-to-real mismatch region.
Training controllers that are safe and robust in simulation, and systematically assessing their readiness for real-world deployment, remain key challenges in sim-to-real transfer. To address this, we propose LyEvO, a physics-grounded framework that combines constrained Evolutionary Optimization and Statistical Model Checking (SMC)-based verification with Lyapunov-based stability analysis. Leveraging prior knowledge of the system dynamics, LyEvO uses Lyapunov analysis to compute an initial candidate stability region. An iterative loop then uses operational scenarios drawn from this region to jointly optimize and statistically verify a policy, and subsequently expands the region's boundaries based on the verification outcome. This integrated procedure provides a practical criterion for assessing deployment readiness. We evaluate LyEvO on Cartpole and 3D Quadrotor benchmarks through extensive simulations and targeted real-world experiments, demonstrating safe and robust sim-to-real transfer.
Riccardo Curcio, Hongpeng Cao, Marco Caccamo
Department of Computer Science, Sapienza University of Rome, via Salaria 113, 00198 Rome, Italy · School of Engineering and Design, Technical University of Munich, Boltzmannstraße 15, 85748 Garching b. München, Germany
In recent years, reinforcement learning (RL) has shown remarkable success in robotics when a fast and accurate simulator is available for a given task. When using RL and simulation, more simulator realism is generally beneficial but becomes harder to obtain as robots are deployed in increasingly complex and widescale domains. In such settings, simulators will likely fail to model all relevant details of a given target task and this observation motivates the study of sim2real with simulators that leave out key task details. In this paper, we formalize and study the abstract sim2real problem: given an abstract simulator that models a target task at a coarse level of abstraction, how can we train a policy with RL in the abstract simulator and successfully transfer it to the real-world? Our first contribution is to formalize this problem using the language of state abstraction from the RL literature. This framing shows that an abstract simulator can be grounded to match the target task if the grounded abstract dynamics take the history of states into account. Based on the formalism, we then introduce a method that uses real-world task data to correct the dynamics of the abstract simulator. We then show that this method enables successful policy transfer both in sim2sim and sim2real evaluation.
Yunfu Deng, Yuhao Li, Josiah P. Hanna
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA