Probably Safe Sim-to-Real Transfer with Lifetime Validity
Organizations: Machine Learning Research Group, University of Oxford, Oxford, United Kingdom.
Abstract
We present Dejavu, a method that makes sim-to-real transfer of reinforcement learning policies in constrained Markov decision processes probably safe over an unbounded sequence of policy updates. An anytime valid test using per-decision importance sampling and a betting confidence sequence certifies each candidate trained in simulation before deployment using only trajectories collected by a safe baseline policy operating in production. Simulation quality does not affect the safety guarantee, only the production sample efficiency of the method. Finally, a summable per-test failure probability yields the lifetime valid property. Empirical results across three sim-to-real benchmarks support the theoretical work.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Seeds | |
| Root seed | ( for CPO’s production training) |
| Prod param seed | , , (HalfCheetahVelocity, PointGoal, IcyLake) |
| Policy | diagonal Gaussian (HalfCheetahVelocity, PointGoal); categorical (IcyLake) |
| Networks | actor, reward critic and cost critic, each , |
| Optimizer | Adam, , learning rate , gradient-norm clip |
| Setting | HalfCheetahVelocity | PointGoal | IcyLake |
|---|---|---|---|
| Cost trigger | speed m/s | in a hazard or by a wall | on an icy tile |
| Cost limit | |||
| Simulation cost limit | |||
| Cost bound | |||
| Episode length | |||
| Horizon |
| Method | Setting | HalfCheetahVelocity | PointGoal | IcyLake |
|---|---|---|---|---|
| Dejavu | Tests (maximum) | |||
| Simulation steps per test † | M | M | M | |
| Parallel simulation environments † | ||||
| Rollout length, epochs, minibatch † | , , | , , | , , | |
| Ratio clip, reward and cost † | , | , | , | |
| Value-loss coefficient † |
| Method | HalfCheetahVelocity | PointGoal | IcyLake |
|---|---|---|---|
| Dejavu | M ( M) | M ( M) | M ( M) |
| SPiDR | |||
| CPO | M | M | M |
| HCPI | M ( M) | M ( M) | M ( M) |
Sim-to-Real Transfer
Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.
See all 352 papers