A Universal Policy Wrapper With Guarantees
Organizations: Center for Engineering Systems and Sciences, 30 Bolshoy Boulevard, bld. 1, Moscow, 121205, Russia · Central University, 7 Gasheka St., bld. 1, Moscow, 123056, Russia · Sirius University of Science and Technology, 1 Olimpiyskiy Avenue, Sirius, 354340, Russia
Abstract
Many control tasks require driving the system into a prescribed goal set and subsequently keeping it there. The considered setting assumes that a working controller is available that provably reaches and maintains the goal, albeit with suboptimal performance. This controller is referred to as the fallback policy. Conversely, although state-of-the-art reinforcement-learning (RL) algorithms can discover policies with impressive empirical performance, the neural networks at their core act as opaque black boxes learned through data-driven exploration and, as such, provide no theoretical mechanism to guarantee that the system will ever reach---and stay within---the prescribed goal set. This article presents CALF-Wrapper, a method that combines the fallback policy with a pretrained RL base policy and uses the critic estimate associated with the base policy to select between them during deployment. The theoretical analysis establishes that CALF-Wrapper inherits the goal-reaching property of the fallback policy under the stated assumptions and provides probability bounds for the settling time and the maximum distance from the goal set over a prescribed time interval. The experimental study evaluates CALF-Wrapper in four simulated control environments of different complexity. Comparisons include the base policy, fallback policy, and constrained reinforcement-learning methods formulated for the same objective. Sensitivity and ablation studies provide direct hyperparameter recommendations. The experiments show how CALF-Wrapper can provide effective task performance while attaining the maximum possible observed goal-reaching rate. CALF-Wrapper requires no additional dynamics model, online learning, or constrained optimization during execution.
Figures & tables
| Aspect | Original CALF [ 8 ] | CALF-Wrapper (Algorithm 1 ) |
|---|---|---|
| Operation | Online learning | Deployment after training |
| Policy | Actions are obtained by optimizing the critic over the action space or are generated by an actor model | Fixed pretrained base policy |
| Critic | The critic model is updated online through constrained optimization | The critic is loaded from the same saved training checkpoint as the base policy and remains fixed during deployment |
| Online optimization | Critic-loss minimization subject to a decrease constraint and critic bounds | No critic-weight or action optimization |
| Decision at each step | The constrained critic-update problem is solved. The action generated by the learning algorithm is applied if a feasible solution is found; otherwise, the fallback-policy action is applied. | The decision requires only one critic forward pass, one uniform random-number sample, and scalar comparisons. No optimization problem is solved. |
| RL training | Modifies the critic update | Does not modify training |
| Approach | Objective or property | Operation during execution | Principal requirements | Relation to policy training |
|---|---|---|---|---|
| Simplex and shielding [ 9 , 10 , 11 ] | Formal safety-specification satisfaction | Switches controller or rejects or replaces the proposed action | Runtime monitor and certified backup controller, or model or abstraction and safety specification | May be constructed separately or incorporated into training |
| Predictive safety filters [ 12 , 13 ] | State and input constraint satisfaction | Modifies the proposed action through a model-based feasibility or optimization problem | Dynamics model, constraints, and feasible set or certificate | Can wrap a fixed policy and solves an online problem |
| Recovery and intervention methods [ 14 , 15 , 16 ] | Constraint satisfaction or reduced violation probability | Replaces the proposed action with a recovery, backup, or planned action | Safety critic and recovery or backup policy, or dynamics model and planner | Requires additional learning, online planning, or both |
| Constrained safe RL [ 17 , 18 , 19 ] | Expected return subject to expected cumulative cost limits | Executes the trained policy directly | Reward, constraint-cost signal, and cost limit | Includes cost constraints in policy training |
| PPO-Lagrangian and TD3-Lagrangian implemented and used as baselines in this article [ 19 , 7 , 5 ] | Expected return subject to a bound on failure probability under the environment-specific goal-reaching criterion | Executes the trained policy directly | Goal-set predicate, episode horizon, binary cost, and cost limit | Adds cost estimation and Lagrange-multiplier updates to PPO and TD3 training |
| CALF-Wrapper | Goal-reaching property inherited from the fallback policy | Selects the base-policy or fallback-policy action at each step | Base policy, its fixed value-function estimate, and fallback policy satisfying the goal-reaching property | Keeps all components fixed and solves no online optimization problem |
| Environment | Upper clipping value |
|---|---|
| Pendulum | |
| CartPole Swing-Up | |
| Treasure-Collecting Robot | |
| Contaminated-Zone AUV Navigation |
| Mode | Target | ||||
|---|---|---|---|---|---|
| Conservative | 0 | 0.00 | 0 | 0 | 0 |
| Guarded | 1 | 0.10 | 0.950002 | 0.990000 | 0.993334 |
| Moderate | 1 | 0.35 | 0.986695 | 0.997343 | 0.998229 |
| Balanced | 1 | 0.50 | 0.992010 | 0.998406 | 0.998937 |
| High | 1 | 0.70 | 0.996178 | 0.999238 | 0.999492 |
| Almost Open | 1 | 0.95 | 0.999480 | 0.999896 | 0.999931 |
| Environment | |||
|---|---|---|---|
| Pendulum | |||
| CartPole Swing-Up | |||
| Contaminated-Zone AUV | |||
| Treasure-Collecting Robot |
| Method | Low-return checkpoint | Mid-return checkpoint | High-return checkpoint | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Episode return | Goal- reaching rate (%) | Fallback actions (%) | Episode return | Goal- reaching rate (%) | Fallback actions (%) | Episode return | Goal- reaching rate (%) | Fallback actions (%) | |
| Pendulum-v1 (PPO) | |||||||||
| Fallback | |||||||||
| Conservative | |||||||||
| Guarded | |||||||||
| Moderate | |||||||||
| Env. | Mode | Fallback policy actions (%) | Base- policy latency ( s) | Critic latency ( s) | Other operations latency ( s) | CALF- Wrapper latency ( s) | Added latency ( s) (95% CI) |
|---|---|---|---|---|---|---|---|
| Pendulum (PPO) | Conserv. | 89.8 | 73.6 | 64.9 | 0.013 | 68.9 | [ , ] |
| Guarded | 84.6 | 0.014 | 76.3 | [ , ] | |||
| Moderate | 76.0 | 0.016 | 84.6 | [ , ] | |||
| Balanced | 71.3 | 0.014 | 85.4 | [ , ] | |||
| High | 65.2 | 0.015 | 95.2 | [ , ] | |||
| Almost Open | 58.8 | 0.014 | 98.7 | [ , ] |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Comparison with fallback policy | Comparison with base policy | ||||||
|---|---|---|---|---|---|---|---|
| Environment | Mode | Low | Mid | High | Low | Mid | High |
| Pendulum | Conservative | ||||||
| Guarded | |||||||
| Moderate | |||||||
| Balanced | |||||||
| High | |||||||
| Environment | Mode | Low | Mid | High |
|---|---|---|---|---|
| Pendulum | Conservative | |||
| Guarded | ||||
| Moderate | ||||
| Balanced | ||||
| High | ||||
| Almost Open |
RL Control
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
See all 298 papers