Reinforcement Learning (RL) has shown promise in various robotics applications, yet its deployment on real systems is still limited due to safety and operational constraints. The safe RL field has gained considerable attention in recent years, which focuses on imposing safety constraints throughout the learning process. However, real systems often require more complex constraints than just safety, such as periodic recharging or time-bounded visits to specific regions. Imposing such spatio-temporal tasks during learning still remains a challenge. Signal Temporal Logic (STL) is a formal language for specifying temporal properties of real-valued signals and provides a way to express such complex tasks. In this paper, we propose a framework that leverages sequential control barrier functions and model-free RL to ensure that the given STL tasks are satisfied throughout the learning process. Our method extends beyond traditional safety constraints by enforcing rich STL specifications, which can involve visits to dynamic targets with unknown trajectories. We also demonstrate the effectiveness of our framework through various simulations.
Figures & tables
Fig. 1 : Overview of the proposed framework. At each time step, the RL agent outputs an unconstrained control input utRL , which is then adjusted by a corrective input utCBF computed using sequential CBFs to satisfy the given STL task. The final control u~t is applied to the system, and the resulting next state and reward (xt+1,rt) are fed back to the RL agent.
Fig. 2 : Simulation environment setups: (a) Case 1 and (b) Case 2.
Fig. 3 : Learning curve comparisons of SAC models trained without STL tasks (orange) and with the STL specifications Φ1 (blue) and Φ2 (green). Results are averaged over 10 independent training runs, and the shaded regions represent the confidence intervals of two standard deviations.