Tractable Reinforcement Learning for Full Class of Signal Temporal Logic Specifications Using Spatiotemporal Tube Reward

Vaishnavi Jagabathula; P Sangeerth; Pushpak Jagtap

cs.ROSep 23, 2026

Tractable Reinforcement Learning for Full Class of Signal Temporal Logic Specifications Using Spatiotemporal Tube Reward

Authors: Vaishnavi Jagabathula, P Sangeerth, Pushpak Jagtap

Organizations: Centre for Cyber-Physical Systems, IISc, Bangalore, India

Abstract

This paper addresses the control problem for robotic systems, including non-holonomic and underactuated platforms operating under unknown dynamics and strict actuator limits to satisfy complex high-level specifications. We denote these high-level specifications using Signal Temporal Logic (STL) and propose a novel time-aware Reinforcement Learning (RL) framework that leverages the geometric properties of Spatiotemporal Tubes (STTs). While traditional analytical STT controllers often struggle to enforce input constraints, and existing RL approaches rely on memory-intensive state history, our method natively overcomes both limitations. By mapping the logical and temporal complexities of the full class of STL into time-varying geometric boundaries, we directly constrain the multidimensional system state without relying on scalar robustness metrics. Augmenting the state space with time, we train a time-aware Soft Actor-Critic (SAC) agent using a continuous, geometry-aware reward function that eliminates the need to explicitly evaluate complex logical semantics during execution. The proposed framework offers a history-free, computationally efficient approach to learn continuous control policies that ensure robust satisfaction of specifications while strictly adhering to system input constraints.

Figures & tables

Method	Full STL Class	Input Bounds	History- Free	Non- Holonomic
Abstraction [ 4 ]	$\times$	✓	✓	✓
History RL [ 5 ]	$\times$	✓	$\times$	✓
F-MDP [ 6 ]	$\times$	✓	✓	✓
Funnel-Analyt. [ 17 ]	$\times$	$\times$	✓	$\times$
Funnel-RL [ 7 ]	$\times$	✓	✓	✓
Analyt. STT [ 11 , 9 ]	✓	$\times$	✓	$\times$

TABLE I: Comparison with other methods in literature

Fig. 1: Turtlebot satisfying $\phi_{1}$ , by reaching and staying in the goal region $G_{1}$ (blue region) within $[15,20]$ seconds and eventually region $G_{2}$ (Green) within $[33,35]$ seconds. The trained RL policy tries to keep the trajectories inside the circular STT, while satisfying the input constraints.

Fig. 2: Cart-pole satisfying $\phi_{2}$ by reaching and staying in the region $G_{1}=[4,5]$ within $[10,12]$ s, and eventually navigating to the region $G_{2}=[-4,-3]$ within $[18,20]$ s, while balancing the pole angle within $[-15^{\circ},15^{\circ}]$ after the initial swing-up phase between [0,5] s. The trained time-aware SAC policy successfully confines the cart-pole’s state within the STT to guarantee robust satisfaction of the spatial and temporal constraints.

Fig. 3: System trajectory and the generated STTs for the Spacecraft Case Study satisfying the specification $\phi_{3}$ .

Fig. 4: Comparison of the control input effort using the analytical approach from [ 9 ] (Top) versus our proposed time-aware SAC RL approach (Bottom), demonstrating strict adherence to input constraints.

Explore similar work

Reward Machines for Signal Temporal Logic
Aug 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiReward Design for RL RL Control
Department of Computer Science, University of Virginia, Charlottesville, VA, USA
Shielded Reinforcement Learning Under Dynamic Temporal Logic Constraints
Mar 17, 2026Sadık Bera Yüksel, Ali Tevfik Buyukkocak, Derya AksarayConstrained RL Reinforcement Learning
Department of Electrical and Computer Engineering at Northeastern University.
Learning Spatiotemporal Tubes for Full Class of Signal Temporal Logic Tasks for Control of Unknown Systems under Input Constraints
Jul 8, 2026Ahan Basu, Ratnangshu Das, Soumyodipta Nath +2Signal Temporal Logic Multi-Agent Control
Centre for Cyber-Physical Systems Indian Institute of Science, Bengaluru, India · Electrical Engineering Department, Eindhoven Institute of Technology, Netherlands

Aug 13, 2026J/K move · Enter open · S save

Reward Machines for Signal Temporal Logic

Alper Kamil Bozkurt, Shangtong Zhang, Yuichi Motai

Department of Computer Science, University of Virginia, Charlottesville, VA, USA

Reward Design for RL RL Control Signal Temporal Logic Reinforcement Learning

Open paper AskPDF arXiv

Signal temporal logic (STL) provides a formal language for specifying real-time properties of real-valued observations, along with a quantitative robustness score for monitoring satisfaction. Control synthesis from STL specifications is of interest since manual controller design becomes infeasible as real-world systems grow in complexity. Moreover, many modern autonomous and AI-enabled systems lack accurate and complete system models, which makes optimization-based synthesis approaches unsuitable and motivates learning-based control. Prior work uses STL robustness scores as rewards in reinforcement learning (RL) to obtain control policies satisfying given specifications; however, robustness depends on execution history, leading to intractable state space expansion for general long-horizon specifications with arbitrarily nested temporal operators. This work introduces a novel automata-based approach that provides an efficient memory mechanism and associated Markovian rewards suitable for RL frameworks. Our approach constructs a timed alternating automaton from the given STL specifications, augments the state space with automaton locations and clock valuations, and derives rewards from the automaton acceptance condition. We empirically demonstrate that our approach learns policies that achieve higher robustness scores and satisfaction rates than those learned by existing approaches using robustness-based rewards.

Abstract

Explore similar work

Reward Machines for Signal Temporal Logic

Shielded Reinforcement Learning Under Dynamic Temporal Logic Constraints

Learning Spatiotemporal Tubes for Full Class of Signal Temporal Logic Tasks for Control of Unknown Systems under Input Constraints