cs.LGApr 6, 2026

Stratifying Reinforcement Learning with Signal Temporal Logic

Authors: Justin Curry, Alberto Speranzon

Organizations: University at Albany, State University of New York, Department of Mathematics and Statistics, 1400 Washington Ave., Albany, NY 12202 USA · Lockheed Martin, Advanced Technology Labs, 1303 Corporate Center Dr, Eagan, MN 55121 USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this paper, we develop a stratification-based semantics for Signal Temporal Logic (STL) in which each atomic predicate is interpreted as a membership test in a stratified space. This perspective reveals a novel correspondence principle between stratification theory and STL, showing that most STL formulas can be viewed as inducing a stratification of space-time. The significance of this interpretation is twofold. First, it offers a fresh theoretical framework for analyzing the structure of the embedding space generated by deep reinforcement learning (DRL) and relates it to the geometry of the ambient decision space. Second, it provides a principled framework that both enables the reuse of existing high-dimensional analysis tools and motivates the creation of novel computational techniques. To ground the theory, we (1) illustrate the role of stratification theory in Minigrid games and (2) apply numerical techniques to the latent embeddings of a DRL agent playing such a game where the robustness of STL formulas is used as the reward. In the process, we propose computationally efficient signatures that, based on preliminary evidence, appear promising for uncovering the stratification structure of such embedding spaces.

Figures & tables

Fig. 1: A disk with a line attached can be stratified via the face-relation poset, which refines the dimension poset 0<1<20<1<2 .
Fig. 2: Empty (left) and Door-Key (right) Minigrid Environments [ 10 ]
Fig. 3: An empty 3×33\times 3 Minigrid Environment is first translated into a cell complex with 49 cells (9 two-dimensional cells, labeled a through i , 24 edges, and 16 vertices), which can be viewed as a finite “digitized” topological space with 49 points. Such a topologized minigrid environment has a continuous map to the tree poset displayed right.
Fig. 4: Stratifying a “Coin Collection” Game inspired by [ 12 ] .
Fig. 5: Example of an environment where an agent can move within a room or traverse a winding corridor. Dots are agent’s position samples.
Fig. 6: Local dimensions from various algorithms using sklearn library. Top-left: [ 14 ] , top-right: [ 15 ] , bottom-left: [ 13 ] and bottom-right: [ 16 ] .
Fig. 7: Top – VGT curves for three points in the environment in Figure 5 : connection point between room and corridor, point a (blue), center of the room, point b (red), middle minimum of the corridor point c (green). Bottom – Smoothed derivative of the VGT curves.
Fig. 8: Stratification detection via HADES [ 18 ] (top) and stratification signatures via Dimension Induced Clustering (middle) (DIC) [ 17 ] and VGT-dot (bottom). The colorization of the middle and bottom figures is obtained by using Kmeans clustering on the DIC and VGT-dot feature, respectively.
Fig. 9: Tubular Neighborhood around a closed interval.
Fig. 10: Screenshots of the environment and the observation used as an input to the transformer network. The light orange avatar represents the agent. The left images show the empty square occurring before or after the “eventually” operator is active and the right images with the filled square showing the instances when the temporal operator is active. The red square on the bottom is an area that must be avoided to get high reward.
Fig. 11: Main panel: UMAP embedding of the token vectors, colored by the three clusters obtained with agglomerative clustering using VGT‑dot as the feature. The colors trace a pattern that resembles the “hourglass” structure shown at the bottom of Figure 8 (the correspondence is slightly blurred by embedding noise). Inset: the same UMAP projection, now colored by time (0–194). The right‑hand point cloud corresponds to timestamps when the “eventually” operator is active, when the green square is filled.
Fig. 12: UMAP projection to 3D from a 100D DCT projection of the original 256D token space. Note that the point cloud retains the geometric “hourglass” structure from Figure 11 , whose idealized form was studied in the right column of Figure 8 . Purple points are those failing the manifold hypothesis, rejected with statistical certainty by HADES. Note how they concentrate near the “choke point” in a noisy “hourglass.”
Fig. 13: Left: UMAP embedding of the token space. Right: ISOMAP embedding of the token space. Both projections are colored by agglomerative clustering (two clusters) using the VGT‑dot as the feature. The resulting pattern mirrors that of the “eventually” game and the hourglass space, with a symmetric core cluster expanding into an outer cluster.

Explore similar work

CardsList

  1. On Tackling Complex Tasks with Reward Machines and Signal Temporal Logics

    Apr 9, 2026Ana María Gómez Ruiz, Thao Dang, Alexandre DonzéRL ControlReinforcement Learning

    UGA · VERIMAG, Université Grenoble Alpes · VERIMAG - IMAG, CNRS, UGA +1

  2. Reward Machines for Signal Temporal Logic

    Aug 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiReward Design for RLRL Control

    Department of Computer Science, University of Virginia, Charlottesville, VA, USA

  3. Distilling Formal Logic into Neural Spaces: A Kernel Alignment Approach for Signal Temporal Logic

    Mar 5, 2026Sara Candussio, Gabriele Sarti, Gaia Saveri +1Neuro-Symbolic ReasoningSignal Temporal Logic

    AILab, MIGe, University of Trieste, IT · Khoury College of Computing Sciences, Northeastern University, Boston, MA, USA