cs.AINov 19, 2026

Actor-Predictor Architecture for Reinforcement Learning with Formal Specification Objectives

Authors: Josef Brozovich, Qining Zhang, Lei Ying

Organizations: University of Michigan, Ann Arbor

Abstract

One fundamental challenge in reinforcement learning (RL) is enforcing temporal safety constraints, such as satisfying formal specifications. Existing methods often transform logic formulas into automata for state augmentation and reward shaping. However, this approach assumes that automaton construction is both feasible and tractable, which limits its applicability to small specifications and linear temporal logic (LTL). This paper proposes Actor-Predictor, a framework that learns a surrogate representation of logic progress without explicitly constructing an automaton. The predictor continuously estimates whether a partial trajectory will satisfy the formal specification in a later step, and its learned features are provided as auxiliary inputs to the actor, which is then optimized using GRPO and RWR. We show theoretically that, under sufficient predictive accuracy, the learned representation is equivalent to the automaton state. Experiments on grid world and robotic tasks in Isaac Lab demonstrate that Actor-Predictor outperforms automaton-based baselines on LTL-constrained tasks. It also performs well on non-regular formal specifications, such as (anbn)(a^n b^n), for which finite-automaton-based approaches are not applicable.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies

    Jun 6, 2026Ashkan Ansarifard, Matteo Mancanelli, Elena Umili +1Reinforcement LearningTransformer-Based RL

  2. Integrating LTL Constraints into PPO for Safe Reinforcement Learning

    Mar 1, 2026Maifang Zhang, Hang Yu, Qian Zuo +3Safe RLConstrained RL

  3. Tractable Reinforcement Learning for Full Class of Signal Temporal Logic Specifications Using Spatiotemporal Tube Reward

    Sep 23, 2026Vaishnavi Jagabathula, P Sangeerth, Pushpak JagtapRobotic RLReinforcement Learning