Reinforcement Learning with Symbolic Reward Machines

Thomas Krug; Daniel Neider

cs.LGMar 3, 2026

Reinforcement Learning with Symbolic Reward Machines

Authors: Thomas Krug, Daniel Neider

Organizations: TU Dortmund University · Research Center Trustworthy Data Science and Security

Abstract

Reward Machines (RMs) are an established mechanism in Reinforcement Learning (RL) to represent and learn sparse, temporally extended tasks with non-Markovian rewards. RMs rely on high-level information in the form of labels that are emitted by the environment alongside the observation. However, this concept requires manual user input for each environment and task. The user has to create a suitable labeling function that computes the labels. These limitations lead to poor applicability in widely adopted RL frameworks. We propose Symbolic Reward Machines (SRMs) together with the learning algorithms QSRM and LSRM to overcome the limitations of RMs. SRMs consume only the standard output of the environment and process the observation directly through guards that are represented by symbolic formulas. In our evaluation, our SRM methods outperform the baseline RL approaches and generate the same results as the existing RM methods. At the same time, our methods adhere to the widely used environment definition and provide interpretable representations of the task to the user.

Figures & tables

Figure 1: Office World environment. The discrete version is displayed. The labeled environments output the labels shown at the specific positions.

Panel of Figure 2 — Figure 2: RM and SRM for the example task in the Office World. Self-loops with an output of zero are omitted in the RM.

Panel of Figure 3 — Figure 3: RM and SRM of task diagonal_run for the Office World. Self-loops in the RM with zero rewards are omitted.

Panel of Figure 4 — Figure 4: Original Mountain Car environment (left) and our version (right).

Figure 5: SRM for task ‘rml’ for our Mountain Car environment.

Panel of Figure 6 — Figure 6: Mean10 performance values for the tasks post_inner_offices and diagonal_run in the discrete Office World (Rows 1-2), and for the tasks post_inner_offices, diagonal_run, and rml in the infinite environments (Rows 3-5).

Panel of Figure 7 — Figure 7: LSRM mean10 performance values: Finite state space environment in the upper half and infinite state space environments in the lower half. The results for LSRM-GF are in the first row and for LSRM-FT in the second row.

Figure 8: Learned SRM by LSRM-FT for task post_inner_offices in the discrete Office World.

Figure 9: Learned SRM by LSRM-FT for task post_inner_offices in the continuous Office World. For better visualization, we apply our rounding function

\mathrm{round^{\prime}}

that rounds the input value to two decimal places. The function

\mathrm{round^{\prime}}

is defined by

\mathrm{round^{\prime}}(x)=\begin{cases}\frac{\left\lfloor x\cdot 100\right\rfloor}{100}&\text{, if }\left\lfloor x\cdot 100\right\rfloor\bmod 10\geq 5,\\ \frac{\left\lceil x\cdot 100\right\rceil}{100}&\text{, if }\left\lfloor x\cdot 100\right\rfloor\bmod 10<5.\end{cases}

The learned SRM is even closer to the SRM used in the environment than the one visualized in this figure.

Panel of Figure 10 — Figure 10: Learned SRMs by LSRM-GF for task post_inner_offices in the discrete and continuous Office World.

Explore similar work

On Tackling Complex Tasks with Reward Machines and Signal Temporal Logics
Apr 9, 2026Ana María Gómez Ruiz, Thao Dang, Alexandre DonzéRL Control Reinforcement Learning
UGA · VERIMAG, Université Grenoble Alpes · VERIMAG - IMAG, CNRS, UGA +1
Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
Oct 31, 2025Kristina Levina, Nikolaos Pappas, Athanasios Karapantelakis +2Reinforcement Learning Q-Learning
Linköping University, Linköping, Sweden · Ericsson Research, Stockholm, Sweden
Pareto Q-Learning with Reward Machines
Jun 17, 2026Arnaud Lequen, Clément Legrand-Lixon, Léo SaulièresMulti-Objective Reinforcement Learning Reinforcement Learning
Linköping University, Sweden · Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStal, F-59000 Lille, France · Univ. Toulouse, INRAE-MIAT, Toulouse, France

Apr 9, 2026J/K move · Enter open · S save

On Tackling Complex Tasks with Reward Machines and Signal Temporal Logics

Ana María Gómez Ruiz, Thao Dang, Alexandre Donzé

UGA · VERIMAG, Université Grenoble Alpes · VERIMAG - IMAG, CNRS, UGA+1

RL Control Reinforcement Learning Reward Design for RL

Open paper AskPDF arXiv

We propose a Reinforcement Learning (RL) based control design framework for handling complex tasks. The approach extends the concept of Reward Machines (RM) with Signal Temporal Logic (STL) formulas that can be used for event generation. The use of STL allows not only a more efficient representation of rewards for complex tasks but also guiding the training process to converge towards behaviors satisfying specified requirements. We also propose an implementation of the framework that leverages the STL online monitoring algorithms. We illustrate the framework with three case studies (minigrid, cart-pole and high-way environments) with non-trivial tasks.

Abstract

Explore similar work

On Tackling Complex Tasks with Reward Machines and Signal Temporal Logics

Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines

Pareto Q-Learning with Reward Machines