A key challenge in scaling up Reinforcement Learning is generalizing learned behaviour. Without the ability to carry forward acquired knowledge an agent is doomed to learn each task from scratch. In this paper we develop a new formalism for transfer by virtue of state abstraction. Based on task-independent, compact observations (outcomes) of the environment, we introduce Outcome-Predictive State Representations (OPSRs), agent-centered and task-independent abstractions that are made up of predictions of outcomes. We show formally and empirically that they have the potential for optimal but limited transfer, then overcome this trade-off by introducing OPSR-based skills, i.e. abstract actions (based on options) that can be reused between tasks as a result of state abstraction. In a series of empirical studies, we learn OPSR-based skills from demonstrations and show how they speed up learning considerably in entirely new and unseen tasks without any pre-processing. We believe that the framework introduced in this work is a promising step towards transfer in RL in general, and towards transfer through combining state and action abstraction specifically.
Figures & tables
Figure 1: A simple Gridworld labelled to construct a representation. The shaded area is inaccessible to the agent and the goal cell with positive reward is indicated by the green circle.
Figure 2: On the left hand side is depicted an MDP with 5 states and the action set {a,b} . Arcs between states denote deterministic transitions following the labelled action, receiving the annotated reward r . An abstraction is shown as grouping together states. The resulting abstract states are depicted on the right.
Figure 3: Diagram detailing the multi-task transfer setting. Rectangles depict state sets while squares denote states within those sets. The dark gray overlap denotes Φα∩Φβ and the dotted arrows starting from it point at ground states in the transfer cover of ϕβ given ϕα .
Figure 4: Two transfer settings, each showing the same two states with two different state abstractions. In each setting the abstract state ψˉ2 falls outside of the transfer cover.
Figure 5: Two gridworld tasks with different layouts that belong to the same domain.
Figure 6: An outcome sequence–test for the action sequence (left,down) .
Figure 7:
Figure 8: Five tasks of a small Gridworld domain.
Figure 9:
Figure 10: Two tasks combined contain all of the subtasks that make up a third task.
Figure 11: Three tasks are depicted. Each task consists of subtasks, the abstractions of which are shown in the middle. The third task, Mγ , is made up of some of the subtasks that recur in Mα and Mβ . The mapping between subtasks of different tasks is effected by state abstractions.
Figure 12: An example hierarchical trace is depicted, produced by an option-enabled agent with primitive actions {a1,a2,a3} and options {ω1=(Φω1,πω1,βω1),ω2=(Φω2,πω2,βω2)} with corresponding state abstractions ϕω1:S↦Φω1 and ϕω2:S↦Φω2 respectively.
Figure 13: Left: depiction of a Craftworld task. The diagram on the right illustrates crafting dependencies.
Figure 14: Learning curves for an agent without options and agents with three discovered OPSR ϕσ,k options, with k ranging from 1 to 4. A higher accumulated reward is better.
Figure 15: Craftworld for OPSR 1-3.
Figure 16: Option occupancy for 100 traces following the same story. πH denotes when the high-level controller picks a primitive actions instead of an option.
Figure 17: A Lightworld task with three rooms, implemented in VGDL 2.0. The same task description can be found in ( Konidaris and Barto, 2007 ) .
Figure 18: Distribution of areas under the reward curve for target Lightworld tasks with different numbers of rooms. Three settings are shown: without options, with discovered OPSR options and with discovered agent-space options.
Figure 19: Distribution of areas under the reward curve in the Lightworld domain with transfer between and within subdomains. Results labeled ‘extra subdomain’ have always been learned from tasks with only 2 rooms, whereas results labeled ‘intra subdomain’ have been learned from tasks with the same number of rooms as indicated.
While humans readily generalize abstract concepts to more complex or larger tasks, building Reinforcement Learning (RL) systems with this ability remains elusive. Here, we present the first theoretical model of how such Out-of-Distribution (OOD) generalization can be achieved in RL agents. Our approach considers Partially Observable Markov Decision Processes (POMDPs) and assumes that an intelligent agent uses an abstraction function to determine which experiences can be treated as equivalent and which must be distinguished. First, we extend the existing state abstraction framework and proof techniques to POMDPs. Then, we define a successor-weighted model reduction, a model reduction variant that enables compression into smaller abstract spaces than prior definitions allow. We derive a bound on the agent's OOD test performance, thereby defining the conditions under which OOD generalization is achievable. This bound decomposes an agent's performance loss into approximation and estimation errors, revealing how reducing an agent's abstract state space size improves test performance and OOD generalization. Our analysis suggests that constraining an agent to operate over a small, finite set of abstract states is necessary for achieving generalization to more complex tasks. Our results motivate further research into learning RL architectures that scale across tasks of varying complexity levels.
Nasehatul Mustakim, Lucas Lehnert
Department of Computer Science University of Saskatchewan Saskatoon, Saskatchewan, Canada
State abstraction plays a key role in scaling reinforcement learning to complex but structured systems. In studying such systems, a wide range of behavioral structures have been studied in reinforcement learning, including value functions, invariants, bisimulation relations, and behavioral metrics. However, a general principle for determining what structures are provably preserved under state abstraction is still lacking. In this paper, we present a unified framework for defining and analyzing behavioral structures in reinforcement learning. Our framework provides a compositional way to specify behavioral semantics based on local, one-step descriptions of system dynamics. Using this framework, we establish results showing how behavioral structures can be safely transferred between abstract and concrete systems. We further show how to construct quantitative metrics from logical behavioral semantics with soundness guarantees. Together, these results provide a principled foundation for reasoning about behaviors under state abstraction in reinforcement learning and offer reusable definition and proof principles for a broad class of behavioral structures in reinforcement learning.
Yivan Zhang, Ziyan Luo, Manuel Baltieri
The University of Tokyo · RIKEN AIP · Mila - Quebec Artificial Intelligence Institute +3
Reinforcement learning has achieved remarkable success in learning complex control policies, yet its applicability remains limited due to sample inefficiency and poor generalization across tasks. In this work, we propose RepMT-SAC, a framework for multi-task RL that enables efficient knowledge sharing and robust transfer to new tasks. RepMT-SAC uses spectral MDP decomposition to capture transferable dynamics, structuring the value function into a task-agnostic core with a minimal task-specific adjustment. This design allows for strong zero-shot performance on in-distribution tasks and rapid few-shot adaptation to out-of-distribution tasks. We evaluate RepMT-SAC on quadcopter trajectory-following tasks across in-distribution and out-of-distribution contexts, demonstrating that it outperforms baselines by up to 30%.
Aryan Naveen, Haitong Ma, Haldun Balim +1
Massachusetts Institute of Technology · Harvard School of Engineering and Applied Sciences