A key challenge in scaling up Reinforcement Learning is generalizing learned behaviour. Without the ability to carry forward acquired knowledge an agent is doomed to learn each task from scratch. In this paper we develop a new formalism for transfer by virtue of state abstraction. Based on task-independent, compact observations (outcomes) of the environment, we introduce Outcome-Predictive State Representations (OPSRs), agent-centered and task-independent abstractions that are made up of predictions of outcomes. We show formally and empirically that they have the potential for optimal but limited transfer, then overcome this trade-off by introducing OPSR-based skills, i.e. abstract actions (based on options) that can be reused between tasks as a result of state abstraction. In a series of empirical studies, we learn OPSR-based skills from demonstrations and show how they speed up learning considerably in entirely new and unseen tasks without any pre-processing. We believe that the framework introduced in this work is a promising step towards transfer in RL in general, and towards transfer through combining state and action abstraction specifically.
Figures & tables
Figure 1: A simple Gridworld labelled to construct a representation. The shaded area is inaccessible to the agent and the goal cell with positive reward is indicated by the green circle.
Figure 2: On the left hand side is depicted an MDP with 5 states and the action set {a,b} . Arcs between states denote deterministic transitions following the labelled action, receiving the annotated reward r . An abstraction is shown as grouping together states. The resulting abstract states are depicted on the right.
Figure 3: Diagram detailing the multi-task transfer setting. Rectangles depict state sets while squares denote states within those sets. The dark gray overlap denotes Φα∩Φβ and the dotted arrows starting from it point at ground states in the transfer cover of ϕβ given ϕα .
Figure 4: Two transfer settings, each showing the same two states with two different state abstractions. In each setting the abstract state ψˉ2 falls outside of the transfer cover.
Figure 5: Two gridworld tasks with different layouts that belong to the same domain.
Figure 6: An outcome sequence–test for the action sequence (left,down) .
Figure 7:
Figure 8: Five tasks of a small Gridworld domain.
Figure 9:
Figure 10: Two tasks combined contain all of the subtasks that make up a third task.
Figure 11: Three tasks are depicted. Each task consists of subtasks, the abstractions of which are shown in the middle. The third task, Mγ , is made up of some of the subtasks that recur in Mα and Mβ . The mapping between subtasks of different tasks is effected by state abstractions.
Figure 12: An example hierarchical trace is depicted, produced by an option-enabled agent with primitive actions {a1,a2,a3} and options {ω1=(Φω1,πω1,βω1),ω2=(Φω2,πω2,βω2)} with corresponding state abstractions ϕω1:S↦Φω1 and ϕω2:S↦Φω2 respectively.
Figure 13: Left: depiction of a Craftworld task. The diagram on the right illustrates crafting dependencies.
Figure 14: Learning curves for an agent without options and agents with three discovered OPSR ϕσ,k options, with k ranging from 1 to 4. A higher accumulated reward is better.
Figure 15: Craftworld for OPSR 1-3.
Figure 16: Option occupancy for 100 traces following the same story. πH denotes when the high-level controller picks a primitive actions instead of an option.
Figure 17: A Lightworld task with three rooms, implemented in VGDL 2.0. The same task description can be found in ( Konidaris and Barto, 2007 ) .
Figure 18: Distribution of areas under the reward curve for target Lightworld tasks with different numbers of rooms. Three settings are shown: without options, with discovered OPSR options and with discovered agent-space options.
Figure 19: Distribution of areas under the reward curve in the Lightworld domain with transfer between and within subdomains. Results labeled ‘extra subdomain’ have always been learned from tasks with only 2 rooms, whereas results labeled ‘intra subdomain’ have been learned from tasks with the same number of rooms as indicated.