Joint Learning of Hierarchical Neural Options and Abstract World Model
Organizations: Cornell University · Google Deepmind
Abstract
Building agents that can perform new skills by composing existing skills is a long-standing goal of AI agent research. Towards this end, we investigate how to efficiently acquire a sequence of skills, formalized as hierarchical neural options. However, existing model-free hierarchical reinforcement algorithms need a lot of data. We propose a novel method, which we call AgentOWL (Option and World model Learning Agent), that jointly learns -- in a sample efficient way -- an abstract world model (abstracting across both states and time) and a set of hierarchical neural options. We show, on a subset of Object-Centric Atari games, that our method can learn more skills using less data than baseline methods and possesses learning and generalization capabilities that the baselines do not have.
Figures & tables
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Options mastered after 1M steps |
| DQN | |
| Goal-conditioned DQN | |
| Hierarchical DQN | |
| AgentOWL w/ Gemini models ( Figure 5 ) | |
| AgentOWL w/ Qwen3.5-9B |
| Method | Options mastered after 5M steps |
| DQN | |
| Goal-conditioned DQN | |
| Hierarchical DQN | |
| AgentOWL w/ manually ordered goals ( Figure 5 ) | |
| AgentOWL w/ LLM-ordered goals | |
| AgentOWL w/ randomly ordered goals |
| Hyperparameter | Value |
| Number of parallel environments | |
| Batch size | |
| Learning rate | |
| Replay buffer size | |
| Multi-step return | |
| Discount factor |
| Hyperparameter | Value |
| Max option executime time | |
| Gradient steps per sample | |
| Annealing schedule | |
| Hyperparameter | Value |
| Multi-step return | |
| Random exploration rate |
| Here is the current observation of the game {game_name}: {cur_obs} And here is a list of goals we know how to achieve: {achieved_goal_names_and_descriptions} Your task is to list 1 achieved goals that, on its own, can act as a possible stepping stone to achieve the target goal of '{target_goal_name}' -- Description: '{target_goal_description}'. Required reasoning process: First, discuss out loud how to achieve the target goal of '{target_goal_name}', taken into account the current observation. Then, for each achieved goal, discuss out loud how completing that goal would help us achieve the target goal of '{target_goal_name}', taken into account the current observation. Make sure to go through all achieved goals. But also do not keep repeating the same achieved goal. The current position of the player is irrelevant. Final output format: After you are done reasoning, list the achieved goal in a numbered list with the following format: Possible stepping stone 1: <achieved goal> |
| I'll give you an input list of objects. I want you to list 4 possible features that the input list of objects has that allows us to achieve a certain goal. Here's an example: Example input list of objects: player object with at (x=16, y=104, w=8, h=21), wall object with at (x=136, y=148, w=7, h=32), logs object with at (x=125, y=118, w=6, h=14), stairpit object with at (x=76, y=122, w=8, h=6), stair object with at (x=78, y=136, w=4, h=42), platform object with at (x=8, y=179, w=152, h=1), platform object with at (x=8, y=125, w=152, h=1), playerscore object with at (x=38, y=9, w=30, h=8), lifecount object with at (x=23, y=22, w=1, h=8), lifecount object with at (x=21, y=22, w=1, h=8), timer object with at (x=31, y=22, w=37, h=8), portal_0 object with at (x=7, y=85, w=1, h=40), portal_1 object with at (x=155, y=85, w=1, h=40), Interaction -- player object with at (x=16, y=104, w=8, h=21) is touching platform object with at (x=8, y=125, w=152, h=1) Example possible features that allow us to achieve the goal of '{goal}': 1. AnyObjTypeTouching: The player object touches a platform object 2. SpecificObjTouching: The player object touches the platform object located at (x=8, y=125) 3. AnyObjTypeTouching: ... 4. SpecificObjTouching: ... Now, I want you to list 4 possible features of the input list of objects has that allows us to achieve the goal of '{goal}'. Input list of objects: {input} Please follow these rules for your output: 1. Do not explain -- simply list each feature 2. Make the features diverse 3. Do use interactions (what the player is touching), as they usually make good features 4. Each rule should of type 'AnyObjTypeTouching' or 'SpecificObjTouching' |
| I'll give you an input list of objects. I want you to list 4 possible features that the input list of objects has that allows us to achieve a certain goal. Here's an example: Example input list of objects: player object (x=30, y=150, w=8, h=12), car object (x=27, y=163, w=20, h=14), score object (x=75, y=8, w=30, h=8), clock object (x=67, y=19, w=30, h=8), roomnumber_+0 object (x=0, y=0, w=0, h=0), portal_to_prev_room object (x=8, y=27, w=5, h=150), portal_to_next_room object (x=155, y=27, w=5, h=150), platform object (x=8, y=177, w=152, h=1), Example possible features that allow us to achieve the goal of '{goal}': 1. RoomNumberExist: An object with type 'roomnumber_+0' exists 2. ObjTouchingAndRoomNumberExist: The car object touches the platform object and an object with type 'roomnumber_+0' exists Now, I want you to list 2 possible features of the input list of objects has that allows us to achieve the goal of '{goal}'. Input list of objects: {input} Please follow these rules for your output: 1. Do not explain -- simply list each feature 2. Each rule should of type 'RoomNumberExist' or 'ObjTouchingAndRoomNumberExist' 3. Make sure to mention the roomnumber in the feature, e.g., 'an object with type 'roomnumber_+0' exists' |
Agent Skill Learning
Agent skill learning develops reusable procedures and behaviors that help AI agents solve new tasks and adapt from experience.
See all 185 papers