Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

Akhil Bagaria; Anita De Mello Koch; George Konidaris

cs.AISep 29, 2026

Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

Authors: Akhil Bagaria, Anita De Mello Koch, George Konidaris

Organizations: Brown University

Abstract

Temporal abstraction via options can improve exploration in large environments. However, existing option discovery algorithms find subgoals that target all aspects of the state simultaneously. This state-reaching approach produces options that only apply in narrow regions of the state-space, eventually causing an explosion in the number of options that overwhelms the agent, and impedes progress on its primary task of reward maximization. We introduce an algorithm that instead identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration. Our approach learns abstract, transferrable options and achieves rapid exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.

Figures & tables

Figure 2: Illustration of counterfactual substitution. The most boring and most novel states, $s_{B}$ and $s_{N}$ respectively, are input into the Counterfactual Generator, which outputs counterfactual states $c_{1},c_{2},c_{3}$ —each with one feature reset to its value in $s_{B}$ . Only resetting the first feature (black $\rightarrow$ blue) substantially lowers novelty, indicating that the first feature accounts for most of the change in novelty $\Delta_{n}$ . This one-at-a-time reset is effective when features contribute independently.

Panel of Figure 3 — Figure 3: Examples of identifying relevant features in MontezumasRevenge . For each state (black and white image), we show the estimated Shapley value attributed to each feature in the image and the bounding boxes selected for the subgoal classifier. The first classifier checks if the player has entered room 0, while the second classifier checks if the player has the sword near the skulls.

Figure 4: Example of a discovered subgoal classifier that transfers to many different rooms. The leftmost image shows the context in which this subgoal classifier was learned—the bounding box shows that the classifier only attends to the player’s position. The remaining three images show some states in which that classifier was triggered—as long as the player re-appears in that portion of the screen, other factors are all irrelevant for achieving that option’s subgoal.

Figure 5: Learning curves comparing our agent ( Abstract Subgoals ) with a non-hierarchical novelty maximizing RL method ( CFN ), a vanilla RL method ( R2D2 ), and a state-reaching HRL baseline ( Pixel Equality ). Solid lines denote average undiscounted return; shaded regions denote standard deviation over

5

random seeds.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Introduced
MDP and options
$M=(S,A,R,T,\gamma)$	Markov decision process	Sec. 2
$o=(\mathcal{I}_{o},\pi_{o},\beta_{o})$	Option: initiation function, policy, termination condition	Sec. 2
$\mathcal{O}$	The agent’s set of discovered options	Sec. 2
$H$	Option timeout (maximum steps per execution)	Sec. 2
$R_{o}(s)=\beta_{o}(s)$	Option pseudo-reward	Sec. 2

Appendix

Table 1: Summary of notation.

Figure 6: Illustration of the core insight. Suppose an agent observes a state trajectory of length $7$ and a novelty estimator assigns each state $s_{t}$ in that trajectory a novelty score $f_{\phi}^{\text{int}}(s_{t})$ . $s_{N}$ denotes the most novel state, and $s_{B}$ denotes the most boring state in that trajectory. Suppose that each state has $3$ features; the colored boxes represent feature values for those two states. Rather than simply treating $s_{N}$ as a target state, we seek to identify which features in particular were responsible for the large difference in novelty $\Delta_{n}=f_{\phi}^{\text{int}}(s_{N})-f_{\phi}^{\text{int}}(s_{B})$ . Our algorithm can handle input images, but we show state features here for simplicity.

Panel of Figure 7 — Figure 7: Hyperparameter sensitivity sweep in MontezumasRevenge . Each curve is the average undiscounted task return across 3 random seeds; shaded regions denote standard deviation.

Figure 8: Learning curves comparing Abstract Subgoals and Pixel Equality given subgoals from a separate discovery-only stage. Both agents are trained on the same fixed set of novel trigger states for MontezumasRevenge , with subgoal discovery disabled; each method constructs its own classifier for these states, so the comparison isolates the effect of the classifier alone (averaged over 4 seeds).

Figure 9: DeepSHAP attribution is broadly consistent between CFN and RND novelty estimators, but CFN attribution is more concentrated. Each row compares the most-novel state independently identified by RND and by CFN at a similar stage of exploration; because the two estimators judge novelty differently, these are not necessarily the same state. The left pair of columns shows RND’s most-novel state alongside its DeepSHAP attribution map, and the right pair shows CFN’s most-novel state alongside its attribution map. Noticeably, CFN’s attributions are more concentrated around a few features, while RND’s attributions are more diffused across all features. Despite this, the resulting classifiers attend to similar features.

Figure 10: Number of options discovered in MontezumasRevenge over training for the learning curves reported in Figure 5 . Solid line denotes the mean number of options discovered; shaded area denotes standard error.

Figure 11: Discovered subgoal classifiers in MiniGrid-KeyCorridor . Bounding boxes show the patches each classifier attends to; all other pixels are ignored. (Left) Subgoal: reach a location and open the yellow door. (Middle) Subgoal: reach the mid-left room while the key is in the hallway. (Right) Subgoal: enter the locked room containing the ball, regardless of key position or door configuration.

R2D2 agent
Hyperparameter	Value
Number of actors	32
Actor backend	CPU
Sequence batch size	32
Trace length	40
Sequence period	20

Appendix

Table 2: Hyperparameters shared across all agents, domains, and methods, unless overridden in Tables 3 – 5 .

Domain	Method	$\gamma$	LR	TUP	SPI	Reward tx.	Intrinsic coeff.	CFN LR	CFN SPI	CFN min. replay
Montezuma	CFN	0.99	$1\times 10^{-4}$	600	2	signed-hyperbolic	0.01	$1\times 10^{-3}$	Unlimited	2,048
Montezuma	R2D2	0.99	$1\times 10^{-4}$	600	2	signed-hyperbolic	0.0	—	—	—
KeyCorridor	CFN	0.99	$3\times 10^{-4}$	1,200	8	identity	0.001	$1\times 10^{-4}$	Unlimited	12,500
KeyCorridor	R2D2	0.99	$3\times 10^{-4}$	600	2	identity	—	—	—	—
VisualTaxi	CFN	0.99	$3\times 10^{-4}$	600	2	identity	0.03	$1\times 10^{-4}$	8	2,048
VisualTaxi	R2D2	0.99	$3\times 10^{-4}$	600	2	identity	—	—	—	—

Appendix

Table 3: Flat RL baselines: core hyperparameters per domain. TUP = target update period; SPI = samples-per-insert ratio. “Reward tx.” is the transform applied to the R2D2 target (identity or signed-hyperbolic). “Intrinsic coeff.” is the coefficient on the CFN novelty bonus added to the training reward.

Domain	Policy	$\gamma$	LR	TUP	SPI	$\lambda$	CFN LR	CFN TUP	CFN SPI	CFN min. replay
Montezuma	Intra-option ( $\pi_{o}$ )	0.997	$1\times 10^{-4}$	600	2	—	—	—	—	—
Montezuma	Exploration ( $\pi_{\text{int}}$ )	0.99	$3\times 10^{-4}$	600	8	0.01	$1\times 10^{-3}$	600	Unlimited	12,500
KeyCorridor	Intra-option ( $\pi_{o}$ )	0.997	$1\times 10^{-4}$	500	2	—	—	—	—	—
KeyCorridor	Exploration ( $\pi_{\text{int}}$ )	0.99	$3\times 10^{-4}$	600	8	0.01	$1\times 10^{-3}$	600	Unlimited	12,500
VisualTaxi	Intra-option ( $\pi_{o}$ )	0.997	$1\times 10^{-4}$	600	2	—	—	—	—	—
VisualTaxi	Exploration ( $\pi_{\text{int}}$ )	0.99	$3\times 10^{-4}$	600	8	0.01	$1\times 10^{-3}$	600	Unlimited	12,500

Appendix

Table 4: Abstract Subgoals / Pixel Equality: policy hyperparameters. TUP = target update period; SPI = samples-per-insert ratio. Pixel Equality uses identical values (it differs only on the classifier).

\lambda

is the CFN exploration-bonus coefficient from Section 3.1 ; it is 0.01 in all three domains.

Hyperparameter	Montezuma	KeyCorridor	VisualTaxi
Option timeout, $H$	100	50	50
Goal-creation threshold multiplier, $\sigma_{\text{state}}$	1	1	1
Number of goals to replay	5	5	5
Option-initiation threshold, $\delta$	N/A (scaled by $V_{\beta_{o}}(s)$ , Appendix C )	0.1	0.1
Extrinsic-reward weight in $U(o)$ , $\alpha$	0.25	0	0
Classifier match threshold, $\xi$	$0.7$ (with $\mathcal{D}=1-\text{IOU}$ )	see Table 2	see Table 2

Appendix

Table 5: Goal-space and option-discovery hyperparameters for Abstract Subgoals.

Hyperparameter	Value
Saliency method	DeepSHAP
SAM checkpoint	ViT-H ( sam_vit_h_4b8939.pth )
Patch attribution threshold, $\sigma_{\text{patch}}$	1.0
Number of attribution baselines, $M$	15
Baseline window, $w$	50
Classifier trigger window size, $\kappa$	8

Appendix

Table 6: MontezumasRevenge -specific classifier discovery hyperparameters.

Explore similar work

Abstraction for Offline Goal-Conditioned Reinforcement Learning
May 21, 2026Clarisse Wibault, Alexander Goldie, Antonio Villares +2Goal-Conditioned Reinforcement Learning Markov Decision Processes
FLAIR, MLRG · University of Oxford · FLAIR +1
Diversity-Enriched Option-Critic
Nov 4, 2020Anand Kamat, Doina PrecupCritic Learning Reward Functions
McGill University · Mila · 3Google deepmind
Performance-Driven Environment Abstraction with Multi-Timescale Learning
Jun 16, 2026Yue Guan, Dipankar Maity, Panagiotis TsiotrasMarkov Decision Processes Synthetic Environments
School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Department of Electrical and Computer Engineering, University of North Carolina at Charlotte, Charlotte, NC, USA

May 21, 2026, cs.LGJ/K move · Enter open · S save

Abstraction for Offline Goal-Conditioned Reinforcement Learning

Clarisse Wibault, Alexander Goldie, Antonio Villares, Maike Osborne+1

FLAIR, MLRG · University of Oxford · FLAIR+1

Goal-Conditioned Reinforcement Learning Markov Decision Processes Temporal Reasoning

Open paper AskPDF arXiv

Markov Decision Processes (MDPs) often exhibit significant redundancy due to symmetries and shared structure across state-goal pairs in real-world Goal-Conditioned Reinforcement Learning (GCRL). While hierarchical policies have been motivated for horizon reduction via temporal abstraction in offline GCRL, we demonstrate that hierarchy also enables absolute abstraction. By introducing relativised options as well as distinct representations for different levels of the hierarchy, we demonstrate how an agent can reuse experience across similar contexts of the state-space. Based on this framework, we introduce two simple algorithms for learning relativised options and abstracting from the absolute frame of reference. Our experiments show that such inductive biases significantly improve performance in offline GCRL.

Abstract

Explore similar work

Abstraction for Offline Goal-Conditioned Reinforcement Learning

Diversity-Enriched Option-Critic

Performance-Driven Environment Abstraction with Multi-Timescale Learning