When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic

Bingyun Liu; Yuheng Jing

cs.LGAug 30, 2026

When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic

Authors: Bingyun Liu, Yuheng Jing

Organizations: Institute of Automation Chinese Academy of Sciences

Abstract

Option-critic learns options: sub-policies together with a learned rule for when each one hands control back. Its headline result is that performance improves as options are added. We explain that result, with theory and experiment. First, the termination rule option-critic learns by maximising return contributes nothing. When the termination test and the policy that picks options read the same values, the test fires at every step, so the learned rule is identical to always terminating. When that policy explores and the test does not, as in option-critic itself, the rule can block the exploration; there are instances where it suffers $Ω(T)$ regret while always terminating holds to $O(\log T)$ . Forcing termination at every step leaves the option-count curve intact. Second, the policy inside an option barely explores at all, so a state locks onto the first action that looked good and never updates again. We name this policy necrosis, give a state-level test for it, and find three fifths of states necrotic in a typical option. Restoring exploration repairs those states, and one option then solves the task. Third, extra options improve no option; what falls is the chance that all of them fail in the same state, from $59\\%$ to $4\\%$ , and performance follows that joint quantity.

Figures & tables

Figure 1: The benchmark, the agent, and the hypothesis. (a) FourRooms with the goal in the east doorway (star) and, for every non-goal cell, the complete set of actions that reduce the wall-respecting distance to the goal; $65$ of the $103$ cells admit two. (b) The option-critic loop and the three quantities we manipulate; everything else stays at the values of the original work. (c) An option is necrotic in a state when its policy has committed there to an action no optimal policy would take, so option multiplicity acts on $\bigcap_{o}D_{o}$ and not on any individual $D_{o}$ .

Tabular
	Relative effect (%) with 95% CI
Environment	$K=2$	$K=4$	$K=8$
No perturbation	$-23.8$ $[-37.0,\,-7.6]$	$+0.7$ $[-12.0,\,13.8]$	$+11.1$ $[7.6,\,14.2]$
Local slip $p=1/3$	$-11.6$ $[-21.3,\,-0.8]$	$-0.9$ $[-11.3,\,10.1]$	$+29.9$ $[22.1,\,36.3]$
Local slip $p=1/2$	$-18.8$ $[-26.2,\,-10.3]$	$+4.8$ $[1.0,\,9.0]$	$+37.0$ $[33.7,\,40.7]$
Teleport $p=1/3$	$-20.0$ $[-28.6,\,-10.6]$	$+1.8$ $[-4.1,\,7.3]$	$+29.9$ $[24.5,\,34.6]$

Table 1: The measured effect of learned termination reverses as the option set grows. Sparse-reward FourRooms, the setting option-critic was introduced on. Each entry is

100\times(\text{Term}-\beta{\equiv}1)/\beta{\equiv}1

on the mean step count over the last

100

training episodes, so negative favours learned termination, with

95\%

paired bootstrap intervals. The tabular grid resamples

10

rank blocks of

35

seeds per cell, the neural grid

50

individual seeds.

K{=}1

is omitted because a single option cannot change identity on termination.

Figure 2: Restoring lower-level exploration replaces the entire benefit of extra options. (a–d) Frozen exact evaluation of every checkpoint, five-kernel average, per option count. Training shares a

\varepsilon_{\mathrm{lo}}=0

prefix through episode

500

and then forks; both branches are evaluated with

\varepsilon_{\mathrm{lo}}=0

. Bands are

95\%

bootstrap intervals over

350

matched seeds. (e) Paired step reduction at episode

1000

, per kernel and averaged.

	Sparse reward: matched fork at episode 500					Constant cost
$K$	control	rescue	reduction	95% CI	seeds helped	train $\varepsilon_{\mathrm{lo}}=0$	train $\varepsilon_{\mathrm{lo}}=0.1$
1	306.66	55.42	$251.23$	$[242.59,\ 259.99]$	350/350	963.21	36.50
2	152.93	67.18	$85.75$	$[79.68,\ 91.98]$	344/350	822.31	45.21
4	79.14	69.28	$9.86$	$[6.74,\ 13.03]$	211/350	384.37	55.09
8	56.42	60.35	$-3.94$	$[-5.73,\ -2.14]$	138/350	122.46	85.03

Table 2: Repairing the lower level removes the work that extra options were doing. Left: sparse-reward tabular FourRooms, five-kernel average of expected capped transitions at the episode-1000 checkpoint, paired over

350

seeds. Right: constant-cost FourRooms,

\beta\equiv 1

agents trained with and without lower-level exploration, evaluated identically. All evaluations use

\varepsilon_{\mathrm{lo}}=0

Figure 3: Two exploration mechanisms make one option sufficient in the neural implementation. Mean steps over the last

100

training episodes with

\beta\equiv 1

throughout,

50

seeds per point,

95\%

bootstrap intervals, log scale. Numbers mark the one-option values.

Figure 4: More options make simultaneous failure rare without making any option better. Constant-cost FourRooms under teleport

p=\tfrac{1}{2}

, episode-1000 checkpoints,

350

matched seeds, threshold-free necrosis test. (a) Every (state, seed) pair in exactly one class. (b) The rate at which every option is necrotic (solid) against the mean single-option rate (faint). (c) Exact frozen performance of the same checkpoints. Learned option-critic and its

\beta\equiv 1

match coincide throughout.

Agent	$\Delta$ all necrotic	$\Delta$ single-option necrosis	$\Delta$ expected steps
learned OC, train $\varepsilon_{\mathrm{lo}}=0$	$-55.11$ $[-55.65,\,-54.57]$	$+0.31$ $[-0.21,\,0.83]$	$-739.0$ $[-749.6,\,-728.1]$
$\beta\equiv 1$ , train $\varepsilon_{\mathrm{lo}}=0$	$-54.66$ $[-55.22,\,-54.10]$	$+0.57$ $[0.02,\,1.12]$	$-726.6$ $[-739.7,\,-713.2]$
$\beta\equiv 1$ , train $\varepsilon_{\mathrm{lo}}=0.1$	$-35.49$ $[-35.98,\,-34.98]$	$+21.39$ $[20.90,\,21.88]$	$+134.9$ $[125.4,\,144.8]$

Table 3: Going from one option to eight, within seed. Constant-cost FourRooms, teleport

p=\tfrac{1}{2}

, episode-1000 checkpoints,

350

matched seeds. Necrosis rates are in percentage points and steps are expected capped transitions; negative means the eight-option agent is better. Joint necrosis collapses in every row while single-option necrosis does not improve in any of them.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	learned OC	$\beta\equiv 1$	learned OC	$\beta\equiv 1$	mean	95% CI
	Expected steps		Success probability		Paired difference in steps
1	962.49	963.21	0.038	0.037	$-0.71$	$[-2.62,\ 1.19]$
2	821.44	822.31	0.195	0.194	$-0.87$	$[-7.93,\ 6.20]$
4	377.20	384.37	0.687	0.680	$-7.17$	$[-21.34,\ 6.95]$
8	117.35	122.46	0.964	0.959	$-5.11$	$[-12.77,\ 2.57]$

Appendix

Table 4: Learned termination and forced per-step reselection are indistinguishable under exact frozen evaluation. Shortest-path FourRooms, episode-1000 checkpoints, expected capped transitions and success probability under horizon-1000 exact dynamic programming with

\varepsilon_{\mathrm{lo}}=0

, averaged over the five kernels within seed. Differences are paired over

350

matched seeds with

95\%

bootstrap intervals; negative favours learned termination. At

K{=}1

termination cannot change which option runs, so that row measures the noise floor of the seed matching.

Figure 5: Table 1 as a heat map. Last-100-episode step count of learned termination minus that of a matched $\beta\equiv 1$ agent, in absolute steps (top of each cell) and relative to the control (bottom). Blue favours learned termination and red favours $\beta\equiv 1$ . The tabular grid reverses sign completely between $K{=}2$ and $K{=}8$ ; the neural grid has no consistent direction.

Figure 6: The sparse-reward replication of Figure 4 . Episode-1000 control checkpoints of the matched-fork grid, five-kernel average over

350

matched seeds, with necrosis defined by the

1\%

optimal-action mass criterion of Section 3 . Joint necrosis falls

47.5\rightarrow 31.8\rightarrow 14.6\rightarrow 3.3

percent while the mean single-option rate stays near

50\%

, and the router’s miss rate rises to

39.2\%

; expected steps fall

306.7\rightarrow 152.9\rightarrow 79.1\rightarrow 56.4

. The effective independent option count is

\kappa=1.82

2.85

and

4.14

K=2,4,8

Figure 7: Where coverage appears, and where the router loses it. Every non-goal cell of constant-cost FourRooms under teleport $p=\tfrac{1}{2}$ , at the episode-1000 checkpoint of the spatial-medoid seed, chosen by minimum Hamming distance across all twelve panels without reference to performance. Dark grey is wall and yellow is the goal. The middle row is a $\beta\equiv 1$ agent trained without lower-level exploration; its maps are visually and numerically close to the learned-termination agent above it at every option count, which is the coverage-level statement of Section 4 .

Hierarchical RL

Hierarchical reinforcement learning studies policies that organize decisions into reusable skills, subgoals, or actions operating at different timescales.

See all 160 papers

Nov 4, 2020

Diversity-Enriched Option-Critic

Temporal abstraction allows reinforcement learning agents to represent knowledge and develop strategies over different temporal scales. The option-critic framework has been demonstrated to learn temporally extended actions, represented as options, end-to-end in a model-free setting. However, feasibility of option-critic remains limited due to two major challenges, multiple options adopting very similar behavior, or a shrinking set of task relevant options. These occurrences not only void the need for temporal abstraction, they also affect performance. In this paper, we tackle these problems by learning a diverse set of options. We introduce an information-theoretic intrinsic reward, which augments the task reward, as well as a novel termination objective, in order to encourage behavioral diversity in the option set. We show empirically that our proposed method is capable of learning options end-to-end on several discrete and continuous control tasks, outperforms option-critic by a wide margin. Furthermore, we show that our approach sustainably generates robust, reusable, reliable and interpretable options, in contrast to option-critic.

Anand Kamat, Doina Precup

McGill University · Mila · 3Google deepmind

Hierarchical RL Intrinsic Motivation Actor-Critic Methods Behavioral Diversity in RL

Ask PDF

Sep 29, 2026

Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

Temporal abstraction via options can improve exploration in large environments. However, existing option discovery algorithms find subgoals that target all aspects of the state simultaneously. This state-reaching approach produces options that only apply in narrow regions of the state-space, eventually causing an explosion in the number of options that overwhelms the agent, and impedes progress on its primary task of reward maximization. We introduce an algorithm that instead identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration. Our approach learns abstract, transferrable options and achieves rapid exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.

Akhil Bagaria, Anita De Mello Koch, George Konidaris

Brown University

Hierarchical RL RL Exploration Sparse-Reward RL Generalization in RL

Ask PDF

Oct 14, 2026

Non-asymptotic Convergence of Average-reward Q-learning with Options

We study the finite-sample convergence of Q-learning algorithms for the average-reward formulation of Reinforcement Learning (RL) with options. While options allow temporarily extended and hierarchical actions, the average-reward formulation incorporates continual tasks beyond discounted and episodic RL. We establish the first

\tilde{O}(1/\text{iterations})

convergence guarantees, in the mean-square sense, for average-reward option Q-learning algorithms in both the inter- and intra-option settings. First, we prove that both inter- and intra-option Bellman operators satisfy multi-step span contractions under unichain assumptions. Then, we propose synchronous estimators of these multi-step option Bellman operators leading to two variants of option Q-learning. Finally, we extend the stochastic approximation framework to prove convergence rates of these two algorithms under the minimal assumptions used in the literature.

Kintan Saha, Ahana Deb, Anders Jonsson +1

Indian Institute of Science · Universitat Pompeu Fabra · Univ. Lille, Inria, CNRS- CRIStAL, Centrale Lille

Q-Learning Average-Reward RL Hierarchical RL

Ask PDF

Abstract

Hierarchical RL

Explore similar work

Diversity-Enriched Option-Critic

Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

Non-asymptotic Convergence of Average-reward Q-learning with Options