Opportunity-Cost Option-Critic: Adaptive Switching Costs for Robust Hierarchical Reinforcement Learning
Authors: Khaled Ammoura, Aya Kayal, Naseem Daher
Organizations: Department of Electrical and Computer Engineering, American University of Beirut, Beirut, Lebanon · Department of Computer Science, American University of Beirut, Beirut, Lebanon · Department of Electrical and Computer Engineering, Department of Mechanical Engineering, American University of Beirut, Beirut, Lebanon
Deliberation costs are commonly used in hierarchical reinforcement learning (HRL) to prevent options from collapsing to primitive actions. Existing approaches rely on fixed penalties or duration-based adaptations, both of which are indirect proxies for an option's utility. This paper argues that option persistence should instead depend on the opportunity cost of abandoning the current behavior. We propose the Opportunity-Cost Option-Critic (OC2) framework, which defines a state-dependent deliberation cost using the positive value gap between the active option and its best alternative. This encourages persistence when the current option is clearly preferable and allows rapid switching when a better option emerges. We show that directly learning such costs leads to a degenerate zero-cost solution and introduces an auxiliary objective that aligns the learned cost with the value-gap target. The approach is evaluated in Four Rooms and Pinball domains under stationary, goal-shift, and geometry-shift conditions. In the Pinball goal-shift task, the OC2 framework achieves a 52% first-ten-episode success rate compared with 18% for a fixed-cost baseline. The results suggest that opportunity-cost-based persistence improves adaptation in non-stationary environments.
Figures & tables
Method
Steps
Discounted return
Terminations
Switches
Duration
Vanilla OC
26.56±9.06
0.794±0.053
17.11±7.31
7.41±4.51
1.46±0.06
Fixed, η=0.005
32.99±20.55
0.777±0.087
18.45±13.26
10.30±9.39
1.73±0.18
Fixed, η=0.01
30.23±8.55
0.777±0.050
11.47±3.57
5.15±2.49
2.33±0.50
Flexible cost
68.67±36.03
0.623±0.102
1.55±0.58
1.02±0.47
27.28±16.65
OC2
19.11±4.89
0.843±0.032
6.72±1.76
4.95±1.64
2.99±1.45
TABLE I : Four Rooms evaluation after 5,000 training episodes. All methods complete in 100% of evaluation episodes. Values are mean ± sample standard deviation across ten seed-level averages (each seed level averaged across ten evaluation episodes), with 100 evaluation episodes per seed. Steps number primitive transitions per episode. Discounted return is the discounted sum of rewards per episode ( γ=0.99 ). Terminations number option-ending decisions per episode (including option terminations followed by re-selection of the same option). Switches number transitions to a different option per episode. Duration is the mean number of primitive steps per execution of an option.
Fig. 1 : Four Rooms environment where the starting point is colored red and the goal is colored yellow. The choice of the starting point and goal is random.
Fig. 2 : Pinball environment layouts: a) original environment; b) relocated-goal environment preserves the obstacles shape; altered-geometry environment preserves the start and goal.
Metric
Fixed γ=.99
Fixed γ=.999
Flexible γ=.999
OC2γ=.999
Success (%)
82.60±36.46
92.56±10.18
95.28±3.64
99.20±0.38
Undiscounted return
4615.6±7289.8
4491.9±3425.8
4521.0±2384.3
6049.6±963.8
Steps per episode
2111.2±3570.1
1457.7±1144.1
1187.6±544.2
865.9±217.3
TABLE II : Pinball play on original layout in training episodes 1–250. Success is the percentage of episodes that reach the goal. Undiscounted return is the sum of per-episode rewards, steps per episode is the number of primitive transitions. Mean ± sample standard deviation over ten seed-level summaries.
Change
Metric
Fixed γ=.99
Fixed γ=.999
Flexible γ=.999
OC2γ=.999
Goal
First-10 success (%)
18.0±24.4
18.0±38.2
15.0±26.8
52.0±27.4
Success (%)
73.92±39.65
87.80±6.32
89.28±10.99
96.76±3.66
Undiscounted Return
2429.9±7438.5
2635.8±3167.0
3093.2±4160.8
7324.7±1226.4
Episode length
2757.4±3890.7
1501.3±681.4
1403.1±1069.7
561.2±352.0
Recovered seeds
8/10
10/10
10/10
10/10
Geometry
First-10 success (%)
81.0±40.1
87.0±30.9
74.0±31.7
100.0±0.0
TABLE III : Pinball performance in the 250 post-change training episodes following a goal or obstacle change. First-10 success is the percentage of the first ten post-change episodes that attained the goal. Success is the percentage across 250 post-change episodes. Undiscounted return is the sum of per-episode rewards; episode length is the number of primitive transitions in an episode, both averaged across the 250 post-change episodes. Values are mean ± sample standard deviation across ten seed-level summaries. Recovered seeds is the number of ten seeds in which the agent achieved at least 90% success for any window of 20 consecutive post-change episodes.
Fig. 3 : Pinball learning curves during training. The upper plots in each subplot show success rate over a 20-episode rolling window, while the lower plots show undiscounted episodic return. Curves represent means over ten seeds and shaded bands indicate ± one standard deviation. The environment changes after episode 250, so windows near the transition may contain episodes from both layouts.
Hierarchical reinforcement learning studies policies that organize decisions into reusable skills, subgoals, or actions operating at different timescales.
Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states. In principle, Hierarchical Reinforcement Learning (HRL) addresses both challenges through the interaction between action (temporal) and state (spatial) abstraction. First, using an action abstraction to represent temporally extended behaviour as options reduces the effective decision horizon. Second, enabling different state abstractions at each level of the decision process permits greater data aggregation for learning. However, realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction. Current HRL algorithms fail in one of two ways. Some discard distinctions between options needed for optimal control, undermining hierarchy altogether. Others retain unnecessary distinctions, preserving horizon reduction, but forfeiting coarser state abstraction. In this work, we characterise three desiderata for an action abstraction. We introduce Q-Shaped Options (QSO) to address all three. QSO builds on an architecture with distinct state-value functions, Q functions and policies at each level of the hierarchy. It learns the action abstraction between consecutive levels as a shared encoder shaped by their respective Q functions. The low-level Q function uses the option as a goal, encouraging the abstraction to retain distinctions necessary for optimal control. The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded. Across offline goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance in tasks where all other evaluated algorithms fail.
Clarisse Wibault, Antoine Gorceix, Antonio Léon Villares +5
Hierarchical reinforcement learning uses temporally extended subtasks for exploration, yet committing to their execution can restrict both deployment and policy learning. We identify and separate the resulting execution and policy suboptimality. Task and execution trees distinguish reward objectives from policy choices and decision interruption. A Unified Value Function for HRL and a four-stage Generalized Hierarchical Bellman Equation then support a common analysis of both losses. Under bounded rewards and uniform termination, we establish hierarchical policy and execution improvement results. With the remaining node policies fixed, task-subtree compatibility and node-policy optimality under the original execution mode establish when Markov execution is optimal. The resulting decomposition leads to independent execution choices for behavior, targets, and deployment. We instantiate this principle through execution improvement and one-stage or two-stage policy improvement at arbitrary hierarchy depth. Option-based and goal-conditioned experiments demonstrate complementary gains from changing execution and changing the learning target. Controlled stochastic environments show how these gains depend on stochastic transition strength and spatial structure. This framework makes execution design an explicit component of hierarchical policy optimization.
Bingyun Liu, Yuheng Jing
Institute of Automation Chinese Academy of Sciences
Unexpected shocks recur in global operations, requiring decision rules that adapt as market and operating conditions change. Many operational systems also have hierarchical structures in which long-term and short-term decisions pursue a shared objective. We study how hierarchical reinforcement learning can strengthen resilience by adapting these interdependent rules jointly. We develop a two-timescale hierarchical reinforcement learning framework that adapts long-term and short-term policies at their respective time scales. Because the policies are interdependent, we synchronize their updates and prove, to our knowledge, the first convergence guarantees for coupled two-timescale learning. Over T periods, our policies' average gap from an optimal policy pair is O(T−1/2), improving to O(logT/T) when poor decisions produce clearer profit losses. In a used-car case study, inventory replenishment is the long-term decision and customer-arrival pricing the short-term decision. Relative to the strongest partially adaptive benchmark, the framework increases mean profit by 9.2% under joint demand-supply shocks and by 11.8% under a prolonged shock scenario, while maintaining a more stable profit trajectory over time. Short-term adaptation addresses routine seasonality and one-sided disruptions by responding immediately to changing conditions. Under joint demand-supply shocks, however, it is insufficient alone; long-term adaptation is also needed to create favorable conditions for short-term decisions. Joint adaptation thus yields higher and more stable profits through disruption and recovery. Because many organizations already use hierarchical planning, the framework strengthens operational resilience without altering existing decision structures.
Young Hyun Cho, Franz Stoll, Will Wei Sun +2
Department of Statistics, Harvard University · Edwardson School of Industrial Engineering, Purdue University · Mitch Daniels School of Business, Purdue University +3