cs.LGNov 10, 2026

Opportunity-Cost Option-Critic: Adaptive Switching Costs for Robust Hierarchical Reinforcement Learning

Authors: Khaled Ammoura, Aya Kayal, Naseem Daher

Organizations: Department of Electrical and Computer Engineering, American University of Beirut, Beirut, Lebanon · Department of Computer Science, American University of Beirut, Beirut, Lebanon · Department of Electrical and Computer Engineering, Department of Mechanical Engineering, American University of Beirut, Beirut, Lebanon

Abstract

Deliberation costs are commonly used in hierarchical reinforcement learning (HRL) to prevent options from collapsing to primitive actions. Existing approaches rely on fixed penalties or duration-based adaptations, both of which are indirect proxies for an option's utility. This paper argues that option persistence should instead depend on the opportunity cost of abandoning the current behavior. We propose the Opportunity-Cost Option-Critic (OC2OC^2) framework, which defines a state-dependent deliberation cost using the positive value gap between the active option and its best alternative. This encourages persistence when the current option is clearly preferable and allows rapid switching when a better option emerges. We show that directly learning such costs leads to a degenerate zero-cost solution and introduces an auxiliary objective that aligns the learned cost with the value-gap target. The approach is evaluated in Four Rooms and Pinball domains under stationary, goal-shift, and geometry-shift conditions. In the Pinball goal-shift task, the OC2OC^2 framework achieves a 52% first-ten-episode success rate compared with 18% for a fixed-cost baseline. The results suggest that opportunity-cost-based persistence improves adaptation in non-stationary environments.

Figures & tables

Explore similar work

CardsList
  1. Q-Shaped Options for Hierarchical Reinforcement Learning

    Oct 8, 2026Clarisse Wibault, Antoine Gorceix, Antonio Léon Villares +5Hierarchical RLRepresentation Learning for RL

  2. On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning

    Oct 4, 2026Bingyun Liu, Yuheng JingReinforcement LearningSequential Decision Making

  3. Two-Timescale Hierarchical Reinforcement Learning for Resilient Operations

    Jul 26, 2026Young Hyun Cho, Franz Stoll, Will Wei Sun +2Non-Stationary RLHierarchical RL