cs.LGNov 21, 2025

Convergence and stability of Q-learning in Hierarchical Reinforcement Learning

Authors: Massimiliano Manenti, Andrea Iannelli

Organizations: Institute for Systems Theory and Automatic Control, University of Stuttgart, Pfaffenwaldring 9, Stuttgart, 70569, Germany

Abstract

Hierarchical Reinforcement Learning promises, among other benefits, to efficiently capture and utilize the temporal structure of a decision-making problem and to enhance transfer learning capabilities, but theoretical guarantees lag behind practice. In this paper, we propose a Feudal Q-learning scheme and investigate under which conditions its coupled updates converge and are stable. By leveraging the theory of Stochastic Approximation and the ODE method, we present a theorem stating the convergence and stability properties of Feudal Q-learning. This provides a principled convergence and stability analysis tailored to Feudal RL. Moreover, we show that the updates converge to a point that can be interpreted as an equilibrium of a suitably defined game, opening the door to game-theoretic approaches to Hierarchical RL. Lastly, experiments based on the Feudal Q-learning algorithm support the outcomes anticipated by theory.

Figures & tables

Explore similar work

CardsList
  1. Robust Federated Q-Learning with Almost No Communication

    Aug 24, 2026Sreejeet Maity, Aritra MitraFederated RLQ-Learning

  2. Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification

    Jul 31, 2026Anders Jonsson, Emilie Kaufmann, Gianmarco Tedeschi +1Hierarchical RLPolicy Learning

  3. Finite-Time Analysis of Q-Value Iteration for General-Sum Stackelberg Games

    Apr 6, 2026Narim Jeong, Donghwan LeeStackelberg GamesGame-Theoretic RL