cs.LGNov 9, 2026

Reward-Estimated Hypergradient for Bilevel Reinforcement Learning with Black-Box Follower

Authors: Shigeki Kusaka, Mikoto Kudo, Takumi Tanabe, Akifumi Wachi, Youhei Akimoto

Organizations: University of Tsukuba · RIKEN AIP · Institute of Science Tokyo · LY Corporation

Abstract

Hypergradient (HG) methods for bilevel reinforcement learning (RL) typically assume a white-box follower whose reward is known. In practice, however, the leader rarely knows the follower's reward function and can only observe its trajectories. We address this black-box setting and propose Reward-Estimated Hypergradients (RE-HG), which recovers the follower's reward from these trajectories via inverse RL (IRL). Yet IRL identifies the reward only up to potential-based shaping, a transformation that leaves the follower's behavior unchanged but biases the HG. We prove that, for linear rewards, this bias cancels when IRL estimates are aggregated across sufficiently diverse dynamics, which the leader's own updates naturally supply. To handle finite samples, RE-HG truncates small eigenvalues of the aggregated feature covariance, restoring the identifiable subspace and stabilizing the estimation. Experiments show that RE-HG estimates gradients aligned with a white-box Oracle and achieves comparable performance; empirically, the truncation also suppresses HG explosions, letting RE-HG avoid local optima that trap the Oracle in sharp reward landscapes.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity

    Jul 30, 2026Naman Saxena, Mudit Gaur, Vaneet AggarwalReinforcement LearningHypergradient Optimization

  2. Sample-Efficient Hypergradient Estimation for Decentralized Bi-Level Reinforcement Learning

    Mar 16, 2026Mikoto Kudo, Takumi Tanabe, Akifumi Wachi +1Hypergradient OptimizationDecentralized MARL