Hypergradient (HG) methods for bilevel reinforcement learning (RL) typically assume a white-box follower whose reward is known. In practice, however, the leader rarely knows the follower's reward function and can only observe its trajectories. We address this black-box setting and propose Reward-Estimated Hypergradients (RE-HG), which recovers the follower's reward from these trajectories via inverse RL (IRL). Yet IRL identifies the reward only up to potential-based shaping, a transformation that leaves the follower's behavior unchanged but biases the HG. We prove that, for linear rewards, this bias cancels when IRL estimates are aggregated across sufficiently diverse dynamics, which the leader's own updates naturally supply. To handle finite samples, RE-HG truncates small eigenvalues of the aggregated feature covariance, restoring the identifiable subspace and stabilizing the estimation. Experiments show that RE-HG estimates gradients aligned with a white-box Oracle and achieves comparable performance; empirically, the truncation also suppresses HG explosions, letting RE-HG avoid local optima that trap the Oracle in sharp reward landscapes.
Figures & tables
Method
Requires Follower’s
Update Method
Requires Hessian
Requires Repeated
True Reward
(HG Info)
(2nd Order Deriv.)
State Visits
PARL ( Chakraborty et al.,, 2024 )
✓
Deterministic
✓
✗
PBRL ( Shen et al.,, 2024 )
✓
Deterministic
✗
✗
SoBiRL ( Yang et al.,, 2025 )
✓
Stochastic
✗
✓
HPGD ( Thoma et al.,, 2024 )
✓
Stochastic
✗
✓
BC-HG ( Kudo et al.,, 2026 )
✓
Stochastic
✗
✗
Table 1: Comparison of requirements for HG estimation methods in Bilevel RL.
Figure 1: Discrete toy task. (a) State transition diagram. rL | rF represents the rewards. (b) Leader’s discounted return across 3 independent training runs. For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 runs are displayed.
Figure 2: Results on Building Thermal Control Task. (a) Cosine similarity of the HG between the estimated and Oracle models. (b) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (c) Leader’s discounted return. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 5 independent training runs are displayed.
Figure 3: Results on 2-player MG, averaged over 3 independent training runs (10 test rollouts per run). Avg. (line) and std. (shaded area) are displayed. “P” indicates the known transition probability scenario.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Component
Algorithm 1 Line
Implementation Details
Conf-MDP (Common)
QFθ†
7
Since the transition dynamics are linear-Gaussian and the reward functions are quadratic, it is derived directly from the DARE solution.
QLθ†,VLθ†
9
Analytically computed by solving the DARE.
∇QFθ†
13
Obtained via the discrete Lyapunov equation.
Conf-MDP (Easy)
HG Calculation
15
Computed analytically using the known transition dynamics and policies, which bypasses the need for empirical trajectory sampling.
Conf-MDP (Hard)
HG Calculation
15
A variance reduction baseline is adopted to stabilize stochastic gradient estimates, using the expectation E[VLθ†(st+1)] obtained via DARE as a baseline for the ∇θlogpθ term.
2-player MG (Discrete)
Surrogate Policy g^k
5, 6
Computed via Soft Value Iteration (SoftVI) if the transition probability (P) is known, or by incorporating Successor Features (SFs) to calculate expected feature values if unknown.
Appendix
Table 2: Implementation Details by Environment
Figure 6
Parameter
Symbol
Value
Leader Discount Factor
γL
0.95
Follower Discount Factor
γF
0.95
Leader Parameter Learning Rate
αL,actor
1.0×10−2
Initial Leader Parameter
θ0
0.25
Leader Iterations
-
500
Temperature ( β )
-
2.5×10−1
Appendix
Table 3: Hyperparameters
Figure 7: Discrete toy task support material.“P” indicates the known transition probability scenario. (a) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed. (b) Cosine similarity of the HG between the estimated and Oracle models. For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed.
Parameter
Symbol
Value
Leader Discount Factor
γL
0.99
Leader Actor Learning Rate
αL,actor
1.0×10−2
Leader Critic Learning Rate
αL,critic
1.0×10−3
Leader Iterations
-
300
Critic Updates per Iteration
-
10
Follower Discount Factor
γF
0.9
Appendix
Table 4: Hyperparameters
Figure 8: Results on Building Thermal Control. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed.
Name
Symbol
Value
Leader iterations
K
200
Episodes per iteration
-
100
Evaluation episodes
-
50
Leader discount factor
γL
0.9
Follower discount factor
γF
0.9
Leader learning rate
αL
0.1
Appendix
Table 5: Hyperparameters
Figure 9: Topology of the 8-Zone Building Thermal Control Environment. Nodes represent the zones, with their respective HVAC control inputs. Edges represent the heat transfer between adjacent zones, parameterized by the leader’s airflow configuration νk . Note that the VIP office (Zone 4) has no direct HVAC, highlighting the under-actuated nature of the follower’s control task.
Figure 10: Results for the relaxed setting ( WF,44=10.0 ). (a) Leader’s discounted return. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 5 independent training runs are displayed. Solid lines and shaded areas denote the avg. and std., respectively. (b) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (c) Cosine similarity of the HG between the estimated and Oracle
Figure 11: Results for the relaxed setting ( WF,44=20.0 ). (a) Leader’s discounted return. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 5 independent training runs are displayed. Solid lines and shaded areas denote the avg. and std., respectively. (b) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (c) Cosine similarity of the HG between the estimated and Oracle .
Name
Symbol
Value
Leader iterations
K
500
Episodes per iteration
-
100
Evaluation episodes
-
50
Leader discount factor
γL
0.9
Follower discount factor
γF
0.9
Leader learning rate
αL
0.1
Appendix
Table 6: Hyperparameters
Figure 12: (a) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (b) Cosine similarity of the HG between the estimated and Oracle .
Figure 13: This figure illustrates the instability caused by calculating the benefit using the online network for both functions. Specifically, the benefit is calculated as the difference between the online leader’s action-value function ( QLonline ) and the online leader’s value function ( VLonline ). For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed. “P” indicates the known transition probability scenario.
Name
Symbol
Value
Time step
Δt
0.1
Max steps per episode
T
100
Leader iterations
K
800
Episodes per iteration
-
4
Leader discount factor
γL
0.99
Follower discount factor
γF
0.99
Appendix
Table 7: Hyperparameters
Figure 14: Learning curves for three Oracle -side modifications. Lines represent the mean discounted leader return across five random seeds, and shaded regions represent one standard deviation. (a) Tuned learning rate: the Oracle learning rate is varied. (b) Gradient clipping: the hypergradient is clipped and the learning rate is varied. (c) Influence-term scaling: the influence term is multiplied by a global factor while the direct term is left unchanged. The dashed RE-HG curve is included as the baseline in each panel.
Figure 15: Comparison between the joint multi-environment IRL baseline (CAO) and RE-HG in the discrete toy environment with known transition dynamics. Lines represent the mean discounted leader return across three random seeds, and shaded regions represent one standard deviation.
Figure 16: Learning curves under reward-feature misspecification in the continuous Conf-MDP. Lines show mean discounted leader return and shaded regions show one standard deviation. The misspecified RE-HG runs use three random seeds, while the Oracle, No-Influence, and matched-setting RE-HG reference curves use five seeds. (a) Basis mismatch: the true reward contains off-diagonal interaction terms such as sisj , whereas the IRL model is restricted to diagonal quadratic features. (b) Limited update capacity and representation misspecification: the three state-cost features with the largest true weights, accounting for approximately 80.4% of the total state-cost weight, are fixed at their true values to isolate an update-capacity restriction. A complementary curve fixes the three least-excited features at incorrect values, introducing genuine representation misspecification. (c) Redundant-feature over-specification: two or five cross-term features are added to the IRL model even though their true coefficients are zero. The RE-HG reference curve is included in every panel.
Figure 17: Learning curves under suboptimal follower models in the continuous Conf-MDP. Lines represent mean discounted leader return across three random seeds (five for the RE-HG baseline), and shaded regions represent one standard deviation. (a) Myopic follower: varying Riccati iteration budget T . (b) Under-reactive follower: varying gain reduction parameter α . The RE-HG curve is included as the baseline in each panel.
Figure 18: Total-variation distance between the suboptimal follower and the exact best response over the course of training, for the same two follower models and in the same panel order as Figure 17 . (a) Myopic follower. (b) Under-reactive follower. Lines represent the mean across three random seeds, and shaded regions represent one standard deviation.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Bilevel reinforcement learning (RL) is an important framework within the literature of RL that can be used to formalize various categories of problems, such as meta-learning, hierarchical task decomposition, and reinforcement learning from human feedback (RL-HF). Most of the bilevel RL algorithms are either not scalable because of using hypergradient with Hessian, or they suffer from high sample complexity because of using penalty-based approximation methods. In this work, we propose a hypergradient-based bilevel RL algorithm using the optimality of the Boltzmann policy for the entropy regularized discounted RL objective function. Our proposed algorithm is Hessian-free and obtains an iteration complexity of O(ε−1) and state-of-the-art sample complexity of O~(ε−2) under mild regularity conditions. Further, in our convergence analysis, we are able to remove the assumption of the Polyak-Lojasiewicz (PL) condition on the outer-level objective function present in the prior state-of-the-art sample complexity work.
Many strategic decision-making problems, such as environment design for warehouse robots, can be naturally formulated as bi-level reinforcement learning (RL), where a leader agent optimizes its objective while a follower solves a Markov decision process (MDP) conditioned on the leader's decisions. In many situations, a fundamental challenge arises when the leader cannot intervene in the follower's optimization process; it can only observe the optimization outcome. We address this decentralized setting by deriving the hypergradient of the leader's objective, i.e., the gradient of the leader's strategy that accounts for changes in the follower's optimal policy. Unlike prior hypergradient-based methods that require extensive data for repeated state visits or rely on gradient estimators whose complexity can increase substantially with the high-dimensional leader's decision space, we leverage the Boltzmann covariance trick to derive an alternative hypergradient formulation. This enables efficient hypergradient estimation solely from interaction samples, even when the leader's decision space is high-dimensional. Additionally, to our knowledge, this is the first method that enables hypergradient-based optimization for 2-player Markov games in decentralized settings. Experiments highlight the impact of hypergradient updates and demonstrate our method's effectiveness in both discrete and continuous state tasks.
Mikoto Kudo, Takumi Tanabe, Akifumi Wachi +1
University of Tsukuba · RIKEN AIP · LY Corporation
We study constrained bilevel reinforcement learning in which the lower-level problem is a constrained Markov decision process with a generally parameterized policy. The lower-level solution map can be set-valued and need not be Lipschitz continuous in the upper-level variable, so standard hypergradient analyses do not apply. We therefore define the upper-level objective as the expected upper-level loss of the policy returned by a fixed stochastic lower-level algorithm, which we call the approximate response. This objective is well defined without uniqueness of the lower-level solution. To compute its gradient, we randomize the reward and constraint values used during lower-level training without changing their means, so that the upper-level variable affects training only through known probabilities. This yields an unbiased estimator of the upper-level objective's gradient that does not differentiate through the lower-level updates. We establish lower-level reward and feasibility guarantees, upper-level stationarity, and a total sample complexity of O(δℓ−6εo−2) over both levels, where δℓ is the lower-level accuracy and εo is the upper-level stationarity error. Setting δℓ=εo=ε gives O(ε−8).