Hypergradient (HG) methods for bilevel reinforcement learning (RL) typically assume a white-box follower whose reward is known. In practice, however, the leader rarely knows the follower's reward function and can only observe its trajectories. We address this black-box setting and propose Reward-Estimated Hypergradients (RE-HG), which recovers the follower's reward from these trajectories via inverse RL (IRL). Yet IRL identifies the reward only up to potential-based shaping, a transformation that leaves the follower's behavior unchanged but biases the HG. We prove that, for linear rewards, this bias cancels when IRL estimates are aggregated across sufficiently diverse dynamics, which the leader's own updates naturally supply. To handle finite samples, RE-HG truncates small eigenvalues of the aggregated feature covariance, restoring the identifiable subspace and stabilizing the estimation. Experiments show that RE-HG estimates gradients aligned with a white-box Oracle and achieves comparable performance; empirically, the truncation also suppresses HG explosions, letting RE-HG avoid local optima that trap the Oracle in sharp reward landscapes.
Figures & tables
Method
Requires Follower’s
Update Method
Requires Hessian
Requires Repeated
True Reward
(HG Info)
(2nd Order Deriv.)
State Visits
PARL ( Chakraborty et al.,, 2024 )
✓
Deterministic
✓
✗
PBRL ( Shen et al.,, 2024 )
✓
Deterministic
✗
✗
SoBiRL ( Yang et al.,, 2025 )
✓
Stochastic
✗
✓
HPGD ( Thoma et al.,, 2024 )
✓
Stochastic
✗
✓
BC-HG ( Kudo et al.,, 2026 )
✓
Stochastic
✗
✗
Table 1: Comparison of requirements for HG estimation methods in Bilevel RL.
Figure 1: Discrete toy task. (a) State transition diagram. rL | rF represents the rewards. (b) Leader’s discounted return across 3 independent training runs. For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 runs are displayed.
Figure 2: Results on Building Thermal Control Task. (a) Cosine similarity of the HG between the estimated and Oracle models. (b) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (c) Leader’s discounted return. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 5 independent training runs are displayed.
Figure 3: Results on 2-player MG, averaged over 3 independent training runs (10 test rollouts per run). Avg. (line) and std. (shaded area) are displayed. “P” indicates the known transition probability scenario.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Component
Algorithm 1 Line
Implementation Details
Conf-MDP (Common)
QFθ†
7
Since the transition dynamics are linear-Gaussian and the reward functions are quadratic, it is derived directly from the DARE solution.
QLθ†,VLθ†
9
Analytically computed by solving the DARE.
∇QFθ†
13
Obtained via the discrete Lyapunov equation.
Conf-MDP (Easy)
HG Calculation
15
Computed analytically using the known transition dynamics and policies, which bypasses the need for empirical trajectory sampling.
Conf-MDP (Hard)
HG Calculation
15
A variance reduction baseline is adopted to stabilize stochastic gradient estimates, using the expectation E[VLθ†(st+1)] obtained via DARE as a baseline for the ∇θlogpθ term.
2-player MG (Discrete)
Surrogate Policy g^k
5, 6
Computed via Soft Value Iteration (SoftVI) if the transition probability (P) is known, or by incorporating Successor Features (SFs) to calculate expected feature values if unknown.
Appendix
Table 2: Implementation Details by Environment
Figure 6
Parameter
Symbol
Value
Leader Discount Factor
γL
0.95
Follower Discount Factor
γF
0.95
Leader Parameter Learning Rate
αL,actor
1.0×10−2
Initial Leader Parameter
θ0
0.25
Leader Iterations
-
500
Temperature ( β )
-
2.5×10−1
Appendix
Table 3: Hyperparameters
Figure 7: Discrete toy task support material.“P” indicates the known transition probability scenario. (a) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed. (b) Cosine similarity of the HG between the estimated and Oracle models. For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed.
Parameter
Symbol
Value
Leader Discount Factor
γL
0.99
Leader Actor Learning Rate
αL,actor
1.0×10−2
Leader Critic Learning Rate
αL,critic
1.0×10−3
Leader Iterations
-
300
Critic Updates per Iteration
-
10
Follower Discount Factor
γF
0.9
Appendix
Table 4: Hyperparameters
Figure 8: Results on Building Thermal Control. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed.
Name
Symbol
Value
Leader iterations
K
200
Episodes per iteration
-
100
Evaluation episodes
-
50
Leader discount factor
γL
0.9
Follower discount factor
γF
0.9
Leader learning rate
αL
0.1
Appendix
Table 5: Hyperparameters
Figure 9: Topology of the 8-Zone Building Thermal Control Environment. Nodes represent the zones, with their respective HVAC control inputs. Edges represent the heat transfer between adjacent zones, parameterized by the leader’s airflow configuration νk . Note that the VIP office (Zone 4) has no direct HVAC, highlighting the under-actuated nature of the follower’s control task.
Figure 10: Results for the relaxed setting ( WF,44=10.0 ). (a) Leader’s discounted return. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 5 independent training runs are displayed. Solid lines and shaded areas denote the avg. and std., respectively. (b) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (c) Cosine similarity of the HG between the estimated and Oracle
Figure 11: Results for the relaxed setting ( WF,44=20.0 ). (a) Leader’s discounted return. For each run, performance is evaluated as the mean of 50 test rollouts. Avg. (line) and std. (shaded area) across the 5 independent training runs are displayed. Solid lines and shaded areas denote the avg. and std., respectively. (b) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (c) Cosine similarity of the HG between the estimated and Oracle .
Name
Symbol
Value
Leader iterations
K
500
Episodes per iteration
-
100
Evaluation episodes
-
50
Leader discount factor
γL
0.9
Follower discount factor
γF
0.9
Leader learning rate
αL
0.1
Appendix
Table 6: Hyperparameters
Figure 12: (a) Pearson correlation between the recovered and true rewards, weighted by the undiscounted state-action occupancy measure of the surrogate follower policy. (b) Cosine similarity of the HG between the estimated and Oracle .
Figure 13: This figure illustrates the instability caused by calculating the benefit using the online network for both functions. Specifically, the benefit is calculated as the difference between the online leader’s action-value function ( QLonline ) and the online leader’s value function ( VLonline ). For each run, performance is evaluated as the mean of 10 test rollouts. Avg. (line) and std. (shaded area) across the 3 independent training runs are displayed. “P” indicates the known transition probability scenario.
Name
Symbol
Value
Time step
Δt
0.1
Max steps per episode
T
100
Leader iterations
K
800
Episodes per iteration
-
4
Leader discount factor
γL
0.99
Follower discount factor
γF
0.99
Appendix
Table 7: Hyperparameters
Figure 14: Learning curves for three Oracle -side modifications. Lines represent the mean discounted leader return across five random seeds, and shaded regions represent one standard deviation. (a) Tuned learning rate: the Oracle learning rate is varied. (b) Gradient clipping: the hypergradient is clipped and the learning rate is varied. (c) Influence-term scaling: the influence term is multiplied by a global factor while the direct term is left unchanged. The dashed RE-HG curve is included as the baseline in each panel.
Figure 15: Comparison between the joint multi-environment IRL baseline (CAO) and RE-HG in the discrete toy environment with known transition dynamics. Lines represent the mean discounted leader return across three random seeds, and shaded regions represent one standard deviation.
Figure 16: Learning curves under reward-feature misspecification in the continuous Conf-MDP. Lines show mean discounted leader return and shaded regions show one standard deviation. The misspecified RE-HG runs use three random seeds, while the Oracle, No-Influence, and matched-setting RE-HG reference curves use five seeds. (a) Basis mismatch: the true reward contains off-diagonal interaction terms such as sisj , whereas the IRL model is restricted to diagonal quadratic features. (b) Limited update capacity and representation misspecification: the three state-cost features with the largest true weights, accounting for approximately 80.4% of the total state-cost weight, are fixed at their true values to isolate an update-capacity restriction. A complementary curve fixes the three least-excited features at incorrect values, introducing genuine representation misspecification. (c) Redundant-feature over-specification: two or five cross-term features are added to the IRL model even though their true coefficients are zero. The RE-HG reference curve is included in every panel.
Figure 17: Learning curves under suboptimal follower models in the continuous Conf-MDP. Lines represent mean discounted leader return across three random seeds (five for the RE-HG baseline), and shaded regions represent one standard deviation. (a) Myopic follower: varying Riccati iteration budget T . (b) Under-reactive follower: varying gain reduction parameter α . The RE-HG curve is included as the baseline in each panel.
Figure 18: Total-variation distance between the suboptimal follower and the exact best response over the course of training, for the same two follower models and in the same panel order as Figure 17 . (a) Myopic follower. (b) Under-reactive follower. Lines represent the mean across three random seeds, and shaded regions represent one standard deviation.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.