Inverse reinforcement learning (IRL) seeks to recover a reward function from observed behavior, but rewards are typically only partially identified: many reward--value pairs can induce the same behavior policy. We study this problem in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalization constraints, with anchor-action constraints as a special case. This leads to Generalized Policy-to-Q-to-Reward (GenPQR), a modular approach to normalized reward recovery based on policy estimation and Q-evaluation via the Bellman equation; both components can be instantiated with off-the-shelf classification and regression methods. We establish modular finite-sample guarantees under general function approximation, with separate terms for policy-estimation and Q-function estimation error. We also give explicit bounds for GenPQR with fitted Q-evaluation. In experiments, GenPQR improves on DeepPQR in reward recovery. Our guarantees extend beyond anchor actions, accommodate large action spaces, and are not tied to a specific neural-network architecture or training procedure.
Figures & tables
Table 1
Figure 1: Comparison of DeepPQR and GenPQR using the same AIRL policy estimate and neural FQE. Entries are mean ± 95% CI over 100 seeds. Anchor count is the number of anchor-action transitions used by DeepPQR’s anchor- Q step.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
MSE ↓
Corr. ↑
NLL ↓
Time ↓
GenPQR (BC, NN)
0.37 ± 0.06
0.81 ± 0.01
1.546 ± 0.004
4.74 ± 0.06
GenPQR (AIRL, NN)
0.63 ± 0.06
0.72 ± 0.03
1.608 ± 0.006
26.43 ± 0.34
GenPQR (BC, GBT)
1.07 ± 0.19
0.49 ± 0.03
1.546 ± 0.004
2.25 ± 0.03
GenPQR (AIRL, GBT)
1.07 ± 0.19
0.41 ± 0.05
1.608 ± 0.006
23.97 ± 0.32
DeepPQR
0.79 ± 0.07
0.69 ± 0.02
1.608 ± 0.006
38.24 ± 0.42
MaxEnt-IRL
1.17 ± 0.18
0.35 ± 0.03
1.624 ± 0.005
8.57 ± 0.11
Appendix
Figure 2: Higher-sample method comparison. Entries are mean ± 95% confidence interval over 100 seeds. NN = neural FQE; GBT = boosted FQE.
Figure 3: Large-action experiments. Left: reward MSE as the number of actions grows at fixed behavior-sample size. Right: 20-action GenPQR using an independent state–outcome source. Error bars are 95% confidence intervals over 100 seeds.
Figure 4: Higher-sample comparison of reward MSE, reward correlation, and runtime. The linear reward baseline is fitted to training rewards. Error bars denote 95% confidence intervals over 100 seeds.
Figure 5: Realized anchor-action count as the action set grows with the total behavior sample held fixed.
Method
ng
Behavior MSE
Uniform MSE
Uniform corr.
g RMSE
DeepPQR
–
2.253±0.089
2.247±0.089
0.560±0.025
–
DeepPQR, anchor projected
–
2.174±0.086
2.187±0.087
0.566±0.025
–
Anchor GenPQR
–
0.443±0.064
0.454±0.064
0.790±0.011
–
Fusion GenPQR
100
0.452±0.073
0.471±0.074
0.795±0.009
0.388±0.035
Fusion GenPQR
250
0.319±0.047
0.337±0.048
0.841±0.006
0.291±0.026
Fusion GenPQR
500
0.259±0.035
0.277±0.036
0.864±0.006
0.230±0.021
Appendix
Table 1: Twenty-action complementary-source experiment. Entries are mean ± 95% confidence interval over 100 seeds. The oracle-nuisance row uses the true π,g and estimates Q by FQE.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Inverse Reinforcement Learning (IRL) aims to recover a reward function that explains expert demonstrations. Existing IRL methods typically rely on a bi-level optimization procedure that alternates between reward learning and policy optimization, leading to substantial computational burden and training instability. In this work, we introduce a different route that eliminates policy optimization entirely by leveraging diffusion policies. Our key insight is that a diffusion policy encodes the action-gradient structure of the optimal soft Q function, enabling reward learning to be cast as a sequence of value recovery problems, thereby allowing us to bypass reward-policy loops inherent in prior IRL methods. Specifically, our method proceeds in three stages: (I) recovering the optimal soft Q function via action-gradient matching and estimating the corresponding soft value function (LogSumExp of Q values) in a way inspired by Gumbel regression; (II) calibrating these soft values by inferring a state-dependent offset; (III) extracting the reward by enforcing Bellman consistency. This leads to Loop-Free Inverse Reinforcement Learning (LFIRL), a fully offline algorithm that operates in a simple, loop-free, and sequential manner. LFIRL is simple to implement and significantly improves training efficiency while maintaining strong reward recovery performance. Empirically, across Maze, Franka Kitchen, Adroit Hand Pen, and Push-T benchmarks, LFIRL achieves 2-3x speedup over the fastest baselines, while matching or surpassing state-of-the-art methods in reward recovery quality.
Yang Chen, Yitan Zhang, Michael Witbrock +1
Shanghai Artificial Intelligence Laboratory · University of Auckland
Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy π⋆ and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.
Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL--one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.
Anish Diwan, Davide Tateo, Christopher E. Mower +3
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3