Inverse Reinforcement Learning with Just Classification and a Few Regressions
Organizations: Department of Statistics, University of Washington · Netflix · Cornell University
Abstract
Inverse reinforcement learning (IRL) seeks to recover a reward function from observed behavior, but rewards are typically only partially identified: many reward--value pairs can induce the same behavior policy. We study this problem in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalization constraints, with anchor-action constraints as a special case. This leads to Generalized Policy-to--to-Reward (GenPQR), a modular approach to normalized reward recovery based on policy estimation and -evaluation via the Bellman equation; both components can be instantiated with off-the-shelf classification and regression methods. We establish modular finite-sample guarantees under general function approximation, with separate terms for policy-estimation and -function estimation error. We also give explicit bounds for GenPQR with fitted -evaluation. In experiments, GenPQR improves on DeepPQR in reward recovery. Our guarantees extend beyond anchor actions, accommodate large action spaces, and are not tied to a specific neural-network architecture or training procedure.
Figures & tables
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | MSE | Corr. | NLL | Time |
|---|---|---|---|---|
| GenPQR (BC, NN) | 0.37 0.06 | 0.81 0.01 | 1.546 0.004 | 4.74 0.06 |
| GenPQR (AIRL, NN) | 0.63 0.06 | 0.72 0.03 | 1.608 0.006 | 26.43 0.34 |
| GenPQR (BC, GBT) | 1.07 0.19 | 0.49 0.03 | 1.546 0.004 | 2.25 0.03 |
| GenPQR (AIRL, GBT) | 1.07 0.19 | 0.41 0.05 | 1.608 0.006 | 23.97 0.32 |
| DeepPQR | 0.79 0.07 | 0.69 0.02 | 1.608 0.006 | 38.24 0.42 |
| MaxEnt-IRL | 1.17 0.18 | 0.35 0.03 | 1.624 0.005 | 8.57 0.11 |
| Method | Behavior MSE | Uniform MSE | Uniform corr. | RMSE | |
|---|---|---|---|---|---|
| DeepPQR | – | – | |||
| DeepPQR, anchor projected | – | – | |||
| Anchor GenPQR | – | – | |||
| Fusion GenPQR | 100 | ||||
| Fusion GenPQR | 250 | ||||
| Fusion GenPQR | 500 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,847 papers