cs.LGSep 25, 2025

Inverse Reinforcement Learning with Just Classification and a Few Regressions

Authors: Lars van der Laan, Nathan Kallus, Aurelien Bibaut

Organizations: Department of Statistics, University of Washington · Netflix · Cornell University

Abstract

Inverse reinforcement learning (IRL) seeks to recover a reward function from observed behavior, but rewards are typically only partially identified: many reward--value pairs can induce the same behavior policy. We study this problem in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalization constraints, with anchor-action constraints as a special case. This leads to Generalized Policy-to-QQ-to-Reward (GenPQR), a modular approach to normalized reward recovery based on policy estimation and QQ-evaluation via the Bellman equation; both components can be instantiated with off-the-shelf classification and regression methods. We establish modular finite-sample guarantees under general function approximation, with separate terms for policy-estimation and QQ-function estimation error. We also give explicit bounds for GenPQR with fitted QQ-evaluation. In experiments, GenPQR improves on DeepPQR in reward recovery. Our guarantees extend beyond anchor actions, accommodate large action spaces, and are not tied to a specific neural-network architecture or training procedure.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

    Sep 30, 2026Yang Chen, Yitan Zhang, Michael Witbrock +1Reinforcement Learning

  2. Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

    May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement LearningPolicy Optimization