Inferring Transferable Rewards via Active Inverse Reinforcement Learning
Organizations: University of Neuchˆatel · University of Zurich · EPFL
Abstract
Inverse Reinforcement Learning (IRL) aims to recover expert rewards that transfer across environments with differing dynamics. Prior work has shown that such transfer is possible when gathering expert demonstrations across sufficiently different dynamics. In practice, however, expert demonstrations are costly, making it critical to select the right dynamics to reduce the quantity of expert data required for learning transferable rewards. We propose a two-phase approach: we first cycle through the available dynamics to obtain a reliable reward estimate, then actively select dynamics to reduce our uncertainty about the expert reward. Crucially, this yields novel sample complexity guarantees on transfer that hold for any dynamics, with a rate independent of the number of available dynamics. Empirically, our method significantly reduces the number of expert demonstrations needed to learn a transferable reward.
Figures & tables
| Training set type | ||
| Algorithm | Uniform | Arbitrary |
| Active only | ||
| Two-phase | ||
| ED | ||
| Round-robin | ||
| Fixed dynamics | ||
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Problem setting | ||
| finite set of dynamics available for expert demonstrations | ||
| number of rounds, and expert trajectories per round | ||
| expert reward parameters, and | ||
| , optimal regularized policy | ||
| Dynamics set | ||
|---|---|---|
| Equivalent dynamics | ||
| Similar dynamics | ||
| Uniformly diverse dynamics | ||
| Arbitrarily diverse dynamics |
| Worst-case suboptimality ( ) on held-out dynamics | |||
|---|---|---|---|
| criterion | Offline | Uniformly diverse training set | Arbitrarily diverse training set |
| No | |||
| No | |||
| No | |||
| Yes | |||
| Yes | |||
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,835 papers