Inverse Reinforcement Learning (IRL) aims to recover expert rewards that transfer across environments with differing dynamics. Prior work has shown that such transfer is possible when gathering expert demonstrations across sufficiently different dynamics. In practice, however, expert demonstrations are costly, making it critical to select the right dynamics to reduce the quantity of expert data required for learning transferable rewards. We propose a two-phase approach: we first cycle through the available dynamics to obtain a reliable reward estimate, then actively select dynamics to reduce our uncertainty about the expert reward. Crucially, this yields novel sample complexity guarantees on transfer that hold for any dynamics, with a rate independent of the number of available dynamics. Empirically, our method significantly reduces the number of expert demonstrations needed to learn a transferable reward.
Figures & tables
Figure 1: Worst-case suboptimality over the training set P as a function of the number of observed expert trajectories on randomly generated MDPs. Each plot corresponds to a different type of training set.
Figure 2: Number of times each dynamics in the training set P is selected after 100 rounds, for two types of training set. Dynamics are ordered by increasing average dissimilarity to the rest of P , so that P10 corresponds to the most isolated dynamics.
Training set type
Algorithm
Uniform
Arbitrary
Active only
02.86±0.02
12.63±0.14
Two-phase
02.93±0.02
13.63±0.15
ED
03.32±0.02
16.52±0.22
Round-robin
03.15±0.02
19.59±0.19
Fixed dynamics
18.18±0.16
24.36±0.09
Table 1: Worst-case suboptimality ( ×100 ) over 1000 unseen dynamics after learning from n=100 expert trajectories, collected on either a uniformly or an arbitrarily diverse training set of dynamics.
Figure 3: Mazes most frequently selected by Active only after 100 rounds, on a representative seed. The blue triangle marks the initial state and stars mark goal states, labeled with their expert rewards. The heatmap shows the state occupancy of policies optimized for the inferred reward.
Figure 4: Worst-case suboptimality over the training set P of 100 random mazes, and reward error, as a function of the number of observed expert trajectories.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Problem setting
P
:−
finite set of dynamics available for expert demonstrations
n,m
:−
number of rounds, and expert trajectories per round
θE,Θ
:−
expert reward parameters, and Θ:−{θ∈Rd:∥θ∥2≤Bθ}
πP(θ)
:−
argmaxπVPπ(θ) , optimal regularized policy
νh,Pπ(s)
:−
PPπ[sh=s]
μh,Pπ(s,a)
:−
νh,Pπ(s)πh(a∣s)
Appendix
Table 6
Dynamics set
μ2(P)±σ2(P)
μmax(P)±σmax(P)
Equivalent dynamics
0.00±0.00
0.00±0.00
Similar dynamics
0.00±0.00
0.98±0.05
Uniformly diverse dynamics
0.03±0.03
1.15±0.17
Arbitrarily diverse dynamics
0.57±0.11
1.44±0.05
Appendix
Table 2: Principal-angle statistics (in radian) of the controlled sets of dynamics, averaged over 20 runs.
Figure 5: Worst-case suboptimality over the training set P as a function of the number of observed expert trajectories on randomly generated MDPs, and of the criterion J used to pick dynamics online ( 7 ) or offline ( 35 ). Each plot corresponds to a different structure of P .
Worst-case suboptimality ( ×100 ) on held-out dynamics
criterion J
Offline
Uniformly diverse training set
Arbitrarily diverse training set
logdet(I+⋅)
No
02.86±0.02
12.63±0.14
tr
No
02.96±0.02
12.40±0.14
∥⋅∥2
No
03.02±0.19
15.23±0.17
logdet(I+⋅)
Yes
03.13±0.02
13.54±0.15
tr
Yes
03.14±0.20
12.64±0.15
Appendix
Table 3: Worst-case suboptimality on sets of held-out stochastic dynamics after learning from n=100 expert trajectories collected on either uniformly or arbitrarily diverse sets of dynamics.
Figure 6: Worst-case suboptimality over the training set P after n=100 rounds, as the warmup share n0/n increases. The vertical gray dashed line corresponds to the share of 1/3 used in the main paper.
Figure 7: Worst-case suboptimality over the training set P of randomly generated mazes, and the corresponding reward error, as a function of the number of observed expert trajectories, and of the size ∣P∣ of the dynamics set.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reward transfer in Inverse Reinforcement Learning (IRL) is unreliable when policies must generalize to unseen combinations of environment dynamics and task goals. We propose Factorized Contrastive Abstractions for Transferable IRL (ConTraIRL), a framework that enables compositional reward transfer by learning decoupled latent representations of these two factors. ConTraIRL uses a dual-encoder architecture that maps observations into separate dynamics and goal latent spaces, trained with a dual contrastive objective. Temporal alignment encourages the dynamics encoder to learn goal-invariant structure, while the goal encoder captures dynamics-invariant features. This factorization supports reward inference under recombined dynamics-goal settings. Experiments on continuous control benchmarks demonstrate effective few-shot transfer to unseen dynamics-goal pairings, improving sample efficiency and reward recovery over transfer IRL baselines.
Yikang Gui, Bikramjit Banerjee, Prashant Doshi
School of Computing, University of Georgia · School of Computing Sciences & Computer Engineering, The University of Southern Mississippi
Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL--one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.
Anish Diwan, Davide Tateo, Christopher E. Mower +3
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3
Inverse reinforcement learning (IRL) typically assumes demonstrations from a single optimal demonstrator, but in many applications data come from multiple imperfect demonstrators with heterogeneous suboptimality levels. We study reward learning in this setting through a feasible-reward-set framework: for each demonstrator, we encode its declared suboptimality level as a linear constraint and intersect the resulting feasible sets across demonstrators. Our theoretical analysis shows that the joint feasible set shrinks monotonically as data are added, and we give an exact characterization of when a new demonstrator strictly tightens it. We further establish two recovery guarantees for the feasible reward set of the ground-truth optimal demonstrator: one bound depends on closeness to the optimal occupancy, while the other requires only sufficient coverage and no near-optimal demonstrator. On the practical side, we introduce strategies to address the inherent reward ambiguity in the obtained reward set and provide an offline algorithm with function approximation for high-dimensional environments. Experiments in tabular grid-world and large language model (LLM) fine-tuning settings are consistent with the theoretical predictions and demonstrate the effectiveness of the proposed framework over baselines.
Kihyun Kim, Shripad Deshmukh, Nikos Vlassis +1
MIT LIDS · University of Massachusetts, Amherst · Adobe Research +1