Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.
Figures & tables
Fig. 1: PHIRL framework. We randomly sample a subset of demonstrations and ask users to annotate them with progress . We then iteratively use an IRL algorithm to learn a reward model from the demonstrations and align the learned reward model with progress over four dimensions. Optionally, the generated trajectories will be sampled and annotated by a VLM for data augmentation. The generated trajectories will be used for reward shaping only.
Fig. 2: Performance comparison and ablation studies Results in the Lift and PickPlaceCan domains. Error bars represent standard deviation. Note that the standard PHIRL model uses 20% annotation (equivalent to PHIRL-20%). (Top) Average environmental return rewards across all baselines. PHIRL significantly outperforms baselines in both domains, particularly when the task horizon is long or demonstration quality is low (MH). (Middle) Ablation study on the effect of removing individual reward shaping components. Completion shaping is most critical when there are failed demonstrations, while reward shaping is most effective overall. (Bottom) Sensitivity analysis of annotation percentages. PHIRL improves learning significantly with 20% of the demonstrations. The benefits of the offline annotation saturate at 50% if the demonstrations are relatively high quality.
Fig. 3: Real Robot Study Results. The goal is to brush the pom-pom off the stuffed animal. We calculated the number of pom-poms that were brushed off the stuffed animal. PHIRL-online achieves the highest.
Fig. 4: Reward Hacking Scenarios and Reward Curves. The lighter colors and darker colors indicate increases and decreases in rewards. The Lift-Drop Loop was generated by a PPO agent that learns from AIRL rewards. The robot repeatedly lifted the block and then dropped it. We found that AIRL encouraged the robot to collect more rewards by lifting and dropping the block repeatedly, while PHIRL appropriately punished dropping behaviors and rewarded lifting behaviors. The Touch-And-Stay was generated by a PPO agent that learns from Robosuite environmental rewards. The robot navigated to the can and kept poking it instead of picking and placing it. PHIRL gave a negative but near-zero reward, indicating the robot is not making progress.
Fig. 5: Comparisons Between Human Annotations and Auto-Generated Annotations. We compare the performance of the model at different annotation coverages. The auto-generated annotations from an unfine-tuned VLM can improve learning from demonstrations with PHIRL.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Inverse reinforcement learning (IRL) typically assumes demonstrations from a single optimal demonstrator, but in many applications data come from multiple imperfect demonstrators with heterogeneous suboptimality levels. We study reward learning in this setting through a feasible-reward-set framework: for each demonstrator, we encode its declared suboptimality level as a linear constraint and intersect the resulting feasible sets across demonstrators. Our theoretical analysis shows that the joint feasible set shrinks monotonically as data are added, and we give an exact characterization of when a new demonstrator strictly tightens it. We further establish two recovery guarantees for the feasible reward set of the ground-truth optimal demonstrator: one bound depends on closeness to the optimal occupancy, while the other requires only sufficient coverage and no near-optimal demonstrator. On the practical side, we introduce strategies to address the inherent reward ambiguity in the obtained reward set and provide an offline algorithm with function approximation for high-dimensional environments. Experiments in tabular grid-world and large language model (LLM) fine-tuning settings are consistent with the theoretical predictions and demonstrate the effectiveness of the proposed framework over baselines.
Kihyun Kim, Shripad Deshmukh, Nikos Vlassis +1
MIT LIDS · University of Massachusetts, Amherst · Adobe Research +1
Inverse reinforcement learning (IRL) provides a powerful framework for learning from demonstrations. However, real-world tasks often exhibit substantial natural variations (e.g., picking up mugs with varying shapes), making it impractical to collect demonstrations that fully specify a new task under every possible scenario. In practice, while demonstrations for the target task are limited, it is often easier to obtain datasets of heterogeneous but related behaviors. This motivates the problem of few-shot IRL with multi-task demonstrations (FM-IRL), where an agent must learn a new task with substantial variations from only a limited number of target-task demonstrations, together with sufficient demonstrations of related tasks and online agent experience. To do so, we must both recover the expert distribution of the new task and provide guidance when the agent deviates from it. We introduce Multitask discriminator Proximity-Guided IRL (MPG), which learns two complementary reward components: (1) a generalizable discriminator that transfers shared structure across related tasks to identify expert behavior in a new task, and (2) a proximity function that measures how far a state deviates from expert behavior and provides corrective guidance during exploration. We demonstrate the effectiveness of our method on multiple challenging navigation and manipulation tasks under significant variations (e.g., object configurations, table layouts, and initial robot poses), achieving an average success rate of 81.2%, outperforming the strongest per-task baseline by an average of 24.7 percentage points.
Existing approaches to reward inference typically assume that humans provide demonstrations according to specific behavior models. However, humans often indicate their goals through a wide range of behaviors, from actions that are suboptimal due to poor planning or execution to behaviors intended to communicate goals rather than achieve them. One existing solution for inferring rewards from such behavior \unicodex2013 provided it is drawn from the same distribution at training and deployment \unicodex2013 is to construct a dataset of behavior paired with known rewards, and to learn the mapping from behavior to rewards; however, prior methods in this family face notable limitations, such as restrictions to tabular settings. Given such a dataset, we propose instead that supervised learning offers a parsimonious yet powerful solution, which we term Supervised Reward Inference (SRI). Theoretically, we prove that SRI is asymptotically Bayes-optimal under standard assumptions. Empirically, SRI achieves near-ceiling performance on a prior benchmark for reward inference from suboptimal behavior, while on Meta-World robotics tasks, it infers rewards from even arbitrarily suboptimal demonstrations as accurately as those demonstrations allow. Finally, we demonstrate our framework's universality with straightforward generalizations to action- and goal-prediction.
Will Schwarzer, Jordan Schneider, Philip S. Thomas +1
University of Massachusetts · Anthropic · Work done while at the University of Texas.