Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.
Figures & tables
Fig. 1: PHIRL framework. We randomly sample a subset of demonstrations and ask users to annotate them with progress . We then iteratively use an IRL algorithm to learn a reward model from the demonstrations and align the learned reward model with progress over four dimensions. Optionally, the generated trajectories will be sampled and annotated by a VLM for data augmentation. The generated trajectories will be used for reward shaping only.
Fig. 2: Performance comparison and ablation studies Results in the Lift and PickPlaceCan domains. Error bars represent standard deviation. Note that the standard PHIRL model uses 20% annotation (equivalent to PHIRL-20%). (Top) Average environmental return rewards across all baselines. PHIRL significantly outperforms baselines in both domains, particularly when the task horizon is long or demonstration quality is low (MH). (Middle) Ablation study on the effect of removing individual reward shaping components. Completion shaping is most critical when there are failed demonstrations, while reward shaping is most effective overall. (Bottom) Sensitivity analysis of annotation percentages. PHIRL improves learning significantly with 20% of the demonstrations. The benefits of the offline annotation saturate at 50% if the demonstrations are relatively high quality.
Fig. 3: Real Robot Study Results. The goal is to brush the pom-pom off the stuffed animal. We calculated the number of pom-poms that were brushed off the stuffed animal. PHIRL-online achieves the highest.
Fig. 4: Reward Hacking Scenarios and Reward Curves. The lighter colors and darker colors indicate increases and decreases in rewards. The Lift-Drop Loop was generated by a PPO agent that learns from AIRL rewards. The robot repeatedly lifted the block and then dropped it. We found that AIRL encouraged the robot to collect more rewards by lifting and dropping the block repeatedly, while PHIRL appropriately punished dropping behaviors and rewarded lifting behaviors. The Touch-And-Stay was generated by a PPO agent that learns from Robosuite environmental rewards. The robot navigated to the can and kept poking it instead of picking and placing it. PHIRL gave a negative but near-zero reward, indicating the robot is not making progress.
Fig. 5: Comparisons Between Human Annotations and Auto-Generated Annotations. We compare the performance of the model at different annotation coverages. The auto-generated annotations from an unfine-tuned VLM can improve learning from demonstrations with PHIRL.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.