Q-OIL: Optimistic Exploration in Learning from Human Interventions
Organizations: University of Washington · Toyota Research Institute
Abstract
As robots are deployed in the real world, they will inevitably make mistakes. Human interventions provide valuable feedback for improving their behavior, enabling them to adapt online to new scenarios. However, studying human-in-the-loop robot learning is difficult: real-world experiments are slow and expensive, so most algorithm development relies on simulated human models. We show that common human modeling assumptions do not match how real people intervene in practice. Through analyzing intervention data from human participants, we find that humans tend to intervene when agent progress stalls, and that their corrective actions are often suboptimal. Motivated by these findings, we introduce a progress-based model of human intervention that achieves higher rank correlation with real-user algorithm rankings, enabling realistic algorithm benchmarking and development in simulation. Building on these insights, we propose Q-OIL, a method that adds optimistic bonuses to intervention transitions in reinforcement learning and softly guides exploration toward human corrections. We find that decoupled critics are critical for stability because they prevent the bonuses from backing up to pre-intervention states. As a result, Q-OIL learns more efficiently than competitive baselines, achieving 10-50% higher success in simulation and on real robots, and improving pretrained VLAs on two bimanual manipulation tasks. As we ground both evaluation and algorithms in realistic human behavior, this work provides a practical path toward scalable robot learning from interventions.
Figures & tables
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Precision | Recall |
|---|---|---|
| Action Suboptimality | 0.1136 | 0.4489 |
| Random | 0.0837 | 0.0837 |
| PAM (Ours) | 0.1750 | 0.6059 |
| Parameter | Value |
|---|---|
| Core RL Parameters | |
| Architecture | MLP with Gaussian Head |
| Hidden layers | 3 layers of width 256 |
| Optimizer | Adam |
| Learning rate | 3e-4 |
| Discount | 0.99 |
| Parameter | Value |
|---|---|
| Base Policy and Residual Architecture | |
| Base policy | Pretrained VLA (frozen) |
| Base policy demonstrations | 25 |
| Base policy fine-tuning steps | 20k |
| Residual policy | Gaussian MLP + ResNet-10 |
| Actor / critic hidden layers | 2 layers of width 256 |
RL for Robotics
Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.
See all 273 papers