From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors
Organizations: University of California, Berkeley · General Motors
Abstract
Collecting demonstrations for bimanual dexterous manipulation is costly, motivating the use of human videos as a source of robot training data. However, uncertainty in video-derived motion estimates and differences between human and robot embodiments make direct motion transfer challenging. In this work, we introduce PriDex, a framework that transforms human-video Priors into executable robot Dexterous manipulation demonstrations through reinforcement learning in simulation. PriDex preserves task-relevant interaction structure while allowing embodiment-specific behavior to be regenerated for the robot. Human-object interactions provide affordance cues for selecting task-relevant, robot-specific grasps from geometry-based candidates. For motion refinement, object trajectories provide a task-level reference, while relative bimanual wrist trajectories offer complementary guidance for coordinated motion. Since object-motion estimates can become unreliable under occlusion or visual ambiguity, PriDex uses object-motion confidence to adaptively balance these two priors, emphasizing object-level guidance when confidence is high and increasing the contribution of wrist-level guidance as confidence decreases. Reinforcement learning refines these video-derived priors under the robot's kinematic and physical constraints, and the resulting policies are rolled out to generate robot demonstrations for training a vision-based diffusion policy. We evaluate PriDex on four bimanual dexterous manipulation tasks, including pouring, sweeping into a dustpan, bottle-cap unscrewing, and plate cleaning, assessing both simulation-based robot data generation and downstream visuomotor policy learning.
Figures & tables
| Method | Pour | Sweep | Unscrew | Clean | Mean |
|---|---|---|---|---|---|
| H2S2R | 0.0 | 3.0 | 0.0 | 7.3 | 2.6 |
| ViViDex | 0.0 | 6.0 | 0.0 | 0.0 | 1.5 |
| w/o HP & Conf. | 16.0 | 50.7 | 6.7 | 98.7 | 43.0 |
| w/o HP | 41.3 | 64.7 | 16.7 | 99.3 | 55.5 |
| PriDex | 98.0 | 44.7 | 40.0 | 46.0 | 57.2 |
| Data source | Pour | Sweep | Unscrew | Clean | Mean |
|---|---|---|---|---|---|
| w/o HP | 18.0 | 72.0 | 0.7 | 6.0 | 24.2 |
| PriDex | 24.0 | 80.0 | 5.3 | 16.0 | 31.3 |
| Task | Real-only | Sim+Real |
|---|---|---|
| Pour | ||
| Sweep | ||
| Clean | ||
| Mean success |
Robotic Manipulation
Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.
See all 1,831 papers