MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos
Organizations: New York University, New York, NY, USA. · Fudan University, Shanghai, China. · NYU Shanghai, Shanghai, China.
Abstract
Passive human videos of everyday activity offer robots access to a massive amount of human experience without requiring a demonstration for each task. However, using this experience requires recovering knowledge from unstructured video recordings and adapting it to a different body (i.e., the robot). To overcome these challenges, we present mobile code-as-policy (MoCaP), an agentic framework that turns passive human video recordings into guidance for an end-to-end mobile manipulation policy. MoCaP extracts spatial memory and cues of how people prepare to interact with objects, which inform navigation and the search for a manipulation-ready configuration. Guided by passive human experience and current observations, MoCaP uses a closed-loop code-as-policy to coordinate planning and tool use primitives to solve motions under the robot's geometry and kinematic constraints. On a real mobile manipulator, MoCaP achieves 80% success on each of two mobile manipulation tasks in an indoor environment whereas existing approaches cannot succeed at the task. Video-derived spatial memory improves object-navigation success from 70% to 100%, and human approach prior raises the rate of finding a reachable base pose from 0% to 100% at an unfavorable start location. Videos and code are available on the project website: https://mocap-7.github.io/MoCAP-page/.
Figures & tables
| Task | Method | Grasp attempted | Success |
|---|---|---|---|
| Grasp box | MoCaP (ours) | 19/20 | 16/20 (80%) |
| CaP-X+ | 8/20 | 0/20 (0%) | |
| CaP-X [ 3 ] | 2/20 | 0/20 (0%) | |
| Grasp can | MoCaP (ours) | 10/10 | 8/10 (80%) |
| CaP-X+ | 0/10 | 0/10 (0%) | |
| CaP-X [ 3 ] | 0/10 | 0/10 (0%) |
| Method | Successes/trials | Success within 180 s | Success within 300 s |
|---|---|---|---|
| MoCaP (ours) | 10/10 | 6/10 | 9/10 |
| MoCaP (no prior) | 7/10 | 4/10 | 6/10 |
| ApexNav [ 34 ] | 2/10 | 1/10 | 2/10 |
| CoW [ 35 ] | 0/10 | 0/10 | 0/10 |
| Success rate (%) | |||
|---|---|---|---|
| Start | Without prior | With prior | |
| A | 0.0 | 93.3 | |
| B | 26.7 | 80.0 | |
| C | 93.3 | 80.0 | |
| All | 40.0 (18/45) | 84.4 (38/45) | |
Zero-Shot Robotic Manipulation
Zero-shot robotic manipulation enables robots to perform unfamiliar tasks and handle unseen objects, scenes, or embodiments without task-specific training.
See all 103 papers