MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos

Yiyang Cai; Tatsuya Kamijo; Ninghao Lu; Zijin Hu; Sherry Yang

cs.ROOct 14, 2026

MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos

Authors: Yiyang Cai, Tatsuya Kamijo, Ninghao Lu, Zijin Hu, Sherry Yang

Organizations: New York University, New York, NY, USA. · Fudan University, Shanghai, China. · NYU Shanghai, Shanghai, China.

Abstract

Passive human videos of everyday activity offer robots access to a massive amount of human experience without requiring a demonstration for each task. However, using this experience requires recovering knowledge from unstructured video recordings and adapting it to a different body (i.e., the robot). To overcome these challenges, we present mobile code-as-policy (MoCaP), an agentic framework that turns passive human video recordings into guidance for an end-to-end mobile manipulation policy. MoCaP extracts spatial memory and cues of how people prepare to interact with objects, which inform navigation and the search for a manipulation-ready configuration. Guided by passive human experience and current observations, MoCaP uses a closed-loop code-as-policy to coordinate planning and tool use primitives to solve motions under the robot's geometry and kinematic constraints. On a real mobile manipulator, MoCaP achieves 80% success on each of two mobile manipulation tasks in an indoor environment whereas existing approaches cannot succeed at the task. Video-derived spatial memory improves object-navigation success from 70% to 100%, and human approach prior raises the rate of finding a reachable base pose from 0% to 100% at an unfavorable start location. Videos and code are available on the project website: https://mocap-7.github.io/MoCAP-page/.

Figures & tables

Fig. 1: MoCaP turns passive human video into guidance for real-world mobile manipulation. From videos recorded during daily life, MoCaP extracts a spatial memory and an approach direction. A coding agent uses them to find the target and reach a base pose with a feasible grasp. Grey dashed paths show execution without this prior.

Fig. 2: MoCaP overview. Passive human video provides reusable spatial memory and observations of how people prepare to manipulate. At task startup, an advisor VLM relates the spatial memory to the instruction and initial robot view. During execution, the system retrieves a recorded interaction with the target object and estimates where the person in the video stood relative to it from the frame preceding the reach. A closed-loop code policy coordinates navigation and manipulation through planning primitives that check actions against live geometry and robot constraints, revising execution from observations and feedback.

Fig. 3: YOR robot platform used in our experiments, equipped with an RGB-D camera, dual arms with grippers, a vertical lift, and an omnidirectional base.

Panel of Figure 4 — Fig. 4: Example executions of grasping a cardboard box, grasping a can, and navigating to a blue trash bin. Keyframes in each row follow one continuous execution from left to right.

Task	Method	Grasp attempted	Success
Grasp box	MoCaP (ours)	19/20	16/20 (80%)
	CaP-X+	8/20	0/20 (0%)
	CaP-X [ 3 ]	2/20	0/20 (0%)
Grasp can	MoCaP (ours)	10/10	8/10 (80%)
	CaP-X+	0/10	0/10 (0%)
	CaP-X [ 3 ]	0/10	0/10 (0%)

TABLE I: Mobile manipulation results.

Method	Successes/trials	Success within 180 s	Success within 300 s
MoCaP (ours)	10/10	6/10	9/10
MoCaP (no prior)	7/10	4/10	6/10
ApexNav [ 34 ]	2/10	1/10	2/10
CoW [ 35 ]	0/10	0/10	0/10

TABLE II: Blue-trash-bin navigation from the kitchen.

Panel of Figure 7 — Fig. 5: Schematic layout of the base placement study. A, B, and C denote the three starting configurations around the desk, and the dotted lines indicate the direction to the target can. The recorded human interaction occurs on the C side.

Start	Without prior	With prior	$\Delta$
	Success rate (%)
A	0.0	93.3	$+93.3$
B	26.7	80.0	$+53.3$
C	93.3	80.0	$-13.3$
All	40.0 (18/45)	84.4 (38/45)	$+44.4$

TABLE III: Manipulation readiness (fifteen trials per start and condition).

Zero-Shot Robotic Manipulation

Zero-shot robotic manipulation enables robots to perform unfamiliar tasks and handle unseen objects, scenes, or embodiments without task-specific training.

See all 103 papers

MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation
Sep 22, 2026Jinhe Tang, Ruixiao Dai, Weiming ZhiRobot Trajectory Tracking Robot Policy Learning
School of Computer Science, The University of Sydney, Australia
MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment
Sep 10, 2026Guangyu Chen, Qiwei Liang, Shaolong Zhu +7Mobile Manipulation Cross-Modal Alignment
1THU · 2Xspark AI · 3HKUST (GZ) +1
CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
Sep 27, 2026Yiheng Lyu, Xueying Jiang, Wenhao Li +2Robotic Policy Evaluation Robotic Manipulation
Nanyang Technological University, Singapore · Independent Researcher

Sep 22, 2026J/K move · Enter open · S save

MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation

Jinhe Tang, Ruixiao Dai, Weiming Zhi

School of Computer Science, The University of Sydney, Australia

Robot Trajectory Tracking Robot Policy Learning Mobile Manipulation Visuomotor Policy Learning

Open paper AskPDF arXiv

Successful mobile manipulation requires coordinated base and arm motion while maintaining accurate spatial positioning. However, demonstration-trained policies can struggle to realise the intended base motion reliably, leading to spatial misalignment and subsequent manipulation failures. We present MAVP (Map-Aware Visuomotor Policies), a framework that improves execution reliability by predicting explicit base-pose targets and tracking them using localisation feedback. MAVP reconstructs a static map from teleoperated demonstrations and expresses demonstrated base trajectories in a shared map frame, providing consistent spatial supervision across demonstrations. At execution time, the policy receives RGB observations, joint states, and the robot's current map-frame base pose, and jointly predicts target base poses, arm actions, and gripper actions. A low-level controller tracks the predicted base targets using feedforward motion and pose error feedback, enabling correction of execution deviations. We additionally use pose-noise augmentation during training to improve robustness to errors in the policy's pose input. Across six real-world manipulation tasks and three policy families, MAVP achieves higher task success rates than unanchored velocity control in all tasks. Videos and additional results are available at https://123qwedsa123.github.io/mavp/.

Abstract

Zero-Shot Robotic Manipulation

Explore similar work

MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation

MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation