DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations

Rui Zhou; Yibo Yuan; Junkai Zhao; Fangyuan Zhao; Xiaoguang Zhao; Shanghang Zhang; Sirui Han

cs.ROSep 28, 2026

DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations

Authors: Rui Zhou, Yibo Yuan, Junkai Zhao, Fangyuan Zhao, Xiaoguang Zhao, Shanghang Zhang, Sirui Han

Organizations: The Hong Kong University of Science and Technology · Beijing Academy of Artificial Intelligence · Beihang University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Institute of Automation, Chinese Academy of Sciences

Abstract

Mobile bimanual dexterous manipulation requires continuous coordination of locomotion, whole-body motion, and finger-level dexterity within a single trajectory, creating a severe robot demonstration bottleneck. Egocentric human demonstrations offer a scalable alternative, but prior approaches ease the transfer by simplifying human motion, discarding exactly the fine-grained, coupled structure such tasks depend on. We present DexRoam, a complete system for learning mobile bimanual dexterous manipulation from human demonstrations, in which whole-body motion remains continuous and coupled throughout the human-to-robot transfer process. To enable scalable collection of whole-body human manipulation demonstrations, we develop a tracker-free capture system using only a consumer VR headset and a head-mounted stereo camera, without external cameras or motion trackers. We then perform three explicit alignment stages---embodiment, action-semantic, and temporal---to map captured motion into the robot action space, preserving fine-grained whole-body motion and allowing human and robot demonstrations to be jointly learned by standard VLA policies. Real-world experiments with different VLA backbones show that human demonstrations consistently improve policy learning across training paradigms, raising average success from 29% to 56% on GR00T N1.7 and from 32% to 57% on pi0.5, while matching robot-only training with half the robot demonstrations. Ablations confirm that each alignment stage is necessary. These results highlight the potential of human demonstrations for scalable whole-body mobile manipulation with preserved fine-grained motion structure.

Figures & tables

Fig. 2: DexRoam data collection system. (a) Follow-style teleoperation lets one operator drive base, torso, arms, head, and dexterous hands simultaneously via a Meta Quest 3S, VR controllers, and MANUS Pro gloves, with gesture-based episode management. (b) Tracker-free egocentric capture uses a Meta Quest 3 and a head-mounted ZED Mini to record 6-DoF head pose, 33 body joints, 25 joints per hand, and timestamped egocentric stereo RGB, with no external cameras, marker or trackers. Real-time in-headset video streaming keeps the demonstrator’s view identical to the policy observation.

Fig. 3: Explicit human-to-robot alignment in DexRoam. (a) Human motion is expressed in a pelvis-centric root frame projected onto the ground plane, yielding root/chassis, torso, wrist, and head trajectories free of global scene coordinates. (b) Embodiment alignment maps each component to its robot counterpart, and action-semantic alignment re-expresses the results as robot-centric relative transformations. (c) Temporal alignment resamples $N_{h}$ human steps into $N_{r}$ waypoints evenly spaced in task progress, matching the execution timescale of robot demonstrations.

Fig. 4: Real-world mobile bimanual dexterous manipulation tasks. Rollouts of the five evaluation tasks, one per row: Pick Chips Can, Pour Water, Throw Trash, Deliver Fruit, and Push Chair & Close Laptop. They span fine-grained dexterity, base–manipulation coupling, and long-horizon whole-body interaction.

Fig. 5: Human–robot action distribution before and after alignment. PCA visualization of whole-body actions on representative tasks. Top: retargeted absolute human actions and robot actions before action-semantic and temporal alignment, which occupy clearly separated regions. Bottom: fully aligned robot-centric relative actions, where the two distributions largely overlap, with marginals shown along each axis.

Component	Metric	Pick Chips Can	Pour Water	Throw Trash	Average
Torso	Position	13.49	11.82	12.31	12.54
Torso	Rotation	2.32	1.64	1.57	1.84
Left Arm	Position	3.75	4.27	4.02	4.01
Left Arm	Rotation	0.51	0.58	0.57	0.55
Right Arm	Position	5.30	5.93	7.67	6.30
Right Arm	Rotation	0.70	0.84	1.22	0.92

TABLE I: Real-robot replay of transferred human trajectories. Position and rotation errors of retargeted human motions across different body components, measured in millimeters (mm) and degrees (

\degree

Human Motion Capture	Torso		Left Arm		Right Arm		Head
Human Motion Capture	Pos.	Rot.	Pos.	Rot.	Pos.	Rot.	Rot.
Tracker-free (Ours)	12.54	1.84	4.01	0.55	6.30	0.92	1.37
PICO + Motion Trackers	13.16	2.06	4.12	0.57	5.63	0.81	1.23

TABLE II: Tracker-free vs. tracker-based capture. Retargeting accuracy averaged over 15 trajectories across three tasks. Position and rotation errors are reported in millimeters and degrees, respectively.

Task	Strategy	w/o Temporal	Absolute	DexRoam
Task	Strategy	Resampling	Action	(Ours)
Pick Chips Can	Cotrain	0.65	0.00	0.70
	Pretrain + FT	0.30	0.00	0.65
	Pretrain + Cotrain FT	0.20	0.00	0.90
Throw Trash	Cotrain	0.40	0.00	0.65
	Pretrain + FT	0.50	0.00	0.55

TABLE III: Ablation of alignment stages. GR00T N1.7 success rate.

Fig. 9: Generalization under distribution shift. Evaluation on Pick Chips Can under the original setting, a novel scene, and an unseen instruction phrasing. The images above show the original and novel scenes. Human-assisted training degrades far less than Robot-only under both shifts, especially when human data is retained during finetuning.

Component	Metric	Pick Chips Can	Pour Water	Throw Trash	Average
Torso	Position	14.76	11.50	13.22	13.16
Torso	Rotation	1.71	2.74	1.74	2.06
Left Arm	Position	3.85	4.11	4.41	4.12
Left Arm	Rotation	0.54	0.58	0.60	0.57
Right Arm	Position	5.44	5.45	6.02	5.63
Right Arm	Rotation	0.79	0.79	0.85	0.81

TABLE IV: Detailed replay errors of the PICO + Motion Trackers capture system. Position and rotation errors are reported in millimeters (mm) and degrees ( ∘ ), respectively.

Task	Data	LDLJ $\uparrow$	SPARC $\uparrow$
Pick Chips Can	Retargeted Human	-20.2802	-5.8049
Pick Chips Can	Teleoperated Robot	-20.4726	-7.0548
Pour Water	Retargeted Human	-20.5962	-5.9222
Pour Water	Teleoperated Robot	-20.4183	-6.5718
Throw Trash	Retargeted Human	-20.8349	-6.6556
Throw Trash	Teleoperated Robot	-20.3022	-6.9229

TABLE V: Motion smoothness of retargeted human trajectories after temporal resampling and teleoperated robot demonstrations. Higher LDLJ and SPARC indicate smoother motion.

Fig. 10: Representative failure cases across the five tasks. Each row shows one task, with successive frames illustrating a failed rollout from left to right.

Component	Representation	Dim.
Left arm	3D Cartesian position + 6D rotation	9
Right arm	3D Cartesian position + 6D rotation	9
Torso	3D Cartesian position + 6D rotation	9
Left XHand	Joint feedback	12
Right XHand	Joint feedback	12
Active head	Joint state	2

TABLE VI: Composition of the 53-D proprioceptive state.

Component	Representation	Dim.
Left XHand	Joint target	12
Right XHand	Joint target	12
Left arm	Relative translation + 6D rotation	9
Right arm	Relative translation + 6D rotation	9
Torso	Relative translation + 6D rotation	9
Active head	Joint target	2

TABLE VII: Composition of the 56-D unified whole-body action.

Hyperparameter	Value
Base model	$\pi_{0.5}$
Number of GPUs	2
Maximum training steps	30,000
Global batch size	4
Per-GPU batch size	2
Gradient accumulation steps	1

TABLE VIII: Training configuration for

\pi_{0.5}

Hyperparameter	Value
Base model	NVIDIA Isaac-GR00T N1.7-3B
Number of GPUs	4
Maximum training steps	40,000
Global batch size	24
Per-GPU batch size	6
Gradient accumulation steps	1

TABLE IX: Training configuration for GR00T N1.7.

Task	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Pick Chips Can	Move forward to the table (0.2)	Move the hand into the grasping envelope without knocking over the can, with the fingertip center within 3 cm of the can axis (0.2)	Close the fingers and establish a stable grasp with at least two opposing fingers in contact with the can (0.2)	Lift the can above the table by at least the basket height and maintain the grasp for at least 3 s without slipping (0.2)	Successfully place the can into the basket (0.2)
Pour Water	Reach toward the container (0.1)	Establish a stable grasp on the container (0.2)	Lift the container at least 5 cm above the table (0.2)	Move toward the target while maintaining a stable grasp on the container (0.2)	Rotate the wrist and pour water into the target container (0.3)
Throw Trash	Move the hand into the grasping envelope of the chips can without knocking it over, with the fingertip center within 3 cm of the can axis (0.2)	Close the fingers and establish a stable grasp with at least two opposing fingers in contact with the object (0.2)	Move the mobile base until the trash bin is within the reachable workspace (0.2)	Release the object such that it falls into the trash bin (0.3)	Complete the task without displacing or knocking over the trash bin (0.1)
Deliver Fruit	Move both hands into the grasping envelope and establish contact with the fruit basket (0.2)	Lift the fruit basket at least 5 cm above the supporting surface using coordinated bimanual grasping (0.2)	Maintain the lifted basket for at least 3 s without dropping it (0.2)	Complete the required locomotion and turning while keeping the basket and its contents stable (0.2)	Place the fruit basket at the target location (0.2)
Push Chair & Close Laptop	Establish contact between the hand and the back of the chair (0.2)	Successfully push the chair without losing hand–chair contact (0.2)	Move the chair to the specified target position under the table (0.2)	Extend the right hand and successfully reach behind the laptop lid (0.2)	Successfully close the laptop (0.2)

TABLE X: Task Completion Score (TCS) criteria. Each task is decomposed into five ordered milestones with task-specific weights shown in parentheses.

Explore similar work

EgoWild2Dex: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience
Sep 20, 2026Kunyang Lin, Xutao Wen, Jingxi Lin +8Contact-Rich Dexterous Manipulation Egocentric Dataset
The University of Hong Kong · Kinetix AI
DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation
Aug 1, 2026Xiaoyang Chen, Shengcheng Luo, Haoran Guo +4Contact-Rich Dexterous Manipulation Offline Reinforcement Learning
Shanghai Jiao Tong University · ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) +1
EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations
Jun 10, 2026Yangcen Liu, Shuo Cheng, Xinchen Yin +6Egocentric Video Robotic Data Acquisition
Georgia Institute of Technology · Tsinghua University

Sep 20, 2026, cs.ROJ/K move · Enter open · S save

EgoWild2Dex: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience

Kunyang Lin, Xutao Wen, Jingxi Lin, Lanyong Lin+7

The University of Hong Kong · Kinetix AI

Contact-Rich Dexterous Manipulation Egocentric Dataset Robotic Manipulation Bimanual Manipulation In-The-Wild Videos

Open paper AskPDF arXiv

Egocentric human data provide a principled source of supervision for learning dexterous robot manipulation. Unlike prior approaches that often collect such data in constrained or specially constructed environments, we collect in-the-wild egocentric demonstrations in real-world settings, including homes, factories, and pharmacies, etc., where people perform their ordinary tasks while wearing head-mounted cameras. This collection protocol captures diverse workflows and hand-object interactions across long-tailed object and skill distributions, but also yields visually challenging observations due to scene clutter and head-motion-induced viewpoint changes (a mean cumulative rotation of

15.93^{\circ}

/s). To address these issues, we introduce EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, respectively. This work offers three benefits. First, we introduce GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot observations. Second, we design a human-robot training scheme to bridge the embodiment gap, enabling high task success with limited robot supervision. Third, we release EgoWild, a 538.9-hour in-the-wild egocentric human dataset comprising 179,049 episodes, 125,961 unique task descriptions, and 1,282 object categories. On real robots, EgoWild2Dex achieves an average success rate of 96.7% across three long-horizon bimanual dexterous manipulation tasks and an average object-level zero-shot success rate of 33.3%. The data, models, and code will be released.

Abstract

Explore similar work

EgoWild2Dex: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience

DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation

EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations