cs.ROOct 29, 2026

AcrossDEX 1.0: From Human Motion Pretraining to Bimanual Dexterous Manipulation across Hand Embodiments

Authors: Xiaoyu Yang, Nan Wu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Bimanual dexterous manipulation requires continuous coordination of wrist motion, finger contact, and object motion. Human videos provide rich manipulation experience. Using this experience for robot control requires aligning human poses with robot commands and accounting for embodiment-specific kinematics and contact dynamics. We present AcrossDEX 1.0, a learning system that connects human motion pretraining, simulation-based expert learning, robot demonstration collection, and visual closed-loop execution. Pretraining on HOT3D Aria predicts one second of bimanual motion from five images and thirty annotated hand states over the preceding second. Robot adaptation preserves the visual encoder, temporal backbone, and wrist parameters. Shared wrist representations, separate finger fields, and validity masks support both six-channel Inspire hands and sixteen-joint Allegro hands. The robot policy predicts sixty control targets from frontal images, command history, and task conditions. It executes a prefix and replans from updated observations. We compare direct regression, diffusion, and flow matching on lid opening and return and closed-lid object reorientation. Direct regression achieves success rates of 40.6% for opening and return, 100.0% for Inspire reorientation, and 93.3% for Allegro reorientation. Flow matching achieves 90.0% on Allegro reorientation. Paired comparisons further show that constraining replanning with the previous plan improves the continuous execution of generative policies. These results demonstrate a common adaptation pipeline for two hand embodiments and highlight the roles of contact coordination and action-chunk continuity.

Figures & tables

Fig. 1: Overview of AcrossDEX 1.0. (a) Human pretraining and robot expert data support adaptation with human replay. (b) A shared Transformer processes images and motion history. Masks select human or robot finger fields. Each method, task, and hand uses a separately trained policy. (c) The robot policy predicts 60 commands, executes the first KK steps, and updates its history. Blue, green, and orange denote shared, human, and robot components. Gray fields are masked.
Fig. 2: Success rates for lid opening and return and object reorientation with two hand embodiments. Each configuration is post-trained separately and evaluated using ( 11 ).
Fig. 3: A successful closed-loop opening and return trajectory with Inspire hands and Flow. The frontal views show the initial state, contact interaction, full opening, and the final state after lid return.
Fig. 4: Successful closed-loop reorientation. Top: Inspire with BC. Bottom: Allegro with Flow. Columns correspond to 00 , 0.220.22 , 0.430.43 , and 0.67 0.67\, s.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,704 papers

Explore similar work

CardsList
Oct 3, 2026

PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects

In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave. Videos and additional results are available at https://patterndex.github.io/PatternDex/

David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee +2

University of California, San Diego

Robotic ManipulationBimanual Robotic ManipulationArticulated Object ManipulationHand-Object InteractionRobotic RLCross-Embodiment Robot Learning
AskPDF
Sep 28, 2026

DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations

Mobile bimanual dexterous manipulation requires continuous coordination of locomotion, whole-body motion, and finger-level dexterity within a single trajectory, creating a severe robot demonstration bottleneck. Egocentric human demonstrations offer a scalable alternative, but prior approaches ease the transfer by simplifying human motion, discarding exactly the fine-grained, coupled structure such tasks depend on. We present DexRoam, a complete system for learning mobile bimanual dexterous manipulation from human demonstrations, in which whole-body motion remains continuous and coupled throughout the human-to-robot transfer process. To enable scalable collection of whole-body human manipulation demonstrations, we develop a tracker-free capture system using only a consumer VR headset and a head-mounted stereo camera, without external cameras or motion trackers. We then perform three explicit alignment stages---embodiment, action-semantic, and temporal---to map captured motion into the robot action space, preserving fine-grained whole-body motion and allowing human and robot demonstrations to be jointly learned by standard VLA policies. Real-world experiments with different VLA backbones show that human demonstrations consistently improve policy learning across training paradigms, raising average success from 29% to 56% on GR00T N1.7 and from 32% to 57% on pi0.5, while matching robot-only training with half the robot demonstrations. Ablations confirm that each alignment stage is necessary. These results highlight the potential of human demonstrations for scalable whole-body mobile manipulation with preserved fine-grained motion structure.

Rui Zhou, Yibo Yuan, Junkai Zhao +4

The Hong Kong University of Science and Technology · Beijing Academy of Artificial Intelligence · Beihang University +2

Mobile ManipulationBimanual Robotic ManipulationCross-Embodiment Robot LearningWhole-Body Robotic ManipulationMotion RetargetingRobot Imitation Learning
AskPDF
Oct 9, 2026

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

Collecting dexterous manipulation demonstrations for each robot hand requires repeated task-specific data acquisition. We ask whether one shared corpus of natural human demonstrations can instead serve as a reusable data interface for different dexterous hands, without collecting task demonstrations on each target robot. We present Human2Dex, a wrist-centric human demonstration interface that separates shared human observation and motion semantics from embodiment-specific execution. It aligns observations through local hand-appearance adaptation and a shared 21-point hand representation with consistent structural semantics across human and robot observations. It constructs executable supervision by retargeting fused human hand motion to each target hand, and introduces the Grasp-Pocket Interaction Representation (GPIR) to express tracked objects relative to a hand-centered Functional Grasp Pocket. The corpus is converted into separate training sets for Linker O6 and Wuji, while action labels, normalization, training, and diffusion-policy checkpoints remain hand-specific. Across six real-world tasks, Human2Dex achieves 109/120 (90.8%) successes on Linker O6 and 97/120 (80.8%) on Wuji. Across both hands, Human-Raw achieves 63/240 (26.3%) successes, whereas the complete Human2Dex configuration achieves 206/240 (85.8%). These results support the use of wrist-centric human demonstrations as a reusable interface across calibrated dexterous hands while retaining embodiment-specific policy learning. The project website is publicly available at https://human2dex.pages.dev/.

Jiachen Zhang, Yuanyuan Li, Shaqi Luo

Beijing Academy of Artificial Intelligence, Beijing, China.

Robotic ManipulationCross-Embodiment Robot LearningRobot Imitation LearningData-Efficient Robot LearningHand-Object InteractionMotion Retargeting
AskPDF