Bimanual dexterous manipulation requires coordinated arm and hand control, yet specifying such coordination manually is challenging and often task-specific. In this work, we explore whether complex bimanual dexterity can emerge from a simple object-centric learning objective. We train reinforcement learning policies under a shared object-centric reward, varying their training environments rather than tuning separate rewards for each behavior. We find that the policies discover pre-grasp manipulation, grasping and regrasping, in-hand adjustment and handover to reach object-pose goals without hand-designed motion sequences. We then refine the experts on transition states and use DAgger and reinforcement learning to distill them into a single policy that composes these behaviors for diverse bimanual tasks. On real hardware, the same policy succeeds in 61 of 80 trials (76.3%) across four tasks with unseen objects and no task-specific fine-tuning. In simulated barcode scanning, our policy achieves 85.0% success, compared with 60.4% for two single-arm policies without handover. These results suggest that this objective provides a simple and scalable interface for learning general bimanual dexterity, shifting the burden from manually designing robot coordination to specifying what should happen to the object.
Figures & tables
Fig. 1 : System overview (A) Three experts are trained in simulation under a shared object-centric representation and reward, with different goal-sampling regimes: the left and right experts πL and πR learn grasping with in-hand manipulation from goals inside one arm’s workspace, and the handover expert πH learns handover from goals that shift from the giver’s workspace to the receiver’s. (B) The experts are refined from a pool of states collected at expert transitions, then distilled into a unified policy πU with Dagger and RL in scenes that contain both individual expert behaviors and their transitions. (C) At deployment, object trajectories from diverse sources such as VR teleoperation and videos serve as the unified interface, and the single policy πU executes diverse real-world tasks.
Fig. 2 : The object-centric representation This figure shows the representation for the manipulated object duing handover. The object’s surface splits into the target contact region S+ (handle, green) and the non-target contact region S− (head, red); nearest-region displacements δA,i+ (green) and δA,i− (red) relate the keypoints xA,i of the target actor A+ (receiver, blue) and the non-target actor A− (giver, orange) to both regions.
Fig. 3 : Experimental setup (a) Real-world setup from the VR view. Three calibrated RGB-D cameras (orange) surround the workspace, and the human controls the object goal poses, shown as rendered object copies, which are streamed to the policy. (b) The real-world object set. (c) An example rule-based object trajectory for the simulated barcode-scanning benchmark.
Fig. 4 : Emergent behaviors on real hardware (a) Handover: the giver holds a stable grasp, adjusts the brush in-hand to expose a graspable region for the receiver, and releases once the receiver has grasped and settled into a more stable grasp. (b) Pre-grasp manipulation: when the object rests in a pose that is not graspable, the hand first pushes it into a graspable pose. (c) In-hand manipulation: the hand reorients the brush within the grasp to track subsequent object goals. None of these behaviors is specified explicitly; the policy discovers them from object-pose goals alone.
Task
Handover
Two-Object Coordination
Precision
SR
Barcode scanning
✓
✓
15/20
Peg insertion
✓
✓
✓
15/20
Cleaning
✓
✓
16/20
Cooking
✓
15/20
Average (%)
76.25
TABLE I : Real-world results on four tasks combining complementary bimanual capabilities. The same SimDuoReal policy is used for all tasks without task-specific fine-tuning. Success is reported as the number of successful trials out of 20.
TABLE II : Simulation results on barcode scanning. We compare full SimDuoReal policy with baselines and distillation variants. Bold marks the best result per metric.
In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave. Videos and additional results are available at https://patterndex.github.io/PatternDex/
David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee +2
Despite recent progress in dexterous reference tracking, existing models focus primarily on single-hand manipulation and a limited range of interactions, leaving the bimanual dexterous behaviors common in real life unaddressed. The scarcity of contact-rich bimanual manipulation data and the increased complexity of coordinating contacts make developing a generalist bimanual tracker particularly challenging. Therefore, we present 1) DONIC, the first generalist bimanual tracker jointly trained across operation categories and human-motion sources, together with 2) BiHOI, a multi-view dataset of 1,575 sequences spanning 229 objects and emphasizing sustained bimanual interaction, where reconstructed interactions are retargeted into robot references. DONIC combines proximal policy optimization with behavioral cloning in a feedback loop that recycles verified successful executions into training demonstrations. An object-centered per-hand reward maintains object-motion feedback while supervising each hand according to its reference contact role. Clip-Centered Advantages remove reference-dependent advantage offsets while preserving within-trajectory differences. Together, these designs enable one single policy to learn coordinated manipulation across heterogeneous references. Without finetuning, a single checkpoint achieves 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, surpassing the state-of-the-art bimanual tracking baselines by 77.36 and 45.73 percentage points.
Bimanual dexterous manipulation requires continuous coordination of wrist motion, finger contact, and object motion. Human videos provide rich manipulation experience. Using this experience for robot control requires aligning human poses with robot commands and accounting for embodiment-specific kinematics and contact dynamics. We present AcrossDEX 1.0, a learning system that connects human motion pretraining, simulation-based expert learning, robot demonstration collection, and visual closed-loop execution. Pretraining on HOT3D Aria predicts one second of bimanual motion from five images and thirty annotated hand states over the preceding second. Robot adaptation preserves the visual encoder, temporal backbone, and wrist parameters. Shared wrist representations, separate finger fields, and validity masks support both six-channel Inspire hands and sixteen-joint Allegro hands. The robot policy predicts sixty control targets from frontal images, command history, and task conditions. It executes a prefix and replans from updated observations. We compare direct regression, diffusion, and flow matching on lid opening and return and closed-lid object reorientation. Direct regression achieves success rates of 40.6% for opening and return, 100.0% for Inspire reorientation, and 93.3% for Allegro reorientation. Flow matching achieves 90.0% on Allegro reorientation. Paired comparisons further show that constraining replanning with the previous plan improves the continuous execution of generative policies. These results demonstrate a common adaptation pipeline for two hand embodiments and highlight the roles of contact coordination and action-chunk continuity.