Despite recent progress in dexterous reference tracking, existing models focus primarily on single-hand manipulation and a limited range of interactions, leaving the bimanual dexterous behaviors common in real life unaddressed. The scarcity of contact-rich bimanual manipulation data and the increased complexity of coordinating contacts make developing a generalist bimanual tracker particularly challenging. Therefore, we present 1) DONIC, the first generalist bimanual tracker jointly trained across operation categories and human-motion sources, together with 2) BiHOI, a multi-view dataset of 1,575 sequences spanning 229 objects and emphasizing sustained bimanual interaction, where reconstructed interactions are retargeted into robot references. DONIC combines proximal policy optimization with behavioral cloning in a feedback loop that recycles verified successful executions into training demonstrations. An object-centered per-hand reward maintains object-motion feedback while supervising each hand according to its reference contact role. Clip-Centered Advantages remove reference-dependent advantage offsets while preserving within-trajectory differences. Together, these designs enable one single policy to learn coordinated manipulation across heterogeneous references. Without finetuning, a single checkpoint achieves 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, surpassing the state-of-the-art bimanual tracking baselines by 77.36 and 45.73 percentage points.
Figures & tables
Source
Capture
Sequences
Frames
Objects
Bimanual contact (%)
BiHOI (ours)
MV RGB
1,575
≈ 1.60M †
229
65.91
GRAB [ 2020 ]
Marker MoCap
1,334
1.622M
51
32.21
OakInk2 [ 2024 ]
MV RGB + MoCap
627
4.01M
75
24.97
TACO (V1) [ 2024 ]
MV RGB + ego RGB-D
2.5K
5.2M
206
15.69
GigaHands [ 2025 ]
MV RGB
–
183M
417
34.74
H2O [ 2021 ]
MV RGB-D
–
571.6K
8
38.57
Table 1: Dataset overview: source statistics and simultaneous-contact rates on analyzed recordings.
Figure 2: Overview of DONIC . Top: BiHOI captures are reconstructed and retargeted, then combined with public-data references to form the training pool. Middle: A shared bimanual policy learns through simulated interaction and verified demonstrations in an iterative PPO+BC loop. Bottom: An object-centered per-hand reward, Clip-Centered Advantages (CCA), and behavioral cloning provide complementary learning signals.
Figure 3: Qualitative comparisons on two held-out object trajectories. Rows show the reference, Dexplore-bi, DexTrack-bi, and DONIC ; time runs left to right.
Dataset
Method
To↓
Ro↓
Th↓
Rh↓
Efinger↓
SR ↑
Comp ↑
(cm)
(rad)
(cm)
(rad)
(rad)
(%)
(%)
Unseen trajectory
DexTrack-bi
41.86
2.3340
23.03
0.8296
0.6301
0.00
1.30
Dexplore-bi
14.73
0.8597
21.88
0.8412
0.3166
9.43
27.87
w/o CCA
11.27
0.9585
4.34
0.2980
0.0433
81.13
62.72
w/o contact gate
10.26
0.9589
4.00
0.3013
0.0338
81.13
65.03
Ours
10.27
0.9277
3.96
0.2905
0.0372
86.79
67.62
Table 2: Reference tracking without fine-tuning, using one checkpoint per method across both splits. Black and blue bold mark the best and second-best results within each split; ties share highlighting.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Training sequences
Simultaneous contact (%)
GRAB
505
32.21
GigaHands
592
34.74
OakInk2
449
24.97
H2O
127
38.57
TACO
172
15.69
Public-data pool
1,845
—
Appendix
Table S1: Training inventory and simultaneous bimanual contact. Contact proportions pool frames across analyzed recordings, dividing both-hand-contact frames by frames with at least one hand in contact. The public-only model uses the public subsets listed below; Ours+BiHOI adds all BiHOI sequences.
Figure S1: Reconstruction and annotation pipeline. Synchronized multi-view RGB observations are processed to obtain SAM3 object masks and metric depth maps. The object branch performs metric reconstruction followed by robust multi-view 6D tracking, while the hand branch performs participant-specific hand calibration and multi-view hand-pose reconstruction. The two branches are finally refined jointly using depth-guided hand–object optimization.
Parameter
Setting
(βhand,βobj,βcont,βhold)
(0.25,0.45,0.15,0.15)
(κp,κR,κv)
(9,2,14)
(fmin,δmax)
(0.5N,0.03m)
(κc,ζ)
(30m−1,0.5)
Parallel simulation environments
8,192
BC coefficient / pairs per minibatch
50 / 4,096
Appendix
Table S2: Reward and generalist-training settings. Distances in reward exponents use metres, rotations radians, and velocities metres per second.
Setting
Value
Optimizer / learning rate / schedule
Adam / 2×10−5 / constant
Rollout horizon / PPO minibatch
64 / 16,384
PPO passes per rollout / clipping radius
6 / 0.2
Discount / GAE trace parameter
0.99 / 0.95
Value / entropy / action-bound loss coefficients
5 / 0 / 10
Value clipping / gradient truncation
Disabled / disabled
Appendix
Table S3: Optimization and simulation configuration.
Figure S2: Qualitative bimanual tracking comparisons. Rows in each group show the reference, Dexplore-bi, DexTrack-bi, and DONIC (Ours); time runs left to right.
In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave. Videos and additional results are available at https://patterndex.github.io/PatternDex/
David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee +2
Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/
Chuyang Xiao, Peilin Meng, David Held
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.
Bimanual coordination is essential for many real-world manipulation tasks, yet learning bimanual robot policies is limited by the scarcity of bimanual robots and datasets. Single-arm robots, however, are widely available in research labs. Can we leverage them to train bimanual robot policies? We present MonoDuo, a framework for learning bimanual manipulation policies using single-arm robot demonstrations paired with human collaboration. MonoDuo collects data by teleoperating a single-arm robot to perform one side of a bimanual task while a human performs the other, then swapping roles to cover both sides. RGB-D observations from a wrist-mounted and fixed camera are augmented into synthetic demonstrations for target bimanual robots using state-of-the-art hand pose estimation, image and point cloud segmentation, and inpainting. These synthetic demonstrations, grounded in real robot kinematics, are used to train bimanual policies. We evaluate MonoDuo on five tasks: box lifting, backpack packing, cloth folding, jacket zipping, and plate handover. Compared to approaches relying solely on human bimanual videos, MonoDuo enables zero-shot deployment on unseen bimanual robot configurations, achieving success rates up to 70%. With only 25 target robot demonstrations, few-shot finetuning further boosts success rates by 65-70% over training from scratch, demonstrating MonoDuo's effectiveness in efficiently transferring knowledge from single-arm robot data to bimanual robot policies.
Sandeep Bajamahal, Lawrence Yunliang Chen, Toru Lin +3