DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Mingze Sun; Ziang Cao; Yuan Fei; Qiankuan Tao; Zihan Wang; Tianqi Liu; Junhao Chen; Haiwen Diao

cs.ROOct 21, 2026

DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Authors: Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao,

Abstract

Despite recent progress in dexterous reference tracking, existing models focus primarily on single-hand manipulation and a limited range of interactions, leaving the bimanual dexterous behaviors common in real life unaddressed. The scarcity of contact-rich bimanual manipulation data and the increased complexity of coordinating contacts make developing a generalist bimanual tracker particularly challenging. Therefore, we present 1) DONIC, the first generalist bimanual tracker jointly trained across operation categories and human-motion sources, together with 2) BiHOI, a multi-view dataset of 1,575 sequences spanning 229 objects and emphasizing sustained bimanual interaction, where reconstructed interactions are retargeted into robot references. DONIC combines proximal policy optimization with behavioral cloning in a feedback loop that recycles verified successful executions into training demonstrations. An object-centered per-hand reward maintains object-motion feedback while supervising each hand according to its reference contact role. Clip-Centered Advantages remove reference-dependent advantage offsets while preserving within-trajectory differences. Together, these designs enable one single policy to learn coordinated manipulation across heterogeneous references. Without finetuning, a single checkpoint achieves 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, surpassing the state-of-the-art bimanual tracking baselines by 77.36 and 45.73 percentage points.

Figures & tables

Source	Capture	Sequences	Frames	Objects	Bimanual contact (%)
BiHOI (ours)	MV RGB	1,575	$\approx$ 1.60M †	229	65.91
GRAB [ 2020 ]	Marker MoCap	1,334	1.622M	51	32.21
OakInk2 [ 2024 ]	MV RGB + MoCap	627	4.01M	75	24.97
TACO (V1) [ 2024 ]	MV RGB + ego RGB-D	2.5K	5.2M	206	15.69
GigaHands [ 2025 ]	MV RGB	–	183M	417	34.74
H2O [ 2021 ]	MV RGB-D	–	571.6K	8	38.57

Table 1: Dataset overview: source statistics and simultaneous-contact rates on analyzed recordings.

Figure 2: Overview of DONIC . Top: BiHOI captures are reconstructed and retargeted, then combined with public-data references to form the training pool. Middle: A shared bimanual policy learns through simulated interaction and verified demonstrations in an iterative PPO+BC loop. Bottom: An object-centered per-hand reward, Clip-Centered Advantages (CCA), and behavioral cloning provide complementary learning signals.

Figure 3: Qualitative comparisons on two held-out object trajectories. Rows show the reference, Dexplore-bi, DexTrack-bi, and DONIC ; time runs left to right.

Dataset	Method	$T_{o}\downarrow$	$R_{o}\downarrow$	$T_{h}\downarrow$	$R_{h}\downarrow$	$E_{\mathrm{finger}}\downarrow$	SR $\uparrow$	Comp $\uparrow$
		(cm)	(rad)	(cm)	(rad)	(rad)	(%)	(%)
Unseen trajectory	DexTrack-bi	41.86	2.3340	23.03	0.8296	0.6301	0.00	1.30
	Dexplore-bi	14.73	0.8597	21.88	0.8412	0.3166	9.43	27.87
	w/o CCA	11.27	0.9585	4.34	0.2980	0.0433	81.13	62.72
	w/o contact gate	10.26	0.9589	4.00	0.3013	0.0338	81.13	65.03
	Ours	10.27	0.9277	3.96	0.2905	0.0372	86.79	67.62

Table 2: Reference tracking without fine-tuning, using one checkpoint per method across both splits. Black and blue bold mark the best and second-best results within each split; ties share highlighting.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Training sequences	Simultaneous contact (%)
GRAB	505	32.21
GigaHands	592	34.74
OakInk2	449	24.97
H2O	127	38.57
TACO	172	15.69
Public-data pool	1,845	—

Appendix

Table S1: Training inventory and simultaneous bimanual contact. Contact proportions pool frames across analyzed recordings, dividing both-hand-contact frames by frames with at least one hand in contact. The public-only model uses the public subsets listed below; Ours+BiHOI adds all BiHOI sequences.

Figure S1: Reconstruction and annotation pipeline. Synchronized multi-view RGB observations are processed to obtain SAM3 object masks and metric depth maps. The object branch performs metric reconstruction followed by robust multi-view 6D tracking, while the hand branch performs participant-specific hand calibration and multi-view hand-pose reconstruction. The two branches are finally refined jointly using depth-guided hand–object optimization.

Parameter	Setting
$(\beta_{\mathrm{hand}},\beta_{\mathrm{obj}},\beta_{\mathrm{cont}},\beta_{\mathrm{hold}})$	$(0.25,0.45,0.15,0.15)$
$(\kappa_{p},\kappa_{R},\kappa_{v})$	$(9,2,14)$
$(f_{\min},\delta_{\max})$	$(0.5\,\mathrm{N},0.03\,\mathrm{m})$
$(\kappa_{c},\zeta)$	$(30\,\mathrm{m}^{-1},0.5)$
Parallel simulation environments	8,192
BC coefficient / pairs per minibatch	50 / 4,096

Appendix

Table S2: Reward and generalist-training settings. Distances in reward exponents use metres, rotations radians, and velocities metres per second.

Setting	Value
Optimizer / learning rate / schedule	Adam / $2\times 10^{-5}$ / constant
Rollout horizon / PPO minibatch	64 / 16,384
PPO passes per rollout / clipping radius	6 / 0.2
Discount / GAE trace parameter	0.99 / 0.95
Value / entropy / action-bound loss coefficients	5 / 0 / 10
Value clipping / gradient truncation	Disabled / disabled

Appendix

Table S3: Optimization and simulation configuration.

Figure S2: Qualitative bimanual tracking comparisons. Rows in each group show the reference, Dexplore-bi, DexTrack-bi, and DONIC (Ours); time runs left to right.

Bimanual Robotic Manipulation

Bimanual robotic manipulation coordinates two robot arms to handle objects, tools, and tasks that require collaboration.

See all 184 papers

PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects
Oct 3, 2026David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee +2Robotic Manipulation Bimanual Robotic Manipulation
University of California, San Diego
JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation
Sep 21, 2026Chuyang Xiao, Peilin Meng, David Held3D Point Tracking Bimanual Robotic Manipulation
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.
MonoDuo: Using One Robot Arm to Learn Bimanual Policies
May 28, 2026Sandeep Bajamahal, Lawrence Yunliang Chen, Toru Lin +3Bimanual Robotic Manipulation Cross-Embodiment Robot Learning
University of California, Berkeley

Oct 3, 2026J/K move · Enter open · S save

PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects

David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee, Nikolay Atanasov+1

University of California, San Diego

Robotic Manipulation Bimanual Robotic Manipulation Articulated Object Manipulation Hand-Object Interaction Robotic RL Cross-Embodiment Robot Learning

Open paper AskPDF arXiv

In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave. Videos and additional results are available at https://patterndex.github.io/PatternDex/

Abstract

Bimanual Robotic Manipulation

Explore similar work

PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation

MonoDuo: Using One Robot Arm to Learn Bimanual Policies