PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects

David Minkwan Kim; Runfa Blark Li; Beckham Po-Ju Lee; Nikolay Atanasov; Truong Nguyen

cs.ROOct 3, 2026

PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects

Authors: David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee, Nikolay Atanasov, Truong Nguyen

Organizations: University of California, San Diego

Abstract

In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave. Videos and additional results are available at https://patterndex.github.io/PatternDex/

Figures & tables

	$\Delta$ Palm	$\Delta$ Finger	Pattern	Wrist	Contact
Hand	(cm)	(cm)	consist. (%) $\uparrow$	offset (cm)	prec. (%) $\uparrow$
Allegro	+0.7	+5.8	96.5 ${\scriptstyle\pm 2.0}$	5.06 ${\scriptstyle\pm 0.72}$	73.8
Shadow	+1.1	+2.1	99.1 ${\scriptstyle\pm 0.7}$	2.82 ${\scriptstyle\pm 0.58}$	80.1
Paxini	+4.6	+2.4	97.7 ${\scriptstyle\pm 1.7}$	5.22 ${\scriptstyle\pm 0.57}$	75.8
XHand	+1.9	+2.3	97.7 ${\scriptstyle\pm 4.5}$	2.77 ${\scriptstyle\pm 0.39}$	77.4

TABLE I : Quantitative results of the predicted wrist poses and contact points on unseen demonstrations.

\Delta

Palm and

\Delta

Finger are the differences in palm and finger lengths between each robot hand and the human hand (MANO [ 22 ] ).

Fig. 4 : Qualitative results of the guided policy. (a) Guided by the estimated wrist motions and contact points shown in the top row, Allegro hands open the microwave, the laptop, and the notebook. (b) Shadow, Paxini, and XHand hands succeed on the same tasks with the interaction pattern transferred from the Allegro hand.

Method	Object/Traj ID	SR (%) $\uparrow$	Pos. err. (cm) $\downarrow$	Rot. err. ( ∘ ) $\downarrow$	Joint err. ( ∘ ) $\downarrow$	Contact err. (cm) $\downarrow$
ObjDex [ 7 ]	Box/s04_01	50.1 ${\scriptstyle\pm 0.2}$	1.01 ${\scriptstyle\pm 0.05}$	1.58 ${\scriptstyle\pm 0.24}$	6.14 ${\scriptstyle\pm 0.21}$	–
PatternDex (w/o Contact)		87.4 ${\scriptstyle\pm 6.4}$	1.73 ${\scriptstyle\pm 0.19}$	2.21 ${\scriptstyle\pm 0.35}$	1.99 ${\scriptstyle\pm 0.26}$	–
PatternDex (full)		99.9 ${\scriptstyle\pm 0.1}$	2.05 ${\scriptstyle\pm 0.24}$	3.02 ${\scriptstyle\pm 0.32}$	1.92 ${\scriptstyle\pm 0.27}$	2.35 ${\scriptstyle\pm 0.06}$
ObjDex [ 7 ]	Espresso/s01_01	45.1 ${\scriptstyle\pm 14.6}$	2.16 ${\scriptstyle\pm 0.28}$	4.13 ${\scriptstyle\pm 0.92}$	2.08 ${\scriptstyle\pm 0.26}$	–
PatternDex (w/o Contact)		47.8 ${\scriptstyle\pm 15.0}$	2.87 ${\scriptstyle\pm 0.30}$	6.74 ${\scriptstyle\pm 0.95}$	9.87 ${\scriptstyle\pm 4.03}$	–
PatternDex (full)		98.7 ${\scriptstyle\pm 1.3}$	1.99 ${\scriptstyle\pm 0.17}$	4.40 ${\scriptstyle\pm 0.49}$	1.94 ${\scriptstyle\pm 0.39}$	3.61 ${\scriptstyle\pm 0.05}$

TABLE II : Quantitative results of the guided policy for bimanual dexterous manipulation on the ARCTIC dataset [ 18 ] . SR denotes success rate. Pos., Rot., and Joint err. denote the position, rotation, and joint angle errors of the object. Contact err. denotes the average distance from the robot hand to the estimated contact points.

Robot	Box/s04_01	Espresso/s01_01	Laptop/s09_02	Microwave/s04_01	Mixer/s08_01	Notebook/s08_03	Mean
Allegro	99.9 ${\scriptstyle\pm 0.1}$	98.7 ${\scriptstyle\pm 1.3}$	94.2 ${\scriptstyle\pm 5.1}$	98.6 ${\scriptstyle\pm 2.8}$	99.5 ${\scriptstyle\pm 0.7}$	66.0 ${\scriptstyle\pm 2.2}$	92.8 ${\scriptstyle\pm 12.2}$
Shadow	96.3 ${\scriptstyle\pm 4.5}$	88.5 ${\scriptstyle\pm 5.6}$	51.6 ${\scriptstyle\pm 2.3}$	78.0 ${\scriptstyle\pm 3.2}$	97.0 ${\scriptstyle\pm 2.0}$	92.3 ${\scriptstyle\pm 4.7}$	84.0 ${\scriptstyle\pm 15.8}$
Paxini	96.7 ${\scriptstyle\pm 3.2}$	74.9 ${\scriptstyle\pm 2.9}$	93.8 ${\scriptstyle\pm 5.8}$	99.0 ${\scriptstyle\pm 1.3}$	12.6 ${\scriptstyle\pm 0.0}$	64.0 ${\scriptstyle\pm 1.7}$	73.5 ${\scriptstyle\pm 30.0}$
XHand	95.4 ${\scriptstyle\pm 5.0}$	73.8 ${\scriptstyle\pm 3.7}$	40.0 ${\scriptstyle\pm 6.7}$	91.9 ${\scriptstyle\pm 7.0}$	91.0 ${\scriptstyle\pm 3.9}$	37.3 ${\scriptstyle\pm 1.8}$	71.6 ${\scriptstyle\pm 24.3}$

TABLE III : Cross-embodiment success rates (%) of the guided policy. The pattern encoder

\mathcal{E}

is trained with Allegro and kept frozen, and only the decoder

\mathcal{D}

and the embodiment encoder

\phi

are fine-tuned for Shadow, Paxini, and XHand.

Fig. 5 : Sim-to-Real Experiment. (a) The wrist motions and contact points are estimated from a human demonstration video, and the guided policy is trained in the simulator with these estimates. The learned policy is then transferred to the real robot. (b) The real Allegro hand opens the microwave at steps 1, 50, and 100. The camera view and the front view are shown as insets in each snapshot.

Explore similar work

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations
Oct 9, 2026Jiachen Zhang, Yuanyuan Li, Shaqi LuoRobotic Manipulation Cross-Embodiment Robot Learning
Beijing Academy of Artificial Intelligence, Beijing, China.
Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Oct 8, 2026Jusuk Lee, Sungha Kim, Yeonsoo Park +8Robotic Manipulation Learning from Demonstration
Seoul National University · University of Maryland, College Park · All Purpose AI +1
DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation
Aug 1, 2026Xiaoyang Chen, Shengcheng Luo, Haoran Guo +4Dexterous Manipulation Robot Skill Learning
Shanghai Jiao Tong University · ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) +1

Oct 9, 2026J/K move · Enter open · S save

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

Jiachen Zhang, Yuanyuan Li, Shaqi Luo

Beijing Academy of Artificial Intelligence, Beijing, China.

Robotic Manipulation Cross-Embodiment Robot Learning Robot Imitation Learning Data-Efficient Robot Learning Motion Retargeting

Open paper AskPDF arXiv

Collecting dexterous manipulation demonstrations for each robot hand requires repeated task-specific data acquisition. We ask whether one shared corpus of natural human demonstrations can instead serve as a reusable data interface for different dexterous hands, without collecting task demonstrations on each target robot. We present Human2Dex, a wrist-centric human demonstration interface that separates shared human observation and motion semantics from embodiment-specific execution. It aligns observations through local hand-appearance adaptation and a shared 21-point hand representation with consistent structural semantics across human and robot observations. It constructs executable supervision by retargeting fused human hand motion to each target hand, and introduces the Grasp-Pocket Interaction Representation (GPIR) to express tracked objects relative to a hand-centered Functional Grasp Pocket. The corpus is converted into separate training sets for Linker O6 and Wuji, while action labels, normalization, training, and diffusion-policy checkpoints remain hand-specific. Across six real-world tasks, Human2Dex achieves 109/120 (90.8%) successes on Linker O6 and 97/120 (80.8%) on Wuji. Across both hands, Human-Raw achieves 63/240 (26.3%) successes, whereas the complete Human2Dex configuration achieves 206/240 (85.8%). These results support the use of wrist-centric human demonstrations as a reusable interface across calibrated dexterous hands while retaining embodiment-specific policy learning. The project website is publicly available at https://human2dex.pages.dev/.

Abstract

Explore similar work

Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation