UniHM: Unified Dexterous Hand Manipulation with Vision Language Model

Zhenhao Zhang; Jiaxin Liu; Ye Shi; Jingya Wang

cs.ROFeb 28, 2026

UniHM: Unified Dexterous Hand Manipulation with Vision Language Model

Authors: Zhenhao Zhang, Jiaxin Liu, Ye Shi, Jingya Wang

Organizations: ShanghaiTech University · InstAdapt

Abstract

Planning physically feasible dexterous hand manipulation is a central challenge in robotic manipulation and Embodied AI. Prior work typically relies on object-centric cues or precise hand-object interaction sequences, foregoing the rich, compositional guidance of open-vocabulary instruction. We introduce UniHM, the first framework for unified dexterous hand manipulation guided by free-form language commands. We propose a Unified Hand-Dexterous Tokenizer that maps heterogeneous dexterous-hand morphologies into a single shared codebook, improving cross-dexterous hand generalization and scalability to new morphologies. Our vision language action model is trained solely on human-object interaction data, eliminating the need for massive real-world teleoperation datasets, and demonstrates strong generalizability in producing human-like manipulation sequences from open-ended language instructions. To ensure physical realism, we introduce a physics-guided dynamic refinement module that performs segment-wise joint optimization under generative and temporal priors, yielding smooth and physically feasible manipulation sequences. Across multiple datasets and real-world evaluations, UniHM attains state-of-the-art results on both seen and unseen objects and trajectories, demonstrating strong generalization and high physical feasibility. Our project page at https://unihm.github.io/.

Figures & tables

Figure 1: Overview. We introduce UniHM, the first unified hand-manipulation framework conditioned on free-form language. UniHM is trained solely on closed-set HOI datasets to follow target trajectories and execute physically feasible interactions, and generalizes to open-world tasks in real-world interactions.

Figure 2: Pipeline. UniHM converts open-vocabulary instructions and RGB-D inputs into executable dexterous-hand trajectories via three stages: (1) morphology-agnostic motion tokenization; (2) language-guided generation that fuses text, perception, and token history to produce manipulation token sequences; and (3) physics-aware decoding with smoothness/contact priors for feasible, stable execution.

	Method	MPJPE $\downarrow$	FOL $\downarrow$	FPL $\downarrow$	FID $\downarrow$	Diversity $\rightarrow$
Seen	GT	-	-	-	-	125.53
	TM2T ( Guo et al., 2022 )	85.33 ${}^{\pm\scalebox{0.5}{3.41}}$	36.57 ${}^{\pm\scalebox{0.5}{1.46}}$	24.10 ${}^{\pm\scalebox{0.5}{0.96}}$	54.83 ${}^{\pm\scalebox{0.5}{2.19}}$	37.12 ${}^{\pm\scalebox{0.5}{1.48}}$
	MDM ( Tevet et al., 2023 )	88.06 ${}^{\pm\scalebox{0.5}{3.52}}$	33.40 ${}^{\pm\scalebox{0.5}{1.34}}$	23.06 ${}^{\pm\scalebox{0.5}{0.92}}$	52.33 ${}^{\pm\scalebox{0.5}{2.09}}$	33.95 ${}^{\pm\scalebox{0.5}{1.36}}$
	FlowMDM ( Barquero et al., 2024 )	82.75 ${}^{\pm\scalebox{0.5}{3.31}}$	31.24 ${}^{\pm\scalebox{0.5}{1.25}}$	21.55 ${}^{\pm\scalebox{0.5}{0.86}}$	48.05 ${}^{\pm\scalebox{0.5}{1.92}}$	61.25 ${}^{\pm\scalebox{0.5}{2.45}}$
	MotionGPT3 ( Zhu et al., 2025 )	74.80 ${}^{\pm\scalebox{0.5}{2.99}}$	28.76 ${}^{\pm\scalebox{0.5}{1.15}}$	19.32 ${}^{\pm\scalebox{0.5}{0.77}}$	43.35 ${}^{\pm\scalebox{0.5}{1.73}}$	72.51 ${}^{\pm\scalebox{0.5}{2.90}}$
	Ours	61.40 ${}^{\pm\scalebox{0.5}{1.93}}$	23.14 ${}^{\pm\scalebox{0.5}{0.65}}$	12.15 ${}^{\pm\scalebox{0.5}{0.24}}$	31.24 ${}^{\pm\scalebox{0.5}{1.02}}$	39.62 ${}^{\pm\scalebox{0.5}{0.66}}$

Table 1: Main Result on DexYCB. The arrow pointing to the right means closer to the GT.

	Method	MPJPE $\downarrow$	FOL $\downarrow$	FPL $\downarrow$	FID $\downarrow$	Diversity $\rightarrow$
Seen	GT	-	-	-	-	147.40
	TM2T ( Guo et al., 2022 )	71.08 ${}^{\pm\scalebox{0.5}{2.84}}$	91.25 ${}^{\pm\scalebox{0.5}{3.65}}$	34.51 ${}^{\pm\scalebox{0.5}{1.38}}$	311.90 ${}^{\pm\scalebox{0.5}{12.48}}$	277.38 ${}^{\pm\scalebox{0.5}{11.10}}$
	MDM ( Tevet et al., 2023 )	67.55 ${}^{\pm\scalebox{0.5}{2.70}}$	93.8 ${}^{\pm\scalebox{0.5}{3.75}}$	30.06 ${}^{\pm\scalebox{0.5}{1.20}}$	285.22 ${}^{\pm\scalebox{0.5}{11.41}}$	275.42 ${}^{\pm\scalebox{0.5}{11.02}}$
	FlowMDM ( Barquero et al., 2024 )	60.74 ${}^{\pm\scalebox{0.5}{2.43}}$	85.43 ${}^{\pm\scalebox{0.5}{3.42}}$	26.47 ${}^{\pm\scalebox{0.5}{1.06}}$	249.08 ${}^{\pm\scalebox{0.5}{9.96}}$	189.54 ${}^{\pm\scalebox{0.5}{7.58}}$
	MotionGPT3 ( Zhu et al., 2025 )	56.29 ${}^{\pm\scalebox{0.5}{2.25}}$	79.24 ${}^{\pm\scalebox{0.5}{3.17}}$	23.98 ${}^{\pm\scalebox{0.5}{0.96}}$	221.10 ${}^{\pm\scalebox{0.5}{8.84}}$	247.10 ${}^{\pm\scalebox{0.5}{9.88}}$
	Ours	52.73 ${}^{\pm\scalebox{0.5}{2.08}}$	72.32 ${}^{\pm\scalebox{0.5}{0.55}}$	19.86 ${}^{\pm\scalebox{0.5}{1.38}}$	204.91 ${}^{\pm\scalebox{0.5}{7.64}}$	165.47 ${}^{\pm\scalebox{0.5}{6.30}}$

Table 2: Main Result on OakInk. The arrow pointing to the right means closer to the GT.

		Success Rate
Split	Method	Grab	Pick&Place	Pull&Push	Open&Close
Seen	MDM+Dex-Retargeting	20%	10%	0%	5%
	MotionGPT3+Dex-Retargeting	30%	15%	25%	25%
	Ours	65%	50%	60%	55%
Unseen	MDM+Dex-Retargeting	5%	0%	0%	5%
	MotionGPT3+Dex-Retargeting	45%	25%	15%	20%

Table 3: Real-World Experiments

Figure 3: Real-World Results. UniHM achieves higher success rates than prior methods on both seen and unseen objects, producing physically consistent and executable real-world manipulations.

	Method	MPJPE $\downarrow$	FOL $\downarrow$	FPL $\downarrow$	FID $\downarrow$	Diversity $\rightarrow$
Seen	GT	-	-	-	-	125.53
	w/o Depth Input	85.47 ${}^{\pm\scalebox{0.5}{3.42}}$	33.41 ${}^{\pm\scalebox{0.5}{1.34}}$	20.97 ${}^{\pm\scalebox{0.5}{0.84}}$	56.36 ${}^{\pm\scalebox{0.5}{2.25}}$	66.40 ${}^{\pm\scalebox{0.5}{2.66}}$
	w/o Masked Training	73.41 ${}^{\pm\scalebox{0.5}{2.94}}$	28.22 ${}^{\pm\scalebox{0.5}{1.13}}$	14.42 ${}^{\pm\scalebox{0.5}{0.58}}$	44.87 ${}^{\pm\scalebox{0.5}{1.79}}$	73.09 ${}^{\pm\scalebox{0.5}{2.92}}$
	w/o Physical Refinement	65.78 ${}^{\pm\scalebox{0.5}{2.63}}$	25.06 ${}^{\pm\scalebox{0.5}{1.00}}$	15.35 ${}^{\pm\scalebox{0.5}{0.61}}$	33.57 ${}^{\pm\scalebox{0.5}{1.34}}$	38.06 ${}^{\pm\scalebox{0.5}{1.52}}$
	Ours	61.40 ${}^{\pm\scalebox{0.5}{1.93}}$	23.14 ${}^{\pm\scalebox{0.5}{0.65}}$	12.15 ${}^{\pm\scalebox{0.5}{0.24}}$	31.24 ${}^{\pm\scalebox{0.5}{1.02}}$	39.62 ${}^{\pm\scalebox{0.5}{0.66}}$
Unseen	w/o Depth Input	90.12 ${}^{\pm\scalebox{0.5}{3.60}}$	39.77 ${}^{\pm\scalebox{0.5}{1.59}}$	21.70 ${}^{\pm\scalebox{0.5}{0.87}}$	77.38 ${}^{\pm\scalebox{0.5}{3.10}}$	67.53 ${}^{\pm\scalebox{0.5}{2.70}}$

Table 4: Ablation Result on DexYCB. The arrow pointing to the right means closer to the GT.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

		Allegro	Shadow	Schunk	LEAP	Ability	Panda Gripper	Overall
MAE	MLP	0.0268	0.0450	0.0293	0.0348	0.0432	0.0221	0.0350
MAE	1D-Conv	0.0216	0.0297	0.0221	0.0257	0.0327	0.0182	0.0256
RMSE	MLP	0.0545	0.0886	0.0656	0.0656	0.0913	0.0630	0.0736
RMSE	1D-Conv	0.0465	0.0654	0.0551	0.0531	0.0705	0.0519	0.0581

Appendix

Table B1: A comparison of the validation set performance for the MLP and 1D-Conv models.

Figure B1: The function plots under varying values of $\alpha$ and $k$ are displayed. Note that $\alpha$ and $k$ control the curve behavior for $x>0$ and $x<0$ , respectively. For comparison, the curve of $y=|x|$ is plotted using a dashed line.

Figure B2: Optimization results when the point cloud includes noise. Here, the black points represent the object point cloud, and the red/green points denote the positions of the dexterous hand’s fingertips. (A) Optimization using the $f(d)$ kernel, visualized on the noisy input. (B) Optimization using the Euclidean distance, visualized on the noisy input. (C) The $f(d)$ kernel optimization result projected onto the clean noise-free point cloud. (D) The Euclidean distance optimization result projected onto the noise-free point cloud.

Figure D1: The visualization results of our grasping sequence in Sapien. More real world examples could be found in the video.

Figure D2: The visualization results of our grasping sequence in Sapien. More real world examples could be found in the video.

Figure E1: Real-World Cross-embodiment Set up

Explore similar work

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos
Mar 23, 2026Gu Zhang, Qicheng Xu, Haozhe Zhang +16Robotic Manipulation Cross-Embodiment Robot Learning
Tsinghua University · Shanghai Qizhi Institute · Sun Yat-sen University +1
UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data
Jun 9, 2026Dong Fang, Youjun Wu, Yuanxin Zhong +4Representation Learning Dexterous Manipulation
1Fudan University · 2Rimbot · 3Hefei University of Technology +1
UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis
Jul 30, 2026Hui Zhang, Julian Ferchow, Jie Song +1Robotic Grasping Dexterous Manipulation
ETH Zürich, Switzerland · inspire AG, Zürich, Switzerland · HKUST (Guangzhou), China

Mar 23, 2026J/K move · Enter open · S save

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

Gu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma+15

Tsinghua University · Shanghai Qizhi Institute · Sun Yat-sen University+1

Robotic Manipulation Cross-Embodiment Robot Learning Robot Imitation Learning VLMs for Robotics Robotic Data Collection

Open paper AskPDF arXiv

Dexterous manipulation remains challenging due to the cost of collecting real-robot teleoperation data, the heterogeneity of hand embodiments, and the high dimensionality of control. We present UniDex, a robot foundation suite that couples a large-scale robot-centric dataset with a unified vision-language-action (VLA) policy and a practical human-data capture setup for universal dexterous hand control. First, we construct UniDex-Dataset, a robot-centric dataset over 50K trajectories across eight dexterous hands (6--24 DoFs), derived from egocentric human video datasets. To transform human data into robot-executable trajectories, we employ a human-in-the-loop retargeting procedure to align fingertip trajectories while preserving plausible hand-object contacts, and we operate on explicit 3D pointclouds with human hands masked to narrow kinematic and visual gaps. Second, we introduce the Function-Actuator-Aligned Space (FAAS), a unified action space that maps functionally similar actuators to shared coordinates, enabling cross-hand transfer. Leveraging FAAS as the action parameterization, we train UniDex-VLA, a 3D VLA policy pretrained on UniDex-Dataset and finetuned with task demonstrations. In addition, we build UniDex-Cap, a simple portable capture setup that records synchronized RGB-D streams and human hand poses and converts them into robot-executable trajectories to enable human-robot data co-training that reduces reliance on costly robot demonstrations. On challenging tool-use tasks across two different hands, UniDex-VLA achieves 81% average task progress and outperforms prior VLA baselines by a large margin, while exhibiting strong spatial, object, and zero-shot cross-hand generalization. Together, UniDex-Dataset, UniDex-VLA, and UniDex-Cap provide a scalable foundation suite for universal dexterous manipulation.

Abstract

Explore similar work

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data

UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis