UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling

Zhihao Sun; Tong Wu; Ruirui Tu; Daoguo Dong; Zuxuan Wu

cs.CVFeb 25, 2026

UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling

Authors: Zhihao Sun, Tong Wu, Ruirui Tu, Daoguo Dong, Zuxuan Wu

Organizations: Institute of Trustworthy Embodied AI (TEAI), Fudan University · Stanford University

Abstract

Hand motion plays a central role in human interaction, yet modeling realistic 4D hand motion (i.e., 3D hand pose sequences over time) remains challenging. Research in this area is typically divided into two tasks: (1) Estimation approaches reconstruct precise motion from visual observations, but often fail under hand occlusion or absence; (2) Generation approaches focus on synthesizing hand poses by exploiting generative priors under multi-modal structured inputs and infilling motion from incomplete sequences. However, this separation not only limits the effective use of heterogeneous condition signals that frequently arise in practice, but also prevents knowledge transfer between the two tasks. We present UniHand, a unified diffusion-based framework that formulates both estimation and generation as conditional motion synthesis. UniHand integrates heterogeneous inputs by embedding structured signals into a shared latent space through a joint variational autoencoder, which aligns conditions such as MANO parameters and 2D skeletons. Visual observations are encoded with a frozen vision backbone, while a dedicated hand perceptron extracts hand-specific cues directly from image features, removing the need for complex detection and cropping pipelines. A latent diffusion model then synthesizes consistent motion sequences from these diverse conditions. Extensive experiments across multiple benchmarks demonstrate that UniHand delivers robust and accurate hand motion modeling, maintaining performance under severe occlusions and temporally incomplete inputs.

Figures & tables

Figure 1: Overview of the UniHand framework. (I) The Joint VAE aligns motion and condition encoders within a shared latent space. An autoregressive decoder iteratively reconstructs motion to preserve temporal consistency. (II) The latent diffusion model is trained on this latent space, where multimodal conditions are fused, and hand-relevant vision tokens are integrated into the denoiser.

Method	All		Occlusion (25%–50%)		Occlusion (50%–75%)		Occlusion (75%–100%)
Method	PA-MPJPE $\downarrow$	$\text{AUC}_{J}$ $\uparrow$	PA-MPJPE $\downarrow$	$\text{AUC}_{J}$ $\uparrow$	PA-MPJPE $\downarrow$	$\text{AUC}_{J}$ $\uparrow$	PA-MPJPE $\downarrow$	$\text{AUC}_{J}$ $\uparrow$
Spurr et al. (2020)	6.83	0.864	7.22	0.856	8.00	0.840	10.65	0.788
MeshGraphormer	6.41	0.872	6.85	0.863	7.22	0.856	7.76	0.845
SemiHandObj	6.33	0.874	6.70	0.866	7.17	0.857	8.96	0.821
HandOccNet	5.80	0.884	6.22	0.876	6.43	0.872	7.37	0.853
WiLoR	5.01	0.900	-	-	5.42	0.892	5.68	0.887

Table 1: Quantitative comparison of SoTA hand pose and motion modeling methods on the DexYCB test set in the camera coordinate space. Results are reported in terms of MPJPE (mm) and AUC, with statistics across different occlusion levels.

Figure 2: Visualization of generated hand poses and trajectories. The first example shows a static camera scenario where the subject picks up a red bowl , with significant hand occlusion. The second example is recorded with a dynamic camera, where the subject picks up and manipulates a magic cube , involving large hand movements. UniHand produces more accurate hand motion by modeling motions in a canonical coordinate space, even without relying on explicit camera extrinsics.

Table 4

Setups	DexYCB-All		DexYCB-Occlusion		HOT3D
Setups	PA-MPJPE $\downarrow$	$\text{AUC}_{J}$ $\uparrow$	PA-MPJPE $\downarrow$	$\text{AUC}_{J}$ $\uparrow$	PA-MPJPE $\downarrow$	G-MPJPE $\downarrow$	GA-MPJPE $\downarrow$
w/o. Condition Encoders $\mathcal{E}_{c}$	5.21	0.895	5.56	0.889	5.92	75.49	31.03
w/o. Pretrained $\mathcal{E}_{\text{vision}}$	6.52	0.869	6.71	0.865	8.73	146.08	39.53
w/o. Hand Perceptron	7.81	0.843	8.75	0.824	12.46	180.59	48.93
w/o. 3D RoPE	4.65	0.906	4.76	0.904	4.95	69.20	28.94
w. $c_{\text{vision}}$	4.24	0.915	4.27	0.915	4.52	53.49	23.28

Table 4: Ablation studies on the core components, design choices, and different condition configurations during inference, evaluated on the DexYCB and HOT3D datasets. Results are reported in terms of MPJPE (

mm

) under different alignment strategies and AUC scores.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Illustration of hand occlusion level computation on the DexYCB dataset.

Figure 4: Additional visualization of generated hand poses and trajectories.

Figure 5: Qualitative comparison between HaMeR and our UniHand. Our method generates more continuous and accurate hand pose sequences compared to HaMeR.

Figure 6: Qualitative comparison between HaMeR and our UniHand. In cases of severe hand self-occlusion, HaMeR misclassifies the right hand as the left hand, resulting in poor reconstruction quality, whereas UniHand generates reliable and consistent hand motions.

Figure 7: Qualitative comparison between HaMeR and our UniHand. HaMeR fails to estimate valid poses in video frames where the hand is absent, whereas UniHand maintains stable reconstructions by exploiting vision perception and temporal modeling.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction
Jun 29, 2026Yuxi Wang, Chengkai Jin, Yufei Liu +6Articulated Object Reconstruction Video Diffusion Models
1Nanyang Technological University · 2Shanghai Jiao Tong University
HandFlow: Fully Generative 4D Hand Recovery with Flow Matching
Jul 13, 2026Mingxi Xu, Bowen Duan, Yi Gu +3Articulated Object Reconstruction 4D Reconstruction
The Hong Kong University of Science and Technology (Guangzhou), China · Google, USA
4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction
Oct 6, 2026Shiqi Li, Sean Cho, Yijie Li +3Physics-Informed Generative Modeling 4D Reconstruction
Texas A&M University · NVIDIA

Jun 29, 2026J/K move · Enter open · S save

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang+5

1Nanyang Technological University · 2Shanghai Jiao Tong University

Articulated Object Reconstruction Video Diffusion Models 3D Hand Pose Estimation

Open paper AskPDF arXiv

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

Abstract

Human Motion Generation

Explore similar work

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

HandFlow: Fully Generative 4D Hand Recovery with Flow Matching

4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction