PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery

Elkhan Ismayilzada; Yufei Zhang; Zijun Cui

cs.CVMar 27, 2026

PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery

Authors: Elkhan Ismayilzada, Yufei Zhang, Zijun Cui

Organizations: Michigan State University · Independent Researcher

Abstract

Significant advancements made in reconstructing hands from images have delivered accurate single-frame estimates, yet they often lack physics consistency and provide no notion of how confidently the motion satisfies physics. In this paper, we propose a novel physics-aware conditional diffusion framework that refines noisy pose sequences into physically plausible hand motion while estimating the physics variance in motion estimates. Building on a MeshCNN-Transformer backbone, we formulate Euler-Lagrange dynamics for articulated hands. Unlike prior works that enforce zero residuals, we treat the resulting dynamic residuals as virtual observables to more effectively integrate physics. Through a last-layer Laplace approximation, our method produces per-joint, per-time variances that measure physics consistency and offers interpretable variance maps indicating where physical consistency weakens. Experiments on two well-known hand datasets show consistent gains over strong image-based initializations and competitive video-based methods. Qualitative results confirm that our variance estimations are aligned with the physical plausibility of the motion in image-based estimates.

Figures & tables

Figure 2: Overview of PAD-Hand. A sequence of images $\mathcal{I}_{1:T}$ is passed through an image-based pose estimator to obtain per-frame pose $\theta_{1:T}$ and the average shape $\beta_{avg}$ estimates. The pose estimates are then refined via a diffusion process to obtain temporally coherent motion. Simultaneously, we propagate the variance at each diffusion step starting from delta Dirac distribution at diffusion step $N$ to obtain per-frame dynamic variance estimates. At each diffusion step, backbone predicts clean motion $\hat{x}_{1:T}$ which is supervised with data-driven loss $\mathcal{L}_{data}$ and physics-driven loss $\mathcal{L}_{EL}$ during training.

Figure 3: Backbone architecture . At diffusion step

n

, the current pose sequence

{x}^{n}_{1:T}

and image-based estimates

{y}_{1:T}

are converted to meshes and encoded by MeshCNN [ 8 ] , while an MLP encodes

n

. A Transformer encoder-decoder fuses these features, and an LLLA head predicts the refined pose sequence

\hat{{x}}_{1:T}

and its variance

\mathrm{Var}(\hat{{x}}_{1:T})

Method	Input	Type	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$
MaskHand [ 34 ]	Image	P	5.0	11.70	-
WiLoR [ 45 ]	Image	D	4.88	12.75	6.70
S 2 HAND(V) [ 55 ]	Sequence	D	7.27	19.67	-
VIBE [ 31 ]			6.43	16.95	-
TCMR [ 11 ]			6.28	16.03	-
Deformer [ 16 ]			5.22	13.64	6.77

Table 1: Comparison to SOTAs on DexYCB . “*” denotes results obtained from official models; others are from original papers. “P” and “D” denote probabilistic and deterministic models, respectively.

Method	Input	Type	PA-MPJPE $\downarrow$	ACCEL $\downarrow$
AMVUR [ 27 ]	Image	P	10.3	-
WiLoR [ 45 ]	Image	D	7.50	4.98
VIBE [ 31 ]	Sequence	D	9.90	-
TCMR [ 11 ]			11.40	-
TempCLR [ 73 ]			10.60	-
Deformer [ 16 ]			9.40	6.37

Table 2: Comparison to SOTAs on HO3D . “*” denotes results from official models; others are from original papers. “P” and “D” denote probabilistic and deterministic models, respectively.

Figure 4: Refined motion estimates by PAD-Hand with dynamic variance on DexYCB. We visualize three representative sequences (I–III). In each block, row (a) compares the original image-based motion estimates to the trajectories refined by PAD-Hand, while row (b) shows the corresponding variance estimations in terms of joint-level and mesh-level dynamic variance. The red boxes highlight frames where the image-based estimates exhibit strong jitter.

Loss		Evaluation Metrics
$\mathcal{L}_{data}$	$\mathcal{L}_{EL}$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$
*WiLoR [ 45 ]		4.88	12.75	6.70
✓		4.65	10.62	3.36
✓	✓	4.63	10.56	3.34

Table 3: Effectiveness of residual loss on DexYCB.

Loss	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$
*WiLoR [ 45 ]	4.88	12.75	6.70
$\mathcal{L}^{D}_{EL}$	4.66	10.61	3.35
Ours	4.63	10.56	3.34

Table 4: Ablation on physics integration on DexYCB .

Figure 5: Distribution of dynamic variances for PAD-Hand. Bar color encodes the mean Euler–Lagrange residual within each variance bin (blue is low, red is high). Higher variance bins coincide with larger residuals, indicating that the model’s uncertainty aligns with physics violations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Loss				Evaluation Metrics
$\mathcal{L}_{b}$	$\mathcal{L}_{g}$	$\mathcal{L}_{r}$	$\mathcal{L}_{EL}$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$
WiLoR				4.88	12.75	6.70
✓				5.01	12.57	3.83
✓	✓			4.83	10.89	3.57
✓	✓	✓		4.65	10.62	3.36
✓	✓	✓	✓	4.63	10.56	3.34

Appendix

Table 5: Effectiveness of data-driven losses.

Method	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$
HaMeR [ 43 ]	4.70	16.71	7.14
HaMeR+ Ours	4.60	11.09	3.37
HandOccNet [ 42 ]	5.80	14.0	9.03
HandOccNet+ Ours	5.31	10.95	3.38

Appendix

Table 6: Comparison to more SOTAs on DexYCB.

Method	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$
WiLoR	8.37	25.13	5.47
$\mathcal{L}_{data}$	8.24	25.02	2.44
$\mathcal{L}_{data}$ + $\mathcal{L}_{EL}$ ( Ours )	8.02	24.38	1.87

Appendix

Table 7: Generalization to unseen TACO [ 37 ] . S1 testing split is used for evaluation.

Method	PA-MPJPE $\downarrow$	MPJPE $\downarrow$
$\mathcal{L}_{data}$	7.90	27.52
$\mathcal{L}_{data}$ + $\mathcal{L}_{EL}$ ( Ours )	6.67	25.84

Appendix

Table 8: Data efficiency under

\mathcal{L}_{EL}

on DexYCB.

Loss	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$	${}^{\dagger}\mathcal{R}(\mathtt{q})\downarrow$
*WiLoR [ 45 ]	4.88	12.75	6.70	19.12
SmoothFilter [ 61 ]	4.77	12.82	3.10	14.33
$\mathcal{L}_{data}$	4.65	10.62	3.36	15.16
$\mathcal{L}_{data}$ + $\mathcal{L}^{D}_{EL}$	4.66	10.61	3.35	13.45
Ours	4.63	10.56	3.34	9.04

Appendix

Table 9: Ablation on physics integration on DexYCB .

\dagger

denotes that values are scaled by

10^{-3}

Figure 6: Refined motion estimates by PAD-Hand with dynamic variance on HO3D. We visualize three representative sequences (I–III). In each block, row (a) compares the original image-based motion estimates to the trajectories refined by PAD-Hand, while row (b) shows the corresponding variance estimations in terms of joint-level and mesh-level dynamic variance. The red boxes highlight frames where the image-based estimates exhibit strong jitter.

Loss	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	ACCEL $\downarrow$	${}^{\dagger}\mathcal{R}(\mathtt{q})\downarrow$
*WiLoR [ 45 ]	4.88	12.75	6.70	19.12
WiLoR + DIP [ 67 ]	4.81	12.88	3.23	13.22
WiLoR + Ours	4.63	10.56	3.34	9.04

Appendix

Table 10: Comparison to DIP [ 67 ] on DexYCB .

\dagger

denotes that values are scaled by

10^{-3}

Loss	PA-MPJPE $\downarrow$	ACCEL $\downarrow$
*WiLoR [ 45 ]	7.50	4.98
WiLoR + DIP [ 67 ]	7.55	3.03
WiLoR + Ours	7.43	2.71

Appendix

Table 11: Comparison to DIP [ 67 ] on HO3D .

3D Hand Pose Estimation

3D hand pose estimation recovers articulated hand joints or meshes from images, video, depth, and other sensor signals.

See all 69 papers

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction
Jun 29, 2026Yuxi Wang, Chengkai Jin, Yufei Liu +6Articulated Object Reconstruction Video Diffusion Models
1Nanyang Technological University · 2Shanghai Jiao Tong University
UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling
Feb 25, 2026Zhihao Sun, Tong Wu, Ruirui Tu +2Human Motion Generation 3D Hand Pose Estimation
Institute of Trustworthy Embodied AI (TEAI), Fudan University · Stanford University
PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions
May 10, 2026Jihyun Lee, Changmin Lee, Donghwan Kim +13D Reconstruction Hand-Object Interaction Reconstruction
School of Computing, KAIST, Daejeon, South Korea.

Jun 29, 2026J/K move · Enter open · S save

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang+5

1Nanyang Technological University · 2Shanghai Jiao Tong University

Articulated Object Reconstruction Video Diffusion Models 3D Hand Pose Estimation

Open paper AskPDF arXiv

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

Abstract

3D Hand Pose Estimation

Explore similar work

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling

PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions