Significant advancements made in reconstructing hands from images have delivered accurate single-frame estimates, yet they often lack physics consistency and provide no notion of how confidently the motion satisfies physics. In this paper, we propose a novel physics-aware conditional diffusion framework that refines noisy pose sequences into physically plausible hand motion while estimating the physics variance in motion estimates. Building on a MeshCNN-Transformer backbone, we formulate Euler-Lagrange dynamics for articulated hands. Unlike prior works that enforce zero residuals, we treat the resulting dynamic residuals as virtual observables to more effectively integrate physics. Through a last-layer Laplace approximation, our method produces per-joint, per-time variances that measure physics consistency and offers interpretable variance maps indicating where physical consistency weakens. Experiments on two well-known hand datasets show consistent gains over strong image-based initializations and competitive video-based methods. Qualitative results confirm that our variance estimations are aligned with the physical plausibility of the motion in image-based estimates.
Figures & tables
Figure 2: Overview of PAD-Hand. A sequence of images I1:T is passed through an image-based pose estimator to obtain per-frame pose θ1:T and the average shape βavg estimates. The pose estimates are then refined via a diffusion process to obtain temporally coherent motion. Simultaneously, we propagate the variance at each diffusion step starting from delta Dirac distribution at diffusion step N to obtain per-frame dynamic variance estimates. At each diffusion step, backbone predicts clean motion x^1:T which is supervised with data-driven loss Ldata and physics-driven loss LEL during training.
Figure 3: Backbone architecture . At diffusion step n , the current pose sequence x1:Tn and image-based estimates y1:T are converted to meshes and encoded by MeshCNN [ 8 ] , while an MLP encodes n . A Transformer encoder-decoder fuses these features, and an LLLA head predicts the refined pose sequence x^1:T and its variance Var(x^1:T) .
Method
Input
Type
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
MaskHand [ 34 ]
Image
P
5.0
11.70
-
WiLoR [ 45 ]
D
4.88
12.75
6.70
S 2 HAND(V) [ 55 ]
Sequence
D
7.27
19.67
-
VIBE [ 31 ]
6.43
16.95
-
TCMR [ 11 ]
6.28
16.03
-
Deformer [ 16 ]
5.22
13.64
6.77
Table 1: Comparison to SOTAs on DexYCB . “*” denotes results obtained from official models; others are from original papers. “P” and “D” denote probabilistic and deterministic models, respectively.
Method
Input
Type
PA-MPJPE ↓
ACCEL ↓
AMVUR [ 27 ]
Image
P
10.3
-
WiLoR [ 45 ]
D
7.50
4.98
VIBE [ 31 ]
Sequence
D
9.90
-
TCMR [ 11 ]
11.40
-
TempCLR [ 73 ]
10.60
-
Deformer [ 16 ]
9.40
6.37
Table 2: Comparison to SOTAs on HO3D . “*” denotes results from official models; others are from original papers. “P” and “D” denote probabilistic and deterministic models, respectively.
Figure 4: Refined motion estimates by PAD-Hand with dynamic variance on DexYCB. We visualize three representative sequences (I–III). In each block, row (a) compares the original image-based motion estimates to the trajectories refined by PAD-Hand, while row (b) shows the corresponding variance estimations in terms of joint-level and mesh-level dynamic variance. The red boxes highlight frames where the image-based estimates exhibit strong jitter.
Loss
Evaluation Metrics
Ldata
LEL
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
*WiLoR [ 45 ]
4.88
12.75
6.70
✓
4.65
10.62
3.36
✓
✓
4.63
10.56
3.34
Table 3: Effectiveness of residual loss on DexYCB.
Loss
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
*WiLoR [ 45 ]
4.88
12.75
6.70
LELD
4.66
10.61
3.35
Ours
4.63
10.56
3.34
Table 4: Ablation on physics integration on DexYCB .
Figure 5: Distribution of dynamic variances for PAD-Hand. Bar color encodes the mean Euler–Lagrange residual within each variance bin (blue is low, red is high). Higher variance bins coincide with larger residuals, indicating that the model’s uncertainty aligns with physics violations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Loss
Evaluation Metrics
Lb
Lg
Lr
LEL
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
WiLoR
4.88
12.75
6.70
✓
5.01
12.57
3.83
✓
✓
4.83
10.89
3.57
✓
✓
✓
4.65
10.62
3.36
✓
✓
✓
✓
4.63
10.56
3.34
Appendix
Table 5: Effectiveness of data-driven losses.
Method
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
HaMeR [ 43 ]
4.70
16.71
7.14
HaMeR+ Ours
4.60
11.09
3.37
HandOccNet [ 42 ]
5.80
14.0
9.03
HandOccNet+ Ours
5.31
10.95
3.38
Appendix
Table 6: Comparison to more SOTAs on DexYCB.
Method
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
WiLoR
8.37
25.13
5.47
Ldata
8.24
25.02
2.44
Ldata + LEL ( Ours )
8.02
24.38
1.87
Appendix
Table 7: Generalization to unseen TACO [ 37 ] . S1 testing split is used for evaluation.
Method
PA-MPJPE ↓
MPJPE ↓
Ldata
7.90
27.52
Ldata + LEL ( Ours )
6.67
25.84
Appendix
Table 8: Data efficiency under LEL on DexYCB.
Loss
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
†R(q)↓
*WiLoR [ 45 ]
4.88
12.75
6.70
19.12
SmoothFilter [ 61 ]
4.77
12.82
3.10
14.33
Ldata
4.65
10.62
3.36
15.16
Ldata + LELD
4.66
10.61
3.35
13.45
Ours
4.63
10.56
3.34
9.04
Appendix
Table 9: Ablation on physics integration on DexYCB . † denotes that values are scaled by 10−3 .
Figure 6: Refined motion estimates by PAD-Hand with dynamic variance on HO3D. We visualize three representative sequences (I–III). In each block, row (a) compares the original image-based motion estimates to the trajectories refined by PAD-Hand, while row (b) shows the corresponding variance estimations in terms of joint-level and mesh-level dynamic variance. The red boxes highlight frames where the image-based estimates exhibit strong jitter.
Loss
PA-MPJPE ↓
MPJPE ↓
ACCEL ↓
†R(q)↓
*WiLoR [ 45 ]
4.88
12.75
6.70
19.12
WiLoR + DIP [ 67 ]
4.81
12.88
3.23
13.22
WiLoR + Ours
4.63
10.56
3.34
9.04
Appendix
Table 10: Comparison to DIP [ 67 ] on DexYCB . † denotes that values are scaled by 10−3 .
4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.
Yuxi Wang, Chengkai Jin, Yufei Liu +6
1Nanyang Technological University · 2Shanghai Jiao Tong University
Hand motion plays a central role in human interaction, yet modeling realistic 4D hand motion (i.e., 3D hand pose sequences over time) remains challenging. Research in this area is typically divided into two tasks: (1) Estimation approaches reconstruct precise motion from visual observations, but often fail under hand occlusion or absence; (2) Generation approaches focus on synthesizing hand poses by exploiting generative priors under multi-modal structured inputs and infilling motion from incomplete sequences. However, this separation not only limits the effective use of heterogeneous condition signals that frequently arise in practice, but also prevents knowledge transfer between the two tasks. We present UniHand, a unified diffusion-based framework that formulates both estimation and generation as conditional motion synthesis. UniHand integrates heterogeneous inputs by embedding structured signals into a shared latent space through a joint variational autoencoder, which aligns conditions such as MANO parameters and 2D skeletons. Visual observations are encoded with a frozen vision backbone, while a dedicated hand perceptron extracts hand-specific cues directly from image features, removing the need for complex detection and cropping pipelines. A latent diffusion model then synthesizes consistent motion sequences from these diverse conditions. Extensive experiments across multiple benchmarks demonstrate that UniHand delivers robust and accurate hand motion modeling, maintaining performance under severe occlusions and temporally incomplete inputs.
Zhihao Sun, Tong Wu, Ruirui Tu +2
Institute of Trustworthy Embodied AI (TEAI), Fudan University · Stanford University
While existing methods for reconstructing hand-object interactions have made impressive progress, they either focus on rigid or part-wise rigid objects-limiting their ability to model real-world objects (e.g., cloth, stuffed animals) that exhibit highly non-rigid deformations-or model deformable objects without full 3D hand reconstruction. To bridge this gap, we present PhysHanDI (Physics-based Reconstruction of Hand and Deformable Object Interactions), a framework that enables full 3D reconstruction of both interacting hands and non-rigid objects. Our key idea is to physically simulate object deformations driven by forces induced from densely reconstructed 3D hand motions, ensuring that the reconstructed object dynamics are both physically plausible and coherent with the interacting hand movements. Furthermore, we demonstrate that such simulation of object deformations can, in turn, refine and improve hand reconstruction via inverse physics. In experiments, PhysHanDI outperforms the state-of-the-art baseline across reconstruction and future prediction.