We propose, HaNDF, object-conditional neural distance fields as data-driven priors for modeling the plausible hand-object configurations, in the product manifold of hand articulations. While recent unconditional generative priors showed success in modeling human pose, hands are versatile articulated bodies often found in interaction with other objects. This makes it crucial to develop a generative prior which can model the plausible hand poses conditioned on the object under interaction. To this end, our HaNDF proposes a geometry-aware, object-conditioned neural distance field, representing the plausible articulations in the zero-level set of a conditional field. We leverage tools from Riemannian geometry to (i) represent plausible hands in the zero level-set of an object-induced neural field, and (ii) project a given pose onto this field. Using HaNDF, we can also optimize for a plausible condition, determining the object under interaction. Our extensive evaluations demonstrate that our HaNDF provides a flexible and powerful prior for hand-object interaction, suitable for applications such as pose refinement, reconstruction under occlusion, and physically plausible manipulation synthesis.
Figures & tables
Figure 1: We introduce HaNDF , an object-conditioned generative hand prior. HaNDF is a geometric neural distance field representing the plausible hand configurations under interaction, and enables numerous tasks from denoising and completion to recovering hands or even objects from image data.
Figure 2: Manifold of quaternions.
Figure 3: HaNDF jointly models the plausible configurations of joints given the object code.
Figure 4: Pose generation on HOI4D: FID vs. dNN (lower is better); bubble size encodes APD.
Figure 5: Hand-pose denoising on HOI4D under different noise levels ( σ∈ [0.2, 1.5] rad). HaNDF (ours, red) achieves the best Δq and m2m at every noise level, with margins growing under heavier corruption.
Figure 6: Qualitative pose denoising iterations for hand-object interaction on HOI4D. Starting from noisy initializations ( t=0 ), our model iteratively refines the hand poses. By t=1000 , the hand reaches a physically plausible configuration that closely matches the Ground Truth (GT).
Figure 7: Qualitative comparison of pose denoising across different methods on HOI4D. We evaluate our method (HaNDF) against HMP and H-NRDF on a variety of object geometries. Ground Truth (GT) is provided for reference.
Missing Fingers
Missing Finger Parts
Method
Δq↓
dNN ↓
APD ↑
FID ↓
Δq↓
dNN ↓
APD ↑
FID ↓
HMP Duran et al. (2024)
24.37 ∘
18.77 ∘
36.86 ∘
0.7407
16.60 ∘
14.45 ∘
26.79 ∘
0.9448
H-PoseNDF
20.80 ∘
16.40 ∘
32.57 ∘
1.0318
12.87 ∘
8.83 ∘
20.03 ∘
0.7267
H-NRDF
16.60 ∘
8.01 ∘
24.48 ∘
0.1558
8.47 ∘
4.52 ∘
12.17 ∘
0.0453
HaNDF
12.57 ∘
5.44 ∘
20.79 ∘
0.0596
6.60 ∘
3.57 ∘
12.23 ∘
0.0106
Table 1: Hand part completion across two settings: missing fingers and partly missing fingers on HOI4D dataset.
HOI4D Liu et al. (2022)
OakInk Yang et al. (2022)
DexYCB Chao et al. (2021)
Method
MPVPE ↓
MPJPE ↓
I.V. ↓
MPVPE ↓
MPJPE ↓
I.V. ↓
MPVPE ↓
MPJPE ↓
I.V. ↓
IHoI Ye et al. (2022)
19.42
20.12
41.44
14.75
15.17
1.66
29.85
30.73
1.26
EasyHOI Liu et al. (2025)
16.64
17.06
13.28
13.17
13.48
0.93
29.64
30.08
2.61
EasyHOI + HaNDF
13.98
14.33
11.40
12.85
13.12
0.89
28.31
28.78
2.43
AlignSDF Chen et al. (2022b)
10.40
10.60
5.39
10.34
10.50
0.82
16.82
17.19
0.99
AlignSDF + HaNDF
10.22
10.43
4.39
9.98
10.17
0.87
16.59
16.97
0.98
Table 2: In-the-wild hand recovery on HOI4D, OakInk, and DexYCB. For hand-focused, fair comparisons, the object ground truth is provided during inference. Lower is better.
Figure 8: Qualitative results for in-the-wild hand recovery on HOI4D. Our HaNDF consistently improves the reconstruction quality of both EasyHOI and AlignSDF, alleviating common failure modes such as lacking contact, hand-object penetration, and implausible hand rotation.
Method
Obj retrieval
Obj recovery
Δq↓
MPJPE ↓
MPVPE ↓
obj-CD ↓
obj-F@1% ↑
obj-F@2% ↑
AlignSDF Chen et al. (2022b)
✗
N/A
14.77
12.08
11.90
0.0909
0.0296
0.1187
AlignSDF + HaNDF
✗
N/A
9.60
11.74
11.52
0.0909
0.0296
0.1187
✓
K=1
9.59
11.50
11.30
0.0896
0.0449
0.1568
✓
K=8
9.57
11.45
11.25
0.1083
0.0387
0.1360
Table 3: Hand-object recovery. We evaluate different hand priors built upon the hand–object reconstruction results from AlignSDF baseline Chen et al. (2022b) on the HOI4D dataset. We condition HaNDF on the object predicted by AlignSDF. We also refine the object by retrieving the nearest neighbor in the object latent space (with PCA), possibly with further object recovery by optimization with different K , which show advanced performance. CD refers to Chamfer Distance. The best and second-best results are highlighted in bold and underlined text , respectively.
Backbone / conditioning
Field geometry
Δq(∘)
dNN (∘)
m2m (mm)
MLP, unconditional
Euclidean
8.29
5.88
9.54
MLP, unconditional
Riemannian
7.16
6.52
8.60
MLP + object condition
Euclidean
8.15
5.12
9.06
MLP + object condition
Riemannian
5.98
4.57
7.14
Transformer, unconditional
Riemannian
6.10
5.75
7.40
Transformer + object condition
Euclidean
10.27
8.09
9.92
Table 4: Ablation over backbone, conditioning, and field geometry on HOI4D.
Δq ( ∘ ) ↓
dNN ( ∘ ) ↓
m2m ↓
Noisy input
11.74
12.43
11.13
HaNDF w/ wrong condition
5.80
5.25
7.05
HaNDF w/o object token
6.10
5.75
7.40
HaNDF
5.52
4.96
6.58
Table 5: Object-conditioning ablation on hand denoising with noise level σ=0.8 rad on HOI4D dataset.
Method
# Steps
FID ↓
dNN ↓
Runtime (ms) ↓
LatentHOI Li et al. (2025)
1000
2.7843
6.41 ∘
64.5
HaNDF
200 †
0.5482
5.07 ∘
12
HaNDF
500
0.5466
4.77 ∘
29
HaNDF
1000
0.5297
4.66 ∘
59
Table 6: Comparison of inference costs between the proposed HaNDF and LatentHOI Li et al. (2025) for object-conditioned generation on HOI4D. Runtime is reported per sample. † With 200 steps, the applied step size is doubled.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Diagram of HaNDF’s transformer backbone, which combines hand, object, and distance tokens to predict the distance.
(θ′,z,j)∈DO∑fΦ(θ′∣z)−minθ∈DjdMH(θ′,θ)
Appendix
Algorithm 1 Training HaNDF
Model
FID ↓
APD ↑
dNN ↓
HMP [ 18 ]
4.9338
44.72 ∘
30.66 ∘
H-PoseNDF
2.685
21.48 ∘
30.8 ∘
H-NRDF
0.7769
20.86 ∘
10.70 ∘
G-HOP [ 51 ]
0.7316
9.34 ∘
10.45 ∘
LatentHOI [ 29 ]
2.7843
6.01 ∘
6.41 ∘
Text2HOI [ 7 ]
7.05
31.15 ∘
26.85 ∘
Appendix
Table 7: Quantitative evaluation of pose generation on HOI4D dataset.
σ=0.4 rad
σ=0.8 rad
σ=1.5 rad
Model
dNN ↓
Δq↓
m2m ↓
dNN ↓
Δq↓
m2m ↓
dNN ↓
Δq↓
m2m ↓
HMP [ 18 ]
7.29 ∘
5.33 ∘
5.90
11.23 ∘
10.26 ∘
10.37
17.64 ∘
17.76 ∘
16.38
H-PoseNDF
3.81 ∘
5.67 ∘
6.62
5.88 ∘
8.29 ∘
9.54
10.97 ∘
14.10 ∘
15.59
H-NRDF
5.57 ∘
4.86 ∘
5.89
6.52 ∘
7.16 ∘
8.60
7.66 ∘
10.15 ∘
12.49
HaNDF
4.74 ∘
3.67 ∘
4.38
4.96 ∘
5.52 ∘
6.58
5.20 ∘
8.00 ∘
10.08
Appendix
Table 8: Quantitative evaluation of hand denoising on HOI4D dataset across three representative noise levels. All methods receive identical noisy inputs.
Figure 10: Extended qualitative comparison of pose denoising across different methods on HOI4D. We provide further comparison cases to supplement the results in the main paper. HaNDF (ours) is evaluated against HMP and H-NRDF, with Ground Truth (GT) provided for reference.
Figure 11: Additional qualitative pose denoising iterations for hand-object interaction on HOI4D. We show two further examples of the refinement steps from t=0 to t=1000 , with Ground Truth (GT) provided for reference.
Object input
Setting
+Δq ( ∘ )
+dNN ( ∘ )
+m2m (mm)
Point noise σp
0.01
+0.00
+0.00
+0.02
0.03
+0.02
+0.06
+0.07
0.05
+0.05
+0.06
+0.09
Partial visibility
25% occluded
+0.03
+0.05
+0.06
50% occluded
+0.04
+0.07
+0.10
Point-cloud sparsity
75% subsample (control)
+0.00
+0.00
+0.01
Appendix
Table 9: Robustness to object-input perturbations. We evaluate robustness to three types of perturbations: point-wise noise at different noise levels σp , partial visibility, and point-cloud sparsity. The results demonstrate good robustness across these perturbations.
Method
Δq ( ∘ )
MPVPE (mm)
MPJPE (mm)
IntVol ( cm3 )
EasyHOI
20.82
16.64
17.06
13.28
EasyHOI + HaNDF
18.60
15.49
15.88
8.11
Improvement (%)
+10.6
+6.9
+6.9
+38.9
Appendix
Table 10: EasyHOI-based end-to-end experiment. We use EasyHOI-predicted objects to match to the nearest training object by shape, and then conditioning its latent on HaNDF to do the hand refinement.
Object encoder
Δq↓
dNN ↓
m2m (mm) ↓
HaNDF+ COD-VAE [ 15 ]
6.11 ∘
5.68 ∘
7.46
HaNDF+ Tencent-XR-3DGen [ 49 ]
5.70 ∘
5.17 ∘
6.78
HaNDF+ DeepSDF-style auto-decoder
5.52∘
4.96∘
6.58
Appendix
Table 11: Ablation on object conditioning representations. We compare three pretrained object encoders as the source of object tokens fed into HaNDF on HOI4D.
Variant
Δq↓
dNN ↓
Tokens(N)
Params
FLOPs (G)
2 layers
4.95 ∘
6.08 ∘
17
1.63
0.027
4 layers
5.52 ∘
4.96 ∘
17
3.21
0.054
8 layers
5.73 ∘
5.07 ∘
17
6.36
0.109
1 object tokens
5.52 ∘
4.96 ∘
17
3.21
0.054
64 object tokens
6.40 ∘
6.09 ∘
80
3.21
0.266
Appendix
Table 12: Ablation on transformer architecture design choices on HOI4D dataset.
Metric
Without obj. optim.
With obj. optim.
Gain
Δq(∘)
9.5985±0.0008
9.5920±0.0011
0.0065±0.0011
MPJPE
11.735±0.002
11.499±0.006
0.236±0.005
MPVPE
11.516±0.002
11.300±0.005
0.216±0.005
Appendix
Table 13: Hand reconstruction results for AlignSDF + HaNDF over five random seeds on HOI4D. The gain is computed as the error without object optimization minus the error with object optimization.
PCA dimension K
Δq(∘)↓
MPJPE (mm) ↓
MPVPE (mm) ↓
2
9.68
11.73
11.53
8
9.74
11.70
11.50
16
9.63
11.63
11.42
32 (default)
9.59
11.50
11.30
48
9.60
11.53
11.33
64 (full latent)
9.59
11.52
11.32
Appendix
Table 14: Effect of the PCA dimension on object retrieval on HOI4D. The 32-dimensional setting is used in Tab. 3 . Note that 64 dimensions correspond to the full latent space.
Iterations
10
50
100
200
350
500
750
1000
Δq(∘)↓
10.18
6.97
5.86
5.48
5.49
5.52
5.56
5.57
dNN(∘)↓
10.88
7.44
5.95
5.16
4.97
4.96
4.98
4.98
Time (ms/pose) ↓
0.6
3
6
12
21
29
44
59
Appendix
Table 15: Denoising accuracy and runtime for different numbers of Riemannian projection iterations. The default setting uses 500 iterations.
Model / Setting
Δq(∘^)↓
dNN(∘^)↓
m2m(mm)↓
Noisy input
11.78
11.46
11.25
HOI4D-HaNDF (unseen category)
8.67
7.50
9.28
OakInk-HaNDF (in-category)
5.01
3.77
6.58
Appendix
Table 16: Comparison of different models and settings.
Figure 12: Generalization to Unseen Object Categories (by testing HOI4D trained HaNDF on OakInk unseen Categories).
Method
Left
Right
Both-hand mean
HaMeR
10.00 / 9.84
10.38 / 10.38
10.19 / 10.11
+ right-hand-only prior (15 joints)
–
9.75 / 9.77
–
+ bimanual prior (30 joints)
9.13 / 8.98
9.97 / 9.99
9.55 / 9.49
Appendix
Table 17: Effect of hand priors on bimanual hand refinement using the OakInk2 dataset. Results are reported for the left hand, right hand, and both hands. The best results are shown in bold.
How can we reconstruct 3D hand poses when large portions of the hand are heavily occluded by itself or by objects? Humans often resolve such ambiguities by leveraging contextual knowledge -- such as affordances, where an object's shape and function suggest how the object is typically grasped. Inspired by this observation, we propose a generative prior for hand pose refinement guided by affordance-aware textual descriptions of hand-object interactions (HOI). Our method employs a diffusion-based generative model that learns the distribution of plausible hand poses conditioned on affordance descriptions, which are inferred from a large vision-language model (VLM). This enables the refinement of occluded regions into more accurate and functionally coherent hand poses. Extensive experiments on HOGraspNet, a 3D hand-affordance dataset with severe occlusions, demonstrate that our affordance-guided refinement significantly improves hand pose estimation over both recent regression methods and diffusion-based refinement lacking contextual reasoning.
Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.
Mingyeong Song, Jungbin Cho, Jisoo Kim +5
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1
Understanding how humans interact with the world requires a representation of physical interactions at the hand surface. Tactile sensing captures these interactions, but collecting tactile data at scale requires specialized hardware. We introduce TactoHand, a model that predicts three dense hand-surface interaction fields---contact, proximity, and 3D force---from monocular video. We derive geometry-based contact and proximity labels and motion-derived force labels from eligible training data within a curated collection spanning approximately 50 million source frames. TactoHand uses a temporal transformer to aggregate visual evidence across frames, while three learned queries decode the corresponding hand-surface fields. A differentiable physics layer further constrains force predictions using observed object motion. Across in-domain and out-of-domain benchmarks, TactoHand improves contact and proximity estimation over prior methods. In addition, physics supervision reduces position and rotation errors in motion rollouts, and the predicted contact and proximity fields improve downstream hand-object reconstruction. Together, these results show a scalable, low-cost path to learning hand-surface interaction fields without collecting tactile measurements. Code will be released at https://github.com/zhenqis123/Tactohand.
Ziheng Xi, Weibo Li, Jiayi Yu +6
Tsinghua University · Peking University · BeingBeyond