Dense hand contact estimation from a single RGB image is a foundational primitive for dexterous manipulation and 3D interaction analysis. Direct predictors typically decode the entire per-vertex contact map from one shared query, leaving individual vertices ungrounded in the image and smearing contact across adjacent, non-touching surfaces. We present VACO, a vertex-aligned decoder in which each MANO vertex samples local image features at its projected location. A hand-reconstruction backbone, jointly trained for reconstruction and contact, predicts these locations at inference without an auxiliary reconstructor. Training uses noise-perturbed ground-truth projections and end-to-end feature adaptation to improve robustness and contact-sensitive features. A single VACO checkpoint sets a new state of the art on five benchmarks spanning hand-object, hand-human, hand-scene, and bimanual interactions. It improves F1 by +20.7 pp over the strongest prior direct predictor on Hi4D and generalizes to the held-out HOT3D and HOCap, with gains of +5.5 and +2.7 pp. The resulting contact predictions also improve downstream HOI reconstruction.
Figures & tables
Figure 1: Dense hand contact paradigms. (a) Geometric proximity after reconstruction. (b) Shared-query direct prediction. (c) VACO samples image evidence at projected vertices.
Figure 2: VACO pipeline. A co-trained WiLoR Potamias et al. (2025) backbone predicts MANO vertices and image features. Multi-scale sampling at vertex projections forms contact tokens; four decoder blocks combine mesh-wide self-attention with image cross-attention before per-vertex prediction.
Figure 3: Qualitative comparison. Input, ground truth, and predictions from VACO, HACO, DECO, and FollowMyHold. Green denotes contact; red circles mark corresponding regions across methods, exposing baseline smearing or omissions.
DexYCB
Hi4D
RICH
HOT3D ∗
HOCap ∗
Hand–Object
Hand–Human
Hand–Scene
Hand–Object
Hand–Object
Method
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
DECO
0.224
0.314
0.199
0.390
0.394
0.343
0.167
0.123
0.126
0.211
0.352
0.205
0.085
0.120
0.075
FollowMyHold †
0.228
0.521
0.245
–
–
–
–
–
–
0.127
0.317
0.128
0.132
0.512
0.180
HACO
0.590
0.800
0.657
0.668
0.709
0.653
0.741
0.899
0.781
0.502
0.727
0.560
0.351
0.775
0.461
VACO (Ours)
0.606
0.859
0.693
0.811
0.943
0.860
0.818
0.908
0.840
0.579
0.723
0.615
0.367
0.842
0.488
Table 1: Contact prediction across interaction paradigms. Per-image P/R/F1 under HACO’s proximity protocol. The first three datasets are in-domain; ∗ denotes held-out datasets. † FollowMyHold is evaluated on 100 samples per split at its reported thresholds (10 mm for DexYCB/HOT3D; 5 mm for HOCap); “–” denotes inapplicable settings. Best results are bold .
Variant
P ↑
R ↑
F1 ↑
(a)
Full VACO (Ours)
0.636
0.855
0.699
training recipe
(b)
w/o reconstruction co-training
0.616
0.846
0.685
(c)
and w/o projection noise
0.618
0.844
0.684
model and features
(d)
w/o encoder fine-tuning
0.481
0.688
0.486
Table 2: Ablations averaged over five benchmarks. Rows (b)–(c) are cumulative; rows (d)–(g) independently modify the full model (a). ‡ Row (e) replaces the sampled features with random features at test time without retraining; all other variants are trained. Text deltas use unrounded values.
Figure 4: Qualitative HOI reconstruction. Left: input frames and VACO contacts (green). Right: AGILE without and with the contact prior, each shown from four viewpoints. The prior yields a tighter grasp with fewer floating-finger and interpenetration artifacts.
Method (HaMeR backbone)
P ↑
R ↑
F1 ↑
HACO Jung and Lee (2025)
0.570
0.782
0.622
VACO (Ours)
0.619
0.825
0.677
Table 3: Matched HaMeR-backbone comparison over five benchmarks. Both encoders are fine-tuned with the same initialization, data pool, augmentations, losses, optimizer and schedule; HACO uses its released weights. This VACO variant omits reconstruction co-training and projection noise, and takes its sampling locations from official WiLoR predictions.
Vertices projected at inference
P ↑
R ↑
F1 ↑
Our own prediction (default)
0.636
0.855
0.699
Ground-truth projections (oracle)
0.630
0.866
0.699
ground-truth projections perturbed at inference
σ=0.02
0.637
0.849
0.697
σ=0.05
0.634
0.838
0.691
σ=0.10
0.625
0.802
0.673
Table 4: Inference scaffold quality over five benchmarks. Only 2D sampling locations vary. The noise σ , measured relative to crop size, is 1× , 2.5× , or 5× the training level.
Metric
WiLoR
VACO
AUC-J ↑
0.839
0.870
PA-MPJPE (mm) ↓
8.05
6.52
AUC-V ↑
0.846
0.866
PA-MPVPE (mm) ↓
7.72
6.70
F@5 ↑
0.642
0.704
F@15 ↑
0.977
0.983
Table 5: Hand reconstruction over five benchmarks. We compare the co-trained VACO backbone with its WiLoR Potamias et al. (2025) initialization; the last block separates evaluation regimes.
CD ↓
F@5 ↑
F@10 ↑
AGILE
23.11±22.77
12.07±8.15
30.43±16.75
+ VACO contact
12.46±8.65
15.32±4.79
37.87±11.15
Table 6: HOI reconstruction on HO3D-v3. Right-hand CD (mm) and F-scores (%), reported as mean ±σ across sequences.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
# Frames
τ
ARCTIC Fan et al. [2023]
bimanual hand–object
147,235
10 mm
Decaf Shimada et al. [2023]
single-hand hand–face
65,310
native
DexYCB Chao et al. [2021]
single-hand hand–object
29,640
10 mm
H2O Kwon et al. [2021]
bimanual hand–object
33,465
10 mm
H2O3D Hampali et al. [2022]
bimanual hand–object
54,840
10 mm
Hi4D Yin et al. [2023]
two-person, four hands
24,315
native
Appendix
Table 7: Training datasets used by VACO. “Domain” summarizes the interaction type and the number of hands annotated per frame. “# Frames” lists the number of frames used per dataset after HACO’s frame subsampling. “ τ ” is the per-category proximity threshold (mm) used to derive per-vertex contact labels following HACO; “native” indicates that the dataset ships its own per-vertex contact ground truth.
Figure 5: Additional qualitative comparisons on DexYCB, HOCap, and HOT3D. For each row we show the input image, the ground-truth contact map, and per-vertex contact predictions from HACO Jung and Lee [2025] , DECO Tripathi et al. [2023] , and VACO (Ours). VACO consistently produces sharper, vertex-localized contact patterns, while shared-query baselines smear activations across adjacent fingers.
Figure 6: Multi-hand contact predictions on in-the-wild scenes. Each example contains 2–3 hands of mixed handedness; VACO is invoked once per detected hand, with left hands routed through the mirroring trick and right hands run natively. Predictions remain vertex-localized and consistent across hands within the same scene.
Dense hand contact estimation requires both high-level semantic understanding and fine-grained geometric reasoning of human interaction to accurately localize contact regions. Recently, multi-modal large language models (MLLMs) have demonstrated strong capabilities in understanding visual semantics, enabled by vision-language priors learned from large-scale data. However, leveraging MLLMs for dense hand contact estimation remains underexplored. There are two major challenges in applying MLLMs to dense hand contact estimation. First, encoding explicit 3D hand geometry is difficult, as MLLMs primarily operate on vision and language modalities. Second, capturing fine-grained vertex-level contact remains challenging, as MLLMs tend to focus on high-level semantics rather than detailed geometric reasoning. To address these challenges, we propose ContactPrompt, a training-free and zero-shot approach for dense hand contact estimation using MLLMs. To effectively encode 3D hand geometry, we introduce a detailed hand-part segmentation and a part-wise vertex-grid representation that provide structured, localized geometric information. To enable accurate and efficient dense contact prediction, we develop a multi-stage contact reasoning with part conditioning, progressively bridging global semantics and fine-grained geometry. Therefore, our method effectively leverages the reasoning capabilities of MLLMs to enable precise dense hand contact estimation. Surprisingly, the proposed approach achieves performance comparable to the previous state-of-the-art method trained on large-scale dense contact datasets, without requiring any task-specific training.
Daniel Sungho Jung, Kyoung Mu Lee
IPAI · Dept. of ECE & ASRI, Seoul National University, Korea
As dexterous robotic manipulation advances, scalable annotations of human hand-object interactions are becoming increasingly important. Furthermore, previous hand contact estimation methods suffered from issues such as dorsal-side misclassifications and insufficient precision, while lacking a unified framework for integrating multiple hand to environment interaction modalities, including distance fields, contact thresholds, and pressure. To address that, we present DenseHap, a unified end-to-end approach to estimate hand distance field recognition, comprehensive hand contact detection, hand pressure analysis, and hand reconstruction from a single monocular image. By incorporating three distinctly defined heads, along with a pretrained 3D reconstructed skeleton and an enhanced dataset, DenseHap outperforms previous state of the art models on four evaluation datasets, improving the F1 score by 8.5 percentage points (a 16% relative improvement) on untrained datasets and by 3 percentage points (5% relative improvement) on in-distribution datasets. Furthermore, DenseHap has effectively improved short-range hand estimate and pressure evaluation. Code and models will be publicly available upon the paper's acceptance.
Understanding how humans interact with the world requires a representation of physical interactions at the hand surface. Tactile sensing captures these interactions, but collecting tactile data at scale requires specialized hardware. We introduce TactoHand, a model that predicts three dense hand-surface interaction fields---contact, proximity, and 3D force---from monocular video. We derive geometry-based contact and proximity labels and motion-derived force labels from eligible training data within a curated collection spanning approximately 50 million source frames. TactoHand uses a temporal transformer to aggregate visual evidence across frames, while three learned queries decode the corresponding hand-surface fields. A differentiable physics layer further constrains force predictions using observed object motion. Across in-domain and out-of-domain benchmarks, TactoHand improves contact and proximity estimation over prior methods. In addition, physics supervision reduces position and rotation errors in motion rollouts, and the predicted contact and proximity fields improve downstream hand-object reconstruction. Together, these results show a scalable, low-cost path to learning hand-surface interaction fields without collecting tactile measurements. Code will be released at https://github.com/zhenqis123/Tactohand.
Ziheng Xi, Weibo Li, Jiayi Yu +6
Tsinghua University · Peking University · BeingBeyond