Understanding how humans interact with the world requires a representation of physical interactions at the hand surface. Tactile sensing captures these interactions, but collecting tactile data at scale requires specialized hardware. We introduce TactoHand, a model that predicts three dense hand-surface interaction fields---contact, proximity, and 3D force---from monocular video. We derive geometry-based contact and proximity labels and motion-derived force labels from eligible training data within a curated collection spanning approximately 50 million source frames. TactoHand uses a temporal transformer to aggregate visual evidence across frames, while three learned queries decode the corresponding hand-surface fields. A differentiable physics layer further constrains force predictions using observed object motion. Across in-domain and out-of-domain benchmarks, TactoHand improves contact and proximity estimation over prior methods. In addition, physics supervision reduces position and rotation errors in motion rollouts, and the predicted contact and proximity fields improve downstream hand-object reconstruction. Together, these results show a scalable, low-cost path to learning hand-surface interaction fields without collecting tactile measurements. Code will be released at https://github.com/zhenqis123/Tactohand.
Figures & tables
Figure 1: Overview of TactoHand. A shared video encoder extracts frame-level features, aggregated by a temporal transformer. Three learned queries decode the contact, proximity, and 3D force interaction fields at corresponding MANO vertices. Predicted vertex forces are aggregated into an object-level wrench for differentiable rigid-body supervision. Contact, proximity, force, and pose losses are applied according to the supervision available for each sample.
Figure 2: Contact-label construction: normal filtering (left) and inside-mesh inclusion (right).
Figure 3: Force allocation geometry and weighting. Upper left: a local polyhedral friction cone constrains each active vertex force. Upper right: signed distance s distinguishes penetrated ( s<0 ), contacting ( s=0 ), and separated ( s>0 ) vertices. Lower left: vertex-local force directions on the hand surface. Lower right: the penalty weight (q(s)2+ϵ)−1 rises toward the relaxed 20 mm candidate cut-off.
Figure 4: Differentiable rigid-body supervision. Indices i,i+1,i+2 are 10 Hz image timestamps; each transition contains three 30 Hz force intervals. Predicted vertex forces determine the object wrench at the image-aligned interval, while the two intermediate intervals use detached motion-derived reference forces. Pose errors supervise the propagated state.
Proximity
Contact
Dataset
Method
ADE ↓
ADE-100 ↓
Accel. ↓
Method
P ↑
R ↑
Macro-F1 ↑
Micro-F1 ↑
HOCap ‡
InterField-SF
132.51
38.15
1.06
HACO
11.1
77.7
18.0
19.5
InterField-LSTM
134.69
40.10
0.47
EgoPHI
10.3
37.3
11.4
16.2
Ours (SF)
46.09
6.35
1.35
Ours (SF)
68.9
57.7
50.1
62.8
Ours (T)
42.51
6.40
1.07
Ours (T)
68.8
60.6
50.5
64.5
ARCTIC (s05, ego)
InterField-SF
29.65
9.81
0.82
HACO
42.2
82.2
42.2
55.8
Table 1: Proximity (left) and contact (right). ADE is in mm, Accel. in m/s 2 , and P/R/F1 in percent. ADE-100 clips predictions and targets at 100 mm; Macro-F1 averages F1 over frame–hand pairs with ground-truth or predicted contact. Bold marks the best per dataset. ‡ HOCap is excluded from training. See Appendix D for metrics.
Figure 5: Contact comparison showing the input scene, hand crop, predicted contact maps, and geometry-derived reference.
Model
Physics
Vertex MAE ↓
Active MAE ↓
Force angle ↓
Rollout ADE ↓
Rollout angle ↓
SF
Off
12.6
124.9
28.2
5.3
45.5
SF
On
12.9
126.7
29.8
3.8
27.4
T
Off
13.7
126.4
31.0
5.7
45.2
T
On
12.8
122.7
27.5
3.3
29.4
Table 2: Force and rollout ablation on DexYCB (DINOv3 ViT-L). SF/T: single-frame/temporal; MAE: mN; rollout ADE: cm; angles: degrees. Active targets exceed 0.05 N. The 0.4 s rollout uses reference-assisted intervals and excludes divergent windows (Appendix D.3 ); bold indicates column minima.
PA-VPE ↓
CDho↓
Dataset
Method
Base
+ HACO
+ Ours
Base
+ HACO
+ Ours
DexYCB
EasyHOI
9.937
9.516
9.077
24.140
12.552
2.004
ArtHOI
7.317
6.027
4.626
11.587
13.627
1.431
HOCap
EasyHOI
9.179
9.051
7.592
59.416
68.943
11.461
ArtHOI
6.984
6.684
6.073
5.234
5.199
1.104
MOW
EasyHOI
14.832
14.276
13.939
69.080
72.638
15.266
Table 3: Controlled reconstruction refinement. PA-VPE: mm; CDho : bidirectional mean squared hand–object Chamfer in 10−3m2 . Bold marks each method–dataset best.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Sequences
Frames
Force labels: articulated objects
ARCTIC ( Fan et al., 2023 )
300
1 903 869
HOI4D ( Liu et al., 2022 )
2 973
891 900
Force labels: rigid objects
GigaHands † ( Fu et al., 2025 )
1 424
9 449 724
Show3D ( Rim et al., 2026 )
1 682
6 931 884
Appendix
Table 4: Data sources and labels in the curated collection.
Figure 6: Comparison of contact labels generated by HACO’s distance-only rule and our geometric rule. Normal filtering removes back-facing false positives, while inside-mesh inclusion recovers penetrating vertices missed by distance thresholding.
Methods
PA-VPE ↓
PA-JPE ↓
CDho↓
F-5ho↑
F-10ho↑
ACC ↓
DexYCB
EasyHOI
9.937
10.483
24.140
0.3609
0.5359
–
EasyHOI + HACO
9.516
10.032
12.552
0.3609
0.5392
–
EasyHOI + Ours
9.077
9.559
2.004
0.3817
0.5606
–
ArtHOI
7.317
7.520
11.587
0.3930
0.5240
4311.6
ArtHOI + HACO
6.027
6.190
13.627
0.3970
0.5300
3978.9
Appendix
Table 5: Full reconstruction results. PA-VPE/JPE: mm; CDho : 10−3m2 ; F-5/F-10: 5/10 mm; ACC: mm/s2 . Bold marks the best per method–dataset block; shading highlights PA-VPE and CDho .
Figure 7: HOCap ( Wang et al., 2025 ) reconstructions. Red circles highlight hand–object contact artifacts in the HACO variants.
Figure 8: EasyHOI ( Liu et al., 2025 ) reconstruction ablation. + Contact: contact-normal term; + Proximity: distance term; + Both: both terms. Values are per-example PA-VPE (mm).
Setting
Value
Hardware and batch
One node with eight A800 GPUs; 7 samples per GPU; two accumulation steps (112 samples per update)
Optimizer
AdamW; learning rate 2×10−5 for backbone and heads; weight decay 10−4 ; cosine decay with two-epoch warmup
Window sampling
32 frames, stride 32; each epoch samples 20% of the windows allocated to each dataset
Tactile signals provide direct contact and force measurements that are essential for understanding physical interactions and enabling dexterous robotic manipulation. However, tactile sensing requires direct measurement at contact interfaces, making large-scale data collection reliant on intrusive, costly, and restrictive instrumentation. We present TouchSight, a monocular egocentric vision framework for dense full-hand contact force prediction that leverages 500 hours of pressure-glove recordings and extensive hand-object interaction (HOI) data. To address the appearance gap between gloved training data and bare-hand real-world scenarios, we construct TwinTouch-20H: 20 hours of paired visual data in which generative video models re-render gloved recordings as bare-hand observations against new backgrounds while preserving the original measured tactile labels. TouchSight predicts dense force from both gloved and generated bare-hand videos, outperforms prior contact prediction methods on OakInk2, qualitatively generalizes to natural bare-hand egocentric videos from unseen datasets, and improves consistently as glove supervision scales. These results demonstrate that dense tactile signals can be recovered from egocentric vision alone, without tactile instrumentation at capture time.
As dexterous robotic manipulation advances, scalable annotations of human hand-object interactions are becoming increasingly important. Furthermore, previous hand contact estimation methods suffered from issues such as dorsal-side misclassifications and insufficient precision, while lacking a unified framework for integrating multiple hand to environment interaction modalities, including distance fields, contact thresholds, and pressure. To address that, we present DenseHap, a unified end-to-end approach to estimate hand distance field recognition, comprehensive hand contact detection, hand pressure analysis, and hand reconstruction from a single monocular image. By incorporating three distinctly defined heads, along with a pretrained 3D reconstructed skeleton and an enhanced dataset, DenseHap outperforms previous state of the art models on four evaluation datasets, improving the F1 score by 8.5 percentage points (a 16% relative improvement) on untrained datasets and by 3 percentage points (5% relative improvement) on in-distribution datasets. Furthermore, DenseHap has effectively improved short-range hand estimate and pressure evaluation. Code and models will be publicly available upon the paper's acceptance.
Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.
Minseong Kweon, Qiao Feng, Yiming Huang +1
University of Minnesota · University of Pennsylvania