TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video

Ziheng Xi; Weibo Li; Jiayi Yu; Sipeng Zheng; Wanpeng Zhang; Haiweng Xu; Hao Luo; Chuan Mao

cs.CVOct 29, 2026

TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video

Authors: Ziheng Xi, Weibo Li, Jiayi Yu, Sipeng Zheng, Wanpeng Zhang, Haiweng Xu, Hao Luo, Chuan Mao,

Organizations: Tsinghua University · Peking University · BeingBeyond

Abstract

Understanding how humans interact with the world requires a representation of physical interactions at the hand surface. Tactile sensing captures these interactions, but collecting tactile data at scale requires specialized hardware. We introduce TactoHand, a model that predicts three dense hand-surface interaction fields---contact, proximity, and 3D force---from monocular video. We derive geometry-based contact and proximity labels and motion-derived force labels from eligible training data within a curated collection spanning approximately 50 million source frames. TactoHand uses a temporal transformer to aggregate visual evidence across frames, while three learned queries decode the corresponding hand-surface fields. A differentiable physics layer further constrains force predictions using observed object motion. Across in-domain and out-of-domain benchmarks, TactoHand improves contact and proximity estimation over prior methods. In addition, physics supervision reduces position and rotation errors in motion rollouts, and the predicted contact and proximity fields improve downstream hand-object reconstruction. Together, these results show a scalable, low-cost path to learning hand-surface interaction fields without collecting tactile measurements. Code will be released at https://github.com/zhenqis123/Tactohand.

Figures & tables

Figure 1: Overview of TactoHand. A shared video encoder extracts frame-level features, aggregated by a temporal transformer. Three learned queries decode the contact, proximity, and 3D force interaction fields at corresponding MANO vertices. Predicted vertex forces are aggregated into an object-level wrench for differentiable rigid-body supervision. Contact, proximity, force, and pose losses are applied according to the supervision available for each sample.

Figure 2: Contact-label construction: normal filtering (left) and inside-mesh inclusion (right).

Figure 3: Force allocation geometry and weighting. Upper left: a local polyhedral friction cone constrains each active vertex force. Upper right: signed distance $s$ distinguishes penetrated ( $s<0$ ), contacting ( $s=0$ ), and separated ( $s>0$ ) vertices. Lower left: vertex-local force directions on the hand surface. Lower right: the penalty weight $(q(s)^{2}+\epsilon)^{-1}$ rises toward the relaxed $20$ mm candidate cut-off.

Figure 4: Differentiable rigid-body supervision. Indices

i,i+1,i+2

are 10 Hz image timestamps; each transition contains three 30 Hz force intervals. Predicted vertex forces determine the object wrench at the image-aligned interval, while the two intermediate intervals use detached motion-derived reference forces. Pose errors supervise the propagated state.

Dataset	Method	ADE $\downarrow$	ADE-100 $\downarrow$	Accel. $\downarrow$	Method	P $\uparrow$	R $\uparrow$	Macro-F1 $\uparrow$	Micro-F1 $\uparrow$
	Proximity				Contact
HOCap ‡	InterField-SF	$132.51$	$38.15$	$1.06$	HACO	$11.1$	77.7	$18.0$	$19.5$
	InterField-LSTM	$134.69$	$40.10$	0.47	EgoPHI	$10.3$	$37.3$	$11.4$	$16.2$
	Ours (SF)	$46.09$	6.35	$1.35$	Ours (SF)	68.9	$57.7$	$50.1$	$62.8$
	Ours (T)	42.51	$6.40$	$1.07$	Ours (T)	$68.8$	$60.6$	50.5	64.5
ARCTIC (s05, ego)	InterField-SF	$29.65$	$9.81$	$0.82$	HACO	$42.2$	82.2	$42.2$	$55.8$

Table 1: Proximity (left) and contact (right). ADE is in mm, Accel. in m/s 2 , and P/R/F1 in percent. ADE-100 clips predictions and targets at 100 mm; Macro-F1 averages F1 over frame–hand pairs with ground-truth or predicted contact. Bold marks the best per dataset.

\ddagger

HOCap is excluded from training. See Appendix D for metrics.

Figure 5: Contact comparison showing the input scene, hand crop, predicted contact maps, and geometry-derived reference.

Model	Physics	Vertex MAE $\downarrow$	Active MAE $\downarrow$	Force angle $\downarrow$	Rollout ADE $\downarrow$	Rollout angle $\downarrow$
SF	Off	12.6	124.9	28.2	5.3	45.5
SF	On	12.9	126.7	29.8	3.8	27.4
T	Off	13.7	126.4	31.0	5.7	45.2
T	On	12.8	122.7	27.5	3.3	29.4

Table 2: Force and rollout ablation on DexYCB (DINOv3 ViT-L). SF/T: single-frame/temporal; MAE: mN; rollout ADE: cm; angles: degrees. Active targets exceed

0.05

N. The

0.4

s rollout uses reference-assisted intervals and excludes divergent windows (Appendix D.3 ); bold indicates column minima.

		PA-VPE $\downarrow$			$\mathrm{CD}_{\mathrm{ho}}$ $\downarrow$
Dataset	Method	Base	+ HACO	+ Ours	Base	+ HACO	+ Ours
DexYCB	EasyHOI	9.937	9.516	9.077	24.140	12.552	2.004
	ArtHOI	7.317	6.027	4.626	11.587	13.627	1.431
HOCap	EasyHOI	9.179	9.051	7.592	59.416	68.943	11.461
	ArtHOI	6.984	6.684	6.073	5.234	5.199	1.104
MOW	EasyHOI	14.832	14.276	13.939	69.080	72.638	15.266

Table 3: Controlled reconstruction refinement. PA-VPE: mm;

\mathrm{CD}_{\mathrm{ho}}

: bidirectional mean squared hand–object Chamfer in

10^{-3}\,\mathrm{m}^{2}

. Bold marks each method–dataset best.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Sequences	Frames
Force labels: articulated objects
ARCTIC ( Fan et al., 2023 )	300	1 903 869
HOI4D ( Liu et al., 2022 )	2 973	891 900
Force labels: rigid objects
GigaHands † ( Fu et al., 2025 )	1 424	9 449 724
Show3D ( Rim et al., 2026 )	1 682	6 931 884

Appendix

Table 4: Data sources and labels in the curated collection.

Figure 6: Comparison of contact labels generated by HACO’s distance-only rule and our geometric rule. Normal filtering removes back-facing false positives, while inside-mesh inclusion recovers penetrating vertices missed by distance thresholding.

Methods	PA-VPE $\downarrow$	PA-JPE $\downarrow$	$\mathrm{CD}_{\mathrm{ho}}$ $\downarrow$	$\mathrm{F\text{-}5}_{\mathrm{ho}}$ $\uparrow$	$\mathrm{F\text{-}10}_{\mathrm{ho}}$ $\uparrow$	ACC $\downarrow$
DexYCB
EasyHOI	$9.937$	$10.483$	$24.140$	$0.3609$	$0.5359$	–
EasyHOI + HACO	$9.516$	$10.032$	$12.552$	$0.3609$	$0.5392$	–
EasyHOI + Ours	9.077	9.559	2.004	0.3817	0.5606	–
ArtHOI	$7.317$	$7.520$	$11.587$	$0.3930$	$0.5240$	$4311.6$
ArtHOI + HACO	$6.027$	$6.190$	$13.627$	$0.3970$	$0.5300$	$3978.9$

Appendix

Table 5: Full reconstruction results. PA-VPE/JPE: mm;

\mathrm{CD}_{\mathrm{ho}}

10^{-3}\,\mathrm{m}^{2}

; F-5/F-10: 5/10 mm; ACC:

\mathrm{mm}/\mathrm{s}^{2}

. Bold marks the best per method–dataset block; shading highlights PA-VPE and

\mathrm{CD}_{\mathrm{ho}}

Figure 7: HOCap ( Wang et al., 2025 ) reconstructions. Red circles highlight hand–object contact artifacts in the HACO variants.

Figure 8: EasyHOI ( Liu et al., 2025 ) reconstruction ablation. + Contact: contact-normal term; + Proximity: distance term; + Both: both terms. Values are per-example PA-VPE (mm).

Setting	Value
Hardware and batch	One node with eight A800 GPUs; 7 samples per GPU; two accumulation steps (112 samples per update)
Optimizer	AdamW; learning rate $2\times 10^{-5}$ for backbone and heads; weight decay $10^{-4}$ ; cosine decay with two-epoch warmup
Window sampling	32 frames, stride 32; each epoch samples 20% of the windows allocated to each dataset
Rollout-loss weight	$10^{-4}$ with physics; $0$ without physics
Dataset sampling multipliers	$4$ : MOW; $2$ : DexYCB; $0.5$ : HOI4D, OakInk-v2; $0.1$ : DECAF, ReInterHand; $0.05$ : HOGraspNet, ObMan, GigaHands; $0.02$ : InterHand2.6M

Appendix

Table 6: Training settings and dataset sampling multipliers. Unlisted training sources use a multiplier of

1

Stage	Updates	Optimized variables	Learning rate
ArtHOI alignment	250	Hand translation	$10^{-2}\!\to\!5\times 10^{-4}$
ArtHOI hand refinement	399	Finger pose, hand translation	$10^{-3}\!\to\!10^{-4}$
ArtHOI penetration correction	300	Hand translation, global orientation	$10^{-3}$
ArtHOI smoothing	300	Hand translation	$5\times 10^{-4}$
EasyHOI global alignment	$20\times 100$	Hand-global parameters, object translation	$5\times 10^{-2}$
EasyHOI hand refinement	1,000	Pose residual, object translation; MANO shape	$10^{-2}$ ; $10^{-4}$ for shape

Appendix

Table 7: Reconstruction stages and optimization settings shared across the compared variants.

Visual-to-Tactile Prediction

See all 19 papers

TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation
Sep 17, 2026Danyan Zhou, Jinxuan Lu, Jiawei Lin +3Vision-Based Tactile Sensing Visual-to-Tactile Prediction
THU · Xspark AI · HKUST (GZ) +1
DenseHap: Dense Hand Haptic Estimation from Monocular Images
Oct 19, 2026Junhan Xue, Rolandos Alexandros PotamiasHand-Object Interaction Detection Hand-Object Interaction
Imperial College London
Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision
Oct 20, 2026Minseong Kweon, Qiao Feng, Yiming Huang +1Visuo-Tactile Representation Learning Hand-Object Interaction
University of Minnesota · University of Pennsylvania

Sep 17, 2026J/K move · Enter open · S save

TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation

Danyan Zhou, Jinxuan Lu, Jiawei Lin, Tianxing Chen+2

THU · Xspark AI · HKUST (GZ)+1

Vision-Based Tactile Sensing Visual-to-Tactile Prediction Tactile Sensing Synthetic Data Generation for Robotics Contact Force Estimation

Open paper AskPDF arXiv

Tactile signals provide direct contact and force measurements that are essential for understanding physical interactions and enabling dexterous robotic manipulation. However, tactile sensing requires direct measurement at contact interfaces, making large-scale data collection reliant on intrusive, costly, and restrictive instrumentation. We present TouchSight, a monocular egocentric vision framework for dense full-hand contact force prediction that leverages 500 hours of pressure-glove recordings and extensive hand-object interaction (HOI) data. To address the appearance gap between gloved training data and bare-hand real-world scenarios, we construct TwinTouch-20H: 20 hours of paired visual data in which generative video models re-render gloved recordings as bare-hand observations against new backgrounds while preserving the original measured tactile labels. TouchSight predicts dense force from both gloved and generated bare-hand videos, outperforms prior contact prediction methods on OakInk2, qualitatively generalizes to natural bare-hand egocentric videos from unseen datasets, and improves consistently as glove supervision scales. These results demonstrate that dense tactile signals can be recovered from egocentric vision alone, without tactile instrumentation at capture time.

Abstract

Visual-to-Tactile Prediction

Explore similar work

TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation

DenseHap: Dense Hand Haptic Estimation from Monocular Images

Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision