Learning Tactile-Aware Hand Representations from Human Hand-Object Interaction Data

Yeonguk Yu; Jaehong Kim; Byungok Han

cs.ROOct 20, 2026

Learning Tactile-Aware Hand Representations from Human Hand-Object Interaction Data

Authors: Yeonguk Yu, Jaehong Kim, Byungok Han

Organizations: The authors are with the Electronics and Telecommunications Research Institute (ETRI), Republic of Korea

Abstract

Tactile representation learning for dexterous manipulation often requires substantial robot-side data collection, limiting its scalability across different hands and sensors. We explore whether large-scale human hand-object interaction (HOI) data can provide a useful prior for robotic tactile learning. To this end, we propose HuTact, a framework that pretrains tactile-aware hand representations from human HOI data without requiring real tactile signals during pretraining. HuTact transfers the interaction structure learned from human hand motions to robotic tactile observations and can be further adapted with a small amount of unlabeled robot interaction data. We evaluate HuTact on tactile perception and manipulation tasks, including grasp prediction, slip detection, object classification, and tactile-conditioned policy learning. Across these tasks, human HOI pretraining improves data efficiency and provides a strong initialization for downstream learning. Notably, with HOI pretraining followed by midtraining on unlabeled robot play data, HuTact achieves 88.26% Macro-F1 on grasp prediction using only 25% of the labeled downstream data, comparable to training from scratch on the full labeled dataset (88.23%). These results suggest that human interaction data can serve as a scalable complementary source for robotic tactile representation learning.

Figures & tables

Fig. 1: Overview of HuTact. Stage 1 pretrains the encoder on human HOI data using fingertip coordinates and geometry-derived pseudo-contact signals. Stage 2 optionally adapts the representation to real robotic tactile signals using unlabeled play data, and Stage 3 fine-tunes the encoder for downstream robotic tasks.

Fig. 2: Illustration of backbone architecture. Each frame is encoded from separate tactile and wrist-local fingertip coordinate token streams. The resulting frame-level CLS tokens are then aggregated across 15 frames by a temporal Transformer to produce a window-level representation for downstream tasks.

Fig. 3: Hardware setup and downstream tasks. We use a Unitree G1 humanoid equipped with a BrainCo Revo2 dexterous hand. Downstream tasks includes grasp prediction, slip detection, object classification, cup destacking, and single-card sliding.

Task	Method	ID Last	ID Avg	OOD Last	OOD Avg
Grasp Prediction	From Scratch	88.23 $\pm$ 0.78	86.39 $\pm$ 0.38	74.92 $\pm$ 9.78	74.12 $\pm$ 9.78
	Revo2 Pretraining	92.15 $\pm$ 0.40	92.09 $\pm$ 0.35	77.84 $\pm$ 9.31	77.51 $\pm$ 9.15
	HOI Pretraining (Ours)	92.61 $\pm$ 0.74	92.18 $\pm$ 0.75	79.21 $\pm$ 10.06	78.46 $\pm$ 10.05
	HOI + Revo2 Midtraining (Ours)	93.29 $\pm$ 0.53	93.24 $\pm$ 0.54	80.27 $\pm$ 9.52	80.02 $\pm$ 9.49
Slip Detection	From Scratch	75.68 $\pm$ 1.73	73.14 $\pm$ 0.73	69.28 $\pm$ 4.55	67.39 $\pm$ 3.24
	Revo2 Pretraining	72.37 $\pm$ 1.69	71.55 $\pm$ 1.96	65.75 $\pm$ 4.38	65.80 $\pm$ 3.87

TABLE I: Comparison of pretraining strategies on downstream perception tasks. Macro-F1 scores are mean

\pm

standard deviation over 9 runs. “Last” denotes the final epoch and “Avg” the mean over evaluations every 10 epochs.

Model	Tactile Encoder	Cup Destacking	Single-Card Sliding	Avg.
ACT	None	30.0	37.5	33.8
	From Scratch	5.0	60.0	32.5
	HOI + Revo2 Midtraining (Ours)	37.5	62.5	50.0
Diffusion Policy	None	27.5	17.5	22.5
	From Scratch	37.5	40.0	38.8
	HOI + Revo2 Midtraining (Ours)	47.5	52.5	50.0

TABLE II: Success rates (%) on tactile-sensitive manipulation tasks.

Encoder Architecture	Pretraining Source	Grasp Pred.	Slip Det.	Object Cls.
Sparsh-Skin [ 8 ]	Allegro + Xela uSkin	87.8	70.6	81.0
HuTact (Ours)	HOI	92.2	73.8	87.3

TABLE III: Cross-sensor transfer performance on downstream tasks.

(a) Architecture ablation
	Temp. Encoder	Token Sep.	HOI Pretrain.	Grasp Pred.	Slip Det.	Object Cls.
A	–	–	–	84.8	60.8	81.8
B	–	–	$\checkmark$	87.7	60.0	83.7
C	–	$\checkmark$	–	84.3	61.8	83.7
D	–	$\checkmark$	$\checkmark$	91.0	60.1	87.9
E	$\checkmark$	$\checkmark$	–	86.4	73.1	83.0

TABLE IV: Ablation studies on in-distribution downstream tasks. Macro-F1 (%) is averaged every 10 epochs from 10 to 100.

Fig. 4: Downstream data efficiency on (a) grasp prediction, (b) slip detection, and (c) object classification. Macro-F1 (%) is reported as the downstream training data varies from 5% to 100%. Error bars denote standard deviation.

Robotic Tactile Sensing

Robotic tactile sensing equips machines to detect contact, force, slip, shape, and material properties for physical interaction.

See all 100 papers

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision
Jun 17, 2026Yuzhe Huang, Jiaping Wu, Jiaming Jiang +6Representation Learning Visuo-Tactile Representation Learning
Beihang University · Rimbot · BUPT +3
Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation
Jul 1, 2026Chi Zhang, Penglin Cai, Ziheng Xi +6Contact-Rich Robotic Manipulation Generalization in Robotic Manipulation
1Peking University · 2BeingBeyond · 3Tsinghua University
TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning
Oct 7, 2026Dayou Li, Hao Wang, Qianqian Yang +23Visual-to-Tactile Prediction Visuo-Tactile Representation Learning
Texas A&M University · Google DeepMind · CMU +12

Jun 17, 2026J/K move · Enter open · S save

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

Yuzhe Huang, Jiaping Wu, Jiaming Jiang, Hezhe Lin+5

Beihang University · Rimbot · BUPT+3

Representation Learning Visuo-Tactile Representation Learning Robot Manipulation Benchmarks Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

Establishing a universal benchmark for tactile representation learning in robotic manipulation remains challenging due to the diversity of tactile sensor designs, data formats, and robot embodiments. Rather than seeking to establish such, we explore a scalable and promising direction for future development: egocentric vision paired with full-hand tactile data. To this end, we introduce HT-Bench, a large-scale multi-task benchmark for dexterous full-hand tactile sensing, comprising 10M RGB frames and 7.8M tactile frames collected across 226 tasks. HT-Bench evaluates tactile representations from three key perspectives: whether they encode meaningful contact geometry, whether they can align tactile observations with visual information, and whether they generalize to unseen tasks. To assess these capabilities, HT-Bench includes four tasks: fine-grained tactile similarity retrieval, masked tactile inpainting, vision-to-tactile synthesis, and multimodal tactile frame prediction. We further propose HandTouch, a vector-quantized vision--tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training. Across HT-Bench, HandTouch consistently outperforms representative tactile encoder baselines, improving Recall@5 on fine-grained tactile similarity retrieval from 74.65% to 85.23%, reducing RMSE on masked tactile inpainting from 0.022 to 0.010, and increasing OOD cIoU on vision-to-tactile synthesis from 0.628 to 0.705. These results demonstrate the effectiveness of HandTouch and suggest that large-scale egocentric full-hand tactile data provides a scalable basis for evaluating and advancing tactile representation learning in dexterous manipulation.

Abstract

Robotic Tactile Sensing

Explore similar work

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning