TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

Dayou Li; Hao Wang; Qianqian Yang; Zihao Zhu; Haoquan Fang; Ziyao Zeng; Yan Han; Zihan Wang

cs.CVOct 7, 2026

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

Authors: Dayou Li, Hao Wang, Qianqian Yang, Zihao Zhu, Haoquan Fang, Ziyao Zeng, Yan Han, Zihan Wang, +18 more

Organizations: Texas A&M University · Google DeepMind · CMU · Stanford University · Yale University · Microsoft · Overfit Lab · NVIDIA · University of Liverpool · Meta · University of Washington · Northwestern University · Sony · Georgia Tech · UC Berkeley

Abstract

Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, yet video alone leaves the contact and pressure that characterize physical interaction unrecorded. Recent visual-tactile datasets provide this missing supervision, but their synchronized tactile data remain far smaller in volume than human video. Moreover, the largest resources often merge recordings from different sensors or annotation procedures, which makes the effect of data scale difficult to isolate. We therefore introduce TouchScale, a 500-hour dataset of contact-rich human interaction recorded with a single unified wearable setup. Its approximately 2K predefined task descriptions span everyday activities and structured manipulation, and each recording temporally aligns egocentric RGB-D video with wrist RGB video and dense full-hand bimanual tactile measurements. Compared with prior tactile data, training on the full TouchScale raises zero-shot contact IoU on data from an unseen tactile sensor from 0.134 to 0.383. Pretraining a visual encoder on TouchScale also yields the highest action recognition accuracy on three benchmarks among the compared visual-tactile datasets. Used for visual-tactile mid-training of a robot policy, TouchScale improves the average real-world success rate across four contact-rich manipulation tasks from 22.5% to 57.5%. With the sensor and collection protocol held fixed, both zero-shot tactile prediction and robot success show an overall upward trend as more TouchScale data is used. These results suggest that human visual-tactile data collected at scale with consistent sensing benefits both perception and robot manipulation. We will publicly release TouchScale, including all synchronized visual-tactile recordings and reconstructed object models, to support future research on scalable visual-tactile learning.

Figures & tables

Figure 1 : Overview. TouchScale is a large-scale human visual–tactile dataset collected with a unified wearable system. Capture: Human interactions are recorded with synchronized head RGB-D, wrist RGB from both hands, and bimanual tactile measurements. Dataset: TouchScale contains 500 hours of data, approximately 87K episodes, and approximately 2K task descriptions spanning diverse everyday interactions. Evaluation: We evaluate it on tactile prediction, action recognition, and robotic manipulation.

Dataset	Hours	Samples	Vision	Views	Hands	Sensor	Components	Taxels/hand	Coverage	Objects
STAG Sundaram et al. (2019)	NR	135K frames	–	–	Single	Glove	Normal	548	Hand	26
ActionSense DelPreto et al. (2022)	12.97	NR	RGB-D	H+6Exo	Both	Glove	Normal	682	Hand excl. tips	21
PressureVisionDB Grady et al. (2022)	16	$\sim$ 3M frames	RGB	4Exo	Single	Pad	Normal	–	Surface	–
ContactLabelDB Grady et al. (2024)	NR	500K pressure frames	RGB	Exo	Single	Pad	Normal	–	Tips	–
EgoPressure Zhao et al. (2025)	5	4.3M frames	RGB-D	H+7Exo	Single	Pad	Normal	–	Surface	–
OpenTouch Song et al. (2025)	5.1	2.9K clips	RGB	H	Single	Glove	Normal	169	Hand	$\sim$ 800 types

Table 1 : Comparison of human tactile and visual–tactile datasets. H, W, and Exo denote head, wrist, and external views, respectively; NR denotes not reported and – denotes not applicable. We report the pressure-labeled subset of ContactLabelDB and the directly recorded human subset of H-Tac as DeskTask-Tac. Among these datasets, TouchScale offers the most recorded hours and the densest glove-based tactile sensing.

Figure 2 : TouchScale data collection. Left: the unified wearable setup used throughout the collection, consisting of a head-mounted RGB-D camera, two wrist-mounted RGB cameras, and bimanual tactile gloves. Right: time-aligned observations from an example interaction, including head RGB-D, bimanual wrist RGB, and bimanual tactile measurements at different stages of the task.

Figure 3 : Task and interaction diversity in TouchScale. Left: representative tasks across different collection settings. Right: task-description verb frequencies, example task instructions, the collection-setting composition of the 500-hour dataset, and representative tactile contact patterns.

Evaluation	Training data	cIoU $\uparrow$	vIoU $\uparrow$	CoP Loc. $\downarrow$	Press. Mag. $\downarrow$
Zero-shot (EgoTactile)	EgoTouch	0.134	0.185	0.444	0.184
Zero-shot (EgoTactile)	TouchScale	0.181	0.243	0.416	0.148
Unseen split	EgoTouch	0.403	0.302	0.272	0.095
Unseen split	TouchScale	0.422	0.299	0.261	0.061

Table 2 : Tactile prediction. We compare models trained on EgoTouch and TouchScale under zero-shot transfer to EgoTactile and in-distribution evaluation on each source dataset’s unseen split. Zero-shot metrics are averaged over 12 anatomical hand regions.

Figure 4 : Scaling of zero-shot tactile prediction. Increasing TouchScale training data generally improves zero-shot performance on EgoTactile across contact, localization, and pressure metrics.

	MECCANO		SSv2		Ego-Exo4D		Avg.
Pretraining Data	Linear	FT	Linear	FT	Linear	FT	Linear	FT
OpenTouch	28.76	42.07	20.39	62.82	19.76	39.94	22.97	48.28
FEEL	17.71	21.60	1.51	12.90	4.32	5.37	7.85	13.29
EgoTouch	20.16	24.16	1.74	29.87	5.06	11.22	8.99	21.75
TouchScale	28.97	43.71	22.82	63.27	21.55	41.89	24.45	49.62

Table 3 : Downstream action recognition. TouchScale achieves the highest average accuracy under both linear probing and end-to-end fine-tuning. FT denotes end-to-end fine-tuning.

Figure 5 : Real-world robot setup and manipulation tasks. Left: the xArm6 platform with a BrainCo Revo 2 robotic hand, tactile sensing, and an external RGB-D camera and a wrist camera, together with the teleoperation setup used to collect robot demonstrations. Right: four contact-rich manipulation tasks covering material-dependent sorting, small-object manipulation, precise transfer, and sustained surface contact. We collect 50 teleoperated demonstrations for each task.

Mid-training	Soft/Hard Sorting	Bottle-Cap Removal	Test-Tube Transfer	Whiteboard Wipe	Avg.
No	10%	40%	30%	10%	22.5%
Yes	60%	70%	60%	40%	57.5%

Table 4 : Real-world robot manipulation. TouchScale mid-training improves success across all four tasks. Both variants start from the same

\mathcal{N}_{0}

-VTLA checkpoint and use the same 50 robot demonstrations per task for robot post-training. Each task is evaluated over 20 trials.

Figure 6 : Scaling of TouchScale mid-training. Left: average post-training success rate across four real-world manipulation tasks with different amounts of TouchScale mid-training data. Right: future-tactile prediction accuracy on robot data unseen during TouchScale mid-training.

Figure 7 : Cross-sensor anatomical mapping. Dataset-specific anatomical assignments pool the TouchScale, EgoTouch, and EgoTactile tactile layouts into the same 12 anatomical regions for cross-dataset evaluation.

Figure 8 : Qualitative tactile prediction. Across diverse human interactions, the predictions closely reproduce the ground-truth contact regions and pressure distributions for both hands, showing consistent tactile prediction across objects and interaction patterns.

Figure 9 : Tactile glove hardware. The Tachin Glove used in TouchScale contains an acquisition module on the dorsal side, while tactile sensing regions cover the five fingers and palm on the palmar side. The glove is connected to an external power module for wearable tactile recording.

Figure 10 : Postprocessing for data quality. Contact events and engaged fingers are first identified from wrist video without tactile input, then compared with tactile responses to detect missing signals. The resulting consistency check assigns PASS, REVIEW, or REJECT, with uncertain cases sent for human inspection.

	Train		Val		Pooled
Mid-training Data	Top-10 $\uparrow$	MRR $\uparrow$	Top-10 $\uparrow$	MRR $\uparrow$	Top-10 $\uparrow$	MRR $\uparrow$	Pool NCE $\downarrow$
None	0.18	0.0016	0.62	0.0039	0.14	0.0012	9.031
20%	2.87	0.0083	2.93	0.0165	2.34	0.0063	8.890
40%	3.11	0.0090	2.80	0.0170	2.63	0.0070	8.889
60%	3.18	0.0108	3.13	0.0154	2.58	0.0075	8.884
80%	3.24	0.0094	3.13	0.0158	2.69	0.0066	8.890

Table 5 : Future-tactile prediction accuracy on robot observations. TouchScale mid-training improves prediction accuracy, with a modest positive scaling trend in Train Top-10.

Figure 11 : Successful executions with TouchScale mid-training. Representative real-world rollouts across the four manipulation tasks. Each row shows the progression from initial interaction to successful task completion.

Figure 12 : Failure cases without TouchScale mid-training. Representative Whiteboard Wipe rollouts from the baseline $\mathcal{N}_{0}$ -VTLA policy that fail to complete the manipulation task.

Explore similar work

May 13, 2026cs.RO

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.

Jianyi Zhou, Ziteng Gao, Feiyang Hong +11

Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University

Tactile Egocentric Dataset 3D Hand Bimanual Manipulation Egocentric Video

Ask PDF

Apr 22, 2026cs.RO

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging. This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale. To address these limitations, we introduce the VTOUCH dataset. It leverages vision based tactile sensing to provide high-fidelity physical interaction signals, adopts a matrix-style task design to enable systematic learning, and employs automated data collection pipelines covering real-world, demand-driven scenarios to ensure scalability. To further validate the effectiveness of the dataset, we conduct extensive quantitative experiments on cross-modal retrieval as well as real-robot evaluation. Finally, we demonstrate real-world performance through generalizable inference across multiple robots, policies, and tasks.

Qianxi Hua, Xinyue Li, Zheng Yan +4

Humanoid Robot (Shanghai) Co., Ltd., Shanghai, China · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · School of Astronautics, Harbin Institute of Technology, Harbin, China

Tactile Bimanual Manipulation Large-Scale Robot Demonstration Datasets Multimodal Dataset Embodied Artificial Intelligence Cross-Modal

Ask PDF

Jul 24, 2026cs.RO

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/

Yunao Huang, Shiyu Sang, Suting Ni +6

ShanghaiTech University · InstAdapt

Tactile World Model Robotic Manipulation Efficient World-Action Model World Models Robot Systems

Ask PDF