TacClaw-Infinity: A Large-Scale Vision-Tactile-Language-Action Dataset for Dexterous Manipulation

Fucai Zhu; Xidong Zhang; Yichi Zhang; Yipan Wang; Fubo Wang; Zibin Wu; Kaiqi Li; Jianhao Jiao

cs.ROOct 14, 2026

TacClaw-Infinity: A Large-Scale Vision-Tactile-Language-Action Dataset for Dexterous Manipulation

Authors: Fucai Zhu, Xidong Zhang, Yichi Zhang, Yipan Wang, Fubo Wang, Zibin Wu, Kaiqi Li, Jianhao Jiao,

Organizations: Daimon Robotics · Great Bay University · Southern University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) ·

Abstract

Large-scale robot learning increasingly relies on diverse human demonstrations, yet tactile manipulation data remain difficult to scale, and existing multimodal datasets often lack consistent quality control and structured supervision. We present TacClaw-Infinity, a large-scale structured Vision-Tactile-Language-Action (VTLA) dataset for contact-rich manipulation, collected with TacClaw, a tactile-enabled UMI-style handheld interface that records synchronized visual observations, fingertip tactile signals, and end-effector trajectories without requiring a target robot during data collection. TacClaw-Infinity is built through a distributed worldwide collection across homes, laboratories, industrial workplaces, retail stores, and outdoor environments including tasks contributed by academic and industrial collaborators. Together, these efforts yield over 10,000 hours, 1 million episodes, and 1 billion frames spanning more than 1,000 task types. We develop a scalable pipeline for trajectory reconstruction, multimodal quality control, embodiment-aware filtering, and structured annotation, covering contact-centric segmentation, subtask description, and spatial grounding. We validate TacClaw-Infinity at both the data and model levels: At the data level, we assess the collection interface, the action-space compatibility of the curated demonstrations, and the structured annotations, while for the model level, we demonstrate cross-embodiment transfer across three heterogeneous robots without robot-specific demonstrations.

Figures & tables

Dataset	Collection	Distributed	Episodes	Hours	Tasks	Tactile	Temporal	Semantic	Spatial
Open X-Embodiment [ 1 ]	Real-Robot	$\checkmark$	1M+	–	527	$\times$	$\times$	$\times$	$\times$
DROID [ 2 ]	Real-Robot	$\checkmark$	76K	350	86	$\times$	$\times$	$\times$	$\times$
UMI [ 3 ]	UMI	✓	1.4K	–	4	$\times$	$\times$	$\times$	$\times$
ViTaMIn [ 4 ]	Tactile UMI	$\times$	747	–	6	$\checkmark$	$\times$	$\times$	$\times$
exUMI [ 5 ]	Tactile UMI	$\times$	1.7K	–	9	$\checkmark$	$\times$	$\times$	$\times$
Touch in the Wild [ 6 ]	Tactile UMI	$\checkmark$	2.7K+	–	43	$\checkmark$	$\times$	$\times$	$\times$

TABLE I: Comparison with representative large-scale, UMI-style, and tactile manipulation datasets. “Distributed” indicates collection across multiple physical locations or contributing institutions; temporal, semantic, and spatial annotations denote within-episode structured supervision.

Fig. 2: Contact-anchored structured annotation. (a) Tactile contact and release events partition a demonstration into manipulation stages and provide boundaries for segment-level language annotation. (b) The inter-finger point at contact provides a spatial anchor for identifying the manipulated object. Because the anchor is a single point in space, it projects into every camera of the rig, so each view is segmented independently from the same physical prior. The URDF silhouettes of both devices, rendered per view, serve as negative regions.

Fig. 3: Statistics of TacClaw-Infinity. (a) Hierarchical distribution of manipulation actions and their associated object categories. (b) Unique action–object pairs under a matched collection-time budget, multi-site versus single-site; shaded bands are $95\%$ resampling intervals over 2,000 draws. (c, d) Action and object vocabularies, with word size proportional to frequency. (e) t-SNE of tactile features colored by object category.

Fig. 4: Haptic subject experiment ( $n=9$ ). (a) Setup: a blindfolded participant closes the jaws onto a force gauge and says “stop” at first perceived contact; the three interfaces, which differ in how the fingers drive the jaws, are shown at right. (b) Perception threshold per interface. Gray markers denote participant means with within-subject standard deviation; black markers denote group mean $\pm$ standard deviation. The vertical axis uses a symlog scale (linear below $20$ gf, logarithmic above). All interfaces were fabricated in-house; UMI follows the authors’ released design files.

Fig. 5: Representative execution sequences of the three bimanual long-horizon tasks: Cup Stacking , Bottle Sorting , and Vase Wiping . The tasks respectively evaluate pick-and-place coordination, tactile-dependent state discrimination, and dynamic contact regulation.

Embodiment	Left Reach.	Right Reach.	Bimanual Reach.	Low Manip. L/R
Tianji	99.20	98.02	97.23	2.5776 / 3.1611
DM-Sparky (full-body)	100.0	100.0	100.0	0.0 / 0.0
RealMan	97.38	96.79	95.15	4.3127 / 4.3526

TABLE II: Cross-embodiment reachability & manipulability (%).

Method	F1@25	Edit	Acc	C-F1@25
MS-TCN++	$48.0\pm 2.6$	$56.9\pm 3.7$	$50.2\pm 2.3$	$43.0\pm 2.4$
FACT	$57.7\pm 0.7$	$63.8\pm 4.1$	$56.8\pm 4.4$	$45.9\pm 3.1$
LTContext	$55.4\pm 4.7$	$63.3\pm 4.8$	$62.7\pm 4.9$	$45.4\pm 4.1$
ASFormer (no tactile)	$57.6\pm 0.5$	$63.3\pm 1.2$	$69.5\pm 0.8$	$47.7\pm 1.0$
Ours	$\mathbf{63.9\pm 0.7}$	$\mathbf{67.3\pm 2.0}$	$\mathbf{70.4\pm 0.9}$	$\mathbf{54.3\pm 0.6}$

TABLE III: Contact-phase segmentation on 30 held-out per-arm sequences. The metrics of mean

\pm

standard deviation over 3 seeds are presented (%).

Policy	Embodiment	Cup Stacking	Bottle Sorting	Vase Wiping	Avg.
$\pi_{0.5}$	Tianji	90	44	28	54.0
	RealMan	86	42	16	48.0
	DM Sparky	82	44	18	48.0
$\pi_{0.5}$ + Tactile	Tianji	94	96	54	81.3
	RealMan	80	76	38	64.7
	DM Sparky	100	92	36	76.0

TABLE IV: Zero-robot cross-embodiment transfer results. All results are reported as success rates (SR, %).

Robot Manipulation Datasets

See all 43 papers

RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation
Jun 30, 2026Xinyi Wang, Donghan Li, Zi'Ang Chen +5Robotic Manipulation Bimanual Robotic Manipulation
Fudan University · ByteDance Intelligent Creation · Shanghai AI Laboratory
TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning
Oct 7, 2026Dayou Li, Hao Wang, Qianqian Yang +23Visual-to-Tactile Prediction Visuo-Tactile Representation Learning
Texas A&M University · Google DeepMind · CMU +12
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
Mar 18, 2026Xitong Chen, Yifeng Pan, Min Li +1Robotic Data Collection Contact-Rich Robotic Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China · Wuhan Huaweike Intelligent Technology Co., Ltd., Wuhan, China

Jun 30, 2026J/K move · Enter open · S save

RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation

Xinyi Wang, Donghan Li, Zi'Ang Chen, Chong Yu+4

Fudan University · ByteDance Intelligent Creation · Shanghai AI Laboratory

Robotic Manipulation Bimanual Robotic Manipulation Robotic Data Collection Humanoid Robotics Robot Imitation Learning Robot Manipulation Benchmarks+1

Open paper AskPDF arXiv

In the field of robot learning, large-scale and diverse demonstration trajectories provide the fundamental basis for enhancing robotic manipulation ability. We introduce RoboTacDex, a large, multi-modal, and diverse dataset of dexterous manipulation behaviors performed with a humanoid robot. Built on the publicly accessible humanoid robot Unitree G1, RoboTacDex consists of 6k trajectories covering 19 tasks, 23 skills, and interactions with 22 objects. RoboTacDex provides comprehensive records including multi-view RGB and depth information, tactile feedback, and detailed semantic annotations. Furthermore, the dataset features a variety of relatively challenging tasks that can only be completed by dual arms and dexterous hands, aiming to mimic human-like operational logic and simulate real-world manipulation complexity. To ensure data collection quality, we develop an improved multi-camera synchronization system to enable millisecond data synchronization and recording of modalities. In our experiments, we evaluate three representative imitation learning models on our dataset, analyzing their performance as well as their respective strengths and limitations across different task categories. Successful trial results and a moderate level of generalization capabilities across a suite of tasks indicate the effectiveness and diversity of the collected dataset. Our dataset will be open-sourced soon.

Abstract

Robot Manipulation Datasets

Explore similar work

RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation