Organizations: Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan. · CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan.
Tactile sensing plays an important role in imitation learning for contact-rich manipulation by providing direct observations of physical contact with objects. Recently, several tactile foundation models (TFMs) have been proposed to learn general-purpose tactile representations through pretraining on large-scale tactile datasets. However, how these pretrained tactile representations should be leveraged for end-to-end imitation learning remains insufficiently explored. In this work, we integrate multiple TFMs into Diffusion Policy and systematically evaluate their effectiveness for contact-rich manipulation under common experimental settings. We further investigate adaptation strategies, including encoder fine-tuning and task-specific mid-training, to understand how pretrained tactile representations can be effectively used for downstream imitation learning. Through extensive evaluation across diverse manipulation tasks in both real-world and simulation environments, including more than 5,000 real-world policy rollouts, we provide empirical insights into the use of TFMs for end-to-end imitation learning. Project page: https://mmurooka.github.io/tfm4il-project-page/
Figures & tables
Fig. 1: Imitation learning with tactile foundation models.
Fig. 2: Overview of the policy architecture with a pretrained tactile encoder.
Fig. 3: Contact-rich manipulation tasks used in the experiments. The top and bottom rows show the front-view and tactile images, respectively. The five left columns correspond to real-world tasks, while the two right columns correspond to simulation tasks.
Real world
Simulation
Policy
MarkerDrawing
HeavyBoxSorting
PaperBoxHandling
SpoonSorting
Avg.
PlateInsertion
SpoonSorting
Avg.
DP-ResNetCamera
8.3
47.2
47.2
55.6
39.6
3.3
62.5
32.9
DP-ResNetTactile
38.9
50.0
55.6
72.2
54.2
66.7
87.5
77.1
DP-SparshMAE
47.2
63.9
50.0
69.4
57.6
51.7
95.8
73.8
DP-SparshDINO
27.8
58.3
66.7
80.6
58.3
41.7
100.0
70.8
DP-SparshDINOv2
58.3
72.2
58.3
80.6
67.4
70.0
100.0
85.0
TABLE I: Task success rates (%) with different tactile representations in real-world and simulation experiments.
Fig. 4: Effect of fine-tuning. Each bar shows the mean task success rate over 15 training seeds, with 12 evaluation rollouts per seed. Error bars indicate the standard deviation across training seeds. Average denotes the mean success rate across the four real-world tasks.
Fig. 5: Effect of task-specific mid-training. Each bar shows the mean task success rate over 10 training seeds, with 12 evaluation rollouts per seed. Error bars indicate the standard deviation across training seeds. Average denotes the mean success rate across the four real-world tasks.
Despite the success of vision-based generalist robotic policies, existing tactile-based policies remain tied to fixed embodiments and sensor setups. This is because tactile signals are highly heterogeneous across hardware, making cross-sensor generalization difficult. We present FTP-1,the first generalist foundation tactile policy pretrained to acquire transferable tactile manipulation abilities across diverse sensors and embodiments. FTP-1 supports varied tactile inputs, including image-, array-, and state-based signals, by using heterogeneous encoders to project them into unified morphology-aware latent tokens that are jointly modeled by a shared tactile Transformer expert. Pretrained on around 3,000 hours of tactile manipulation data aggregated from 26 data sources, spanning human and robot demonstrations across 21 sensors, FTP-1 learns tactile skills that transfer beyond the sensors seen during pretraining. Across downstream finetuning experiments spanning 5 hardware configurations, FTP-1 improves contact-rich manipulation on seen sensor setups by +17.2% and, surprisingly, transfers to two previously unseen tactile-sensor setups, achieving a +31% gain in success rate. FTP-1 establishes the first unified foundation baseline for tactile manipulation, providing future tactile policies with a shared model-level starting point. Pretrained models, datasets, training code and more visualization at https://ftp1-policy.github.io.
Chengbo Yuan, Zicheng Zhang, Mingjie Zhou +14
Tsinghua University · Shanghai Qi Zhi Institute · Sharpa +5
Imitation learning for contact-rich manipulation requires high-quality expert data that is expensive to obtain. This makes learning a sample-efficient policy a key issue. To address this, we propose VISTA, a workspace-level equivariant visuotactile diffusion policy for data-efficient contact-rich imitation learning. VISTA projects visual and tactile observations into spherical tokens, injects tactile contact cues into visual spherical directions through permutation-equivariant spherical fusion, and rotates the fused harmonic representation using the end-effector orientation. The resulting representation conditions an equivariant diffusion policy to predict spatially consistent actions. Extensive experiments in both simulation and real-world robotic settings show that VISTA substantially improves data efficiency over strong visuotactile imitation learning baselines. Project website: https://vista-paper.github.io/
Lik Hang Kenny Wong, Yiyao Ma, Xiu-Shen Wei +5
Department of Computer Science and Engineering, The Chinese University of Hong Kong · School of Computer Science and Engineering, Southeast University
Touch sensing is beneficial for solving a wide variety of manipulation tasks. While there exists a wide range of tactile sensors with different properties, exploiting the fusion of multiple heterogeneous tactile sensors to improve manipulation learning remains underexplored. We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks. We propose an effective architecture using modality-specific convolutional stems and transformer-based fusion that effectively integrates information from an RGB camera stream, a vision-based GelSight Mini sensor and a high-frequency event-based Evetac sensor. This multisensory representation then conditions a flow-matching policy for solving downstream tasks. Our systematic study across five contact-rich manipulation tasks demonstrates the effectiveness of multi-resolution tactile features in imitation learning wherein MiTaS achieves an average success rate of 80 %, while vision-only (35.5,%) and visuotactile (55.5,%, 63,%) baselines cannot solve the task reliably. Project Page: http://mitas-touch.github.io.
Rickmer Krohn, Erik Helmut, Niklas Funk +3
Interactive Robot Perception & Learning, TU Darmstadt · Hessian AI · Robotics Institute Germany +1