cs.ROOct 30, 2026
SaveTacMind: Benchmarking and Modeling Tactile Interaction States for Robotic Manipulation
Organizations: Shanghai Jiao Tong University · Daimon Robotics · The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology ·
Abstract
Tactile sensing provides robots with direct observations of contact physics, local geometry, and interaction dynamics, making it a vital source of feedback for contact-rich manipulation. However, existing tactile manipulation studies are often developed and evaluated within specific task settings, making it difficult to systematically compare how tactile information contributes under different interaction conditions. In this work, we develop a benchmark in simulation comprising 128 tasks organized around three complementary evaluation regimes: Precise, Recognition, and Dynamic. The benchmark further includes controlled static and dynamic task variants and 12 perturbation settings for evaluating robustness under changes in visual and physical conditions. These settings motivate studying tactile information as a temporally evolving interaction representation rather than only as an instantaneous observation. We therefore propose an action-conditioned latent predictive world model that integrates tactile history and learns a tactile state through prediction of future tactile latents. Building on this representation, we design a model that conditions action generation on the learned tactile state and uses predicted future tactile information to refine candidate actions through predictive tactile reflection. Experiments on the proposed benchmark show consistent gains from incorporating tactile information across the evaluated regimes. Project Page: https://tacmind-bench.github.io/
Figures & tables
Figure 1: Overview of TacMind-Bench and TacMind-0 . (a) TacMind-Bench comprises 28 standard and 100 additional tasks spanning precise contact, recognition under ambiguity, and dynamic interaction, together with controlled perturbations for robustness evaluation. (c) TacMind-0 is our VTLA model that maintains a streaming tactile state for action generation. (d) Predictive tactile reflection leverages Tac-LeWM to estimate future tactile states conditioned on candidate actions and uses them to refine action generation. (e) Tac-LeWM is pretrained on large-scale multi-sensor tactile manipulation demonstrations collected from both simulation and the real world. (b) TacMind-0 achieves competitive performance on TacMind-Bench Standard.
Figure 2: Overview of TacMind-Bench. The benchmark comprises 128 tasks (28 Standard and 100 Extra), 287 object assets, two tactile sensors (GSmini and Daimon), and 12 controlled perturbation settings covering illumination, camera viewpoint, and object scale. (a) Task composition and contact-behavior coverage. The inner ring shows the three evaluation regimes within Standard and the Extra suite; the outer ring and accompanying table summarize contact-behavior annotations. These annotations are non-exclusive: a task may involve multiple contact behaviors. (b)–(l) Representative tasks illustrating the 11 contact behaviors, each shown alongside tactile visualizations from the left (L) and right (R) fingers.
| Benchmark | Benchmark Objective | Tasks | E2E | SD. | Dyn. | Perturb. |
| ManiSkill-ViTac | Precision contact-rich manipulation | 3 | – | – | – | – |
| ManiFeel | Tactile-aware exploration and contact-rich manipulation | 9 | – | – | ||
| UniVTAC | Unified visuo-tactile manipulation | 8 | – | – | – | |
| Tabero | Gentle manipulation with tactile force regulation | 40 | – | – | – | |
| TacMind-Bench | Roles of touch throughout manipulation | 128 |
Table 1: Comparison with representative tactile manipulation benchmarks. denotes an explicit benchmark design axis, denotes partial coverage, and – denotes that the dimension is not explicitly evaluated. E2E denotes manipulation that includes object acquisition rather than only the terminal contact stage. SD. denotes State Disambiguation. Dyn. denotes externally driven source or target motion during execution. Perturb. denotes an orthogonal controlled robustness axis.
Figure 3: Overview of Tac-LeWM. (a–b) Marker-motion directions vary with contact conditions, while the left and right fingertip sensors have mirrored spatial layouts. These properties motivate tactile-specific spatial encoding that accounts for directional deformation and cross-fingertip symmetry. (c) Tac-LeWM learns a predictive tactile state by combining a streaming tactile encoder with an action-conditioned latent predictor trained to predict future tactile embeddings. (d) The streaming encoder reuses historical features through cached keys and values, enabling efficient online updates. (e) Causal attention integrates preceding tactile context while preventing access to future observations.
| Model | Precision | Recognition | Dynamic | Avg | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| InD | Light | Cam | Size | InD | Light | Cam | Size | InD | Light | Cam | Size | InD | OOD | |
| Single-Task Specialist Policies (trained separately per task) | ||||||||||||||
| ACT+Tactile | 30.0 | 26.0 | 2.5 | 2.5 | 69.1 | 48.9 | 31.7 | 13.9 | 15.9 | 10.5 | 1.8 | 1.6 | 39.8 | 16.4 |
| DP | 23.3 | 23.3 | 0.0 | 2.9 | 27.7 | 23.2 | 6.1 | 5.0 | 13.2 | 9.8 | 2.1 | 1.1 | 21.1 | 8.1 |
| Multi-Task Generalist Policies (jointly trained across all 28 tasks) | ||||||||||||||
| OpenPI0.5 | 5.0 | 0.0 | 0.0 | 0.0 | 30.0 | 24.2 | 1.2 | 15.9 | 0.5 | 0.2 | 0.0 | 0.2 | 13.0 | 5.5 |
Table 2: Performance on TacMind-Bench Standard (GelSight Mini). OOD performance is categorized into lighting, camera, and size perturbations.
Figure 4: Comparison between N0-VTLA and TacMind-0 on the Dynamic suite of TacMind-Bench.
| Tactile | Reflection | Precise | Recog. | Dynamic | Avg | ||||
|---|---|---|---|---|---|---|---|---|---|
| InD | OOD | InD | OOD | InD | OOD | InD | OOD | ||
| 20.0 | 8.7 | 38.6 | 21.7 | 8.2 | 2.7 | 22.7 | 11.4 | ||
| 35.8 | 12.4 | 47.3 | 30.2 | 9.1 | 3.6 | 29.8 | 15.9 | ||
| 40.0 | 22.4 | 50.0 | 35.6 | 10.5 | 4.9 | 32.3 | 20.7 | ||
Table 3: Ablation study of TacMind-0. We evaluate the contributions of streaming tactile features and predictive tactile reflection across different evaluation regimes and distributions.
Figure 5: Analysis of Tac-LeWM. (a) Effect of pretraining on current-frame tactile reconstruction. (b–c) Linear probing of proprioception, gripper state, actions, and tactile pad energy, evaluated using MSE z (↓) and Pearson correlation (↑). (d) Tactile encoder inference latency across batch sizes, with and without streaming. (e) Future tactile reconstruction quality with different cache lengths. (f) Future tactile reconstruction quality under clean, shuffled, and polluted memory.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Skill | Functional Scope |
|---|---|
| expert-trajectory | Development and refinement of expert policies for our benchmark, including tactile grasping, cuRobo-based free-space motion planning, insertion and placement, success-condition specification, and trajectory-smoothness optimization. |
| dynamic-collect | Collection and same-environment replay-gated evaluation of dynamic conveyor-belt tasks tac_dynamic , together with systematic diagnosis of Kit execution hangs and zero-episode dataset failures. |
| reset-randomization | Design and validation of reset-time perturbations for the benchmark suite, including pose_range , object permutation, and layout_xy , with consistency verification between JSON specifications and runtime perturbation bounds. |
| generate-tactile-task | Generation and validation of collectible tactile manipulation environments from natural-language task specifications. |
| object-expert-repair | Diagnosis and remediation of geometry and collision issues, insertion failures, false-positive success conditions, and expert trajectory collection consistent with teleoperated behaviour. |
Table 4: Summary of data collection skills.
| Function | Role in the Data Processing Pipeline |
|---|---|
| collect | Performs data collection, validation, and persistence while exposing diagnostic information and runtime task traces to support agent-driven analysis of collection failures and their underlying patterns. |
| expert | Dispatches the task-specific expert procedure to generate demonstrations for the target manipulation task. |
| replay_gate | Replays collected trajectories in the corresponding environment and evaluates execution correctness as a quality-control gate for dataset admission. |
| capture | Captures images from arbitrary tasks in Isaac Sim for visual inspection, debugging, and downstream multimodal processing. |
| validate_dataset | Inspects dataset metadata and verifies individual episodes against the predefined episode schema to ensure structural consistency. |
| convert_dataset | Converts datasets between formats, processes action sequences, and removes redundant or unnecessary frames to produce training-ready datasets. |
Table 5: Built-in Functions and Their Roles in the Data Processing Pipeline.
Figure 6: The 28 standard tasks of TacMind-Bench (GelSight Mini).
Figure 7: The 28 standard tasks of TacMind-Bench (Daimon).
Figure 8: Visualization of tactile information of Gelsight Mini.
Figure 9: Visualization of tactile information of Daimon.
Figure 10: Visualization of the extra suite task (0-63) of TacMind-Bench.
Figure 11: Visualization of the extra suite task (64-99) of TacMind-Bench.
| Regime | Task | Language Instruction |
| Precise | 00 | Pick up the tube and insert it into the black rubber hole base. |
| 01 | Pick up the key and insert it into the blue plastic lock. | |
| 02 | Pick up the connector and insert it into the orange port. | |
| 03 | Pick up the gear and insert it into the white painted base. | |
| 04 | Insert the three-prong power plug into the power socket. | |
| 05 | Insert the bulb into the bulb socket. |
Table 6: All 28 TacMind-Bench Standard tasks, grouped by evaluation regime. For the Dynamic regime, the fixture is mounted on the moving conveyor platform unless stated otherwise.
| Regime | Task | Language Instruction |
|---|---|---|
| Extra | 00 | Pull the drawer open by at least 12 cm. |
| Extra | 01 | Grasp the handle and open the hinged door by at least 60 degrees. |
| Extra | 02 | Turn the valve counterclockwise by 90 degrees relative to its initial angle. |
| Extra | 03 | Press the spring button at least 8 mm for 0.3 seconds, then release it. |
| Extra | 04 | Move the bistable lever from the negative well to the positive well, then let go. |
| Extra | 05 | Pull the cap completely out of its spring-loaded retaining shoes and move it away by at least 3 cm. |
Table 7: All 100 TacMind-Bench Extra tasks. Task IDs correspond to the zero-based IDs in the Extra task catalog.
| Variant | Proprio ( / ) | Gripper ( / ) | Action ( / ) | Pad ( / ) |
|---|---|---|---|---|
| Full | 0.1816 / 0.8981 | 0.1958 / 0.9053 | 0.1841 / 0.8976 | 0.2771 / 0.7968 |
| Camera | 0.1748 / 0.9027 | 0.1958 / 0.9053 | 0.1770 / 0.9022 | 0.2905 / 0.7855 |
| Causal Attn | 0.1733 / 0.9035 | 0.1958 / 0.9053 | 0.1756 / 0.9030 | 0.3557 / 0.7282 |
| CTR | 0.1831 / 0.8971 | 0.1958 / 0.9053 | 0.1851 / 0.8968 | 0.3366 / 0.7476 |
Table 8: Ablation study on Tac-LeWM: linear probing results.
| Current Reconstruction | Future Reconstruction | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Variant | PSNR | MSE | RMSE | MAE | SSIM | PSNR | MSE | RMSE | MAE | SSIM |
| Full | 31.83 | 0.00157 | 0.0341 | 0.0223 | 0.8748 | 28.98 | 0.00175 | 0.0391 | 0.0263 | 0.8593 |
| Camera | 32.66 | 0.00177 | 0.0348 | 0.0230 | 0.8649 | 26.52 | 0.00255 | 0.0489 | 0.0342 | 0.8034 |
| Causal Attn | 31.45 | 0.00208 | 0.0385 | 0.0252 | 0.8451 | 23.07 | 0.00558 | 0.0724 | 0.0494 | 0.7116 |
| CTR | 30.85 | 0.00190 | 0.0372 | 0.0242 | 0.8620 | 23.42 | 0.00567 | 0.0714 | 0.0518 | 0.7088 |
Table 9: Ablation study on Tac-LeWM on TacMind-Bench Standard 28 tasks: current and future reconstruction results.
| Inference backend | Single round (ms) | Tac-FRS three rounds (ms) |
|---|---|---|
| PyTorch | 115.4 | 300.9 |
| CUDA Graph | 16.5 | 57.0 |
| TensorRT | 11.9 | 32.4 |
| TensorRT + CUDA Graph | 4.9 | 14.0 |
Table 10: Action integration latency on an NVIDIA RTX 5090. Values are median latencies over 30 measurements.
Figure 12: Representative trajectories for the six real-world tasks. From top to bottom, the rows show USB insertion, plug insertion, small-gear assembly, large-gear assembly, heavy-bottle discrimination, and light-bottle discrimination. Each row contains eight frames arranged in chronological order from left to right.
Figure 13: Success rates on six real-world manipulation tasks. We compare , N0-VTLA, and TacMind-0 on two insertion tasks (USB and Plug), two assembly tasks (GearS and GearL), and two weight discrimination tasks (Heavy and Light). Each method is evaluated over 20 trials per task. The numbers above the bars denote success rates in percent, and Avg is the mean across the six tasks.
More in this topic
Contact-Rich Robotic Manipulation
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
See all 436 papers