Estimating human pose, classifying actions, and predicting movement progress are essential for human-robot interaction. While vision-based methods suffer from occlusion and privacy concerns in realistic environments, tactile sensing avoids these issues. However, prior tactile-based approaches handle each task separately, leading to suboptimal performance. In this study, we propose a Shared COnvolutional Transformer for Tactile Inference (SCOTTI) that learns a shared representation to simultaneously address three separate prediction tasks: 3D human pose estimation, action class categorization, and action completion progress estimation. To the best of our knowledge, this is the first work to explore action progress prediction using foot tactile signals from custom wireless insole sensors. This unified approach leverages the mutual benefits of multi-task learning, enabling the model to achieve improved performance across all three tasks compared to learning them independently. Experimental results demonstrate that SCOTTI outperforms existing approaches across all three tasks. Additionally, we introduce a novel dataset collected from 15 participants performing various activities and exercises, with 7 hours of total duration, across eight different activities.
Figures & tables
Fig. 1 : Overview of the proposed method. The proposed model simultaneously performs 3D pose estimation, action progress prediction, and action classification by learning a shared representation from foot tactile signals collected with wearable insole sensors.
Model
Pose Estimation
Action Classification
Progress Prediction
MPJPE (mm) ↓
Accuracy (%) ↑
APP ↑
MSE ↓
Voxel-CNN
104.91
-
-
-
Sep-CNN
100.57
-
-
-
Res-CNN
98.51
-
-
-
GCN-Transformer
117.81
-
-
-
STAT
-
84.56
-
-
TABLE I : Comparison of SCOTTI with baselines. Arrows indicate the desired direction for each metric, and bold numbers indicate the best performance.
Model
Pose Estimation
Action Classification
Progress Prediction
MPJPE (mm) ↓
Accuracy (%) ↑
APP ↑
MSE ↓
Voxel-CNN
103.55
80.38
0.8770
0.0288
Sep-CNN
101.05
78.18
0.8668
0.0330
Res-CNN
99.00
82.59
0.8778
0.0278
GCN-Transformer
124.27
61.02
0.8191
0.0537
STAT
99.61
86.36
0.8894
0.0242
TABLE II : Comparison of SCOTTI with multi-task learning versions of baselines. Arrows indicate the desired direction for each metric, and bold numbers indicate the best performance.
Fig. 2 : Qualitative result of pose estimation and progress prediction results for lunges (left) and squats (right). The results demonstrate SCOTTI’s ability to accurately predict both pose and progress for different activities.
Fig. 3 : Progress Precision-Margin (PM) curve. PM-curve with random prediction baseline (left) . PM-curve with multi-task version of baselines (right) .
Fig. 4 : Confusion matrix for action classification. While SCOTTI achieves high accuracy overall, misclassifications are observed between actions with similar tactile signal patterns. In particular, step-up exercises are sometimes misclassified as SideWalking or Lunge, and conversely, SideWalking samples are often confused with step-up exercises.
Model
Pose Estimation
Action Classification
Progress Prediction
MPJPE (mm) ↓
Accuracy (%) ↑
APP ↑
MSE ↓
Pose Only
101.61
-
-
-
Action Only
-
87.39
-
-
Progress Only
-
-
0.8866
0.0264
All Tasks
96.63
90.06
0.8952
0.0223
TABLE III : Comparison of SCOTTI with single-task versions.
Fig. 5 : Analysis of SCOTTI. (a) Importance of different foot regions for SCOTTI across tasks, highlighting the central foot regions’ significance. (b) t-SNE visualization of shared features based on progress values, showing structured patterns for different actions. (c) t-SNE visualization of shared features based on action labels, demonstrating well-clustered features for individual actions.
Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning. Related resources can be found at https://egotac.github.io/ .
Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1
Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/
Seongjin Bien, Carlo Kneissl, Tobias Jülg +6
Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany · Ludwig Maximilian University of Munich · TUM School of Computation, Information and Technology, Technical University of Munich, Germany
Tactile sensors provide critical information for contact-rich manipulation, yet tactile representations and policies remain tightly coupled to each specific sensor, limiting transferability across robots and hardware platforms. We propose TactX, a framework for learning a transferable tactile representation across sensors spanning three fundamentally different transduction modalities: resistive, magnetic, and vision-based. TactX maps heterogeneous tactile observations into a shared latent space through modality-specific encoders trained on paired contact data. Such paired interactions provide a natural alignment signal across modalities, and the encoders are jointly trained across all sensor pairs, inducing a consistent latent space for all sensor types. Our experiments show that TactX aligns tactile representations across sensors while preserving object-level contact information, as evidenced by sensor-identity prediction and object classification in the learned latent space. We evaluate TactX on four contact-rich manipulation tasks: pick-and-place, plug insertion, board wiping, and object reorientation, and show that policies trained with one sensor transfer zero-shot to physically distinct sensors through the shared latent. This improves the average success rate from 27.5% for vision-only policy to 45.9%, providing a step toward sensor-agnostic tactile manipulation.
Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2
UC San Diego · Seoul National University · Amazon FAR