Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals

Isaac Han; Seoyoung Lee; Sangyeon Park; Ecehan Akan; Yiyue Luo; Joseph DelPreto; Kyung-Joong Kim

cs.CVMar 26, 2026

Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals

Authors: Isaac Han, Seoyoung Lee, Sangyeon Park, Ecehan Akan, Yiyue Luo, Joseph DelPreto, Kyung-Joong Kim

Organizations: Gwangju Institute of Science and Technology (GIST) · University of Washington · MIT CSAIL

Abstract

Estimating human pose, classifying actions, and predicting movement progress are essential for human-robot interaction. While vision-based methods suffer from occlusion and privacy concerns in realistic environments, tactile sensing avoids these issues. However, prior tactile-based approaches handle each task separately, leading to suboptimal performance. In this study, we propose a Shared COnvolutional Transformer for Tactile Inference (SCOTTI) that learns a shared representation to simultaneously address three separate prediction tasks: 3D human pose estimation, action class categorization, and action completion progress estimation. To the best of our knowledge, this is the first work to explore action progress prediction using foot tactile signals from custom wireless insole sensors. This unified approach leverages the mutual benefits of multi-task learning, enabling the model to achieve improved performance across all three tasks compared to learning them independently. Experimental results demonstrate that SCOTTI outperforms existing approaches across all three tasks. Additionally, we introduce a novel dataset collected from 15 participants performing various activities and exercises, with 7 hours of total duration, across eight different activities.

Figures & tables

Fig. 1 : Overview of the proposed method. The proposed model simultaneously performs 3D pose estimation, action progress prediction, and action classification by learning a shared representation from foot tactile signals collected with wearable insole sensors.

Model	Pose Estimation	Action Classification	Progress Prediction
Model	MPJPE (mm) $\downarrow$	Accuracy (%) $\uparrow$	APP $\uparrow$	MSE $\downarrow$
Voxel-CNN	104.91	-	-	-
Sep-CNN	100.57	-	-	-
Res-CNN	98.51	-	-	-
GCN-Transformer	117.81	-	-	-
STAT	-	84.56	-	-

TABLE I : Comparison of SCOTTI with baselines. Arrows indicate the desired direction for each metric, and bold numbers indicate the best performance.

Model	Pose Estimation	Action Classification	Progress Prediction
Model	MPJPE (mm) $\downarrow$	Accuracy (%) $\uparrow$	APP $\uparrow$	MSE $\downarrow$
Voxel-CNN	103.55	80.38	0.8770	0.0288
Sep-CNN	101.05	78.18	0.8668	0.0330
Res-CNN	99.00	82.59	0.8778	0.0278
GCN-Transformer	124.27	61.02	0.8191	0.0537
STAT	99.61	86.36	0.8894	0.0242

TABLE II : Comparison of SCOTTI with multi-task learning versions of baselines. Arrows indicate the desired direction for each metric, and bold numbers indicate the best performance.

Fig. 2 : Qualitative result of pose estimation and progress prediction results for lunges (left) and squats (right). The results demonstrate SCOTTI’s ability to accurately predict both pose and progress for different activities.

Panel of Figure 5 — Fig. 3 : Progress Precision-Margin (PM) curve. PM-curve with random prediction baseline (left) . PM-curve with multi-task version of baselines (right) .

Fig. 4 : Confusion matrix for action classification. While SCOTTI achieves high accuracy overall, misclassifications are observed between actions with similar tactile signal patterns. In particular, step-up exercises are sometimes misclassified as SideWalking or Lunge, and conversely, SideWalking samples are often confused with step-up exercises.

Model	Pose Estimation	Action Classification	Progress Prediction
Model	MPJPE (mm) $\downarrow$	Accuracy (%) $\uparrow$	APP $\uparrow$	MSE $\downarrow$
Pose Only	101.61	-	-	-
Action Only	-	87.39	-	-
Progress Only	-	-	0.8866	0.0264
All Tasks	96.63	90.06	0.8952	0.0223

TABLE III : Comparison of SCOTTI with single-task versions.

Panel of Figure 8 — Fig. 5 : Analysis of SCOTTI. (a) Importance of different foot regions for SCOTTI across tasks, highlighting the central foot regions’ significance. (b) t-SNE visualization of shared features based on progress values, showing structured patterns for different actions. (c) t-SNE visualization of shared features based on action labels, demonstrating well-clustered features for individual actions.

3D Human Pose Estimation

3D human pose estimation recovers people’s body-joint positions or full-body shape and motion from images, video, and other sensors.

See all 113 papers

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision
Aug 15, 2026Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2Visual-to-Tactile Prediction Contact Force Estimation
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1
FlowTouch: View-Invariant Visuo-Tactile Prediction
Mar 9, 2026Seongjin Bien, Carlo Kneissl, Tobias Jülg +6Visual-to-Tactile Prediction Vision-Based Tactile Sensing
Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany · Ludwig Maximilian University of Munich · TUM School of Computation, Information and Technology, Technical University of Munich, Germany
TactX: Learning Shared Tactile Representations Across Diverse Sensors
Jun 30, 2026Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2Cross-Modal Representation Learning Contact-Rich Robotic Manipulation
UC San Diego · Seoul National University · Amazon FAR

Aug 15, 2026J/K move · Enter open · S save

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

Wenkang Zhang, Chengbo Yuan, Zicheng Zhang, Zhengxue Cheng+1

Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University+1

Visual-to-Tactile Prediction Contact Force Estimation Robotic Tactile Sensing Contact-Rich Manipulation

Open paper AskPDF arXiv

Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning. Related resources can be found at https://egotac.github.io/ .

Abstract

3D Human Pose Estimation

Explore similar work

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

FlowTouch: View-Invariant Visuo-Tactile Prediction

TactX: Learning Shared Tactile Representations Across Diverse Sensors