Temporal Visuo-Tactile Learning for Dexterous Grasp Stability

Ken Nakahara; Aleksei Buvailik; Prokhor Kotov; Roberto Calandra

cs.ROOct 7, 2026

Temporal Visuo-Tactile Learning for Dexterous Grasp Stability

Authors: Ken Nakahara, Aleksei Buvailik, Prokhor Kotov, Roberto Calandra

Organizations: LASR Lab, Faculty of Computer Science, TU Dresden, Germany

Abstract

Humans can grasp everyday objects with almost perfect success rates using fingertip tactile feedback, yet much of the robotic grasping literature emphasizes vision-based grasp selection with parallel grippers. In this work, we systematically investigate how high-resolution, dynamic tactile sensing contributes to grasp stability prediction and model-guided grasping in dexterous robotic hands. To this end, we collected a dataset of 10,000 grasp trials across 200 objects using a multi-fingered robotic hand equipped with four Digit 360 tactile sensors, recording external vision, proprioception, and tactile streams throughout each grasp. With this dataset, we trained end-to-end temporal multimodal models to predict post-lift stability from pre-lift grasp observations and compared sensing modalities and encoding backbones. Experimental results and controlled input ablations show that incorporating touch, and particularly high-resolution, dynamic touch, improves grasp stability prediction. Finally, we deployed the learned predictor as an online stability gate on the real robot, where visuo-tactile model-guided regrasping improved the success rate among executed lifts by 10.5 percentage points over a non-tactile gate. These results show how rich fingertip sensing and expressive temporal models that capture the dynamics of touch can support learned grasping with multi-fingered hands without explicit contact or force modeling, providing a scalable data-driven path from tactile experience toward stable dexterous manipulation. The dataset is publicly available at https://lasr-lab.github.io/dexterous-grasp-stability/.

Figures & tables

Figure 1 : Temporal multimodal stability prediction for dexterous grasping. During grasp execution, the robot observes the object and fingertip contacts through external vision, proprioception, and four Digit 360 tactile sensors. A temporal stability predictor fuses these signals to estimate post-lift grasp stability and gate lift-or-regrasp decisions online, leading to higher success among executed lifts.

Figure 2 : Hardware setup. The system continuously records external visual observations, proprioception, and tactile streams from four Digit 360 sensors throughout each grasp trial.

Figure 3 : Model architecture for temporal multimodal grasp stability prediction. An observation window from the pre-lift grasp phase, containing RGB frames, proprioception, and tactile streams from four Digit 360 sensors, is encoded by modality-specific Transformer-based branches. Finger-level tactile features are integrated across modalities, time, and fingers, then combined with RGB and proprioceptive features through attention-based fusion to predict post-lift grasp stability.

Figure 4 : The 200 objects used to collect 10,000 grasping trials.

Figure 5 : Automated self-supervised data collection timeline. Each trial comprises object localization, randomized reaching and grasping, lifting, and stability labeling from object height changes, with continuous multimodal recording.

Figure 6 : Mean accuracy with SD error bars for modality and backbone ablations under 5-fold CV. Touch-enabled models achieve the strongest performance. Multi-frame models consistently outperform the static single-frame baseline, with the VideoMAEv2-based model performing best, suggesting the value of expressive temporal models for capturing grasp dynamics.

Figure 7 : Mean accuracy with SD shading across data scales for VideoMAEv2-based models. Trial-only subsampling (solid) reduces trials per object while retaining the object set; object-and-trial subsampling (dashed) also reduces the number of objects by the same ratio. The leftmost panel compares modalities; the others compare subsampling protocols. Accuracy generally improves with more data, while touch-enabled models remain comparatively robust to reduced object coverage.

Figure 8 : Mean and SD of accuracy across tactile-image spatial resolutions, with the non-tactile baseline shown as a horizontal band. Higher tactile resolution improves accuracy.

Panel of Figure 9 — Figure 9 : Mean and SD of accuracy under temporal input ablations for (a) single-modality and (b) vision–proprioception–touch models. Final-observation repetition removes temporal variation, while joint vision–touch permutation in (b) disrupts temporal order. Removing tactile dynamics lowers accuracy, highlighting the importance of temporal tactile information.

Figure 10 : Mean and SD of accuracy under finger-specific input ablations; joint angles and, when enabled, tactile streams are restricted to selected fingers. Vision largely compensates for restricted joint-state inputs, whereas touch, particularly from the thumb as the principal opposing finger, provides stability cues not fully captured by external vision.

Figure 11 : Grasp success rates among executed lifts for four policies on 20 unseen objects. Object mass and dimensions ( $W\times D\times H$ in $\mathrm{mm}$ ) are shown below each image. The vision–proprioception–touch gate outperforms the vision–proprioception gate, direct lifting, and the random gate by 10.5, 32.5, and 34.5 percentage points, respectively. Error bars show object-cluster bootstrap $95\%$ confidence intervals for overall success rates; per-object rates are based on 10 executed lifts per policy.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure 12 : Physical-property and material distributions of the 200 dataset objects. (a) Each point represents one object, positioned by its measured mass and dimension-derived bounding-box volume; both axes use logarithmic scales, the top and right histograms show the marginal distributions, and the dashed lines mark their medians. (b) Object counts per primary material, split into rigid and deformable classes; primary materials represented by at most two objects (e.g., leather, ceramic, and stone) are grouped as Others.

		Manual label
		1 (Stable)	0 (Unstable)	Total
Automatic label	1 (Stable)	4361	415	4776
Automatic label	0 (Unstable)	351	4873	5224
	Total	4712	5288	10000

Appendix

Table I : Confusion matrix of the automatic labels against the manually verified labels, which agree on

92.34\%

of the 10,000 trials.

Finger	Joint	Min. [ ∘ ]	Max. [ ∘ ]	Finger	Joint	Min. [ ∘ ]	Max. [ ∘ ]
Thumb	IP	25	45	Index	DIP	25	45
	MCP	25	45		PIP	25	45
	ABD	50	70		MCP	45	90
	CMC	75	90		ABD	0	15
Middle	DIP	25	45	Ring	DIP	25	45
	PIP	25	45		PIP	25	45

Appendix

Table II : Calibrated per-joint ranges from which the 16 hand joint targets are sampled independently and uniformly during data collection. IP, MCP, CMC, DIP, and PIP denote the interphalangeal, metacarpophalangeal, carpometacarpal, distal interphalangeal, and proximal interphalangeal joints, and ABD denotes the abduction degree of freedom.

Digit 360 inputs	Acc. [%]
Full modalities	$83.55\pm 1.58$
Camera-only	$83.41\pm 1.30$
Audio-only	$79.60\pm 1.46$
IMU-only	$79.94\pm 1.50$
Pressure-only	$79.93\pm 1.10$
No tactile streams (i.e., Vision + Proprio)	$79.56\pm 1.60$

Appendix

Table III : Mean accuracy with SD over 5-fold CV for Digit 360 submodality ablations in the touch branch, with vision and proprioception retained.

	Acc. [%]
Finger inputs included	Proprio	Vision + [1.5pt] Proprio	Proprio + [1.5pt] Touch	Vision + Proprio [1.5pt] + Touch
Thumb + Index + Middle + Ring	$79.99\pm 0.78$	$79.56\pm 1.60$	$82.34\pm 2.26$	$83.55\pm 1.58$
Thumb + Index + Middle	$79.48\pm 1.45$	$79.34\pm 0.88$	$82.23\pm 1.80$	$83.31\pm 1.41$
Thumb + Index	$79.00\pm 0.86$	$79.93\pm 1.50$	$81.15\pm 1.70$	$82.63\pm 1.40$
Thumb + Middle	$79.54\pm 0.88$	$79.83\pm 1.75$	$81.78\pm 1.52$	$83.41\pm 0.95$
Thumb	$79.38\pm 1.09$	$79.49\pm 1.24$	$81.17\pm 2.15$	$82.55\pm 1.34$

Appendix

Table IV : Mean accuracy with SD over 5-fold CV when varying which fingers provide input. Finger-specific inputs (joint angles in every setting, and Digit 360 streams when touch is enabled) are restricted to the listed fingers, while vision is not finger-specific and remains unchanged; all grasps are still executed with the full four-fingered hand.

Panel of Figure 13 — Figure 13 : Fingertip contacts and grasp success across all 10,000 grasp trials in the dataset. (a) Success rate by detected-contact count; $n$ denotes the number of trials at each contact count. Trials with no or a single detected contact can still succeed when the object is supported outside the fingertip sensing surfaces or when contact falls below the detection threshold. (b) Success rate by non-empty contact set; solid and cross-hatched bars indicate sets with and without the thumb, respectively. Both panels exclude geometric finger–finger self-contact, using a $5\text{\,}\mathrm{mm}$ clearance threshold on the posed 3D hand model. Shading and error bars show object-cluster bootstrap $95\%$ confidence intervals; the dashed line is the overall success rate.

Explore similar work

SECOND-Grasp: Semantic Contact-guided Dexterous Grasping
May 13, 2026Han Yi Shin, Heeju Ko, Jaewon Mun +6Robotic Grasping Robotic Manipulation
1Korea University · University of Texas at Austin · University of Michigan +1
AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion
Aug 6, 2026Xirui Liang, Jiaqi Liang, Jingkai Xu +6Robotic Grasping Grasp Adaptation
Peking University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · University of California, Berkeley, CA, USA
ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction
Aug 3, 2026Shiqi Zhang, Xin Zhang, Yedong Shen +9Contact-Rich Dexterous Manipulation Tactile
University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK

May 13, 2026, cs.ROJ/K move · Enter open · S save

SECOND-Grasp: Semantic Contact-guided Dexterous Grasping

Han Yi Shin, Heeju Ko, Jaewon Mun, Qixing Huang+5

1Korea University · University of Texas at Austin · University of Michigan+1

Robotic Grasping Robotic Manipulation

Open paper AskPDF arXiv

Achieving reliable robotic manipulation, such as dexterous grasping, requires a synergy between physically stable interactions and semantic task guidance, yet these objectives are often treated as separate, disjoint goals. In this paper, we investigate how to integrate dexterous grasping techniques, i.e., physically stable grasps for object lifting and language-guided grasp generation, to achieve both physical stability and semantic understanding. To this end, we propose SECOND-Grasp (SEmantic CONtact-guided Dexterous Grasping), a unified framework that enables robotic hands to dynamically adjust grasping strategies based on semantic reasoning while ensuring physical feasibility. We begin by obtaining coarse contact proposals through vision-language reasoning to infer where contacts should occur based on object properties, followed by segmentation to localize these regions across views. To further ensure consistency across multiple viewpoints, we introduce Semantic-Geometric Consistency Refinement (SGCR), which refines initial contact predictions by enforcing semantic consistency across views and removing geometrically invalid regions, yielding reliable 3D contact maps. Then, we derive a feasible hand pose for each contact map via inverse kinematics, generating a supervision signal for policy learning. Our approach, trained on DexGraspNet, consistently outperforms baselines in lifting success rate on both seen and unseen categories, achieving 98.2% and 97.7%, respectively, while also improving intent-aware grasping by 12.8% and 26.2%. We further show promising results on additional datasets and robotic hands, including Shadow Hand and Allegro Hand.

Abstract

Explore similar work

SECOND-Grasp: Semantic Contact-guided Dexterous Grasping

AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion

ReTouch: Empowering Contact-Rich Dexterous Manipulation with Online-Refined Tactile Prediction