cs.RONov 2, 2026

Visuo-tactile Diffusion Policy for Robotic Manipulation with Auxiliary Tactile Dynamics Prediction

Authors: Kaihong Ouyang, Zihao Su, Chengxuan Zheng, Weiyi Lai, Dong Zhou

Organizations: School of Astronautics, Harbin Institute of Technology, Harbin, Heilongjiang, China

Abstract

Long-horizon robotic assembly requires reliable perception and action across grasping, alignment, and insertion. Vision captures global object pose and scene context, where as fingertip touch provides local contact cues that are especially useful as contact states change. We present a visuotactile diffusion policy that combines four-step histories from two camera views and two fingertip tactile images to condition action-sequence generation. A recurrent temporal representation integrates recent observations from both modalities. To encourage it to capture how contact evolves under action, an auxiliary objective predicts fingertip-image changes two steps ahead from the temporal representation and demonstrated actions. This objective is used only during training and adds no inference-time computation. Across three independent training seeds in MuJoCo assembly experiments, the full model achieves an average success rate of 59.0%±2.6%59.0\% \pm 2.6\%, improving on the visual-only baseline (42.7%±1.5%42.7\% \pm 1.5\%) by 16.3 percentage points.

Explore similar work

CardsList
  1. Equivariant Visual-Tactile Diffusion Policy for Contact-Rich Manipulation

    Oct 2, 2026Lik Hang Kenny Wong, Yiyao Ma, Xiu-Shen Wei +5Robot Imitation LearningRobot Learning

  2. GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

    Oct 20, 2026Ziyong Wang, Yupei Lin, Runxuan Huang +3Tactile-Vision-Language-Action ModelsRobot Policy Learning

  3. OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

    Mar 19, 2026Yuhang Zheng, Songen Gu, Yupeng Zheng +11Contact-Rich Robotic ManipulationTactile World Models