Equivariant Visual-Tactile Diffusion Policy for Contact-Rich Manipulation

Lik Hang Kenny Wong; Yiyao Ma; Xiu-Shen Wei; Zelong Tan; Zhuheng Song; Dongsheng Xie; Kai Chen; Qi Dou

cs.ROOct 2, 2026

Equivariant Visual-Tactile Diffusion Policy for Contact-Rich Manipulation

Authors: Lik Hang Kenny Wong, Yiyao Ma, Xiu-Shen Wei, Zelong Tan, Zhuheng Song, Dongsheng Xie, Kai Chen, Qi Dou

Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong · School of Computer Science and Engineering, Southeast University

Abstract

Imitation learning for contact-rich manipulation requires high-quality expert data that is expensive to obtain. This makes learning a sample-efficient policy a key issue. To address this, we propose VISTA, a workspace-level equivariant visuotactile diffusion policy for data-efficient contact-rich imitation learning. VISTA projects visual and tactile observations into spherical tokens, injects tactile contact cues into visual spherical directions through permutation-equivariant spherical fusion, and rotates the fused harmonic representation using the end-effector orientation. The resulting representation conditions an equivariant diffusion policy to predict spatially consistent actions. Extensive experiments in both simulation and real-world robotic settings show that VISTA substantially improves data efficiency over strong visuotactile imitation learning baselines. Project website: https://vista-paper.github.io/

Figures & tables

Figure 1: Model architecture of VISTA. (a) Vision and tactile inputs are encoded into spherical tokens, fused via permutation-equivariant spherical fusion, projected onto spherical harmonics, and aligned with the workspace using the end-effector orientation. (b) The aligned representation conditions a finite-group-equivariant diffusion policy to denoise action chunks and predict 6-DoF actions plus gripper commands.

Figure 2: Permutation-Equivariant Spherical Fusion . Bottom : visual spherical tokens before and after a token-space rotation, with tactile tokens fixed. Top : fused visuotactile tokens rotate coherently with the visual token structure, showing that tactile cross-attention preserves the spherical indexing.

Figure 3: Simulation environments from UniVTAC [ 1 ] . The benchmark includes insertion, extraction, lifting, placement, and contact-based classification tasks.

Method	Grasp Classify	Insert HDMI	Insert Hole	Insert Tube	Lift Bottle	Lift Can	Pull Out Key	Put Bottle in Shelf	Mean
ManiFeel	100.0	12.5	32.5	45.0	55.0	10.0	95.0	40.0	48.8
VITAL	100.0	7.5	27.5	55.0	20.0	17.5	65.0	57.5	43.8
UniVTAC	60.0	5.0	17.5	32.5	0.0	0.0	37.5	17.5	21.3
EquiDiff + Tac	95.0	10.0	42.5	92.5	57.5	37.5	97.5	87.5	65.0
VISTA	100.0	40.0	55.0	95.0	80.0	92.5	97.5	87.5	80.9

Table 1: Success rates in simulation. All methods are trained with 50 demonstrations per task and evaluated over 40 rollouts.

Figure 4: Summary of simulation ablations. Left: eight-task-mean fusion ablation over ISP, VISTA-global-

l0

, Early fusion, ISP+Concat-Tac, and VISTA (definitions in Appendix A.6 ). Right: demo-budget ablation over all eight simulation tasks with 50 or 100 demonstrations. Detailed per-task results are provided in Appendix A.10 .

Figure 5: Task-level ISP–VISTA comparison. Success rates on all eight simulation tasks, grouped into grasp classification, insertion/pull, and tasks requiring gripper-closure control.

Method	Erase	Insert USB	Insert pencil	Mean
EquiDiff + Tac	1/10	3/10	0/10	1.33/10
VISTA	9/10	5/10	8/10	7.33/10

Table 2: Success rates on real-robot contact-rich manipulation tasks. Mean reports the average number of successful trials out of 10 across tasks.

Figure 6: Real-robot task rollouts from VISTA, covering tight insertion and sustained surface-contact manipulation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Shape
Wrist RGB input	$[B,H_{o},3,H,W]$
Tactile RGB input	$[B,H_{o},3,H,W]$
Flattened image inputs	$[N,3,H,W]$
Visual feature map	$[N,C_{I},7,7]$
Tactile feature map	$[N,C_{T},7,7]$
Visual sphere tokens	$[N,P,C_{I}]$

Appendix

Table 3: Main tensor shapes in VISTA. Here

N=BH_{o}

P

is the number of sampled sphere points, and

N_{G}

is the number of finite-group samples.

Variant	Tactile representation	Fusion stage	Fusion operator
ISP	none	–	–
VISTA-global- $l0$	invariant global vector	$\ell=0$ harmonic coefficients	addition
Early Fusion	2D tactile feature map	before spherical lifting	$1\times 1$ conv on concatenated maps
ISP+Concat-Tac	spherical tokens	on the sphere, before harmonics	channel-wise concat + linear
VISTA	spherical tokens	on the sphere, before harmonics	cross-attention (visual $Q$ , tactile $K/V$ )

Appendix

Table 4: Ablation variants. Fusion is the only intended architectural difference; the visual spherical backbone and decoder are held fixed.

Task	VISTA group	Main randomized factors
Grasp classify	$C_{8}$	object class, grasp height, gripper closing pose
Insert HDMI	$I_{60}$	slot $x/y$ , in-hand pitch/yaw, staging pose noise
Insert hole	$I_{60}$	slot $x/y$ , slot yaw, target pitch, grasp height
Insert tube	$I_{60}$	slot $x/y$ , grasp height, staging pose noise
Lift bottle	$C_{8}$	bottle $x/y$ , bottle yaw, grasp pitch
Lift can	$C_{8}$	can size, can $x/y$ , grasp pitch

Appendix

Table 5: Simulation task randomization and VISTA group instantiation. The ranges summarize the data-generation protocol used by the benchmark.

Method	Group	Latency (ms)
ISP	$C_{8}$	61.25
ISP	$I_{60}$	65.66
VISTA	$C_{8}$	63.74
VISTA	$I_{60}$	71.54
EquiDiff + Tac	$C_{8}$	79.88

Appendix

Table 6: Offline observation-to-chunk inference latency on one RTX 3090 with DDIM-8. EquiDiff + Tac uses its

C_{8}

decoder.

Method	Grasp Classify	Insert HDMI	Insert Hole	Insert Tube	Lift Bottle	Lift Can	Pull Out Key	Put Bottle in Shelf	Mean
ManiFeel	100.0	12.5	32.5	45.0	55.0	10.0	95.0	40.0	48.8
VITAL	100.0	7.5	27.5	55.0	20.0	17.5	65.0	57.5	43.8
UniVTAC	60.0	5.0	17.5	32.5	0.0	0.0	37.5	17.5	21.3
EquiDiff + Tac	95.0	10.0	42.5	92.5	57.5	37.5	97.5	87.5	65.0
VISTA	100.0	40.0	55.0	95.0	80.0	92.5	97.5	87.5	80.9

Appendix

Table 7: Main simulation comparison. Success rates in simulation. All methods are trained with 50 demonstrations per task and evaluated over 40 rollouts. Missing entries are marked as – and are excluded from the mean.

Method	Grasp Classify	Insert HDMI	Insert Hole	Insert Tube	Lift Bottle	Lift Can	Pull Out Key	Put Bottle in Shelf	Mean
ISP	87.5	32.5	55.0	92.5	15.0	92.5	92.5	77.5	68.1
VISTA	100.0	40.0	55.0	95.0	80.0	92.5	97.5	87.5	80.9

Appendix

Table 8: ISP versus VISTA. Backbone ablation in simulation. ISP is the image-only spherical policy baseline; VISTA adds tactile spherical lifting and tactile-conditioned spherical fusion.

Method	Grasp Classify	Insert HDMI	Insert Hole	Insert Tube	Lift Bottle	Lift Can	Pull Out Key	Put Bottle in Shelf	Mean
ISP	87.5	32.5	55.0	92.5	15.0	92.5	92.5	77.5	68.1
VISTA-global- $l0$	100.0	17.5	52.5	90.0	77.5	77.5	57.5	82.5	69.4
Early Fusion	100.0	42.5	37.5	90.0	75.0	77.5	82.5	77.5	72.8
ISP+Concat-Tac	100.0	37.5	67.5	85.0	67.5	92.5	77.5	65.0	74.1
VISTA	100.0	40.0	55.0	95.0	80.0	92.5	97.5	87.5	80.9

Appendix

Table 9: Fusion ablation. Fusion ablation on all eight simulation tasks. VISTA-global-

l0

injects tactile information as invariant context, Early Fusion fuses 2D feature maps before spherical projection, ISP+Concat-Tac concatenates visual and tactile spherical tokens, and VISTA performs tactile-conditioned spherical fusion before harmonic projection.

Method	#Demo	Grasp Classify	Insert HDMI	Insert Hole	Insert Tube	Lift Bottle	Lift Can	Pull Out Key	Put Bottle in Shelf	Mean
ManiFeel	50	100.0	12.5	32.5	45.0	55.0	10.0	95.0	40.0	48.8
ManiFeel	100	100.0	25.0	60.0	92.5	70.0	77.5	95.0	67.5	73.4
VITAL	50	100.0	7.5	27.5	55.0	20.0	17.5	65.0	57.5	43.8
VITAL	100	100.0	10.0	20.0	72.5	50.0	45.0	72.5	62.5	54.1
UniVTAC	50	60.0	5.0	17.5	32.5	0.0	0.0	37.5	17.5	21.3
UniVTAC	100	60.0	0.0	15.0	42.5	0.0	10.0	65.0	60.0	31.6

Appendix

Table 10: Data efficiency. Detailed data-efficiency comparison in simulation. Missing entries are marked as – and are excluded from the mean. VISTA is reported as the task-appropriate instantiation selected from task geometry before training.

Explore similar work

VISTA: Visually Inferred Spatial ConTact Attention for Contact-Rich Manipulation
Aug 26, 2026Jiayi Chen, Wenlong Dong, Yan Huang +5Contact-Rich Robotic Manipulation Visuomotor Policy Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China. · School of Future Technology, Harbin Institute of Technology, Harbin, China.
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
Mar 19, 2026Yuhang Zheng, Songen Gu, Yupeng Zheng +11Contact-Rich Robotic Manipulation Tactile World Models
National University of Singapore · Fudan University · CASIA +4
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Jan 31, 2026Yaohua Liu, Rong Fu, Amir H. Gandomi +2Representation Learning Cross-Modal Representation Learning
Chinese Medicine Guangdong Laboratory, Hengqin, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Macau 999078, China · Faculty of Engineering & IT, University of Technology Sydney, Ultimo, NSW 2007, Australia +3

Aug 26, 2026J/K move · Enter open · S save

VISTA: Visually Inferred Spatial ConTact Attention for Contact-Rich Manipulation

Jiayi Chen, Wenlong Dong, Yan Huang, Xianglin Chen+4

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China. · School of Future Technology, Harbin Institute of Technology, Harbin, China.

Contact-Rich Robotic Manipulation Visuomotor Policy Learning Robot Imitation Learning Vision-Based Tactile Sensing Generalization in Robotic Manipulation

Open paper AskPDF arXiv

Contact-rich manipulation requires precise interaction feedback. While vision-centric imitation learning is prevalent, external visual observations provide indirect and ambiguous cues about contact states, particularly under occlusion or subtle object--gripper interactions; dedicated tactile or force sensors can provide rich contact information but introduce additional hardware complexity, calibration requirements, and deployment costs. To bridge this gap, we propose VISTA-Policy, an imitation learning paradigm that utilizes the Visual Deformation Field (VDF), a 3D displacement representation of a compliant gripper, as high-dimensional visuo-physical feedback. The framework integrates: 1) a Physics-Aware Encoding Engine for real-time VDF decoding; 2) an Energy Aggregation Denoising Mechanism to isolate true interaction signals; and 3) a Deformation-Augmented Policy Network with incremental gripper actions for precise closed-loop correction. Extensive evaluations on Cross-Scale Object Grasping, Cap Unscrewing, and Calligraphy Writing demonstrate that VISTA-Policy outperforms the strong pure-vision baseline 3D Diffusion Policy and the tactile baseline. VISTA-Policy further demonstrates substantial out-of-distribution generalization to unseen object scales and robustness against dynamic disturbances, offering a durable and cost-effective route toward general-purpose fine-grained manipulation in unstructured environments. Project videos and supplementary materials are available at: https://sites.google.com/view/vista-policy.

Abstract

Explore similar work

VISTA: Visually Inferred Spatial ConTact Attention for Contact-Rich Manipulation

OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation