Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding

Zhengtong Xu; Yeping Wang; Ben Abbatematteo; Jom Preechayasomboon; Sonny Chan; Nick Colonnese; Amirhossein H. Memar

cs.ROMar 5, 2026

Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding

Authors: Zhengtong Xu, Yeping Wang, Ben Abbatematteo, Jom Preechayasomboon, Sonny Chan, Nick Colonnese, Amirhossein H. Memar

Organizations: Purdue University · University of Wisconsin–Madison · Meta Reality Labs Research

Abstract

Contact-rich dexterous manipulation with multi-finger hands remains an open challenge in robotics because task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile-informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low-level controller dynamics. We present Contact-Grounded Policy (CGP), a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller. CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping that converts the predicted robot state-tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays. Across a range of dexterous tasks including in-hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile diffusion-policy baselines.

Figures & tables

Method Category	Executable Contact Modeling	Multi-Finger Hand	Extensible to Distributed Contacts
Adaptive Compliance Policies [ 12 , 31 , 5 ]	✓	✗	✗
Sparse Fingertip Force Policies [ 39 , 3 ]	✓	✓	✗
Contact-Grounded Policy (Ours)	✓	✓	✓

TABLE I: Comparison of representative policy paradigms with executable contact modeling.

Fig. 1: Schematic of contact grounding using a 3-DoF revolute finger, illustrating the actual robot state, target robot state, and the resulting contact patches. We assume that each joint is controlled by a low-level proportional-derivative (PD) controller, which can be viewed as a virtual spring-damper that maps the tracking error between target and actual joint angles to motor torques, enabling compliant motion.

Fig. 2: Overview of Contact-Grounded Policy (CGP). CGP grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller.

Fig. 3: Teleoperation pipeline. We use a Meta Quest 3 headset for VR-based hand tracking in simulation and an OptiTrack system for mocap-based hand tracking on the real robot. Despite different tracking front-ends, both settings share the same retargeting and controller-stack architecture.

	Sim			Real
Method	In-Hand Box Flipping 60 demos	Fragile Egg Grasping 100 demos	Dish Wiping 100 demos	Jar Opening 45 demos	Real In-Hand Box Flipping 90 demos
Contact-Grounded Policy	66.0%	74.8%	58.4%	93.3%	80.0%
Visuotactile DP	58.0%	70.0%	43.6%	66.7%	60.0%
Visuomotor DP	53.2%	53.2%	42.4%	73.3%	60.0%

TABLE II: Success rate over 5 challenging contact-rich, dexterous manipulation tasks. DP is short for diffusion policy. For the simulation tasks (left three), we report the mean success rate over the last 5 checkpoints, evaluated with 250 rollouts in total. For the real-world tasks, we report the success rate over 15 consecutive rollouts.

Fig. 4: Snapshots of Contact-Grounded Policy (CGP) rollouts on three simulated tasks, showing time-aligned predicted and observed tactile feedback. At each inference step, the diffusion model predicts the next 16 steps of tactile feedback and actual states, which are mapped to target states and executed for 8 steps before the next inference. Predicted tactile is time-aligned with subsequent observations after execution, and the close match indicates that CGP executes contact-grounded targets and realizes the predicted contact evolution. Full rollout videos are provided in the supplementary material.

Fig. 5: Hand configuration predictions by the contact-consistency mapping for unseen grasps (Section V-B ). This figure provides high-level evidence that contact can be consistently represented through this mapping in a way that generalizes across diverse contact configurations.

Input Modality	Tactile Encoder	Abs. Mode $\downarrow$	Residual Mode $\downarrow$
State + Tactile	ResNet1D	8.80 $\pm\,0.24$	5.94 $\pm\,0.20$
	MLP	12.50 $\pm\,0.32$	8.33 $\pm\,0.32$
	Transformer	14.39 $\pm\,0.38$	9.58 $\pm\,0.48$
State Only	-	16.05 $\pm\,0.39$	10.64 $\pm\,0.38$
Tactile Only	ResNet1D	35.93 $\pm\,0.89$	12.15 $\pm\,0.20$
	MLP	36.86 $\pm\,0.25$	12.72 $\pm\,0.25$

TABLE III: Hand configuration prediction results (Section V-B ). MAE denotes mean absolute error. Values are reported in joint-angle space as MAE (

\times 10^{-3}

rad), shown as mean

\pm

std over 3 seeds. Model initializations and the train/validation splits are determined by the random seed. “Abs.” stands for absolute.

	Box			Egg			Dish
Method	MAE $\downarrow$	Active MAE $\downarrow$	KL $\downarrow$	MAE $\downarrow$	Active MAE $\downarrow$	KL $\downarrow$	MAE $\downarrow$	Active MAE $\downarrow$	KL $\downarrow$
MLP (w/ KL)	1.37	12.24	0.21	0.77	6.70	0.15	1.86	8.14	0.15
MLP (w/o KL)	1.12	9.93	0.59	0.54	4.51	0.40	1.40	6.20	0.42
ResNet1D (w/ KL)	1.26	12.07	0.12	0.69	5.95	0.22	1.54	6.80	0.24
ResNet1D (w/o KL)	0.97	9.91	0.73	0.45	3.92	0.43	1.02	4.49	0.45
Transformer (w/ KL)	1.66	14.88	0.19	1.58	10.58	0.28	3.19	13.57	0.21

TABLE IV: Tactile reconstruction and compression results on the validation set. MAE stands for mean absolute error. MAE averages the per-unit force error over all 748 tactile units, where each unit’s force magnitude is computed as the resultant of the 3-axis forces. Active MAE computes the same error but only over tactile units whose ground-truth force is non-zero (i.e., active contact units). MAE and Active MAE are reported in

10^{-2}\,\mathrm{N}

Fig. 6: Ablation results of KL regularization for tactile compression. MAE stands for mean absolute error. “Recon.” stands for reconstruction.

Fig. 7: Inference time comparison. Average time over 50 runs for visuomotor diffusion-policy baseline, visuotactile diffusion-policy baseline, and Contact-Grounded Policy (CGP) using 8-step DDIM denoising. To guarantee fairness, shared network modules utilize the same architecture and size across all methods. For architectural details, please refer to Section III-E .

Fig. 8: VAE reconstruction examples on the validation set for tactile arrays. We show ground truth (top) and reconstruction (bottom).

Fig. 9: VAE reconstruction examples on the validation set for Digit360 tactile images. We show ground truth (top) and reconstruction (bottom).

Task	Domain	Demos	${T}$	${T_{o}}$	${T_{a}}$	Frequency	Action	State	Tactile	Vision	Latent	KL Weight
In-Hand Box Flipping	Sim	60	16	2	8	5 Hz 8 DDIM steps	29	27	$748{\times}3$ (Tactile Array)	$2{\times}$ RGB $180{\times}320$	32	$1{\times}10^{-5}$
Fragile Egg Grasping	Sim	100	16	2	8	5 Hz 8 DDIM steps	29	27	$748{\times}3$ (Tactile Array)	$2{\times}$ RGB $180{\times}320$	32	$1{\times}10^{-5}$
Dish Wiping	Sim	100	16	2	8	5 Hz 8 DDIM steps	29	27	$748{\times}3$ (Tactile Array)	$2{\times}$ RGB $180{\times}320$	32	$1{\times}10^{-4}$
Jar Opening	Real	45	16	2	8	5 Hz 8 DDIM steps	25	23	$4{\times}$ RGB $72{\times}72$ (Digit360)	$2{\times}$ RGB $240{\times}320$	80	$5{\times}10^{-5}$
Real In-Hand Box Flipping	Real	90	16	2	8	5 Hz 8 DDIM steps	25	23	$4{\times}$ RGB $72{\times}72$ (Digit360)	$2{\times}$ RGB $240{\times}320$	80	$5{\times}10^{-5}$

TABLE V: Summaries of the task, training, and inference specifications. Here,

T

is the prediction horizon,

T_{o}

is the temporal length of conditioned observations, and

T_{a}

is the execution horizon; Frequency reports both the policy rollout frequency and the denoising sampling steps: we rollout the policy at 5 Hz for all tasks and use 8 DDIM steps per inference. The specific inference time is reported in Fig. 8 of the paper. In simulation, the action space has 29 dimensions (arm position + rot6D [ 4 , 42 ] + 20 hand joints) and the robot state observation has 27 dimensions (arm position + quaternion + 20 hand joints). On real hardware, the action space has 25 dimensions (arm position + rot6D + 16 hand joints) and the robot state observation has 23 dimensions (arm position + quaternion + 16 hand joints). rot6D facilitates stable action regression [ 4 , 42 ] . Latent denotes the tactile latent dimension: 32 for tactile arrays and 80 for Digit360 (20 per sensor, 4 sensors). Vision uses two RGB views (agent and wrist) with resolutions shown in the table. KL Weight is the weighting coefficient of the KL term in the training objective.

Fig. 10: 11 objects with different shapes and sizes used for the hand configuration prediction experiment.

Fig. 11: Snapshot of real-world Contact-Grounded Policy rollouts, with overlaid target and actual robot states. Before contact, the small target-actual mismatch primarily reflects the steady-state tracking error of the low-level PD controller (e.g., due to gravity and friction; see the first column). During contact, the gap grows as the joint-space compliance controller yields to contact forces; in this phase, the gap reflects the physical interaction. When a finger loses contact, the contact forces disappear and the gap quickly shrinks.

Latent Dim	Model	Box				Jar
Latent Dim	Model	MAE $\downarrow$	KL Loss $\downarrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	KL Loss $\downarrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$
80	w/ KL	9.01	0.0694	35.2050	0.9870	8.10	0.0647	35.9445	0.9883
80	w/o KL	4.55	0.7781	40.4590	0.9939	3.88	0.6247	42.9845	0.9957
160	w/ KL	9.41	0.0341	35.2768	0.9874	8.40	0.0337	36.5394	0.9891
160	w/o KL	4.66	0.7902	40.4800	0.9941	3.98	0.6899	42.9143	0.9957
320	w/ KL	9.02	0.0171	35.2693	0.9870	8.29	0.0155	36.1177	0.9885

TABLE VI: Validation results for tactile compression on real in-hand box flipping and jar opening tasks. MAE denotes mean absolute error (reported in

\times 10^{-3}

), computed on tactile images normalized to

[0,1]

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Fig. 12: Robustness of the contact-consistency mapping to tactile noise. Gaussian noise is injected into tactile inputs with different standard deviations, expressed as percentages of the dataset-level tactile standard deviation. The prediction error remains stable across a wide range of noise levels, indicating that the mapping is robust to tactile perturbations.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models
Oct 20, 2026Ziyong Wang, Yupei Lin, Runxuan Huang +3Tactile-Vision-Language-Action Models Robot Policy Learning
Sun Yat-sen University · X-Era AI Lab · The Hong Kong Polytechnic University
ForeTac: Predictive Contact Guidance for Generative Robot Policies
Oct 13, 2026Zhiyuan Chen, Shuai Chen, Haoran Zhou +7Contact-Rich Robotic Manipulation Tactile World Models
PredTac: Learning Contact-Rich Manipulation with Predicted Touch
Sep 14, 2026Weijia Fan, Daqiang GuoVisual-to-Tactile Prediction Tactile Sensing
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Oct 20, 2026J/K move · Enter open · S save

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

Ziyong Wang, Yupei Lin, Runxuan Huang, Liang Lin+2

Sun Yat-sen University · X-Era AI Lab · The Hong Kong Polytechnic University

Tactile-Vision-Language-Action Models Robot Policy Learning Contact-Rich Robotic Manipulation Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

General-purpose multimodal models have shown promise as direct robot action policies, yet whether they grasp the physics underlying real-world interactions and can apply that understanding to guide action remains underexplored. We investigate this question using contact-rich manipulation as a representative testbed, examining tactile understanding as a concrete instance of this broader capability. Contact-rich manipulation requires interpreting tactile feedback and adjusting actions after obstruction. Existing learned tactile policies acquire these capabilities through specialized pretraining and per-task optimization, with substantial data and training costs. We present GPT-as-Tactile-Policy, which summarizes expert demonstrations into task guidance and tactile references for contact assessment and corrective action. GPT-6-Astra combines this context with multimodal observations and interaction history to generate control commands and adjust actions through contact feedback. Without additional robotics-specific pretraining or per-task policy fine-tuning, our method achieves 93.8% mean success across five UniVTAC tasks. Overall performance is comparable to state-of-the-art policies, with higher success on selected tasks. On Insert Hole, tactile feedback improves success from 31% to 99%. Execution analysis further demonstrates contact-guided probing and correction, supporting manipulation progress through explicit expert experience and online feedback.

Abstract

Robotic Manipulation

Explore similar work

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

ForeTac: Predictive Contact Guidance for Generative Robot Policies

PredTac: Learning Contact-Rich Manipulation with Predicted Touch