cs.ROMar 5, 2026

Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding

Authors: Zhengtong Xu, Yeping Wang, Ben Abbatematteo, Jom Preechayasomboon, Sonny Chan, Nick Colonnese, Amirhossein H. Memar

Organizations: Purdue University · University of Wisconsin–Madison · Meta Reality Labs Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Contact-rich dexterous manipulation with multi-finger hands remains an open challenge in robotics because task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile-informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low-level controller dynamics. We present Contact-Grounded Policy (CGP), a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller. CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping that converts the predicted robot state-tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays. Across a range of dexterous tasks including in-hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile diffusion-policy baselines.

Figures & tables

Method CategoryExecutable Contact ModelingMulti-Finger HandExtensible to Distributed Contacts
Adaptive Compliance Policies [ 12 , 31 , 5 ]✓✗✗
Sparse Fingertip Force Policies [ 39 , 3 ]✓✓✗
Contact-Grounded Policy (Ours)✓✓✓
TABLE I: Comparison of representative policy paradigms with executable contact modeling.
Fig. 1: Schematic of contact grounding using a 3-DoF revolute finger, illustrating the actual robot state, target robot state, and the resulting contact patches. We assume that each joint is controlled by a low-level proportional-derivative (PD) controller, which can be viewed as a virtual spring-damper that maps the tracking error between target and actual joint angles to motor torques, enabling compliant motion.
Fig. 2: Overview of Contact-Grounded Policy (CGP). CGP grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller.
Fig. 3: Teleoperation pipeline. We use a Meta Quest 3 headset for VR-based hand tracking in simulation and an OptiTrack system for mocap-based hand tracking on the real robot. Despite different tracking front-ends, both settings share the same retargeting and controller-stack architecture.
 SimReal
MethodIn-Hand Box Flipping 60 demosFragile Egg Grasping 100 demosDish Wiping 100 demosJar Opening 45 demosReal In-Hand Box Flipping 90 demos
Contact-Grounded Policy66.0%74.8%58.4%93.3%80.0%
Visuotactile DP58.0%70.0%43.6%66.7%60.0%
Visuomotor DP53.2%53.2%42.4%73.3%60.0%
TABLE II: Success rate over 5 challenging contact-rich, dexterous manipulation tasks. DP is short for diffusion policy. For the simulation tasks (left three), we report the mean success rate over the last 5 checkpoints, evaluated with 250 rollouts in total. For the real-world tasks, we report the success rate over 15 consecutive rollouts.
Fig. 4: Snapshots of Contact-Grounded Policy (CGP) rollouts on three simulated tasks, showing time-aligned predicted and observed tactile feedback. At each inference step, the diffusion model predicts the next 16 steps of tactile feedback and actual states, which are mapped to target states and executed for 8 steps before the next inference. Predicted tactile is time-aligned with subsequent observations after execution, and the close match indicates that CGP executes contact-grounded targets and realizes the predicted contact evolution. Full rollout videos are provided in the supplementary material.
Fig. 5: Hand configuration predictions by the contact-consistency mapping for unseen grasps (Section V-B ). This figure provides high-level evidence that contact can be consistently represented through this mapping in a way that generalizes across diverse contact configurations.
Input ModalityTactile EncoderAbs. Mode ↓\downarrowResidual Mode ↓\downarrow
State + TactileResNet1D8.80 ± 0.24\pm\,0.245.94 ± 0.20\pm\,0.20
 MLP12.50 ± 0.32\pm\,0.328.33 ± 0.32\pm\,0.32
 Transformer14.39 ± 0.38\pm\,0.389.58 ± 0.48\pm\,0.48
State Only-16.05 ± 0.39\pm\,0.3910.64 ± 0.38\pm\,0.38
Tactile OnlyResNet1D35.93 ± 0.89\pm\,0.8912.15 ± 0.20\pm\,0.20
 MLP36.86 ± 0.25\pm\,0.2512.72 ± 0.25\pm\,0.25
TABLE III: Hand configuration prediction results (Section V-B ). MAE denotes mean absolute error. Values are reported in joint-angle space as MAE ( ×10−3\times 10^{-3} rad), shown as mean ±\pm std over 3 seeds. Model initializations and the train/validation splits are determined by the random seed. “Abs.” stands for absolute.
 BoxEggDish
MethodMAE ↓\downarrowActive MAE ↓\downarrowKL ↓\downarrowMAE ↓\downarrowActive MAE ↓\downarrowKL ↓\downarrowMAE ↓\downarrowActive MAE ↓\downarrowKL ↓\downarrow
MLP (w/ KL)1.3712.240.210.776.700.151.868.140.15
MLP (w/o KL)1.129.930.590.544.510.401.406.200.42
ResNet1D (w/ KL)1.2612.070.120.695.950.221.546.800.24
ResNet1D (w/o KL)0.979.910.730.453.920.431.024.490.45
Transformer (w/ KL)1.6614.880.191.5810.580.283.1913.570.21
TABLE IV: Tactile reconstruction and compression results on the validation set. MAE stands for mean absolute error. MAE averages the per-unit force error over all 748 tactile units, where each unit’s force magnitude is computed as the resultant of the 3-axis forces. Active MAE computes the same error but only over tactile units whose ground-truth force is non-zero (i.e., active contact units). MAE and Active MAE are reported in 10−2 N10^{-2}\,\mathrm{N} .
Fig. 6: Ablation results of KL regularization for tactile compression. MAE stands for mean absolute error. “Recon.” stands for reconstruction.
Fig. 7: Inference time comparison. Average time over 50 runs for visuomotor diffusion-policy baseline, visuotactile diffusion-policy baseline, and Contact-Grounded Policy (CGP) using 8-step DDIM denoising. To guarantee fairness, shared network modules utilize the same architecture and size across all methods. For architectural details, please refer to Section III-E .
Fig. 8: VAE reconstruction examples on the validation set for tactile arrays. We show ground truth (top) and reconstruction (bottom).
Fig. 9: VAE reconstruction examples on the validation set for Digit360 tactile images. We show ground truth (top) and reconstruction (bottom).
TaskDomainDemosT{T}To{T_{o}}Ta{T_{a}}FrequencyActionStateTactileVisionLatentKL Weight
In-Hand Box FlippingSim6016285 Hz 8 DDIM steps2927748×3748{\times}3 (Tactile Array)2×2{\times} RGB 180×320180{\times}320321×10−51{\times}10^{-5}
Fragile Egg GraspingSim10016285 Hz 8 DDIM steps2927748×3748{\times}3 (Tactile Array)2×2{\times} RGB 180×320180{\times}320321×10−51{\times}10^{-5}
Dish WipingSim10016285 Hz 8 DDIM steps2927748×3748{\times}3 (Tactile Array)2×2{\times} RGB 180×320180{\times}320321×10−41{\times}10^{-4}
Jar OpeningReal4516285 Hz 8 DDIM steps25234×4{\times} RGB 72×7272{\times}72 (Digit360)2×2{\times} RGB 240×320240{\times}320805×10−55{\times}10^{-5}
Real In-Hand Box FlippingReal9016285 Hz 8 DDIM steps25234×4{\times} RGB 72×7272{\times}72 (Digit360)2×2{\times} RGB 240×320240{\times}320805×10−55{\times}10^{-5}
TABLE V: Summaries of the task, training, and inference specifications. Here, TT is the prediction horizon, ToT_{o} is the temporal length of conditioned observations, and TaT_{a} is the execution horizon; Frequency reports both the policy rollout frequency and the denoising sampling steps: we rollout the policy at 5 Hz for all tasks and use 8 DDIM steps per inference. The specific inference time is reported in Fig. 8 of the paper. In simulation, the action space has 29 dimensions (arm position + rot6D [ 4 , 42 ] + 20 hand joints) and the robot state observation has 27 dimensions (arm position + quaternion + 20 hand joints). On real hardware, the action space has 25 dimensions (arm position + rot6D + 16 hand joints) and the robot state observation has 23 dimensions (arm position + quaternion + 16 hand joints). rot6D facilitates stable action regression [ 4 , 42 ] . Latent denotes the tactile latent dimension: 32 for tactile arrays and 80 for Digit360 (20 per sensor, 4 sensors). Vision uses two RGB views (agent and wrist) with resolutions shown in the table. KL Weight is the weighting coefficient of the KL term in the training objective.
Fig. 10: 11 objects with different shapes and sizes used for the hand configuration prediction experiment.
Fig. 11: Snapshot of real-world Contact-Grounded Policy rollouts, with overlaid target and actual robot states. Before contact, the small target-actual mismatch primarily reflects the steady-state tracking error of the low-level PD controller (e.g., due to gravity and friction; see the first column). During contact, the gap grows as the joint-space compliance controller yields to contact forces; in this phase, the gap reflects the physical interaction. When a finger loses contact, the contact forces disappear and the gap quickly shrinks.
Latent DimModelBoxJar
MAE ↓\downarrowKL Loss ↓\downarrowPSNR (dB) ↑\uparrowSSIM ↑\uparrowMAE ↓\downarrowKL Loss ↓\downarrowPSNR (dB) ↑\uparrowSSIM ↑\uparrow
80w/ KL9.010.069435.20500.98708.100.064735.94450.9883
w/o KL4.550.778140.45900.99393.880.624742.98450.9957
160w/ KL9.410.034135.27680.98748.400.033736.53940.9891
w/o KL4.660.790240.48000.99413.980.689942.91430.9957
320w/ KL9.020.017135.26930.98708.290.015536.11770.9885
TABLE VI: Validation results for tactile compression on real in-hand box flipping and jar opening tasks. MAE denotes mean absolute error (reported in ×10−3\times 10^{-3} ), computed on tactile images normalized to [0,1][0,1] .
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Fig. 12: Robustness of the contact-consistency mapping to tactile noise. Gaussian noise is injected into tactile inputs with different standard deviations, expressed as percentages of the dataset-level tactile standard deviation. The prediction error remains stable across a wide range of noise levels, indicating that the mapping is robust to tactile perturbations.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Explore similar work

CardsList
Oct 20, 2026

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

General-purpose multimodal models have shown promise as direct robot action policies, yet whether they grasp the physics underlying real-world interactions and can apply that understanding to guide action remains underexplored. We investigate this question using contact-rich manipulation as a representative testbed, examining tactile understanding as a concrete instance of this broader capability. Contact-rich manipulation requires interpreting tactile feedback and adjusting actions after obstruction. Existing learned tactile policies acquire these capabilities through specialized pretraining and per-task optimization, with substantial data and training costs. We present GPT-as-Tactile-Policy, which summarizes expert demonstrations into task guidance and tactile references for contact assessment and corrective action. GPT-6-Astra combines this context with multimodal observations and interaction history to generate control commands and adjust actions through contact feedback. Without additional robotics-specific pretraining or per-task policy fine-tuning, our method achieves 93.8% mean success across five UniVTAC tasks. Overall performance is comparable to state-of-the-art policies, with higher success on selected tasks. On Insert Hole, tactile feedback improves success from 31% to 99%. Execution analysis further demonstrates contact-guided probing and correction, supporting manipulation progress through explicit expert experience and online feedback.

Ziyong Wang, Yupei Lin, Runxuan Huang +3

Sun Yat-sen University · X-Era AI Lab · The Hong Kong Polytechnic University

Tactile-Vision-Language-Action ModelsRobot Policy LearningContact-Rich Robotic ManipulationTactile Sensing for Robotic Manipulation
AskPDF
Oct 13, 2026

ForeTac: Predictive Contact Guidance for Generative Robot Policies

In contact-rich manipulation, actions that appear equally plausible from vision can produce different contact outcomes, while reactive control based on current tactile feedback may respond too late to prevent failure. We present ForeTac, a predictive contact-guidance framework organized as a represent-predict-evaluate-guide loop. ForeTac encodes the current tactile state, predicts future tactile latent trajectories conditioned on candidate action sequences, and evaluates each predicted trajectory with a contact-quality objective learned from successful and failed physical interactions. At inference, gradients of this objective are backpropagated through the tactile predictor to guide action generation before execution while keeping all model parameters frozen. We evaluate ForeTac on four real-robot tasks (planar and curved-surface wiping, card swiping, and fragile-object grasping) with diffusion and flow-matching policies. Compared with vision-only Diffusion Policy, ForeTac improves the average task success rate (SR) from 36.3% to 71.3% and the average contact success rate (CSR) from 20.0% to 66.3% across these tasks, and yields gains in both metrics with a flow-matching policy. Component analyses support the learned tactile representation, action-conditioned foresight, and generation-time gradient guidance, while predicted tactile latent trajectories preserve contact-quality ordering. Project page: https://foretac.github.io/.

Zhiyuan Chen, Shuai Chen, Haoran Zhou +7

Contact-Rich Robotic ManipulationTactile World Models
AskPDF
Sep 14, 2026

PredTac: Learning Contact-Rich Manipulation with Predicted Touch

Contact-rich manipulation benefits from tactile feedback, yet physical tactile sensors introduce hardware, calibration, synchronization, and maintenance costs that complicate policy learning and deployment. We formulate predicted touch as an alternative to measured tactile input and present PredTac, a framework that learns to infer tactile states from causal visual observations and robot states and uses the predicted touch as an explicit interface for policy learning and execution. A tactile predictor is first trained with tactile supervision and then used to provide contact information without requiring measured tactile input during downstream policy training or execution. We evaluate PredTac across three contact-rich manipulation tasks in simulation and on a real robot, and further examine how policy performance depends on the predicted contact content. In simulation goal-offset evaluations, predicted-touch policies achieve 27.0%, 52.0%, and 44.7% success on USB, Barbed-spike, and Valve, respectively, improving over the visual baseline by 8.0-13.7 percentage points. On the real robot, predicted-touch ACT achieves 70.0%, 50.0%, and 90.0% success on USB insertion, Barbed extraction, and Valve rotation, respectively, with a three-task mean of 70.0%, approaching measured-touch ACT at 72.2% and substantially outperforming visual ACT at 21.1%. Fixed-policy interventions further show that performance is sensitive to the spatial structure of predicted contact, with spatial rearrangement at fixed value distributions reducing Valve success by 10.7 percentage points. These results demonstrate that predicted touch can provide useful contact information for contact-rich manipulation without requiring tactile sensing as a policy input.

Weijia Fan, Daqiang Guo

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Visual-to-Tactile PredictionTactile SensingContact-Rich Robotic ManipulationRobotic RL
AskPDF