Contact-rich manipulation requires policies to generate precise actions by reasoning over contact forces, robot configurations, and interaction histories beyond visual observations. Existing methods passively condition on force feedback rather than actively predicting future contact dynamics, limiting their ability to generate high-precision actions. To address this problem, we introduce Predictive Sensorimotor Representation (PSR) learning, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy. Specifically, during a pretraining stage, a multimodal Transformer is trained to learn a hierarchy of predictive representations by jointly forecasting future interaction dynamics. The learned hierarchy subsequently augments the action stream, enabling the resulting policy to exploit contact-relevant cues at multiple depths. We further instantiate PSR within a Vision-Language-Action (VLA) model, resulting in PSR-VLA, and evaluate it on six real-world contact-rich manipulation tasks. Experimental results show that PSR-VLA achieves 91.7% overall success, improving over π0.5, ForceVLA-π0.5, and ForceVLA2-π0.5 by 30.0, 22.5, and 19.2 percentage points, respectively. These results demonstrate the effectiveness of the proposed PSR for force-aware, contact-rich manipulation. Videos of the tasks and stability tests are available at https://psr-vla.pages.dev/.
Figures & tables
Fig. 1: An overview of PSR-VLA and the six real-world contact-rich manipulation tasks. PSR first learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy for improved precision and robustness.
Fig. 2: PSR learns a hierarchy of predictive sensorimotor representations through multi-target prediction in Stage 1. In Stage 2, PSR integrates the resulting representations into the action stream of a policy via gated cross-attention, influencing action generation across multiple depths.
Fig. 3: Multi-depth action augmentation through gated cross-attention.
Method
Hard
Medium
Easy
Overall
Three-Way Interlocking Joint Assembly
Water-Hose Insertion
Three-Hole Plug Insertion
LEGO Assembly
Two-Hole Plug Insertion
Blackboard Wiping
π0.5
9/20
13/20
12/20
13/20
13/20
14/20
74/120 (61.7%)
ForceVLA- π0.5
12/20
14/20
12/20
14/20
15/20
16/20
83/120 (69.2%)
ForceVLA2- π0.5
13/20
15/20
17/20
11/20
16/20
15/20
87/120 (72.5%)
PSR-VLA
18/20
19/20
16/20
18/20
19/20
20/20
110/120 (91.7%)
TABLE I: Main results on six real-world tasks.
Fig. 4: Stage 1 predictions on held-out offline sequences. The decoder predicts all 12 F/T channels, 20 joint-effort channels, and terminal joint-state displacement. The horizontal axes show time (s); the vertical axes show force (N) or joint effort (N m), as indicated by the panel titles. Representative curves are selected by event-associated ground-truth variation rather than prediction accuracy; measured and predicted signals are filtered identically.
Variant
Water-Hose Insertion
Two-Hole Plug Insertion
MSE ↓
L1 ↓
MSE ↓
L1 ↓
Q1: Predictive Pretraining
w/o Stage-1 Pretraining
0.005190
0.026432
0.004378
0.033470
PSR-VLA
0.003378
0.021413
0.003816
0.035424
Q2: Representations and Depth Alignment
Shared z6 at All Layers
0.004269
0.030268
0.004066
0.036335
TABLE II: Offline action-generation ablations.
Fig. 5: Ablation study on Water-Hose Insertion and Two-Hole Plug Insertion. Bars show successes out of 20 attempts per task.
Contact-rich manipulation benefits from tactile feedback, yet physical tactile sensors introduce hardware, calibration, synchronization, and maintenance costs that complicate policy learning and deployment. We formulate predicted touch as an alternative to measured tactile input and present PredTac, a framework that learns to infer tactile states from causal visual observations and robot states and uses the predicted touch as an explicit interface for policy learning and execution. A tactile predictor is first trained with tactile supervision and then used to provide contact information without requiring measured tactile input during downstream policy training or execution. We evaluate PredTac across three contact-rich manipulation tasks in simulation and on a real robot, and further examine how policy performance depends on the predicted contact content. In simulation goal-offset evaluations, predicted-touch policies achieve 27.0%, 52.0%, and 44.7% success on USB, Barbed-spike, and Valve, respectively, improving over the visual baseline by 8.0-13.7 percentage points. On the real robot, predicted-touch ACT achieves 70.0%, 50.0%, and 90.0% success on USB insertion, Barbed extraction, and Valve rotation, respectively, with a three-task mean of 70.0%, approaching measured-touch ACT at 72.2% and substantially outperforming visual ACT at 21.1%. Fixed-policy interventions further show that performance is sensitive to the spatial structure of predicted contact, with spatial rearrangement at fixed value distributions reducing Valve success by 10.7 percentage points. These results demonstrate that predicted touch can provide useful contact information for contact-rich manipulation without requiring tactile sensing as a policy input.
Weijia Fan, Daqiang Guo
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Tactile-enhanced vision-language-action (VLA) policies have been introduced for contact-rich manipulation, where critical interaction states are often hidden from vision. Future tactile prediction is a promising way to use touch because it turns tactile outcomes into supervision for action-induced contact dynamics. Yet VLA policies contain representations with different roles, from perceptual encoding to motor prediction, making it unclear where this supervision should be applied. We study this as a representation-alignment problem. Through a linear probe analysis, we find that future tactile states are most predictable from intermediate action-expert features, rather than from vision-language features or final action states. Motivated by this observation, we introduce a lightweight Latent Tactile Predictor (LTP), which predicts compact future tactile embeddings from the identified intermediate representation. By avoiding direct prediction of noisy raw tactile signals, LTP provides an action-outcome grounding signal that aligns intermediate action representations with future contact consequences. Experiments on real-world contact-rich manipulation tasks show that representation-aligned tactile grounding outperforms less aligned or multi-interface tactile prediction, highlighting the importance of where tactile supervision is applied.
Ruilin Chen, Jingkai Jia, Tong Yang +8
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · TeleAl, China Telecom +2
Vision-Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs coupled with wrist-camera local vision and third-person macro vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial Alignment (HSA) loss that aligns tactile tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic agents.