Tactile sensing is particularly valuable for contact-rich robotic manipulation. Recent work has made substantial progress in tactile representation learning for robotic manipulation. However, similar tactile observations can arise from interaction conditions with very different task-risk implications, such as sensor noise, task-necessary variations, and emerging undesirable contact. Without context-grounded risk information, these cases can be ambiguous to downstream policies, leading to unnecessary corrections to benign variations or delayed responses to genuinely risky contact. To address this limitation, we propose Risk-Aware Tactile Encoding (RATE), which learns tactile representations that encode task-conditioned interaction risk. Specifically, history-conditioned prediction captures interaction context, while alert supervision associates this context with task-conditioned risk. The learned risk-aware representation complements conventional tactile features through a lightweight residual adapter. Experiments in both simulation and the real world demonstrate substantial improvements in task success, with controlled ablations confirming the complementary benefits of alert-guided learning and predictive temporal modeling.
Figures & tables
Figure 1: RATE in contact-rich manipulation: corrective behavior and task performance. (a) Representative interaction sequence: when the interaction reaches critical contact, baseline methods may continue toward failure, whereas RATE enables a corrective response toward successful execution. (b) Success rates across all eight simulation tasks, comparing RATE with representative tactile and large-scale policy baselines. RATE achieves the best or jointly best performance on six of the eight tasks. (c) Success rates on three real-robot tasks, including USB insertion, bottle-cap screwing, and plug insertion, where RATE achieves the highest success rate on all three tasks.
Figure 2: Overview of RATE representation learning and policy adaptation. RATE learns a complementary risk-aware representation from tactile history, jointly shaped by short-horizon future contact-change prediction and future alert supervision. During policy adaptation, the learned representation is mapped to a bounded residual and fused with conventional tactile features through a lightweight residual pathway. The resulting tactile representation, together with camera observations and robot state, conditions the downstream manipulation policy. The tactile encoders and policy backbone remain frozen during adaptation, and all auxiliary prediction heads are used only for training.
Method
Params.
Contact-Rich Interaction
Pose Reasoning
Shape Perception
Avg.
Pull-out Key
Insert Hole
Insert HDMI
Insert Tube
Lift Bottle
Lift Can
Put Bottle
Grasp Classify
ACT (Vision Only) [ 22 ]
0.08B
28.0
19.0
15.0
45.0
42.0
20.0
28.0
50.0
30.9
ACT+UniVTAC [ 4 ]
0.10B
46.0
24.0
28.0
56.0
71.0
29.0
31.0
99.0
48.0
VITaL [ 2 ]
0.10B
47.0
25.0
6.0
34.0
72.0
8.0
32.0
100.0
40.5
π0.5 [ 23 ]
3.3B
35.0
25.0
8.0
74.0
100.0
6.0
34.0
49.0
41.4
StarVLA- α [ 24 ]
4B
51.0
52.0
24.0
69.0
32.0
65.0
88.0
68.0
56.1
Table I: Simulation success rates (%) on the eight-task UniVTAC benchmark. Best results are bold. Published ACT, ACT+UniVTAC, and VITaL success rates are reported in UniVTAC [ 4 ] . The success rates of π0.5 , StarVLA- α , and GigaWorld-Policy are reported in N0 -VTLA [ 13 ] . Published results are included for reference and may differ in demonstration count and observation configuration. ACT+UniVTAC (Retrained) is retrained and evaluated by us under the same configuration as RATE.
Alert Supervision
Context
Contact-Rich Interaction
Pose Reasoning
Shape Perception
Avg.
Discrete
Continuous
Transition- Aware
Pull-out Key
Insert Hole
Insert HDMI
Insert Tube
Lift Bottle
Lift Can
Put Bottle
Grasp Classify
–
–
–
–
51.0
30.0
16.0
49.0
51.0
49.0
7.0
99.0
44.0
✓
–
–
–
65.0
31.0
31.0
59.0
67.0
62.0
34.0
100.0
56.1
–
✓
–
–
81.0
32.0
18.0
66.0
61.0
67.0
35.0
100.0
57.5
–
–
✓
–
78.0
41.0
14.0
66.0
68.0
52.0
29.0
100.0
56.0
–
–
–
✓
68.0
46.0
25.0
56.0
73.0
68.0
12.0
100.0
56.0
Table II: Ablation results for RATE on the eight-task UniVTAC benchmark. All variants share the same task-specific tactile base. The table evaluates alert supervision using Discrete Ordinal, Continuous Severity, and Transition-Aware Continuous formulations, while context denotes interaction context learned through history-conditioned prediction. The final row corresponds to Full RATE, combining continuous severity alert supervision with context. Avg. denotes the eight-task macro average. Best results are bold.
Method
USB Insertion
Bottle Cap Screwing
Plug Insertion
SR
CSSR
SR
CSSR
SR
CSSR
ACT (Vision Only)
0.0
0.00
15.0
33.33
30.0
33.33
ACT+UniVTAC
30.0
0.00
35.0
85.71
50.0
70.00
π0.5
40.0
62.50
45.0
55.56
35.0
57.14
RATE (Ours)
75.0
93.33
85.0
100.00
60.0
83.33
Table III: Real-world evaluation over 20 trials per task. Task Success Rate (SR) and Contact-Safe Success Rate (CSSR) are reported in separate columns, in percent. Best results are bold.
Figure 3: Real-world contact-rich manipulation tasks. Rows from top to bottom show USB Insertion, Bottle Cap Screwing, and Plug Insertion. Columns show successive stages of each task from left to right.
Tactile information is essential for contact-rich manipulation tasks in robotics. Vision-based tactile sensors make it particularly easy to design end-to-end manipulation policies with tactile sensing, as they enable the use of existing encoders from computer vision. However, this has led to a huge variety of architectures, training datasets, and evaluation protocols, making it difficult to determine which design choices best encode touch. In this work, we address this gap and present a comprehensive study of tactile encoders and fusion strategies across various contact-rich manipulation tasks in real-world experiments. To enable a controlled comparison, we train and evaluate all models under the same pipeline and experimental setup, comprising more than 2000 real-world rollouts. Our results go beyond other studies that only compare simulation performance, which does not necessarily translate to real-world settings, where large-scale evaluations are needed to obtain reliable statistics. Our key finding is that there is no universally optimal representation or fusion strategy for encoding visual-tactile. Instead, the best encoder backbone and fusion scheme depend strongly on the task.
Seongjin Bien, Débora Oliveira Makowski, Carlo Kneissl +6
University of Technology Nuremberg, Germany. · Ludwig-Maximilians University Munich, Germany. · Munich Center for Machine Learning (MCML), Germany. +3
Contact-rich manipulation benefits from tactile feedback, yet physical tactile sensors introduce hardware, calibration, synchronization, and maintenance costs that complicate policy learning and deployment. We formulate predicted touch as an alternative to measured tactile input and present PredTac, a framework that learns to infer tactile states from causal visual observations and robot states and uses the predicted touch as an explicit interface for policy learning and execution. A tactile predictor is first trained with tactile supervision and then used to provide contact information without requiring measured tactile input during downstream policy training or execution. We evaluate PredTac across three contact-rich manipulation tasks in simulation and on a real robot, and further examine how policy performance depends on the predicted contact content. In simulation goal-offset evaluations, predicted-touch policies achieve 27.0%, 52.0%, and 44.7% success on USB, Barbed-spike, and Valve, respectively, improving over the visual baseline by 8.0-13.7 percentage points. On the real robot, predicted-touch ACT achieves 70.0%, 50.0%, and 90.0% success on USB insertion, Barbed extraction, and Valve rotation, respectively, with a three-task mean of 70.0%, approaching measured-touch ACT at 72.2% and substantially outperforming visual ACT at 21.1%. Fixed-policy interventions further show that performance is sensitive to the spatial structure of predicted contact, with spatial rearrangement at fixed value distributions reducing Valve success by 10.7 percentage points. These results demonstrate that predicted touch can provide useful contact information for contact-rich manipulation without requiring tactile sensing as a policy input.
Weijia Fan, Daqiang Guo
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Tactile-enhanced vision-language-action (VLA) policies have been introduced for contact-rich manipulation, where critical interaction states are often hidden from vision. Future tactile prediction is a promising way to use touch because it turns tactile outcomes into supervision for action-induced contact dynamics. Yet VLA policies contain representations with different roles, from perceptual encoding to motor prediction, making it unclear where this supervision should be applied. We study this as a representation-alignment problem. Through a linear probe analysis, we find that future tactile states are most predictable from intermediate action-expert features, rather than from vision-language features or final action states. Motivated by this observation, we introduce a lightweight Latent Tactile Predictor (LTP), which predicts compact future tactile embeddings from the identified intermediate representation. By avoiding direct prediction of noisy raw tactile signals, LTP provides an action-outcome grounding signal that aligns intermediate action representations with future contact consequences. Experiments on real-world contact-rich manipulation tasks show that representation-aligned tactile grounding outperforms less aligned or multi-interface tactile prediction, highlighting the importance of where tactile supervision is applied.
Ruilin Chen, Jingkai Jia, Tong Yang +8
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · TeleAl, China Telecom +2