Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions. However, real-world reinforcement learning often overfits to the visual appearance of training scenes. We propose Semantics-to-Contact (S2C), a stagewise framework for visually robust contact-rich manipulation. The insight is to decompose the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using force feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional variations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation. Our demos can be seen in https://edem-ai.github.io/S2C/.
Figures & tables
Figure 1: S2C Framework : S2C decomposes contact-rich manipulation into two stages: semantic localization and contact refinement . The semantic focusing stage uses the VLM to predict a task-relevant pre-contact goal g^ and motion planning to move the robot near the desired contact region. The contact refinement stage then uses human-in-the-loop real-world residual RL to learn local force-aware corrections.
Figure 2: Different visual robustness settings.
Method
Charger
USB-Easy
USB-Hard
Tube-Easy
Tube-Medium
Tube-Hard
Avg. SR
TT
SR
TT
SR
TT
SR
TT
SR
TT
SR
TT
SR
ACT
–
0/20
–
1/20
–
0/20
–
2/20
–
0/20
–
0/20
2.50%
SERL
120
0/20
120
0/20
120
0/20
120
0/20
120
0/20
120
0/20
0.00%
Octo-small
–
0/20
–
2/20
–
0/20
–
5/20
–
0/20
–
0/20
5.83%
HIL-SERL
30
19/20
60
20/20
120
18/20
30
20/20
120
16/20
120
16/20
90.83%
HIL-ConRFT
30
18/20
40
20/20
60
19/20
20
20/20
100
16/20
100
17/20
91.76%
Table 1: Training time and success rate under training setting. Each setting is tested 20 times. We implement HIL-SERL and HIL-ConRFT following the experimental settings described in their original papers; therefore, they did not use the same VLM-based pre-contact localization and motion-planning initialization as S2C. To ensure consistency and comparison of the success rate and training time, all methods are compared within the same workspace and start from the same initial region in each experiment. We also use the same evaluation protocol and success criterion for all methods, such that a trial is counted as successful only when the object is fully inserted into the target slot. For training time, we report the learner-side actor–critic optimization time because real-robot rollout collection and actor–critic optimization run as separate processes.
Method
Charger
USB-Easy
USB-Hard
Tube-Easy
Tube-Medium
Tube-Hard
Avg. SR
Patterned tablecloth + Position
HIL-SERL
0/20
0/20
0/20
0/20
0/20
0/20
0.00%
HIL-ConRFT
0/20
0/20
0/20
0/20
0/20
0/20
0.00%
S2C
18/20
15/20
11/20
13/20
14/20
12/20
69.17%
Strong sun light + Position
HIL-SERL
0/20
0/20
0/20
0/20
0/20
0/20
0.00%
Table 2: Success rate under visual and positional robustness. Only HIL-based RWRL methods and S2C are evaluated. Each setting is evaluated 20 times.
Method
Charger
USB (Hard)
Avg. SR
Strong sun light + Red tablecloth + Position
w/o HIL
0/20
0/20
0.00%
w/o VLM
10/20
12/20
55.00%
w/o OC-Aug
0/20
0/20
0.00%
w/o Contact
0/20
0/20
0.00%
S2C
16/20
14/20
75.00%
Table 3: Fine-grained ablation of the semantic components. The ablation is conducted on Charger Insertion and USB Insertion (hard). “w/o VLM” removes VLM-guided motion planning, whereas “w/o OC-Aug” removes object-centric augmentation. Each setting is evaluated 20 times.
Perception/VLM Backbone
Inference Time (s) ↓
Success Rate ↑
YOLOE
0.07
0/10 (0%)
SAM 3
0.12
3/10 (30%)
Qwen3-VL-8B
2.00
5/10 (50%)
GPT-5.5 API
15.00
10/10 (100%)
MolmoPoint-8B
2.00
10/10 (100%)
Table 4: Comparison of perception and VLM backbones on the charger insertion task. Inference time denotes the time required to localize the target in each episode. Success rate is evaluated over 10 real-robot trials.
Task
A
B
C
D
E
F
G
H
I
Charger
10/10
9/10
0/10
10/10
8/10
10/10
10/10
10/10
0/10
USB-Easy
10/10
8/10
3/10
10/10
8/10
10/10
10/10
10/10
0/10
Table 5: Scalability evaluation of S2C across different tasks. Each task is tested 10 times.
Task
Training Time (min)
Success Rate (%)
Test-tube Removal
25
90 (9/10)
Hard-drive Removal
35
100 (10/10)
Table 6: Performance of S2C on additional contact-rich removal tasks
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist + 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
30
Appendix
Table 7: Policy training details for the Tube Insertion (easy) task.
Figure 4: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist+ 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
40
Appendix
Table 8: Policy training details for the Tube Insertion (medium) task.
Figure 5: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist + 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
38
Appendix
Table 9: Policy training details for the Charger grasp-insertion.
Figure 6: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist + 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
40
Appendix
Table 10: Policy training details for the Tube Insertion (hard) task.
Figure 7: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
35
Appendix
Table 11: Policy training details for the USB Insertion (easy) task.
Figure 8: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
35
Appendix
Table 12: Policy training details for the USB Insertion (hard) task.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
When using reinforcement learning (RL) for contact-rich robotic manipulation, vision can provide task-relevant information that complements robot proprioception. However, vision-enabled policies tend to overfit to the visual conditions seen during training, limiting their robustness and transferability. We present a human-in-the-loop RL framework that employs teacher-student distillation to achieve robust performance across multiple task variants, trained entirely in the real world without requiring domain randomization or data augmentation. A vision-enabled teacher distills its knowledge into a vision-free student that relies solely on pose, twist, and wrench sensing, combining fast training with strong task generalization. On the real-world NIST assembly benchmark board, our approach achieves 95% overall success after approximately 50 minutes of training on 3 representative tasks, including robust generalization to 8 unseen task variants. Fine-tuning with distillation achieves full success on the most challenging task. We demonstrate that the resulting policies outperform baselines in both robustness and adaptability. Page: https://tuwien-asl.github.io/VE2VF/.
Victor Kowalski, Chengxi Li, Dongheui Lee
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany
Contact-rich manipulation underlies many everyday human activities, yet evaluating robots on such tasks remains difficult: the tasks involve diverse contact behaviors and are hard to reproduce across laboratories. Drawing on observations of everyday human manipulation, we identify eight essential skills that provide a practical framework for studying contact-rich manipulation. We introduce ContactBench 1.0, an extensible benchmark that instantiates these skills through 36 reference tasks and more than 40 object assets. The assets capture task-relevant geometry and contact behavior, can be fabricated reproducibly on consumer-grade 3D printers, and are each paired with a matching simulation model. Controlled task variations in both the real world and simulation enable evaluation of how well robot systems generalize. We evaluate recent representative vision-only and visual-tactile policy learning methods on six benchmark tasks. The results reveal that while vision-only policies struggle to perceive or react to contact events, visual-tactile policies do not meaningfully improve their performance, highlighting that contact-rich manipulation remains a significant challenge for current policy learning methods. By providing common task definitions alongside physical and simulated assets, ContactBench 1.0 offers a shared testbed for developing, evaluating, and comparing contact-rich manipulation systems. The benchmark will be open-sourced at https://jingyi-xiang.github.io/contactbench.
Jingyi Xiang, Yixuan Wang, Jeannette Bohg +1
University of Illinois Urbana-Champaign, Urbana, IL, USA · Stanford University, Palo Alto, CA, USA
Contact-rich manipulation requires policies to generate precise actions by reasoning over contact forces, robot configurations, and interaction histories beyond visual observations. Existing methods passively condition on force feedback rather than actively predicting future contact dynamics, limiting their ability to generate high-precision actions. To address this problem, we introduce Predictive Sensorimotor Representation (PSR) learning, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy. Specifically, during a pretraining stage, a multimodal Transformer is trained to learn a hierarchy of predictive representations by jointly forecasting future interaction dynamics. The learned hierarchy subsequently augments the action stream, enabling the resulting policy to exploit contact-relevant cues at multiple depths. We further instantiate PSR within a Vision-Language-Action (VLA) model, resulting in PSR-VLA, and evaluate it on six real-world contact-rich manipulation tasks. Experimental results show that PSR-VLA achieves 91.7% overall success, improving over π0.5, ForceVLA-π0.5, and ForceVLA2-π0.5 by 30.0, 22.5, and 19.2 percentage points, respectively. These results demonstrate the effectiveness of the proposed PSR for force-aware, contact-rich manipulation. Videos of the tasks and stability tests are available at https://psr-vla.pages.dev/.