Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions. However, real-world reinforcement learning often overfits to the visual appearance of training scenes. We propose Semantics-to-Contact (S2C), a stagewise framework for visually robust contact-rich manipulation. The insight is to decompose the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using force feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional variations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation. Our demos can be seen in https://edem-ai.github.io/S2C/.
Figures & tables
Figure 1: S2C Framework : S2C decomposes contact-rich manipulation into two stages: semantic localization and contact refinement . The semantic focusing stage uses the VLM to predict a task-relevant pre-contact goal g^ and motion planning to move the robot near the desired contact region. The contact refinement stage then uses human-in-the-loop real-world residual RL to learn local force-aware corrections.
Figure 2: Different visual robustness settings.
Method
Charger
USB-Easy
USB-Hard
Tube-Easy
Tube-Medium
Tube-Hard
Avg. SR
TT
SR
TT
SR
TT
SR
TT
SR
TT
SR
TT
SR
ACT
–
0/20
–
1/20
–
0/20
–
2/20
–
0/20
–
0/20
2.50%
SERL
120
0/20
120
0/20
120
0/20
120
0/20
120
0/20
120
0/20
0.00%
Octo-small
–
0/20
–
2/20
–
0/20
–
5/20
–
0/20
–
0/20
5.83%
HIL-SERL
30
19/20
60
20/20
120
18/20
30
20/20
120
16/20
120
16/20
90.83%
HIL-ConRFT
30
18/20
40
20/20
60
19/20
20
20/20
100
16/20
100
17/20
91.76%
Table 1: Training time and success rate under training setting. Each setting is tested 20 times. We implement HIL-SERL and HIL-ConRFT following the experimental settings described in their original papers; therefore, they did not use the same VLM-based pre-contact localization and motion-planning initialization as S2C. To ensure consistency and comparison of the success rate and training time, all methods are compared within the same workspace and start from the same initial region in each experiment. We also use the same evaluation protocol and success criterion for all methods, such that a trial is counted as successful only when the object is fully inserted into the target slot. For training time, we report the learner-side actor–critic optimization time because real-robot rollout collection and actor–critic optimization run as separate processes.
Method
Charger
USB-Easy
USB-Hard
Tube-Easy
Tube-Medium
Tube-Hard
Avg. SR
Patterned tablecloth + Position
HIL-SERL
0/20
0/20
0/20
0/20
0/20
0/20
0.00%
HIL-ConRFT
0/20
0/20
0/20
0/20
0/20
0/20
0.00%
S2C
18/20
15/20
11/20
13/20
14/20
12/20
69.17%
Strong sun light + Position
HIL-SERL
0/20
0/20
0/20
0/20
0/20
0/20
0.00%
Table 2: Success rate under visual and positional robustness. Only HIL-based RWRL methods and S2C are evaluated. Each setting is evaluated 20 times.
Method
Charger
USB (Hard)
Avg. SR
Strong sun light + Red tablecloth + Position
w/o HIL
0/20
0/20
0.00%
w/o VLM
10/20
12/20
55.00%
w/o OC-Aug
0/20
0/20
0.00%
w/o Contact
0/20
0/20
0.00%
S2C
16/20
14/20
75.00%
Table 3: Fine-grained ablation of the semantic components. The ablation is conducted on Charger Insertion and USB Insertion (hard). “w/o VLM” removes VLM-guided motion planning, whereas “w/o OC-Aug” removes object-centric augmentation. Each setting is evaluated 20 times.
Perception/VLM Backbone
Inference Time (s) ↓
Success Rate ↑
YOLOE
0.07
0/10 (0%)
SAM 3
0.12
3/10 (30%)
Qwen3-VL-8B
2.00
5/10 (50%)
GPT-5.5 API
15.00
10/10 (100%)
MolmoPoint-8B
2.00
10/10 (100%)
Table 4: Comparison of perception and VLM backbones on the charger insertion task. Inference time denotes the time required to localize the target in each episode. Success rate is evaluated over 10 real-robot trials.
Task
A
B
C
D
E
F
G
H
I
Charger
10/10
9/10
0/10
10/10
8/10
10/10
10/10
10/10
0/10
USB-Easy
10/10
8/10
3/10
10/10
8/10
10/10
10/10
10/10
0/10
Table 5: Scalability evaluation of S2C across different tasks. Each task is tested 10 times.
Task
Training Time (min)
Success Rate (%)
Test-tube Removal
25
90 (9/10)
Hard-drive Removal
35
100 (10/10)
Table 6: Performance of S2C on additional contact-rich removal tasks
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist + 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
30
Appendix
Table 7: Policy training details for the Tube Insertion (easy) task.
Figure 4: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist+ 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
40
Appendix
Table 8: Policy training details for the Tube Insertion (medium) task.
Figure 5: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist + 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
38
Appendix
Table 9: Policy training details for the Charger grasp-insertion.
Figure 6: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist + 1D gripper
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
40
Appendix
Table 10: Policy training details for the Tube Insertion (hard) task.
Figure 7: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
35
Appendix
Table 11: Policy training details for the USB Insertion (easy) task.
Figure 8: Sample input images from cameras used as inputs to the policy.
Parameter
Value
base policy
ACT
Observation space
wrist_1, wrist_2, tcp_pose, tcp_vel, tcp_f/t
Action space
6D twist
Classifier views
wrist_1, wrist_2
Classifier accuracy
99%
Initial offline demonstrations
35
Appendix
Table 12: Policy training details for the USB Insertion (hard) task.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany