Embedded Sim-to-Real Visual Servoing: Learned Reaching and Force-Guided Placement
Organizations: Robotics and Mechatronics, Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands.
Abstract
Embedded perception-to-control systems must combine visual targeting with reliable contact handling under tight computational constraints. We present a shared architecture on an NVIDIA Jetson AGX Orin controlling a Franka Research 3 through Cartesian impedance control. RealSense D435i observations, YOLO11-seg segmentation, depth back-projection, and hand--eye calibration provide metric robot-frame targets. We compare two strategies: compact proximal policy optimization (PPO) policies transferred from simulation for object- and hole-region reaching, and force-guided Cartesian placement using approximately 2 N normal contact and spiral search. The PPO policies achieved 87.2% and 83.1% success over 1000 simulation episodes each. Across 50 real-robot trials per strategy, the learned hybrid and force-guided analytical pipelines achieved 70.0% and 74.0% completion, respectively. Mean perception latency with TensorRT was 34.55 ms. These results support an embedded architecture combining learned free-space reaching with explicit contact correction.
Figures & tables
| Item | Configuration |
|---|---|
| Hardware | |
| Robot / control | Franka Research 3 / Cartesian impedance |
| Camera | Intel RealSense D435i |
| Embedded compute | NVIDIA Jetson AGX Orin |
| Perception | |
| Detector / format | YOLO11-seg / TensorRT |
| Class | P | R | mAP50 | mAP50-95 |
|---|---|---|---|---|
| Blue circle | 0.9613 | 0.8500 | 0.9360 | 0.8056 |
| Green triangle | 0.9858 | 0.7500 | 0.9160 | 0.6627 |
| Nijntje/Miffy | 1.0000 | 0.8478 | 0.9855 | 0.8184 |
| Yellow square | 1.0000 | 0.9270 | 0.9736 | 0.8786 |
| Macro | 0.987 | 0.844 | 0.953 | 0.791 |
| Property | Learned hybrid | Force-guided analytical |
|---|---|---|
| Shared input | Class-labelled target in the FR3 base frame | Class-labelled target in the FR3 base frame |
| Motion representation | Incremental joint commands mapped to Cartesian equilibrium targets | Deterministic Cartesian equilibrium targets |
| Learned components | Policy A object reach; Policy B hole-region reach | None |
| Scripted components | Grasp, lift, release, retraction | Approach, grasp, transport, release, retraction |
| Contact feedback | Not included in policy state | Normal force and end-effector height |
| Local correction | No explicit insertion-state correction | Force regulation and bounded spiral search |
Vision-Based Robot Control
Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.
See all 237 papers