cs.RONov 11, 2026

Embedded Sim-to-Real Visual Servoing: Learned Reaching and Force-Guided Placement

Authors: Dezhi Sun, Duy Khang Banh, Kenan Niu

Organizations: Robotics and Mechatronics, Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands.

Abstract

Embedded perception-to-control systems must combine visual targeting with reliable contact handling under tight computational constraints. We present a shared architecture on an NVIDIA Jetson AGX Orin controlling a Franka Research 3 through Cartesian impedance control. RealSense D435i observations, YOLO11-seg segmentation, depth back-projection, and hand--eye calibration provide metric robot-frame targets. We compare two strategies: compact proximal policy optimization (PPO) policies transferred from simulation for object- and hole-region reaching, and force-guided Cartesian placement using approximately 2 N normal contact and spiral search. The PPO policies achieved 87.2% and 83.1% success over 1000 simulation episodes each. Across 50 real-robot trials per strategy, the learned hybrid and force-guided analytical pipelines achieved 70.0% and 74.0% completion, respectively. Mean perception latency with TensorRT was 34.55 ms. These results support an embedded architecture combining learned free-space reaching with explicit contact correction.

Figures & tables

Explore similar work

CardsList
  1. InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality

    Sep 21, 2026Zhenghua Ma, Xinpan Meng, Zeyu Liu +4RL for RoboticsContact-Rich Robotic Manipulation

  2. Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation

    Jul 6, 2026Zhe Zhao, Zhibin Li, Yilin Ou +1Robotic GraspingDexterous Manipulation