Residual Reinforcement Learning for Visual Servo Control with Zero-Shot Sim-to-Real Transfer
Organizations: Automatic Control Laboratory, ETH Zürich, Zurich, Switzerland · ZHAW Center for AI, Zurich University of Applied Sciences, Winterthur, Switzerland · Automation Institute, FHNW, Windisch, Switzerland
Abstract
We address tracking a moving object on a conveyor with a Franka Emika Panda robot arm, despite an imprecise digital twin, measurement errors, and noise. Our novel approach augments a loop-shaped inverse-Jacobian PI controller with a residual policy learned via Soft Actor-Critic in the digital twin. A delay-compensated Kalman filter fuses out-of-sequence camera measurements to provide latency-aware moving target position estimates. We compare the achieved performance to a performance bound for a class of model-based controllers derived from a frequency-domain sensitivity analysis, quantifying the minimum achievable error imposed by uncertain kinematic model mismatch. Simulation and hardware experiments indicate that our residual policy drives tracking error below the bound. Across conveyor speeds (27.9-41.8 mm/s), our method reduces mean tracking errors from 1.1-1.4 mm (baseline inverse-Jacobian PI) to 0.3-0.7 mm. These results show that learning residual corrections in a limited-fidelity simulator, combined with delay compensation, can substantially improve model-based control under modeling and measurement uncertainties while maintaining reliable sim-to-real transfer.
Figures & tables
| Parameter | Value |
|---|---|
| Hidden layers (actor/each critic) | , ReLU |
| Adam rates (actor/critics; ) | ; |
| Discount | |
| Initial ; target entropy | ; |
| Minibatch; replay capacity | ; |
| Warm-up / learning starts | interactions |
| Controller | ||||
|---|---|---|---|---|
| inverse–Jacobian PI | 27.9 | 1.1 | 0.1 | 1.2 |
| residual SAC–PI | 27.9 | 0.7 | 0.1 | 0.7 |
| inverse–Jacobian PI | 34.9 | 1.2 | 0.1 | 1.2 |
| residual SAC–PI | 34.9 | 0.3 | 0.1 | 0.4 |
| inverse–Jacobian PI | 41.9 | 1.4 | 0.1 | 1.5 |
Robot Trajectory Tracking
Robot trajectory tracking develops methods that make robots follow desired paths or motions accurately despite disturbances, uncertainty, and physical constraints.
See all 152 papers