Residual Reinforcement Learning for Visual Servo Control with Zero-Shot Sim-to-Real Transfer
Authors: Mahdi Nobar, Alisa Rupenyan, John Lygeros, Jürg Keller
Organizations: Automatic Control Laboratory, ETH Zürich, Zurich, Switzerland · ZHAW Center for AI, Zurich University of Applied Sciences, Winterthur, Switzerland · Automation Institute, FHNW, Windisch, Switzerland
We address tracking a moving object on a conveyor with a Franka Emika Panda robot arm, despite an imprecise digital twin, measurement errors, and noise. Our novel approach augments a loop-shaped inverse-Jacobian PI controller with a residual policy learned via Soft Actor-Critic in the digital twin. A delay-compensated Kalman filter fuses out-of-sequence camera measurements to provide latency-aware moving target position estimates. We compare the achieved performance to a performance bound for a class of model-based controllers derived from a frequency-domain sensitivity analysis, quantifying the minimum achievable error imposed by uncertain kinematic model mismatch. Simulation and hardware experiments indicate that our residual policy drives tracking error below the bound. Across conveyor speeds (27.9-41.8 mm/s), our method reduces mean tracking errors from 1.1-1.4 mm (baseline inverse-Jacobian PI) to 0.3-0.7 mm. These results show that learning residual corrections in a limited-fidelity simulator, combined with delay compensation, can substantially improve model-based control under modeling and measurement uncertainties while maintaining reliable sim-to-real transfer.
Figures & tables
Fig. 1 : Components of the closed-loop robot conveyor tracking system and its digital twin
Fig. 2 : Residual learning control framework combining SAC and model-based controller for zero-shot transfer on real robot
Fig. 3 : Closed-loop system with an inverse–Jacobian PI controller.
Fig. 4 : Delay-compensated Kalman filter: each delayed camera measurement p∗(k−d) is inserted at its true acquisition index k−d , corrected once, and then propagated forward to the current control time k to yield a prediction p~∗(k) .
Parameter
Value
Hidden layers (actor/each critic)
2×256 , ReLU
Adam rates (actor/critics; α )
10−3 ; 10−4
Discount γ
0.99
Initial α ; target entropy
0.1 ; −6
Minibatch; replay capacity
136 ; 1.36×106
Warm-up / learning starts
13600 interactions
TABLE I : SAC training configuration.
Fig. 5 : Learning curve of the SAC controller. The maximum return is 110 . The dashed line shows the inverse–Jacobian PI controller performance.
Fig. 6 : Monte Carlo study tracking error mean and 95% confidence interval in the simulation.
Fig. 7 : Simulated error PSD deviation from the optimal PI-class lower bound for the inverse–Jacobian PI class of controllers
Fig. 8 : Tracking error comparison on a real robot system between inverse–Jacobian PI controllers and our residual SAC–PI controller. Shaded area shows the 95% confidence interval.
Fig. 9 : Measured error PSD deviation from the optimal PI-class lower bound in ( 23 ) for the inverse–Jacobian PI controller and our residual SAC–PI controller
p˙⋆
ε
η
eRMS
Controller
mm/s
mm
mm
mm
inverse–Jacobian PI
27.9
1.1
0.1
1.2
residual SAC–PI
27.9
0.7
0.1
0.7
inverse–Jacobian PI
34.9
1.2
0.1
1.2
residual SAC–PI
34.9
0.3
0.1
0.4
inverse–Jacobian PI
41.9
1.4
0.1
1.5
TABLE II : Tracking error metrics of different controllers given different conveyor belt speeds p˙⋆
Robot trajectory tracking develops methods that make robots follow desired paths or motions accurately despite disturbances, uncertainty, and physical constraints.
Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.
Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim +2
School of Mechanical Engineering, Hanyang University, Seoul 04763, Republic of Korea · Department of Mechanical System Design, Kyonggi University, Suwon 16227, South Korea · School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea
Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge, particularly when dealing with low-level torque control, where even small modelling inaccuracies can lead to unstable or unsafe behaviours. In this work, we propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms by combining trajectory matching, parameter optimization via genetic algorithms, and domain randomization. Using the 7-DOF Franka Emika Panda robot, we first identify friction, inertia, and gravity compensation parameters by minimizing the error between real and simulated joint trajectories. These calibrated dynamics are then used to train a TQC-based reinforcement learning agent in simulation. The trained policy is evaluated in both Gazebo and MuJoCo environments, and finally deployed on the real robot. Our results demonstrate a significant improvement in tracking accuracy and policy robustness after parameter tuning, with smooth policy transfer from simulation to the Real-World across multiple target-reaching tasks. This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.
Davide Bargellini, Alex Pasquali, Andrea Govoni +2
Department of Electrical, Electronic, and Information Engineering (DEI), University of Bologna, 40136 Bologna, Italy · Robotics and Mechatronics (RaM) group, University of Twente, Enschede, The Netherlands
Sim-to-real transfer in robot learning is often limited by discrepancies between the ideal actuator dynamics assumed during policy training and the nonlinear, hardware-dependent behavior of physical motors. While conventional approaches attempt to bridge this gap by increasing simulator fidelity through system identification, domain randomization, or learned actuator models, we introduce an alternative paradigm: actuator reality shaping. Instead of modifying the simulator to match the real world, our method shapes the closed-loop behavior of physical actuators to match the idealized second-order reference dynamics used in simulation. By equipping each joint with a two-degree-of-freedom feedforward--feedback controller, we decouple reference-response shaping from robust stabilization, thereby providing a standardized actuator interface for reinforcement learning policies. As a result, policies trained only with the prescribed reference model can be deployed zero-shot on real hardware without task-level fine-tuning or learned actuator models. We validate the approach on a single-joint high-gear-ratio servo under external loads and a 7-DOF robotic arm reaching task, where actuator reality shaping substantially reduces sim-to-real tracking error and improves zero-shot task performance compared with standard servo-control and representative real-to-sim-to-real baselines. We further demonstrate zero-shot transfer on a wheeled-legged robot driving over a slope and a humanoid robot walking, suggesting that actuator reality shaping can serve as a reusable interface for robot learning across diverse hardware platforms. Project page: https://syamamori.github.io/ActuatorRealityShaping.github.io/
Satoshi Yamamori, Koji Ishihara, Kenjiro Minamikawa +4
Graduate School of Informatics, Kyoto University, Kyoto, Japan · Dept. of Brain Robot Interface, Computational Neuroscience Labs, ATR, Kyoto, Japan