Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning

Davide Bargellini; Alex Pasquali; Andrea Govoni; Riccardo Zanella; Gianluca Palli

cs.ROAug 23, 2026

Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning

Authors: Davide Bargellini, Alex Pasquali, Andrea Govoni, Riccardo Zanella, Gianluca Palli

Organizations: Department of Electrical, Electronic, and Information Engineering (DEI), University of Bologna, 40136 Bologna, Italy · Robotics and Mechatronics (RaM) group, University of Twente, Enschede, The Netherlands

Abstract

Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge, particularly when dealing with low-level torque control, where even small modelling inaccuracies can lead to unstable or unsafe behaviours. In this work, we propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms by combining trajectory matching, parameter optimization via genetic algorithms, and domain randomization. Using the 7-DOF Franka Emika Panda robot, we first identify friction, inertia, and gravity compensation parameters by minimizing the error between real and simulated joint trajectories. These calibrated dynamics are then used to train a TQC-based reinforcement learning agent in simulation. The trained policy is evaluated in both Gazebo and MuJoCo environments, and finally deployed on the real robot. Our results demonstrate a significant improvement in tracking accuracy and policy robustness after parameter tuning, with smooth policy transfer from simulation to the Real-World across multiple target-reaching tasks. This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.

Figures & tables

Fig. 1: Diagram illustrating the workflow for validating and executing an RL policy in both simulation and Real-World scenarios. The process includes parameter estimation (e.g., torque/position curves, friction, inertias, and gravity compensation), training via RL (TQC with GridSearch optimization), policy validation, and execution for Real2Sim and Sim2Real applications.

Fig. 2: Episode mean reward comparison across DDPG, SAC, TD3, and TQC algorithms. TQC demonstrates consistently higher rewards, underscoring its superior performance and robustness in our environment.

Panel of Figure 3 — Fig. 3: Comparison of the Sim2Real RL model transfer for the Panda robot. (a) Joint 1 torque comparison without parameter tuning. (b) Joint 2 torque comparison without parameter tuning. (c) Joint 1 torque comparison with parameter tuning. (d) Joint 2 torque comparison with parameter tuning.

	J1	J2	J3	J4	J5	J6	J7
Torque (Nm)	-0.13	-0.73	0.03	0.53	0	-0.06	0

TABLE I: Offset Values for Gravity Compensation and static friction.

Torque	J1	J2	J3	J4	J5	J6	J7
Pos. $\boldsymbol{\tau}$ (Nm)	0.65	0.88	0.51	0.61	0.44	0.40	0.32
Neg. $\boldsymbol{\tau}$ (Nm)	-0.62	-0.81	-0.42	-0.51	-0.67	-0.23	-0.35

TABLE II: Positive and Negative Friction Values for Each Joint.

	Inertia	Link 1	Link 2	Link 3	Link 4	Link 5	Link 6	Link 7
OLD	$I_{xx}$	$0.703\,37$	$0.007\,962$	$0.037\,24$	$0.025\,85$	$0.035\,55$	$0.001\,964$	$0.012\,52$
	$I_{yy}$	$0.706\,61$	$0.028\,11$	$0.036\,16$	$0.019\,55$	$0.029\,47$	$0.004\,354$	$0.010\,03$
	$I_{zz}$	$0.009\,117$	$0.025\,995$	$0.010\,83$	$0.028\,32$	$0.008\,627$	$0.005\,433$	$0.004\,815$
	$I_{xy}$	$-0.000\,139$	$-0.003\,925$	$-0.004\,761$	$0.007\,796$	$-0.002\,117$	$0.000\,109$	$-0.000\,428$
	$I_{xz}$	$0.006\,772$	$0.010\,25$	$-0.011\,40$	$-0.001\,332$	$-0.004\,037$	$-0.001\,158$	$-0.001\,196$
	$I_{yz}$	$0.019\,169$	$0.000\,704$	$-0.012\,81$	$0.008\,641$	$0.000\,229$	$0.000\,341$	$-0.000\,741$

TABLE III: Inertia tensor components before (Old) and after (New) optimization for the 7 links.

	Friction	J1	J2	J3	J4	J5	J6	J7
OLD	$\varphi_{1,j}$	0.55	0.87	0.64	1.28	0.84	0.30	0.56
	$\varphi_{2,j}$	5.12	9.07	10.14	5.59	8.35	17.13	10.34
	$\varphi_{3,j}$	0.04	0.03	-0.05	0.04	0.03	-0.02	0.01
NEW	$\varphi_{1,j}$	3.40	1.98	2.51	2.44	2.60	1.19	1.55
	$\varphi_{2,j}$	9.67	28.67	9.83	18.78	4.19	5.75	1.70
	$\varphi_{3,j}$	0.01	0.01	-0.01	0.01	0.01	-0.01	0.01

TABLE IV: Friction parameters before (Old) and after (New) optimization.

Panel of Figure 8 — Fig. 4: Comparison of the TQC agent’s performance in the MuJoCo and Gazebo simulators. The plots display the rewards obtained across four random goal scenarios, emphasizing the performance variations resulting from the distinct dynamics and properties of each simulation environment.

Fig. 5: Visualization of the Panda robot reaching the goal in the MuJoCo simulator. The image depicts the simulated trajectory and final pose, showcasing task completion in the virtual environment.

Fig. 6: Visualization of the Panda robot reaching the goal position in the Gazebo simulator. The image highlights the robot’s path and final pose, demonstrating successful performance in the simulated setting.

Fig. 7: Visualization of the Panda robot reaching the goal position in the Real-World environment. The image captures the robot’s trajectory and final pose, emphasizing successful task execution in a physical setup.

Panel of Figure 12 — Fig. 8: Comparison between MuJoCo and Real-World for four random goals. MuJoCo shows smoother transitions while Gazebo introduces disturbances mimicking physical inconsistencies, serving as an intermediate test-bed before real deployment.

Explore similar work

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer
Mar 13, 2025Yuxuan Xu, Shiyu Wang, Jinhao Huang +6Robotic Data Collection Sim-to-Real Transfer
Institute for AI Industry Research (AIR), Tsinghua University · Beijing Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou)
Abstract Sim2Real through Approximate Information States
Apr 16, 2026Yunfu Deng, Yuhao Li, Josiah P. HannaSim-to-Real Transfer State Abstraction
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA
Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)
May 30, 2026Kyle Morgenstein, Bharath Masetty, Stephen Welch +1Robot Policy Learning Robotics Simulation
Apptronik · University of Texas at Austin

Mar 13, 2025, cs.ROJ/K move · Enter open · S save

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer

Yuxuan Xu, Shiyu Wang, Jinhao Huang, Wenhao Zhao+5

Institute for AI Industry Research (AIR), Tsinghua University · Beijing Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou)

Robotic Data Collection Sim-to-Real Transfer RL Exploration Robot Policy Generalization

Open paper AskPDF arXiv

The sim-to-real gap remains a critical challenge in robotics, hindering the deployment of algorithms trained in simulation to real-world systems. We propose a flexible Real-to-Sim-to-Real (RSR) framework whose central contribution is an information-theoretic cost function that explicitly accounts for sim-to-real discrepancies. This cost balances two objectives, completing the task and steering the policy to collect real-world samples that are maximally informative for improving transfer. It can be integrated seamlessly into existing reinforcement learning algorithms (e.g., PPO, SAC) and ensures a balanced exploration of critical regions in the real domain. The framework treats differentiable simulation as optional: when a differentiable simulator is available, the collected informative data can also be used to tune simulator parameters. We implement the framework with the MuJoCo MJX platform and demonstrate its generality by evaluating on both manipulation tasks with a 6-DOF robotic arm and locomotion tasks on a legged robot. Empirical results show that our RSR loop yields more efficient data acquisition and substantially improves task performance in real-world that achieves a smoother sim-to-real transfer.

Abstract

Explore similar work

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer

Abstract Sim2Real through Approximate Information States

Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)