An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer

Yuxuan Xu; Shiyu Wang; Jinhao Huang; Wenhao Zhao; Yufei Jia; Zike Yan; Weibin Gu; Lu Shi

cs.ROMar 13, 2025

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer

Authors: Yuxuan Xu, Shiyu Wang, Jinhao Huang, Wenhao Zhao, Yufei Jia, Zike Yan, Weibin Gu, Lu Shi, +1 more

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Beijing Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

The sim-to-real gap remains a critical challenge in robotics, hindering the deployment of algorithms trained in simulation to real-world systems. We propose a flexible Real-to-Sim-to-Real (RSR) framework whose central contribution is an information-theoretic cost function that explicitly accounts for sim-to-real discrepancies. This cost balances two objectives, completing the task and steering the policy to collect real-world samples that are maximally informative for improving transfer. It can be integrated seamlessly into existing reinforcement learning algorithms (e.g., PPO, SAC) and ensures a balanced exploration of critical regions in the real domain. The framework treats differentiable simulation as optional: when a differentiable simulator is available, the collected informative data can also be used to tune simulator parameters. We implement the framework with the MuJoCo MJX platform and demonstrate its generality by evaluating on both manipulation tasks with a 6-DOF robotic arm and locomotion tasks on a legged robot. Empirical results show that our RSR loop yields more efficient data acquisition and substantially improves task performance in real-world that achieves a smoother sim-to-real transfer.

Figures & tables

Fig. 1: Overview of the proposed RSR (Real–Sim–Real) loop framework. The red outer loop (arrows labeled “R”) is the main Real–Sim–Real cycle: a simulation-trained policy is deployed on the real robot, the collected data are used to update the InfoGap cost (and, optionally, the simulator), and a new policy is then trained in simulation. The blue inner loop is the policy-training loop, which uses the current simulator and the adaptive InfoGap loss. The green inner loop is the optional simulator-parameter-tuning loop (indexed by i). Together, these loops iteratively improve both the policy and, when applicable, the simulator.

\theta_{i,k-1}\leftarrow\theta_{i-1,k-1}-\alpha\nabla_{\theta}\mathcal{L}(\theta_{i-1,k-1})

Algorithm 1 The RSR Loop Framework

Fig. 2: The process of bridging the sim-to-real gap in robot training. When the discrepancy between the simulation (blue domain) and real robot (orange domain) is large, the policy prioritizes collecting informative data (marked as crosses) from the real domain to better characterize its properties other than the task trajectory (black dashed line).

Stage	KL Divergence ( $x$ )	KL Divergence ( $y$ )
PPO+DR	16.3509	36.3168
1st_RSR	1.6903	3.4206
2nd_RSR	5.0462	2.3946
3rd_RSR	0.9739	0.8384

TABLE I: Distribution Deviation Results in the Block-Pushing Task

Fig. 3: Real-world block-pushing trajectories across different iterations.

Fig. 4: 1- $\sigma$ bounds of real trajectories for the yaw angle in the T-shaped block pushing trials for different iterations, where the shaded area represents the bounds.

Stage	KL Divergence
Stage	$v_{x}$ (m/s)	$v_{y}$ (m/s)	$\omega_{z}$ (rad/s)
PPO+DR	48.2941	5.1624	56.7724
1st_RSR	38.3659	3.5171	47.5421
2nd_RSR	9.2554	2.6966	19.5236
3rd_RSR	4.3988	1.9778	3.9715

TABLE II: Distribution Deviation of the Circular-Tracking Task

Fig. 5: Real-world legged trajectories across different iterations.

Stage	$v_{x}$ (m/s)	$v_{y}$ (m/s)	$\omega_{z}$ (rad/s)
PPO+DR	0.194	0.077	0.334
1st_RSR	0.147	0.050	0.295
2nd_RSR	0.067	0.068	0.086
3rd_RSR	0.051	0.045	0.052

TABLE III: Velocity Tracking RMSE of the Quadrupedal Robot

Payload	Stage	$v_{x}$ (m/s)	$v_{y}$ (m/s)	$\omega_{z}$ (rad/s)
None	SAC+DR	44.8173	4.9362	51.2847
	1st RSR	30.9648	4.0837	35.6724
	2nd RSR	17.3826	2.9145	19.4861
	3rd RSR	8.7519	1.7286	8.9375
Centered	SAC+DR	52.4618	6.2743	63.9052
	1st RSR	37.5831	4.8619	44.7168

TABLE IV: Distribution Deviation across RSR Iterations under Different Payload Conditions

Fig. 6: Velocity-tracking RMSE under different payload conditions across successive RSR and SAC+Real Replay iterations. Both methods use the same real-world transition sets at each adaptation round.

Fig. 7: Trajectories of real-robot deployment with and without visual loss.

Explore similar work

Abstract Sim2Real through Approximate Information States
Apr 16, 2026Yunfu Deng, Yuhao Li, Josiah P. HannaSim-To-Real Reinforcement Learning Sim-To-Real Gap
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA
Provably Safe Sim-to-Real Transfer
Sep 1, 2026Tingting Ni, Maryam KamgarpourSim-To-Real Reinforcement Learning Simulation-Based Reinforcement Learning
SYCAMORE, EPFL
Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience
Jun 25, 2026Raymond Yu, William Huey, Mustafa Mukadam +2Sim-To-Real Reinforcement Learning Robotic Manipulation Policies
University of Washington · Amazon FAR

Apr 16, 2026, cs.ROJ/K move · Enter open · S save

Abstract Sim2Real through Approximate Information States

Yunfu Deng, Yuhao Li, Josiah P. Hanna

Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA

Sim-To-Real Reinforcement Learning Sim-To-Real Gap Real-World Tasks

Open paper AskPDF arXiv

In recent years, reinforcement learning (RL) has shown remarkable success in robotics when a fast and accurate simulator is available for a given task. When using RL and simulation, more simulator realism is generally beneficial but becomes harder to obtain as robots are deployed in increasingly complex and widescale domains. In such settings, simulators will likely fail to model all relevant details of a given target task and this observation motivates the study of sim2real with simulators that leave out key task details. In this paper, we formalize and study the abstract sim2real problem: given an abstract simulator that models a target task at a coarse level of abstraction, how can we train a policy with RL in the abstract simulator and successfully transfer it to the real-world? Our first contribution is to formalize this problem using the language of state abstraction from the RL literature. This framing shows that an abstract simulator can be grounded to match the target task if the grounded abstract dynamics take the history of states into account. Based on the formalism, we then introduce a method that uses real-world task data to correct the dynamics of the abstract simulator. We then show that this method enables successful policy transfer both in sim2sim and sim2real evaluation.

Abstract

Explore similar work

Abstract Sim2Real through Approximate Information States

Provably Safe Sim-to-Real Transfer

Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience