cs.ROFeb 23, 2026
SaveWhat Matters for Simulation to Online Reinforcement Learning on Real Robots
Organizations: ETH Zurich · Google DeepMind
Abstract
We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100 real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yield stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort.
Figures & tables
Figure 1: Robotic platforms studied in this work. We conduct our experiments on three robotic platforms spanning robotic manipulation, locomotion, and navigation tasks. Manipulation. We use a Franka Emika Panda robot to locate, grasp and lift a cube to a goal position. The policy determines the end-effector’s position and the gripper’s opening given grayscale image observations. Locomotion. We use a Unitree Go1 robot to follow joystick commands. The policy maps randomly sampled linear and angular velocity commands (expressed in the robot’s local coordinate frame) to joint position targets. Navigation. Finally, we use a remote-controlled race car that must park at a specified goal position as quickly as possible. This task is particularly challenging due to the system’s high agility, fast control loop (60 Hz) and the difficulty of accurately modeling tire friction and drifting behavior.
Figure 2: Off-policy algorithms may lose stability due to approximation errors in action-value functions, leading to unlearning of the prior policy during online learning.
Figure 3: Downward spiral on a simulated Race Car robot under a mild dynamics mismatch. Left: Performance during online learning with vanilla Soft Actor-Critic (“Unstable”) and our approach (“Stable”). Right: Episode-wise histograms of the critic error on newly collected data, with log counts represented by intensity. Vanilla SAC develops widespread positive errors as performance deteriorates, whereas our approach keeps most errors close to zero.
Figure 4: Normalized performance w.r.t. the best experiment for each task across all experiments. In all tasks the prior policy suffers a performance decrease due to imperfect simulation dynamics.
Figure 5: Comparison of learning performance and runtime for different configurations. Top: Unitree Go1 experiments. Bottom: Franka Emika Panda experiments. Increasing UTD requires fewer environment steps, but at the price of longer training time.
Figure 6: Performance and robustness increase as we accumulate more transitions over trials. Reusing training data across experiments accelerates online learning in all robots.
Figure 7: The blue point marks the episode until which we do not update under the “warmstart” baseline. Prefilling using improves learning stability in the Unitree Go1 and Race Car, while it is not required to obtain strong performance on the Franka Emika Panda.
Figure 8: We warmstart both runs with the same , indicated by the vertical dotted line. Asymmetric updates are crucial for effective transfer across all robots, even when is initialized via a warmstart.
Figure 9: After roughly ten minutes of training, including additional hardware overhead (e.g., resetting the robot and data transmission over the internet), SAC recovers a policy with an almost-perfect success rate for the Franka Emika Panda robot.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Performance on the real robot after training with 8192 and 128 domain-randomized environments. A sufficient number of domain-randomized environments is key to effective transfer.
Figure 11: Normalized performance of PPO and SAC w.r.t. the best performance achieved after online learning.
Figure 12: Learning curves for Soft Actor-Critic under mismatch in the dynamics. In the Franka Emika Panda robot, we replace the cube with a soft red ball. For the Race Car, we first pretrain on a semi-kinematic bicycle model and finetune on more realistic dynamics that account for tire friction [ 18 , see] . In the Unitree G1 [ 53 , humanoid, see] and Go1 robots, we reduce the ground friction. We ablate , showing significant stability improvements as we increase and reduce the learning rate from across all robots. In Appendix B we show similar results using TD3 [ 9 ] .
Figure 13: Learning curves for transfer (in simulation) of TD3. Similarly to the results in Figure 12 , for small values of delay, mismatch in the dynamics can result in significant performance drops.
Figure 14: Dashed lines represent the value of during learning. Robust transfer under different values of the initial mixing parameter .
Figure 15: Online performance when retaining data used for training in simulation. Simulation data stabilizes learning even without priming the online buffer with .
Figure 16: Domain-randomized environments in the simulated Franka Emika Panda robot.
Figure 17: Robustness to commands that were not seen during training in simulation improves over training episodes.
Figure 18: The prior policy overshoots the goal, obtaining low rewards due to the sparse reward in this task. After trials, the finetuned policy reaches the goal faster and with greater precision.
More in this topic
RL for Robotics
Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.
See all 252 papers