We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100 real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yield stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort.
Figures & tables
Figure 1: Robotic platforms studied in this work. We conduct our experiments on three robotic platforms spanning robotic manipulation, locomotion, and navigation tasks. Manipulation. We use a Franka Emika Panda robot to locate, grasp and lift a cube to a goal position. The policy determines the end-effector’s position and the gripper’s opening given grayscale image observations. Locomotion. We use a Unitree Go1 robot to follow joystick commands. The policy maps randomly sampled linear and angular velocity commands (expressed in the robot’s local coordinate frame) to joint position targets. Navigation. Finally, we use a remote-controlled race car that must park at a specified goal position as quickly as possible. This task is particularly challenging due to the system’s high agility, fast control loop (60 Hz) and the difficulty of accurately modeling tire friction and drifting behavior.
Figure 2: Off-policy algorithms may lose stability due to approximation errors in action-value functions, leading to unlearning of the prior policy π0 during online learning.
Figure 3: Downward spiral on a simulated Race Car robot under a mild dynamics mismatch. Left: Performance during online learning with vanilla Soft Actor-Critic (“Unstable”) and our approach (“Stable”). Right: Episode-wise histograms of the critic error ϵ(st,at)≈Qϕπn(st,at)−QMCπn(st,at) on newly collected data, with log counts represented by intensity. Vanilla SAC develops widespread positive errors as performance deteriorates, whereas our approach keeps most errors close to zero.
Figure 4: Normalized performance w.r.t. the best experiment for each task across all experiments. In all tasks the prior policy suffers a performance decrease due to imperfect simulation dynamics.
Figure 5: Comparison of learning performance and runtime for different configurations. Top: Unitree Go1 experiments. Bottom: Franka Emika Panda experiments. Increasing UTD requires fewer environment steps, but at the price of longer training time.
Figure 6: Performance and robustness increase as we accumulate more transitions over trials. Reusing training data across experiments accelerates online learning in all robots.
Figure 7: The blue point marks the episode N∗ until which we do not update π0 under the “warmstart” baseline. Prefilling Donline using π0 improves learning stability in the Unitree Go1 and Race Car, while it is not required to obtain strong performance on the Franka Emika Panda.
Figure 8: We warmstart both runs with the same N∗ , indicated by the vertical dotted line. Asymmetric updates are crucial for effective transfer across all robots, even when Donline is initialized via a warmstart.
Figure 9: After roughly ten minutes of training, including additional hardware overhead (e.g., resetting the robot and data transmission over the internet), SAC recovers a policy with an almost-perfect success rate for the Franka Emika Panda robot.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Performance on the real robot after training with 8192 and 128 domain-randomized environments. A sufficient number of domain-randomized environments is key to effective transfer.
Figure 11: Normalized performance of PPO and SAC w.r.t. the best performance achieved after online learning.
Figure 12: Learning curves for Soft Actor-Critic under mismatch in the dynamics. In the Franka Emika Panda robot, we replace the cube with a soft red ball. For the Race Car, we first pretrain on a semi-kinematic bicycle model and finetune on more realistic dynamics that account for tire friction [ 18 , see] . In the Unitree G1 [ 53 , humanoid, see] and Go1 robots, we reduce the ground friction. We ablate M∈{20,10,5,1} , showing significant stability improvements as we increase M and reduce the learning rate from 3×10−4→1×10−5 across all robots. In Appendix B we show similar results using TD3 [ 9 ] .
Figure 13: Learning curves for transfer (in simulation) of TD3. Similarly to the results in Figure 12 , for small values of delay, mismatch in the dynamics can result in significant performance drops.
Figure 14: Dashed lines represent the value of α during learning. Robust transfer under different values of the initial mixing parameter α .
Figure 15: Online performance when retaining data used for training in simulation. Simulation data stabilizes learning even without priming the online buffer with π0 .
Figure 16: Domain-randomized environments in the simulated Franka Emika Panda robot.
Figure 17: Robustness to commands that were not seen during training in simulation improves over training episodes.
Figure 18: The prior policy π0 overshoots the goal, obtaining low rewards due to the sparse reward in this task. After 20 trials, the finetuned policy πN reaches the goal faster and with greater precision.
Reinforcement learning (RL) research has demonstrated success in both physical and simulated domains; however, the predominant methodology remains rooted in simulations. The predominance of simulations makes translating research to physical reality uncertain for both algorithms and researchers. We propose a physical platform that is designed to simplify the transition. In this paper, we present the Open Ant: a physical variant of the commonly used Gymnasium Ant environment, along with a simulation. We demonstrate that competent walking policies can be learned from scratch in approximately one hour directly from the physical robot's experience for two substantially different RL algorithms: SARSA(λ) and Soft Actor-Critic (SAC). Separately, we show policies that were learned in simulation transfer to reality. We also examine how well the platform supports a nimble experimental ecosystem. Specifically, we observe the speed with which new users from diverse backgrounds achieve their first success with the platform, and how easily the platform can be repaired and updated when hardware issues arise. Both the hardware design and software are available as open-source on GitHub for ease of customization. In summary, we advocate for the use of the Open Ant for RL researchers who frequently use simulated environments, so they can more easily include robot experiments in their evaluations.
Elena Sorina Lupu, Patrick Spieler, Khurram Javed +4
Openmind Research Institute, Canada · Keen Technologies · Department of Computing Science, University of Alberta, Canada
We built a robot called the Robotroller that actuates an Atari CX40+ controller and a device called the Atari Devbox that renders the game frame and the reward signal from the Arcade Learning Environment on a screen. The Robotroller and the Atari Devbox, together with an off-the-shelf camera and a desktop computer, constitute a system that can be used to study reinforcement learning algorithms in the physical world. We call the full system Physical Atari. In this paper, we detail the key decisions that make Physical Atari a robust and accessible platform. To make the system robust, we designed the Robotroller so that all movement is done through bearings, which reduces wear. Additionally, we wrote software that monitors the state of the servos at a high frequency and intervenes to limit stress. To make the system accessible, we used affordable off-the-shelf components and parts that can be manufactured using consumer 3D printers. Physical Atari can be built for under $1,000 and has been used for weeks of non-stop reinforcement learning experiments without any mechanical failures. We used it to validate that reinforcement learning algorithms can learn directly on robots and show that even small distribution shifts between learning and deployment can significantly degrade the performance of policies. Our results underscore the importance of on-device adaptation for strong performance on robots.
Khurram Javed, Joseph Modayil, Gloria Kennickell +2
Keen Technologies · Openmind Research Institute · University of Alberta, Canada
Robots trained on real world data tend to be imprecise, slow, and brittle to perturbations. Improving these policies with reinforcement learning (RL) is an appealing alternative, but this process often requires expensive training in the real world. Performing policy improvement in simulation instead provides a far cheaper alternative, but unconstrained RL in simulation can exploit contact and dynamics mismatches, resulting in unsafe behaviors that do not transfer to hardware. Common forms of regularization can furthermore limit improvement by overconstraining to an imperfect behavior prior. In this work, we propose Support-Constrained Off-Domain REinforcement (SCORE), a real-to-sim-to-real framework that constrains RL in simulation to the support of a generative policy pretrained on real data. We instantiate this constraint through flow steering, restricting SCORE to actions the base policy can already produce, which ensures transferable behaviors while maximizing policy improvement. Improving a policy with SCORE requires minimal effort: it learns from sparse rewards, avoids distillation, and leaves the base policy untouched. Across eight real-world dexterous multi-fingered robotic manipulation tasks, SCORE improves average success rate from 37.8% to 89.9%, compared to 59.5% for the best baseline, and reaches success in 36.8% fewer steps than the base policy. Ultimately, through extensive experiments and ablations, we show that simulation can substantially improve real-world manipulation policies when policy optimization is appropriately constrained, introducing a new paradigm for real-to-sim-to-real policy improvement. Videos and code are available at https://weirdlabuw.github.io/score/.