cs.LGOct 23, 2026
SaveRegularised RL improves sim-to-real transfer
Organizations: Université de Montréal · Mila - Québec AI Institute · McGill University · Polytechnique Montréal ·
Abstract
Regularised reinforcement learning (RL) has provable robustness guarantees that should ease the transfer of policies from simulation to robots, known as sim-to-real. Yet, weakly regularised algorithms like Proximal Policy Optimisation (PPO) remain the de facto choice in sim-to-real pipelines, while the practical benefits of regularisation for transfer remain poorly characterised. In this work, we study the sim-to-real performance of commonly used RL algorithms, with the aim of characterising the effects of regularisation on robotic performance. To that end, we derive Soft MellowMax Actor-Critic (SMM-AC), a novel regularised RL algorithm based on the soft mellowmax value function, which carries strong robustness guarantees. On MuJoCo continuous control benchmarks, SMM-AC is competitive with baselines like Soft Actor-Critic (SAC). On real robots, SMM-AC achieves higher return, while exhibiting greater robustness to perturbations in observations, the environment, and dynamics than baselines. Together, these results demonstrate that regularised RL improves real-world policy deployment through better robustness to the sim-to-real gap, and changing real-world conditions.
Figures & tables
Figure 1: Overview of the Sim-to-Real Robustness Evaluation. Top: Agents trained on the Franka Lift task in simulation are deployed with a cube with a different center-of-mass than the one seen during training. The first frame shows the initial state, while the second shows the outcome, where the policy either successfully lifts the cube or fails to grasp it. Bottom: Agents trained on the ANYmal-D Rough task are deployed in the real world to climb up stairs. With noise added to the observation, non-regularised methods fail to climb the stairs.
| Name | Regulariser | Value | Policy |
| Negative Entropy | |||
| Soft Mellowmax |
Table 1: Optimality equations for various regularised RL formulations, where is an SMM reference policy, and , for inverse temperatures and .
Figure 2: Smoothed episodic returns of SMM-AC, SAC, PPO, and TD3 over the number of steps taken in the environment. The best results were selected according to the performance of the algorithm at the final step.
Figure 3: The four real-world evaluation settings: Franka Reach and Lift, and ANYmal-D velocity tracking on flat and rough terrain. For the ANYmal-D tasks, blue arrows indicate the robot’s direction of commanded velocity.
| SAC | SMM-AC (ours) | |||||||
| Perturbation | PPO | FastTD3 | ||||||
| Franka Reach | ||||||||
| none | 0.69 0.04 | 0.40 0.09 | 0.48 0.12 | 0.58 0.09 | 0.53 0.09 | 0.51 0.08 | 0.61 0.03 | 0.45 0.05 |
| obs. noise ( ) | 0.19 0.05 | 0.24 0.03 | 0.35 0.03 | 0.40 0.03 | 0.35 0.02 | 0.45 0.02 | 0.56 0.02 | 0.33 0.01 |
| Franka Lift | ||||||||
| none | 133.1 54.8 | 123.1 4.1 | 92.9 83.8 | 125.8 69.9 | 94.9 86.0 | 148.5 8.6 | 152.9 2.5 | 123.9 69.4 |
Table 2: Return on hardware with and without perturbations. Entries are the mean and one standard deviation of the return across seeds. Each column is one trained configuration, and reading down a column within a task shows how its return changes under each perturbation. The none row is the deployment without perturbation. PPO and FastTD3 are shown at their best configuration per task. Highlighted entries indicate the highest return in each row. Entries marked with “—” were not deployed, because the configuration failed the simulation safety check ( Subsec. 5.1 ). Task errors for the none rows are reported in Table 13 .
Figure 4: Return kept under real-world deployment-time perturbations. Return under each perturbation as a percentage of the same configuration’s return without it, so means no loss. Bars are seed means, and error bars show one standard deviation. Franka is perturbed with Gaussian observation noise and, on Lift, with a cube of 2 the training mass and with a mm CoM offset. ANYmal is perturbed with lower surface friction, pushes, a payload, and proprioceptive noise. PPO and FastTD3 do not appear on ANYmal because they fail the simulation safety check, which is applied identically to every configuration.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Algorithm | Hyperparameter search space |
| PPO | |
| SAC | |
| TD3 |
Table 3: Hyperparameter search spaces for MuJoCo experiments. ∗ For Pusher-v5 and Swimmer-v4, we additionally evaluated to increase greediness, which was crucial for matching SAC performance. Since such high inverse temperatures were not part of the general sweep, the total number of experiments per method remains comparable.
| Environment | PPO LR / Ent. | SAC Pol. LR / Freq. | SMM-AC Ref. LR / | TD3 LR / Noise |
| Ant-v4 | / 0.0 | / 2.0 | / 15.0 | / 0.3 |
| HalfCheetah-v4 | / 0.01 | / 2.0 | / 15.0 | / 0.3 |
| Hopper-v4 | / 0.0 | / 2.0 | / 15.0 | / 0.2 |
| Humanoid-v4 | / 0.0 | / 2.0 | / 10.0 | / 0.1 |
| Inverted Double Pendulum | / 0.0 | / 6.0 | / 15.0 | / 0.1 |
| Pusher-v5 | / 0.0 | / 6.0 | / 400.0 | / 0.2 |
Table 4: Best hyperparameters for each environment, selected based on final performance. Each entry reports the learning rate and the corresponding algorithm-specific hyperparameter.
Figure 5: Average returns in MuJoCo tasks against environment steps. Each algorithm is run five times, for three million steps, with the shaded areas representing the standard deviation, and the solid line being the mean of the average return curves.
| Environment | FlashSAC | FlashSMM-AC |
| Ant-v4 | 400 | 400 |
| HalfCheetah-v4 | 400 | 400 |
| Hopper-v4 | 400 | 400 |
| Humanoid-v4 | 200 | 400 |
| Walker2d-v4 | 400 | 400 |
Table 5: Best-performing inverse temperature hyperparameters for FlashSAC and FlashSMM-AC.
| Index | (m) | (m) | (m) | yaw (rad) | Joint pos. scale |
| DR 1 (Default) | |||||
| DR 2 (Limited) | |||||
| DR 3 (Severe) |
Table 6: Command (target position of the arm, first 4 columns) and reset-randomization (where the arm can start, last column) ranges swept for Isaac-Franka-Reach-v0. Each index corresponds to one of the three experiments.
| Agent | DR 1 (Default) | DR 2 (Limited) | DR 3 (Severe) | |
| SAC | 400 | 0.52 0.26 | -0.05 0.53 | -1.18 0.63 |
| 800 | 0.52 0.25 | -0.18 0.53 | -1.31 0.61 | |
| 1200 | 0.56 0.23 | 0.04 0.42 | -1.46 0.74 | |
| SMM-AC | 400 | 0.52 0.23 | 0.17 0.23 | -1.64 0.71 |
| 800 | 0.52 0.25 | 0.03 0.36 | -1.49 0.73 | |
| 1200 | 0.53 0.23 | 0.08 0.24 | -1.57 0.75 |
Table 7: Reach: mean standard deviation return over full episodes (pooled over 5 to 10 seeds and 32 envs per cell), where all models are evaluated on the fully randomised environments, DR 1.
| Index | target (m) | target (m) | target (m) | object (m) | object (m) |
| DR 1 (Default) | |||||
| DR 2 (Limited) | |||||
| DR 3 (Severe) |
Table 8: Command (target position of the arm, first 3 columns) and reset-randomization (where the object can start, last two columns) ranges swept for Isaac-Lift-Cube-Franka-v0. Each index corresponds to one of the three experiments.
| Agent | DR 1 | DR 2 | DR 3 | |
| SAC | 400 | 139.17 37.56 | 59.23 73.42 | 62.20 65.76 |
| 800 | 94.20 76.37 | 90.80 74.79 | 64.21 67.76 | |
| 1200 | 157.67 2.57 | 127.06 50.64 | 47.21 63.69 | |
| SMM-AC | 400 | 115.00 65.88 | 111.52 64.29 | 52.96 63.27 |
| 800 | 152.56 3.44 | 108.16 63.19 | 73.16 62.76 | |
| 1200 | 153.05 4.88 | 107.34 66.51 | 39.61 58.51 |
Table 9: Lift-Cube: mean standard deviation return over full episodes (pooled over 5 to 10 seeds and 32 envs per cell), where all models are evaluated on the fully randomised environment, DR 1
| Quantity | DR 1 (Default) | DR 2 (Limited) | DR 3 (Severe) |
| Observation noise (additive uniform) | |||
| Base linear velocity | 0.10 m/s | 0.05 m/s | 0.025 m/s |
| Base angular velocity | 0.20 rad/s | 0.10 rad/s | 0.05 rad/s |
| Projected gravity | 0.05 | 0.025 | 0.0125 |
| Joint position | 0.01 rad | 0.005 rad | 0.0025 rad |
| Joint velocity | 1.5 rad/s | 0.75 rad/s | 0.375 rad/s |
Table 10: Anymal-D domain-randomization (DR) limits by level. Observation noise ranges are additive uniform noise on the policy input; reset ranges control the initial state distribution at episode start.
| Agent | DR 1 | DR 2 | DR 3 | |
| SAC | 400 | 24.75 1.67 | 19.87 8.58 | 9.48 13.51 |
| 800 | 25.35 2.09 | 19.54 10.17 | 4.93 16.92 | |
| 1200 | 25.29 2.82 | 17.48 12.99 | -0.17 22.83 | |
| SMM-AC | 400 | 24.60 2.31 | 20.90 7.93 | 9.53 12.07 |
| 800 | 25.64 1.25 | 19.02 9.96 | 8.98 12.15 | |
| 1200 | 25.83 0.94 | 18.30 11.06 | 3.07 19.64 |
Table 11: AnymalD-Flat: mean standard deviation return over full episodes (pooled over 5 to 10 seeds and 32 envs per cell), where all models are evaluated on the fully randomised environments, DR 1.
| Agent | DR 1 | DR 2 | DR 3 | |
| SAC | 400 | 18.04 6.75 | 20.85 5.80 | 18.59 7.39 |
| 800 | 21.13 4.56 | 19.96 6.26 | 18.48 7.42 | |
| 1200 | 19.61 4.83 | 20.54 5.55 | 17.76 8.88 | |
| SMM-AC | 400 | 17.91 7.21 | 20.18 6.51 | 18.19 7.08 |
| 800 | 20.57 4.87 | 20.21 5.75 | 17.71 8.32 | |
| 1200 | 19.96 3.88 | 20.71 4.91 | 17.45 8.39 |
Table 12: AnymalD-Rough: mean standard deviation return over full episodes (pooled over 5 to 10 seeds and 32 envs per cell), where all models are evaluated on the fully randomised environments, DR 1.
| SAC | SMM-AC (ours) | |||||||
| Task | PPO | FastTD3 | ||||||
| Franka Reach | ||||||||
| Error (cm) | 0.64 0.11 | 0.93 0.14 | 0.93 0.16 | 0.79 0.12 | 0.91 0.15 | 1.00 0.15 | 0.76 0.10 | 1.07 0.17 |
| Return | 0.69 0.04 | 0.40 0.09 | 0.48 0.12 | 0.58 0.09 | 0.53 0.09 | 0.51 0.08 | 0.61 0.03 | 0.45 0.05 |
| Franka Lift | ||||||||
| Error (cm) | 5.79 3.6 | 4.26 0.6 | 16.53 9.4 | 8.54 7.1 | 17.32 9.7 | 1.53 0.6 | 0.82 0.2 | 8.57 6.6 |
Table 13: Hardware deployment without perturbation: task error and return, mean and one standard deviation over seeds. Each column is one trained configuration. On Franka, the error is the distance between the end effector (Reach) or the cube (Lift) and the target at the end of the episode. On ANYmal, it is the RMSE between commanded and measured base velocity. PPO and FastTD3 are shown at their best configuration per task. Highlighted entries indicate the best mean within each task metric.
| SAC | SMM-AC (ours) | |||||||
| DR level | PPO | FastTD3 | ||||||
| Franka Reach | ||||||||
| DR 1 (Default) | 0.69 0.04 | 0.40 0.09 | 0.48 0.12 | 0.58 0.09 | 0.53 0.09 | 0.51 0.08 | 0.61 0.03 | 0.45 0.05 |
| DR 2 (Limited) | 0.24 0.08 | -0.16 0.11 | -0.01 0.14 | -0.11 0.13 | 0.08 0.11 | 0.21 0.07 | 0.12 0.09 | 0.16 0.06 |
| DR 3 (Severe) | -1.18 0.13 | -1.42 0.18 | -1.03 0.16 | -1.15 0.15 | -1.27 0.19 | -1.38 0.14 | -1.29 0.16 | -1.34 0.17 |
| Franka Lift | ||||||||
Table 14: Hardware return of policies trained with limited DR, deployed without perturbation. Each column is one training configuration, and reading down a column within a task shows how its hardware return changes with the training-time DR level of App. E . The DR 1 row corresponds to the none rows of Table 2 . Entries are the mean and one standard deviation of the return across seeds. Highlighted entries indicate the highest return in each column and task. Entries marked with “—” were not deployed, because the configuration failed the simulation safety check ( Subsec. 5.1 ).