Sim-to-real transfer of locomotion policies often leads to performance degradation due to the inevitable sim-to-real gap. Naively fine-tuning these policies directly on hardware is problematic, as it poses risks of mechanical failure and suffers from high sample inefficiency. In this paper, we address the challenge of safely and efficiently fine-tuning reinforcement learning (RL) policies for dynamic locomotion tasks. Specifically, we focus on fine-tuning policies learned in simulation directly on hardware, while explicitly enforcing safety constraints. In doing so, we introduce SLowRL, a framework that combines Low-Rank Adaptation (LoRA) with training-time safety enforcement via a recovery policy. We evaluate our method both in simulation and on a real Unitree Go2 quadruped robot for jump and trot tasks. Experimental results show that our method achieves a 46.5% reduction in fine-tuning time and near-zero safety violations compared to standard proximal policy optimization (PPO) baselines. Notably, we find that a rank-1 adaptation alone is sufficient to recover pre-trained performance in the real world, while maintaining stable and safe real-world fine-tuning. These results demonstrate the practicality of safe, efficient fine-tuning for dynamic real-world robotic applications.
Figures & tables
Fig. 1 : The SLowRL architecture. A frozen main policy and a trainable LoRA adapter operate in parallel to generate the main action. A Safety Filter monitors the robot’s state and selects between the main action and a conservative Recovery Policy action to ensure safe operation during fine-tuning.
Fig. 2 : Detailed schematic of the SLowRL framework. We freeze the pre-trained policy parameters (blue dashed blocks) to retain prior knowledge from IsaacLab. To enable adaptation to the target environment, we inject low-rank trainable adapters (green blocks) in parallel to the frozen weights. The outputs are summed before the ELU activation, ensuring a safe exploration process.
Fig. 3 : The Unitree Go2 robot performs a dynamic jump. Using SLowRL , we fine-tune a low-rank adapter on top of a frozen simulation-trained policy, achieving substantially faster training with near-zero safety violations compared to standard PPO .
Task
Trot
Jump
FFT wo safety
14.25
69.0
FFT w safety
7.5
17.5
SLowRL(Ours)
0.0
2.0
TABLE I : Safety Evaluation: Average failures (falls/crashes) during the training across 4 seeds.
Gait
FFT (Reduction %)
SLowRL (Reduction %)
Trot
38.9%
88.9%
Jump
5.7%
35.7%
TABLE II : Smoothness: Percentage reduction in action rate ∥at−at−1∥ from start to finish.
Fig. 4 : Sim-to-Sim Sample Efficiency during Fine-tuning (Trot Task) Comparative learning curves showing mean reward over wall-clock time for the trotting task. SLowRL (blue) significantly outperforms the FFT and Zero-Shot baselines, achieving a 38% reduction in time-to-convergence while maintaining higher stability.
Fig. 5 : Sim-to-Sim Sample Efficiency during Fine-tuning (Jump Task) Performance comparison for the dynamic jumping task on the Unitree Go2 robot. SLowRL demonstrates superior sample efficiency, reducing convergence time by 55% compared to baselines without incurring the safety violations observed in FFT .
Fig. 6 : Impact of LoRA Rank ( ρ ) on Adaptation An ablation study sweeping rank parameters ρ∈{1,2,4,8} . The results empirically validate that a minimal rank of ρ=1 yields the highest final reward, confirming that effective sim-to-real adaptation can be achieved via a single principal direction of adjustment.
Fig. 7 : Impact of Critic Adaptation A comparison of adaptation performance when applying LoRA to the Actor and Critic versus the Actor Only. The results show that adapting the critic is strictly necessary to realign the value function with real-world dynamics, whereas the Actor Only configuration fails to converge.
Fig. 8 : Evaluation of Adapter Placement Strategy Performance comparison of different LoRA injection depths, including partial layer adaptation. The All Layers configuration achieves the highest performance, demonstrating that deep correction across all levels of abstraction is required to robustly bridge the sim-to-real gap.
Fig. 9 : Sim-to-Real Adaptation Efficiency (Jump Task) A comparison of mean reward over wall-clock time. SLowRL (blue) demonstrates faster convergence and higher asymptotic performance compared to FFT with safety constraints (orange).
Department of Electrical and Computer Engineering, University of Louisville · Faculty of Electrical and Computer Engineering, University of Louisville, Louisville, KY, 40208, USA