Synergistic Action and Style Priors for Learning Legged Locomotion

Oliver Hausdörfer; Alexander von Rohr; Éric Lefort; Angela Schoellig

cs.ROOct 4, 2024

Synergistic Action and Style Priors for Learning Legged Locomotion

Authors: Oliver Hausdörfer, Alexander von Rohr, Éric Lefort, Angela Schoellig

Organizations: Technical University of Munich, Germany; TUM School of Computation, Information and Technology, Department of Computer Engineering, Learning Systems and Robotics Lab; Munich Institute of Robotics and Machine Intelligence.

Abstract

Deep Reinforcement Learning (DRL) enables robots to learn complex behaviors through interaction with the environment. However, due to the unrestricted nature of the learning algorithms, the resulting solutions are often brittle and appear unnatural. This is especially true for learning direct joint-level torque control, as inductive biases are difficult to integrate into the learning process. We propose an inductive bias for learning locomotion that is especially useful for torque control: latent actions learned from a small dataset of expert demonstrations. This prior allows the policy to directly leverage knowledge contained in the expert's actions and facilitates more efficient exploration. We observe that the agent is not restricted to the reward levels of the demonstration, and performance in transfer tasks is improved significantly. Latent action priors combined with style rewards for imitation lead to a closer replication of the expert's behavior. Videos are available at https://sites.google.com/view/latent-action-priors.

Figures & tables

Panel of Figure 1 — Fig. 2: We evaluate the synergies of combining the simple data-driven action prior with a style reward term on (a) - (c) three Gymnasium Mujoco [ 25 ] environments of varying difficulty, and (d) a simulated Unitree A1 quadruped with position and torque control from the loco-mujoco benchmark [ 26 ] . (e) - (f) We then test our approach on a Unitree Go2 for two gait types and terrains using LLM-generated demonstrations, and deploy the policy on hardware for both position and direct $500\text{\,}\mathrm{Hz}$ torque control.

Panel of Figure 2 — Fig. 3: Prompt template we use to generate the gait controllers.

Fig. 4: All motions from the demonstrations can be expressed with only a few Principal components (PCs), motivating the use of a compressed action space that exploits such correlations. PCs left of the vertical red line explain

>

95\text{\,}\mathrm{\%}

of the variance in the actions and the black lines show the cumulative explained variance. Results from data in Tab. II .

Panel of Figure 4 — Fig. 5: Results on benchmark environments. A data-driven action prior significantly outperforms using unbiased PPO and a simple reference position in the action space regarding both sample efficiency during DRL and maximum reward for all environments. This strongly motivates the use of demonstrations for policy learning rather than static joint position references. Adding a single style reward term (+style) creates synergistic effects except for Humanoid, where the demonstration is brittle. We run all experiments on five seeds and report the means and standard deviations on task rewards. BC is omitted as it did not improve upon PPO.

Panel of Figure 5 — Fig. 6: Policy learning for different hyperparameters introduced by the latent action prior: action space weighting $w_{a}$ and the dimension of the latent action space dim( $a_{l}$ ) for the Unitree A1 . Shown are the mean and standard deviation on 5 seeds.

Fig. 7: The combination of a latent action prior with a single style reward term is significantly less sensitive towards reward tuning than baselines with only DeepMimic-style style rewards and reference joint positions. Shown is the ablation on the style reward weights

w_{style}

for Unitree A1 (

w_{\mathrm{task}}=1-w_{\mathrm{style}}

). Shown are the mean and standard deviation on 5 seeds.

Fig. 8: Joint positions for the left hind leg of the Unitree A1 for torque control in simulation. The synergy of both priors leads to joint positions close to the expert. The BC policy tracks the expert demonstration for the first frames but diverges quickly. RMSE [rad]: PPO+latent+style: 0.05, PPO+latent: 0.17, PPO+reference+style: 0.31, PPO+style: 0.52.

Panel of Figure 8 — Fig. 9: LLMs can effectively design open-loop gait controllers for various locomotion scenarios leveraging established methods [ 30 , 31 ] : (a) trot, (b) walk, and (c) stair climbing. Fig. (d) Shows the produced feet contact timing for the trotting and walking gaits when executing the open-loop controllers in simulation and the resulting policies. The feet contact patterns translate to the trained policy indicating imitation.

Fig. 10: While position control can directly be deployed to hardware, torque control on the robot leads to high-frequency vibrations when no penalty for the action rate is added. Shown are the commanded torques sent to the robot’s rear left calf joint.

Env	$\textbf{dim}\bm{(a_{l})}$	$\bm{w_{a}}$	Samples / Update	Policy
HC-v4	0.5 $\dim(a_{full})$	0.1	2048	[64x64]
Ant-v4	0.5 $\dim(a_{full})$	0.1	2048	[64x64]
UnitreeA1/Go2	0.5 $\dim(a_{full})$	0.1	2x2048	[256,256]
Humanoid-v4	0.5 $\dim(a_{full})$	0.5	4x2048	[512x512]

TABLE I: Training parameters.

Env	Frames a	Source
HalfCheetah-v4	10	Open-loop oscillator [ 32 ]
Ant-v4	22	Open-loop oscillator [ 32 ]
Humanoid-v4	212	Off-policy DRL [ 39 ]
Unitree A1	92	loco-mujoco [ 26 ]
Unitree Go2	112–120	Open-loop controller (LLM)

TABLE II: Expert demonstrations.

RL for Legged Locomotion

Reinforcement learning for legged locomotion trains walking robots to move across varied terrain, adapt to disturbances, and carry out tasks.

See all 166 papers

Towards Torque-Driven Reinforcement Learning for Quadruped Locomotion
Jul 20, 2026Jordan Dowdy, Jean Chagas VazReinforcement Learning Terrain-Aware Robot Locomotion
Department of Electrical and Computer Engineering at the University of Louisville. · Faculty of Electrical and Computer Engineering, University of Louisville, Louisville, KY, 40208, USA.
HuMBLE: Human Motion-Driven Behavior Learning for Embodied Locomotion
Oct 7, 2026Mike Zhang, Dongho Kang, Kevin Bergamin +19Human-to-Robot Motion Imitation Humanoid Robot Locomotion
RAI Institute, USA · Boston Dynamics, USA · Carnegie Mellon University, USA
LAMP: Latent Motion Prior-Guided Real-World Learning for Dexterous Hand Manipulation
Jul 7, 2026Xinye Yang, Zhiyuan Ma, Hongze Yu +5Latent Action Learning Robot Skill Learning
Fudan University · PsiBot · Zhongguancun Academy +2

Jul 20, 2026J/K move · Enter open · S save

Towards Torque-Driven Reinforcement Learning for Quadruped Locomotion

Jordan Dowdy, Jean Chagas Vaz

Department of Electrical and Computer Engineering at the University of Louisville. · Faculty of Electrical and Computer Engineering, University of Louisville, Louisville, KY, 40208, USA.

Reinforcement Learning Terrain-Aware Robot Locomotion RL for Legged Locomotion Quadruped Locomotion

Open paper AskPDF arXiv

Reinforcement learning (RL) for legged robots is advancing locomotion, demonstrating its ability to adapt to new and challenging terrain. Traditionally, these RL locomotion frameworks are position-based, making the policy less adaptable to terrain types and requiring state estimation techniques in the observation space, i.e., linear velocity. Moreover, these RL frameworks often use small, lightweight quadrupeds that are limited in their viability for high-complexity tasks due to hardware constraints. This work explores an RL torque control framework for heavyweight high-torque quadrupeds. The RL framework in this paper can traverse rough terrain and effectively track a desired linear velocity without requiring knowledge of the agent's current velocity. Using Nvidia's Isaac Sim and Isaac Lab, simulation results of the RL torque control policy are shown on the Unitree B1 quadruped, achieving speeds of 3.5 m/s and 1.5 rad/s. In addition, the quadruped can walk up and down stairs without the aid of an exteroceptive sensor.

Abstract

RL for Legged Locomotion

Explore similar work

Towards Torque-Driven Reinforcement Learning for Quadruped Locomotion

HuMBLE: Human Motion-Driven Behavior Learning for Embodied Locomotion

LAMP: Latent Motion Prior-Guided Real-World Learning for Dexterous Hand Manipulation