RL for Legged Locomotion

RL: Reinforcement Learning

Ask about this topic

Overview

This field studies how quadruped and humanoid robots can learn coordinated walking, running, jumping, and navigation while staying balanced and managing contact with the ground. Common approaches train actor–critic policies such as PPO in simulation, using reward shaping and curricula; others add motion-imitation priors, perception, learned dynamics, or model-predictive control. Researchers test policies in simulators and on hardware across settings such as stairs, gaps, rough or deformable terrain, payloads, and external pushes, often using teacher–student distillation to transfer privileged simulation training to real sensors. A central challenge is achieving reliable, safe sim-to-real transfer and broad terrain or morphology generalization without sacrificing efficiency, natural motion, or energy use.

Latest papers

CardsList

166 papers33 papers in the last four weeks, up 106% on the four weeks before. 0.2% of all new papers.

Mar 4, 2026

Learning Hip Exoskeleton Control Policy via Predictive Neuromusculoskeletal Simulation

Developing exoskeleton controllers that generalize across diverse locomotor conditions typically requires extensive motion-capture data and biomechanical labeling, limiting scalability beyond instrumented laboratory settings. Here, we present a physics-based neuromusculoskeletal learning framework that trains a hip-exoskeleton control policy entirely in simulation, without motion-capture demonstrations, and deploys it on hardware via policy distillation. A reinforcement learning teacher policy is trained using a muscle-synergy action prior over a wide range of walking speeds and slopes through a two-stage curriculum, enabling direct comparison between assisted and no-exoskeleton conditions. In simulation, exoskeleton assistance reduces mean muscle activation by up to 3.4% and mean positive joint power by up to 7.0% on level ground and ramp ascent, with benefits increasing systematically with walking speed. On hardware, the assistance profiles learned in simulation are preserved across matched speed-slope conditions (r: 0.82, RMSE: 0.03 Nm/kg), providing quantitative evidence of sim-to-real transfer without additional hardware tuning. These results demonstrate that physics-based neuromusculoskeletal simulation can serve as a practical and scalable foundation for exoskeleton controller development, substantially reducing experimental burden during the design phase.

Ilseung Park, Changseob Song, Inseung Kang

Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213 USA.

Hip Exoskeleton ControlRobotic ControlSim-to-Real TransferPhysics-Based SimulationRL for Legged Locomotion
AskPDF
Mar 3, 2026

CMoE: Contrastive Mixture of Experts for Motion Control and Terrain Adaptation of Humanoid Robots

For effective deployment in real-world environments, humanoid robots must autonomously navigate a diverse range of complex terrains with abrupt transitions. While the Vanilla mixture of experts (MoE) framework is theoretically capable of modeling diverse terrain features, in practice, the gating network exhibits nearly uniform expert activations across different terrains, weakening the expert specialization and limiting the model's expressive power. To address this limitation, we introduce CMoE, a novel single-stage reinforcement learning framework that integrates contrastive learning to refine expert activation distributions. By imposing contrastive constraints, CMoE maximizes the consistency of expert activations within the same terrain while minimizing their similarity across different terrains, thereby encouraging experts to specialize in distinct terrain types. We validated our approach on the Unitree G1 humanoid robot through a series of challenging experiments. Results demonstrate that CMoE enables the robot to traverse continuous steps up to 20 cm high and gaps up to 80 cm wide, while achieving robust and natural gait across diverse mixed terrains, surpassing the limits of existing methods. To support further research and foster community development, we release our code publicly.

Shihao Ma, Hongjin Chen, Zijun Xu +6

College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China, 200433

Humanoid Robot LocomotionTerrain-Aware Robot LocomotionRobot LocomotionRL for Legged LocomotionMixture-of-Experts Models
AskPDF
Mar 3, 2026

Watch Your Step: Learning Semantically-Guided Locomotion in Cluttered Environment

Although legged robots demonstrate impressive mobility on rough terrain, using them safely in cluttered environments remains a challenge. A key issue is their inability to avoid stepping on low-lying objects, such as high-cost small devices or cables on flat ground. This limitation arises from a disconnection between high-level semantic understanding and low-level control, combined with errors in elevation maps during real-world operation. To address this, we introduce SemLoco, a Reinforcement Learning (RL) framework designed to avoid obstacles precisely in densely cluttered environments. SemLoco uses a two-stage RL approach that combines both soft and hard constraints. It performs pixel-wise foothold safety inference, which enables more accurate foot placement. Additionally, SemLoco integrates semantic map, allowing it to assign traversability costs instead of relying only on geometric data. SemLoco greatly reduces collisions and improves safety around sensitive objects, enabling reliable navigation in situations where traditional controllers would likely cause damage. Experimental results further show that SemLoco can be effectively applied to more complex, unstructured real-world environments. A demo video can be view at https://youtu.be/FSq-RSmIxOM.

Denan Liang, Yuan Zhu, Ruimeng Liu +3

Legged RoboticsRL for Legged LocomotionRobot Collision Avoidance
AskPDF
Mar 2, 2026

Learning Thermal-Aware Locomotion Policies for an Electrically-Actuated Quadruped Robot

Electrically-actuated quadrupedal robots possess high mobility on complex terrains, but their motors tend to accumulate heat under high-torque cyclic loads, potentially triggering overheat protection and limiting long-duration tasks. This work proposes a thermal-aware control method that incorporates motor temperatures into reinforcement learning locomotion policies and introduces thermal-constraint rewards to prevent temperature exceedance. Real-world experiments on the Unitree A1 demonstrate that, under a fixed 3 kg payload, the baseline policy triggers overheat protection and stops within approximately 7 minutes, whereas the proposed method can operate continuously for over 27 minutes without thermal interruptions while maintaining comparable command-tracking performance, thereby enhancing sustainable operational capability.

Letian Qian, Yuhang Wan, Shuhan Wang +1

State Key Laboratory of Intelligent Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China

Robot LocomotionQuadruped LocomotionRL for Legged LocomotionConstrained RL
AskPDF
Feb 16, 2026

Learning Transferability: A Two-Stage Reinforcement Learning Approach for Enhancing Quadruped Robots' Performance in U-Shaped Stair Climbing

Quadruped robots are employed in various scenarios in building construction. However, autonomous stair climbing across different indoor staircases remains a major challenge for robot dogs to complete building construction tasks. In this project, we employed a two-stage end-to-end deep reinforcement learning (RL) approach to optimize a robot's performance on U-shaped stairs. The training robot-dog modality, Unitree Go2, was first trained to climb stairs on Isaac Lab's pyramid-stair terrain, and then to climb a U-shaped indoor staircase using the learned policies. This project explores end-to-end RL methods that enable robot dogs to autonomously climb stairs. The results showed (1) the successful goal reached for robot dogs climbing U-shaped stairs with a stall penalty, and (2) the transferability from the policy trained on U-shaped stairs to deployment on straight, L-shaped, and spiral stair terrains, and transferability from other stair models to deployment on U-shaped terrain.

Baixiao Huang, Baiyu Huang, Yu Hou

Independent Researcher, 7200 Bollinger Rd, Apt 516, CA 95129. · Independent Researcher, 16506 82nd Pl NE, Kenmore, WA, 98028. · Western New England University, Department of Construction Management, 1215 Wilbraham Rd, Springfield, MA 01119 (corresponding author).

RL for Legged LocomotionQuadruped LocomotionRobot Policy Generalization
AskPDF
Feb 13, 2026

TRAIN: Terrain-Aware Interactive Navigation for Quadruped Robots via Reinforcement Learning

This study introduces TRAIN: TeRrain-Aware Interactive Navigation, a deep reinforcement learning (DRL)-based framework for quadrupedal navigation across unstructured terrain and human-populated environments. Conventional quadrupedal navigation typically separates motion planning from locomotion control, neglecting whole-body constraints and terrain awareness. On the other hand, end-to-end methods are more integrated but require high-frequency sensing, which is often noisy and computationally costly. In addition, most existing approaches assume static environments, limiting their use in human-populated settings. To address these limitations, we propose a two-stage training framework with three DRL pipelines. (1) TRAIN-Loco employs an asymmetric actor-critic (AC) model for quadrupedal locomotion, enabling agile traversal of uneven terrains without explicit terrain or contact observations. (2) TRAIN-Nav applies a symmetric AC framework for navigation with interactive pedestrians, directly mapping transformed LiDAR data to ego-agent actions under differential-drive kinematics. (3) A unified pipeline, TRAIN, integrates TRAIN-Loc and TRAIN-Nav, supporting terrain-aware quadrupedal navigation in uneven and interactive environments. Comprehensive benchmarks against quadrupedal locomotion and robot navigation baselines demonstrate the effectiveness of TRAIN. Hardware experiments further confirm its potential for sim-to-real transfer.

Wei Zhu, Irfan Tito Kurniawan, Ye Zhao +1

Department of Robotics, Graduate School of Engineering, Tohoku University, 980-8579, Sendai, Japan. · Laboratory for Intelligent Decision and Autonomous Robots, Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA 30313, USA.

Quadruped Robot NavigationTerrain-Aware Robot LocomotionRL for Legged LocomotionSocial Navigation
AskPDF
Feb 2, 2026

Flow Policy Gradients for Robot Control

Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients, a recent framework that bypasses likelihood computation, can be made effective for training and finetuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved finetuning robustness over baselines.

Brent Yi, Hongsuk Choi, Himanshu Gaurav Singh +9

Amazon FAR · UC Berkeley · Stanford +2

Robotic ControlRL for RoboticsFlow-Based Policy LearningRL for Legged Locomotion
AskPDF
Jan 22, 2026

PUMA: Perception-driven Unified Foothold Prior for Mobility Augmented Quadruped Parkour

Parkour tasks for quadrupeds have emerged as a promising benchmark for agile locomotion. While human athletes can effectively perceive environmental characteristics to select appropriate footholds for obstacle traversal, endowing legged robots with similar perceptual reasoning remains a significant challenge. Existing methods often rely on hierarchical controllers that follow pre-computed footholds, thereby constraining the robot's real-time adaptability and the exploratory potential of reinforcement learning. To overcome these challenges, we present PUMA, an end-to-end learning framework that integrates visual perception and foothold priors into a single-stage training process. This approach leverages terrain features to estimate egocentric polar foothold priors, composed of relative distance and heading, guiding the robot in active posture adaptation for parkour tasks. Extensive experiments conducted in simulation and real-world environments across various discrete complex terrains, demonstrate PUMA's exceptional agility and robustness in challenging scenarios.

Liang Wang, Kanzhong Yao, Yang Liu +4

Institute of Cyber-Systems and Control, Zhejiang University, 310027, China · Institute of Artificial Intelligence (TeleAI), China Telecom.

Legged RoboticsTerrain-Aware Robot LocomotionRL for Legged LocomotionRobotic PerceptionQuadruped Locomotion
AskPDF
Jan 13, 2026

Agile and Generalized Legged Locomotion via Attention-Based Neural Map Encoding

Achieving agile and generalized legged locomotion across terrains requires tight integration of perception and control, especially under occlusions and sparse footholds. Existing methods have demonstrated agility on parkour courses but often rely on end-to-end sensorimotor models with limited generalization and interpretability. By contrast, methods targeting generalized locomotion typically exhibit limited agility and struggle with visual occlusions. We introduce a unified reinforcement learning (RL) framework for agile and generalized locomotion that incorporates a novel attention-based map encoder in the control policy. This encoder extracts local and global mapping features and uses attention mechanisms to focus on salient regions, producing an interpretable and generalized embedding for RL-based control. We further propose a learning-based mapping pipeline that provides fast, uncertainty-aware terrain representations robust to noise and occlusions, serving as policy inputs. It uses neural networks to convert depth observations into local elevations with uncertainties, and fuses them with odometry. The pipeline also integrates with parallel simulation so that we can train controllers with online mapping, aiding sim-to-real transfer. We validate our framework with the proposed mapping pipeline on a quadruped and a biped robot, and the resulting controllers demonstrate strong agility and generalization to unseen terrains in simulation and in real-world experiments.

Chong Zhang, Victor Klemm, Fan Yang +1

1Robotic Systems Lab, ETH Zurich, Switzerland · 2Secure, Reliable, and Intelligent Systems Lab, ETH Zurich, Switzerland · 3ETH AI Center, Switzerland

Terrain-Aware Robot LocomotionRL for Legged LocomotionRobot Policy Generalization
AskPDF
Jan 12, 2026

Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids

Achieving robust humanoid locomotion in complex, unstructured environments requires transitioning from reactive proprioception to proactive perception. However, integrating exteroception remains a significant challenge: existing approaches often fail to achieve high-speed, stable locomotion across diverse, unstructured terrains. In this work, we present Hiking in the Wild, an extensible, end-to-end perceptive legged parkour framework designed for robust humanoid hiking. Our approach maps depth inputs and proprioception directly to joint actions, without relying on external localization and mapping. To ensure safety and training stability, we introduce two key mechanisms: a foothold safety mechanism combining extensible Terrain Edge Detection with Foot Volume Points to reduce catastrophic slippage on edges, and a Flat Patch Sampling strategy that mitigates reward hacking by generating feasible navigation targets. Extensive experiments on the Unitree G1 humanoid robot demonstrate that our policy enables robust traversal of complex terrains at speeds up to 2.5 m/s.

Shaoting Zhu, Ziwen Zhuang, Mengjie Zhao +2

IIIS, Tsinghua University · Shanghai Qi Zhi Institute · Department of Computer Science and Technology, Tsinghua University

Humanoid Robot LocomotionRobot LocomotionTerrain-Aware Robot LocomotionRL for Legged Locomotion
AskPDF
Nov 30, 2025

Beyond Topology: A Morphological Symmetry Graph Representation for Locomotion Policy Learning

Reinforcement learning has enabled impressive locomotion skills on articulated robots, but common policy representations remain only weakly aligned with robot physics. Generic networks ignore kinematic structure, while graph-based policies encode connectivity without specifying how physical quantities transform across symmetric body parts. We introduce a morphological symmetry graph representation for locomotion policy learning and instantiate it in MS-PPO. Starting from the robot's topological graph, our representation augments each observation and action space with the permutation and sign transformations induced by morphological symmetry. This yields a symmetry-equivariant graph actor and a symmetry-invariant graph critic, enforcing the desired policy and value constraints by construction rather than through reward shaping or data augmentation. We evaluate MS-PPO on a variety of locomotion tasks using both Unitree Go2 quadruped and Unitree G1 humanoid, including command tracking, asymmetric joint failures, out-of-distribution command generalization, and zero-shot sim-to-real deployment. Experiments show improved symmetry generalization, robustness, sample efficiency, and model efficiency over topology- and symmetry-aware baselines.

Sizhe Wei, Xulin Chen, Fengze Xie +3

Georgia Institute of Technology · Syracuse University · California Institute of Technology

RL for Legged LocomotionEquivariant GNNsRobot Policy Generalization
AskPDF
Oct 12, 2025

Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning

Existing robot locomotion learning techniques rely heavily on the offline selection of proper reward weighting gains and cannot guarantee constraint satisfaction (i.e., constraint violation) during training. Thus, this work aims to address both issues by proposing Reward-Oriented Gains via Embodied Regulation (ROGER), which adapts reward-weighting gains online based on penalties received throughout the embodied interaction process. The ratio between the positive reward (primary reward) and negative reward (penalty) gains is automatically reduced as the learning approaches the constraint thresholds to avoid violation. Conversely, the ratio is increased when learning is in safe states to prioritize performance. With a 60-kg quadruped robot, ROGER achieved near-zero constraint violation throughout multiple learning trials. It also achieved up to 50% more primary reward than the equivalent state-of-the-art techniques. In MuJoCo continuous locomotion benchmarks, including a single-leg hopper, ROGER exhibited comparable or up to 100% higher performance and 60% less torque usage and orientation deviation compared to those trained with the default reward function. Finally, real-world locomotion learning of a physical quadruped robot was achieved from scratch within one hour without any falls. Therefore, this work contributes to constraint-satisfying real-world continual robot locomotion learning and simplifies reward weighting gain tuning, potentially facilitating the development of physical robots and those that learn in the real world.

Arthicha Srisuchinnawong, Poramate Manoonpong

Vidyasirimedhi Institute of Science and Technology, Rayong, Thailand. · The University of Southern Denmark, Odense, Denmark.

Constrained RLRL for Legged LocomotionReward Design for RLQuadruped Locomotion
AskPDF
Sep 22, 2025

GeCCo -- a Generalist Contact-Conditioned Policy for Loco-Manipulation Skills on Legged Robots

Most modern approaches to quadruped locomotion focus on using Deep Reinforcement Learning (DRL) to learn policies from scratch, in an end-to-end manner. Such methods often fail to scale, as every new problem or application requires time-consuming and iterative reward definition and tuning. We present Generalist Contact-Conditioned Policy (GeCCo) --- a low-level policy trained with Deep Reinforcement Learning that is capable of tracking arbitrary contact points with a quadruped robot. We shift from task-specific end-to-end learning to a modular hierarchy in which a single planner-agnostic, contact-conditioned tracking policy serves as a robust interface between planning and control. By maintaining stable contact execution under dynamic uncertainty and planner mismatch, the policy enables high-level planners (e.g., handcrafted, learned, or optimization-based) to be composed and swapped without retraining. We demonstrate the scalability and robustness of our method by evaluating on a wide range of locomotion and manipulation tasks in a common framework and under a single generalist policy. These include a variety of gaits, traversing complex terrains (e.g., stairs and slopes) as well as previously unseen stepping-stones and narrow beams, and interacting with objects (e.g., pressing a button, pushing a barrel). Our framework acquires new behaviors more efficiently, simply by combining a task-specific high-level contact planner and the pre-trained generalist policy. Project website: https://vassil-atn.github.io/gecco.github.io

Vassil Atanassov, Wanming Yu, Siddhant Gangapurwala +2

Dynamic Robot Systems Group, Oxford Robotics Institute, University of Oxford · Sony AI · Dyson Institute of Engineering and Technology

Robotic ControlLegged RoboticsRobot Policy LearningRobot Skill LearningRobotic ManipulationRL for Legged Locomotion+1
AskPDF
Aug 28, 2025

Physics-Guided Residual Reinforcement Learning for Humanoid Narrow-Path Traversal

Traversing narrow paths is challenging for humanoid robots due to the sparse and safety-critical footholds required. Purely template-based or end-to-end reinforcement learning-based methods suffer from such harsh terrains. This paper proposes a two stage training framework for such narrow path traversing tasks, coupling a template-based foothold planner with a low-level foothold tracker from Stage-I training and a lightweight perception aided foothold modifier from Stage-II training. With the curriculum setup from flat ground to narrow paths across stages, the resulted controller in turn learns to robustly track and safely modify foothold targets to ensure precise foot placement over narrow paths. This framework preserves the interpretability from the physics-based template and takes advantage of the generalization capability from reinforcement learning, resulting in easy sim-to-real transfer. The learned policies outperform purely template-based or reinforcement learning-based baselines in terms of success rate, centerline adherence and safety margins. Validation on a Unitree G1 humanoid robot yields successful traversal of a 0.2m wide and 3m long beam for 20 trials without any failure.

Tianchen Huang, Sisheng Chen, Wei Zhou +8

Institute of Humanoid Robots, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, Anhui 230026, China

Humanoid Robot LocomotionRobot Skill LearningRobot LocomotionRL for Legged LocomotionResidual Policy Learning
AskPDF
Oct 4, 2024

Synergistic Action and Style Priors for Learning Legged Locomotion

Deep Reinforcement Learning (DRL) enables robots to learn complex behaviors through interaction with the environment. However, due to the unrestricted nature of the learning algorithms, the resulting solutions are often brittle and appear unnatural. This is especially true for learning direct joint-level torque control, as inductive biases are difficult to integrate into the learning process. We propose an inductive bias for learning locomotion that is especially useful for torque control: latent actions learned from a small dataset of expert demonstrations. This prior allows the policy to directly leverage knowledge contained in the expert's actions and facilitates more efficient exploration. We observe that the agent is not restricted to the reward levels of the demonstration, and performance in transfer tasks is improved significantly. Latent action priors combined with style rewards for imitation lead to a closer replication of the expert's behavior. Videos are available at https://sites.google.com/view/latent-action-priors.

Oliver Hausdörfer, Alexander von Rohr, Éric Lefort +1

Technical University of Munich, Germany; TUM School of Computation, Information and Technology, Department of Computer Engineering, Learning Systems and Robotics Lab; Munich Institute of Robotics and Machine Intelligence.

RL for Legged LocomotionRobot Policy LearningData-Efficient Robot LearningLatent Action LearningRobot Imitation Learning
AskPDF
Jul 5, 2024

PA-LOCO: Learning Perturbation-Adaptive Locomotion for Quadruped Robots

Numerous locomotion controllers have been designed based on Reinforcement Learning (RL) to facilitate blind quadrupedal locomotion traversing challenging terrains. Nevertheless, locomotion control is still a challenging task for quadruped robots traversing diverse terrains amidst unforeseen disturbances. Recently, privileged learning has been employed to learn reliable and robust quadrupedal locomotion over various terrains based on a teacher-student architecture. However, its one-encoder structure is not adequate in addressing external force perturbations. The student policy would experience inevitable performance degradation due to the feature embedding discrepancy between the feature encoder of the teacher policy and the one of the student policy. Hence, this paper presents a privileged learning framework with multiple feature encoders and a residual policy network for robust and reliable quadruped locomotion subject to various external perturbations. The multi-encoder structure can decouple latent features from different privileged information, ultimately leading to enhanced performance of the learned policy in terms of robustness, stability, and reliability. The efficiency of the proposed feature encoding module is analyzed in depth using extensive simulation data. The introduction of the residual policy network helps mitigate the performance degradation experienced by the student policy that attempts to clone the behaviors of a teacher policy. The proposed framework is evaluated on a Unitree GO1 robot, showcasing its performance enhancement over the state-of-the-art privileged learning algorithm through extensive experiments conducted on diverse terrains. Ablation studies are conducted to illustrate the efficiency of the residual policy network.

Zhiyuan Xiao, Xinyu Zhang, Xiang Zhou +1

School of Aeronautics and Astronautics, Sun Yat-sen University, Shenzhen 518107, P.R. China.

RL for Legged LocomotionQuadruped LocomotionLearning Using Privileged InformationRobust RLResidual Policy Learning
AskPDF