Learning Structured Locomotion References for Optimization-Based Whole-Body Control
Authors: Ömer Tarık Banuş, Uluç Saranlı, M. Mert Ankaralı
Organizations: Center for Robotics and Artificial Intelligence (ROMER), Middle East Technical University (METU), 06800 Ankara, Türkiye. · Defense Systems Technologies Division, ASELSAN Inc., 06200 Ankara, Türkiye. · Department of Computer Engineering, METU, 06800 Ankara, Türkiye. · Department of Electrical and Electronics Engineering, METU, 06800 Ankara, Türkiye.
Quadruped locomotion requires coordinating body motion,contact forces, foothold placement, and gait timing. We present a hierarchical controller in which reinforcement learning adapts these structured locomotion references while a gait scheduler, reactive footstep planner, and whole-body impulse controller retain responsibility for high-frequency execution. The model-based backbone can sustain a nominal trot independently, allowing the policy to focus on when and how to modify an already functional locomotion controller. We train the complete hierarchy in a vectorized controller-in-the-loop environment using a six-direction disturbance curriculum. Under a common simulator and evaluation protocol, the proposed controller achieves 77.29% mean survival across the tested disturbance grid, compared with 50.35% for the model-based reference controller. Training-time channel ablations identify adaptive foothold placement as the largest marginal contributor to grid survival and reveal direction-dependent effects of gait-rate adaptation. Together, these results demonstrate the viability of the proposed hierarchical controller architecture for integrating learned adaptation with optimization-based whole-body control through structured locomotion references.
Figures & tables
Fig. 1: Proposed controller. A 100 Hz policy supplies structured locomotion commands to a gait scheduler, footstep planner, and WBIC running at 500 Hz. Feedback provides the 54-dimensional observation.
Index
Command
Decoded range
0–1
body vx,vy residual
±0.5 m/s
2
yaw-rate residual
±0.3 rad/s
3–14
four forces (fx,fy,fz)
±50,±50,[0,250] N
15–22
four foothold offsets (x,y)
±0.5 m
23
body-height residual
±0.05 m
24
phase increment
[0,0.01] per tick
TABLE I: Action components; leg order is front-right (FR), front-left (FL), hind-right (HR), and hind-left (HL).
Signal
Dimension
Commanded forward velocity vxcmd
1
Body linear velocity vbb
3
Body angular velocity ωbb
3
Body orientation ϕb (roll, pitch, yaw)
3
Joint positions qj
12
Joint velocities q˙j
12
TABLE II: Observation components.
Term
Penalty pi
wi
Term
Penalty pi
wi
vx
3∣vxw−vxref∣
0.25
Straightness
5∣ybw∣+3∣vyw∣
0.15
vz
3∣vzw∣
0.10
Linear acceleration
∥abw∥22
0.025
Height
20∣zb−zref∣
0.35
Angular velocity
∥ωb∥22
0.025
Orientation
50(1−⟨qb,q0⟩)
0.15
Action variation
3∥at−at−1∥22
0.10
Phase progression
max(0,aθ+1)
0.05
Action magnitude
∑j∈Aat,j2
0.05
TABLE III: Reward penalties and weights. Each healthy-state component is wiexp(−pi) . Superscript w denotes the world frame, and q0 is the neutral-orientation quaternion.
Fig. 2: Go2 simulation environment. The arrow illustrates the disturbance direction; evaluation applies instantaneous velocity changes.
Fig. 3: Training episode length and timeout fraction for three seeds. Shading indicates disturbance ceilings of 1, 2, and 3 m/s, with transitions at 8.1 and 24.3 million transitions. Curves are unsmoothed.
Fig. 4: Survival under additive velocity disturbances, with 100 episodes per cell and policy. Curves and bands show the mean and ±1 sample standard deviation across three training seeds for the proposed and source-recipe-based joint-space controllers. MPC+WBIC is one fixed controller.
Controller
Grid survival (%)
MPC+WBIC
50.35
Source-recipe-based joint-space RL
54.01±5.55
Proposed
77.29±1.15
TABLE IV: Mean survival over the 60 disturbance conditions for the complete configured systems. Learned-controller results report mean ± sample SD across three training seeds; MPC+WBIC is one fixed controller.
Fig. 5: Training-time action-channel ablations: grid-survival change (left) and directional c50 change (right; negative is worse). Backward c50 uses extensions above 5.0 m/s excluded from the grid score.
Fig. 6: Nominal behavior of the retrained ablation arms over five undisturbed episodes. Small circles denote episodes, diamonds denote means, and crosses in the speed panel denote commanded-speed means.
Legged robots equipped with manipulators offer promising capabilities for loco-manipulation tasks in unstructured environments. However, achieving task-consistent end-effector behavior during locomotion remains challenging due to terrain-induced disturbances and the strong coupling between locomotion and manipulation, particularly for small-scale quadruped platforms. A reinforcement-learning based whole-body control framework is proposed for quadruped loco-manipulation on complex terrain. The proposed framework combines a hybrid task representation with task-prioritized whole-body learning to coordinate locomotion and manipulation while preserving task-consistent end-effector behavior. A lightweight quadruped manipulator platform is further developed to validate the proposed framework on a small-scale quadruped system. Real-world experiments demonstrate successful deployment on complex terrains such as slopes up to 30 degrees, stairs, and deformable terrain. These results demonstrate the feasibility of deploying learned whole-body control policy for quadruped loco-manipulation in real-world environments and highlight the importance of task-consistent whole-body coordination for terrain-adaptive loco-manipulation.
Yanzhou Jin, Fei Suo, Chang Liu +5
Graduate School of Engineering, The University of Tokyo, Japan. · Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya, 464-8603, Japan. · Department of Mathematics and Informatics, Ecole Centrale de Lyon, France.
Learning-based quadrupedal locomotion typically relies on complex reward formulations that entangle task specification, operational limits, gait preference, and terrain adaptation within a single optimization objective. We instead treat these functions through distinct mechanisms: rewards for task specification, constraints for operational limits, energy minimization for gait preference, and exteroceptive perception for adapting energy use to terrain difficulty. We show that these components jointly enable efficient, terrain-adaptive locomotion, and that removing each component exposes a distinct failure mode. Our formulation removes explicit gait priors (including air-time, contact-count, and foot-clearance targets) in favor of emergent behavior. Compared to a conventional complex-reward baseline, our formulation achieves comparable terrain traversal while reducing cost of transport by 56% and operational-limit violations by 96%. The resulting policies transfer zero-shot to a physical Unitree Go2 using LiDAR-based elevation mapping. Project website with videos: https://tinyurl.com/locomposition.
Loukas Kordos, Leonard T. Franz, Simon Rappenecker +4
Technical University of Munich. · University of Tübingen. · Hertie Institute for Clinical Brain Research & Center for Integrative Neuroscience. +1
Numerous locomotion controllers have been designed based on Reinforcement Learning (RL) to facilitate blind quadrupedal locomotion traversing challenging terrains. Nevertheless, locomotion control is still a challenging task for quadruped robots traversing diverse terrains amidst unforeseen disturbances. Recently, privileged learning has been employed to learn reliable and robust quadrupedal locomotion over various terrains based on a teacher-student architecture. However, its one-encoder structure is not adequate in addressing external force perturbations. The student policy would experience inevitable performance degradation due to the feature embedding discrepancy between the feature encoder of the teacher policy and the one of the student policy. Hence, this paper presents a privileged learning framework with multiple feature encoders and a residual policy network for robust and reliable quadruped locomotion subject to various external perturbations. The multi-encoder structure can decouple latent features from different privileged information, ultimately leading to enhanced performance of the learned policy in terms of robustness, stability, and reliability. The efficiency of the proposed feature encoding module is analyzed in depth using extensive simulation data. The introduction of the residual policy network helps mitigate the performance degradation experienced by the student policy that attempts to clone the behaviors of a teacher policy. The proposed framework is evaluated on a Unitree GO1 robot, showcasing its performance enhancement over the state-of-the-art privileged learning algorithm through extensive experiments conducted on diverse terrains. Ablation studies are conducted to illustrate the efficiency of the residual policy network.
Zhiyuan Xiao, Xinyu Zhang, Xiang Zhou +1
School of Aeronautics and Astronautics, Sun Yat-sen University, Shenzhen 518107, P.R. China.