Organizations: the Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · the Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.
State of the art quadrupedal locomotion approaches integrate Model Predictive Control (MPC) with Reinforcement Learning (RL), enabling complex motion capabilities with planning and terrain adaptive behaviors. However, they often face compounding errors over long horizons and have limited interpretability due to the absence of physical inductive biases. We address these issues by integrating Lagrangian Neural Networks (LNNs) into an RL MPC framework, enabling physically consistent dynamics learning. At deployment, our inverse dynamics infinite horizon MPC scheme avoids costly matrix inversions, improving computational efficiency by up to 4x with minimal loss of task performance. We validate our framework through multiple ablations of the proposed LNN and its variants. We show improved sample efficiency, reduced long-horizon error, and faster real time planning compared to unstructured neural dynamics. Lastly, we also test our framework on the Unitree Go1 robot to show real world viability.
Figures & tables
Fig. 1 : Overview of our inverse dynamics-based planner. At each control step, the method samples candidate joint trajectories, computes the corresponding torques, evaluates them using learned reward and value functions, and updates the distribution for the next sampling round.
Quadruped Locomotion Methods
Inference Speed (1-5)
Planning
Interpretable Dynamics
Multi-Terrain
Nonlinear MPC [ 1 , 2 , 3 ]
3 to 4
✓
✓
✗
DreamWaQ [ 11 ] , HIMLoco [ 12 ]
5
✗
None
✓
PIP-Loco [ 13 ]
4
✓
✗
✓
Ours
3
✓
✓
✓
TABLE I : Comparison of proprioceptive quadruped locomotion methods, including our approach. The table summarizes key trade-offs across inference speed (higher rank is faster), planning capability, model interpretability, and terrain generalization.
Fig. 2 : Our training framework where observation history is encoded into full-state estimates, and passed through a physics-informed Dreamer module to augment expert actor with future predictions for robust policy learning.
q1:H(j)∼N(μi−1,σi−1),∀j∈[1,M]
Algorithm 2 Inverse Dynamics-Based ∞ Horizon MPC
Hyperparameters
Value
Training
Hidden layers for RL
[512,256,128]
Hidden layers for Dreamer
[256,256]
Activation function
tanh
Number of environments
4096
Number of mini batches
25
TABLE II : Hyperparameters used for training and deployment. Separate configurations are used for the RL and MPC modules to balance performance and efficiency.
Fig. 3 : Training loss curves across different dynamics models. Our method achieves strong sample efficiency while balancing compute cost, compared to LNN Forward and ONN baselines.
Fig. 4 : H-step prediction error of different models. Our approach maintains stable error across horizons, with significantly lower compounding error than ONN and CoM LNN.
Fig. 5 : Trade-off between MPC inference time and control performance across planning horizons. Our inverse-dynamics-based planner achieves competitive returns with significantly lower inference latency.
Method
Flat Ground
Slopes
Rough terrain
Stairs (12 cm)
Stairs (14 cm)
Stairs (16 cm)
HIMLoco
9.19±0.02
9.18±0.05
8.72±0.07
7.32±0.24
7.01±0.38
6.31±0.58
DeLaN (H=9)
9.38±0.1
9.38±0.11
9.1±0.11
8.33±0.29
7.81±0.28
7.31±0.31
CoM LNN (H=5)
9.05±0.26
8.26±0.38
7.98±0.62
6.83±0.59
Failed
Failed
ONN (H=9)
9.19±0.07
9.20±0.12
8.80±0.21
7.48±0.46
7.21±0.48
6.91±0.78
Ours (H=9)
9.23±0.17
8.67±0.42
8.64±0.42
7.74±0.53
7.51±0.58
7.18±0.53
TABLE III : Performance comparison across terrains in simulation. Reported values are average episodic rewards ± standard deviation. DeLaN achieves the highest rewards, while our method closely matches with much lower compute cost.
Fig. 6 : Hardware validation and absolute error tracking across 1200 time steps. Our method maintains significantly lower, more consistent rollout error than baselines (ONN, DeLaN, LNN variants) while navigating diverse terrain, including gravel, ramps, and stairs.
Model-based reinforcement learning (MBRL) is sample-efficient but depends on the accuracy of the learned dynamics, which are often modeled using black-box methods that do not adhere to physical laws. Those methods tend to produce inaccurate predictions when presented with data that differ from the original training set. In this work, we employ Lagrangian neural networks (LNNs), which enforce an underlying Lagrangian structure to train the model within a Dyna-based MBRL framework. Furthermore, we train the LNN using stochastic gradient-based and state-estimation-based optimizers to learn the network's weights. The state-estimation-based method converges faster than the stochastic gradient-based method during neural network training. Simulation results are provided to illustrate the effectiveness of the proposed LNN-based Dyna framework for MBRL.
Shreya Das, Kundan Kumar, Muhammad Iqbal +4
Department of Electrical Engineering and Automation, Aalto University, Finland · VTT Technical Research Centre of Finland · Department of Computer Science, University of Helsinki, Finland +1
Reinforcement learning (RL) has achieved strong performance in robotic control; however, state-of-the-art policy learning methods, such as actor-critic methods, still suffer from high sample complexity and often produce physically inconsistent actions. This limitation stems from neural policies implicitly rediscovering complex physics from data alone, despite accurate dynamics models being readily available in simulators. In this paper, we introduce a novel physics-informed RL framework, called PIPER, that seamlessly integrates physical constraints directly into neural policy optimization with analytical soft physics constraints. At the core of our method is the integration of a differentiable Lagrangian residual as a regularization term within the actor's objective. This residual, extracted from a robot's simulator description, subtly biases policy updates towards dynamically consistent solutions. Crucially, this physics integration is realized through an additional loss term during policy optimization, requiring no alterations to existing simulators or core RL algorithms. Extensive experiments demonstrate that our method significantly improves learning efficiency, stability, and control accuracy, establishing a new paradigm for efficient and physically consistent robotic control.
Quadrupedal locomotion plays a critical role in enabling agile, versatile movement across complex terrains. Understanding and estimating the underlying physical dynamics are essential for achieving efficient and stable quadrupedal locomotion. We propose a novel training framework for quadrupedal locomotion that enables the Control Policy to understand and reason about physical dynamics. In simulation, we concurrently train an Intrinsic Dynamics (ID) Head that learns state-to-torque dynamics alongside the Control Policy, and we define a dynamics reward enabled by the ID Head that encourages the Policy toward more predictable dynamical behavior. We also provide a mechanism to tune the learned dynamics in the resulting Policy by controlling the training coefficients of the ID Head. Our simulation experiments show that this mechanism drives convergence to better optima across a wide range of standard quadrupedal locomotion rewards, yielding more efficient and smoother policies. Our real-robot experiments demonstrate sim-to-real transfer of these improvements, with significant gains in torque efficiency (16.8%), action rate (18.6%), and mechanical power (12.8%), while improving safe torque occupancy by 6.4%.
Aman Arora, Nalini Ratha
Department of Computer Science, University at Buffalo, The State University of New York, Buffalo, USA