Integrating Lagrangian Neural Networks into the Dyna Framework for Reinforcement Learning
Authors: Shreya Das, Kundan Kumar, Muhammad Iqbal, Outi Savolainen, Dominik Baumann, Laura Ruotsalainen, Simo Särkkä
Organizations: Department of Electrical Engineering and Automation, Aalto University, Finland · VTT Technical Research Centre of Finland · Department of Computer Science, University of Helsinki, Finland · ELLIS Institute Finland
Model-based reinforcement learning (MBRL) is sample-efficient but depends on the accuracy of the learned dynamics, which are often modeled using black-box methods that do not adhere to physical laws. Those methods tend to produce inaccurate predictions when presented with data that differ from the original training set. In this work, we employ Lagrangian neural networks (LNNs), which enforce an underlying Lagrangian structure to train the model within a Dyna-based MBRL framework. Furthermore, we train the LNN using stochastic gradient-based and state-estimation-based optimizers to learn the network's weights. The state-estimation-based method converges faster than the stochastic gradient-based method during neural network training. Simulation results are provided to illustrate the effectiveness of the proposed LNN-based Dyna framework for MBRL.
Figures & tables
Fig. 1 : The basic idea behind the developed method of LNN-based MBRL in the Dyna framework. We learn the dynamics model using the LNN from real environment data and use an RK-2 integrator to generate model-based rollouts. The policy and value networks are trained on both real and model-generated data, improving sample efficiency while preserving physical consistency.
Fig. 2 : Model learning using LNN with (q,q˙) as input and L is the output of the network, which is further used to evaluate q¨ using the Euler-Lagrangian equation. The second-order Runge-Kutta (RK-2) method is then used to compute (qt+1,q˙t+1) . The reward is learning from the state-action pair.
Fig. 3 : Average return versus timestep plot for the proposed PIMBRL using LNN with Adam and EKF as optimizers, PIMBRL using DNN with constraints as in [ 32 ] , and MFRL.
State of the art quadrupedal locomotion approaches integrate Model Predictive Control (MPC) with Reinforcement Learning (RL), enabling complex motion capabilities with planning and terrain adaptive behaviors. However, they often face compounding errors over long horizons and have limited interpretability due to the absence of physical inductive biases. We address these issues by integrating Lagrangian Neural Networks (LNNs) into an RL MPC framework, enabling physically consistent dynamics learning. At deployment, our inverse dynamics infinite horizon MPC scheme avoids costly matrix inversions, improving computational efficiency by up to 4x with minimal loss of task performance. We validate our framework through multiple ablations of the proposed LNN and its variants. We show improved sample efficiency, reduced long-horizon error, and faster real time planning compared to unstructured neural dynamics. Lastly, we also test our framework on the Unitree Go1 robot to show real world viability.
the Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · the Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.
This paper proposes a Bayesian filtering-based approach for learning the dynamics of a physical system from partial, noisy measurements. We model the system dynamics using a Lagrangian mechanics formulation. As in Lagrangian neural networks (LNNs), we parameterize the kinetic and potential energies with neural networks. The unknown external forces in the Lagrangian formulation are modeled as white Gaussian noise. The corresponding Euler--Lagrange equations then yield a continuous-time stochastic state-space model (SSM) that describes the system dynamics. The neural network parameters and system states are then jointly learned via a maximum-likelihood method using Gaussian-approximation-based Bayesian filters. The effectiveness of the proposed method is demonstrated on pendulum and Duffing oscillator examples, and its performance is compared with conventional LNNs and with approximate Bayesian filters using known system models.
Kundan Kumar, Shreya Das, Simo Särkkä
K. Kumar is with the School of Engineering and Applied Science, Ahmedabad University, India · K. Kumar and S. Särkkä are with the ELLIS Institute Finland and the Department of Electrical Engineering and Automation, Aalto University, Finland · S. Das is with the Department of Electrical Engineering, Indian Institute of Technology (BHU) Varanasi, India
Model-based reinforcement learning (MBRL) infers information about the environment from a learned dynamics model and bears the potential to address open problems such as data efficient and safe learning in robotics. However, inaccuracies of the learned dynamics model are typically exploited by the agent, substantially hampering the capabilities of MBRL methods. We present a framework for dealing with inaccuracies of probabilistic models through targeted handling of uncertainty that effectively mitigates model exploitation. We present recent successes in learning directly on hardware and safe exploration, and discuss future directions for uncertainty-aware MBRL.
Bernd Frauenknecht, Devdutt Subhasish, Artur Eisele +2
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, 52062 Aachen, Germany