cs.RONov 10, 2026

Koopman Model-Based Reinforcement Learning

Authors: Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou

Organizations: School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN 47907, USA

Abstract

We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.

Figures & tables

Explore similar work

CardsList
  1. Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems

    Apr 21, 2026Wenjian Hao, Yuxuan Fang, Zehui Lu +1Robotic ControlReinforcement Learning

  2. SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learning

    Jul 13, 2026Evelyn D'Elia, Weishu Zhan, Giulio Turrisi +5Reinforcement LearningKoopman Operator Learning