We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.
Figures & tables
Figure 1: Bipedal walking mode of a quadruped robot on outdoor asphalt pavement supported by the proposed method.
Figure 2: A diagram showing the proposed K-MBRL algorithm workflow. The red blocks highlight the approximate policy objective mechanism. Red and blue arrows show the policy learning flow, and black arrows show the policy deployment flow.
Figure 3: Torque-constrained pendulum swing-up.
Figure 4: Ablations in Dynamics (top) and critic (bottom).
Figure 5: Comparison of diminishing and constant step sizes (top) and performance of K-MBRL with different buffer sizes (bottom).
Figure 6: Lifting dimensions (top) and number of neurons (bottom) ablations.
Figure 7: Performance comparisons in MuJoCo. Solid lines and shaded regions denote the mean and standard deviation, respectively, across 5 algorithm runs.
Figure 8: Dynamics learning ablation.
Figure 9: Go1 quadruped robot performing bipedal walking using the proposed K-MBRL in outdoor experiments.
Figure 10: Training curves of the rising, bipedal walking, and landing phases using K-MBRL on Go1 over 3 algorithm runs. Both the straight-walking and turning tasks share the same rising maneuver and feedback landing policy. From-scratch training denotes that each phase runs K-MBRL independently, whereas full-cycle training denotes subsequent sequential refinement, where each phase is initialized from the terminal state of the preceding phase. For the rising phase, the full-cycle curve reports the search fitness of the open-loop trajectory optimization, while the turning policy is initialized from the trained bipedal straight walking policy.
Figure 11: Bipedal straight walking and turning on a marble floor using K-MBRL. Notably, during a bipedal right turn, Go1’s left hind foot steps onto a raised floor socket cover not encountered during training, yet the robot maintains balance and completes the turn.
Parameter
Value
Optimizer
Adam
Critic step size
3×10−4
Lifting/dynamics step size
3×10−4
Discount factor γ
0.99
Data buffer capacity
160,000 data triplets
Data batch size
256
Table 2: Shared K-MBRL training hyperparameters.
Parameter
Policy
Critic
Lifting network
Input dimension
48
60
48
Hidden widths
[256,256]
[256,256]
[400,400]
Hidden activation
ReLU
ReLU
ReLU
Output dimension
12
1
96
Output activation
tanh
Linear
Linear
Table 3: Shared network architectures used by K-MBRL.
Parameter
FS-rising
FS-walking
FS-landing
Bipedal left/right turns
Bipedal straight walking
Shared landing
Policy step size
10−4
10−4
10−4
2×10−6
10−6
10−6
Initial exploration std.
0.20
0.20
0.20
0.05
0.05
0.05
Final exploration std.
0.01
0.05
0.01
0.02
0.02
0.02
Reward scale
0.001
1
0.01
1
1
0.01
Table 4: K-MBRL training settings.
Parameter
Value
Optimized variables
Launch/catch keyframes and timing
Number of generations
20
Candidates per generation
24
Elite candidates
4
Training trials per candidate
12
Held-out seed-set size
24
Table 5: Open-loop rising hyperparameters for the elite-based Gaussian search.
Component
FS-rising
FS-walking
FS-landing
Upright orientation reward
1
–
–
Normalized height reward
0.8
–
–
Valid reared-posture bonus
2
–
–
Rise reference-pose reward
2
–
–
Handoff velocity cost
−0.25
–
–
Handoff pose cost
−1
–
–
Table 6: Reward coefficients for from-scratch training.
Component
Bipedal left/right turns
Bipedal straight walking
Shared landing
Squared action-change cost
−0.02
−0.02
−0.001
Loaded-foot slip cost
−10
−1
−10
Heading-hold cost
−4
−16
–
Trunk/feet twist cost
−3
−6
–
Rear-calf torque cost
–
−1
–
Rear-action change cost
–
−3
–
Table 7: Reward coefficients for full-cycle training.
This paper presents a model-based reinforcement learning (RL) framework for optimal closed-loop control of nonlinear robotic systems. The proposed approach learns linear lifted dynamics through Koopman operator theory and integrates the resulting model into an actor-critic architecture for policy optimization, where the policy represents a parameterized closed-loop controller. To reduce computational cost and mitigate model rollout errors, policy gradients are estimated using one-step predictions of the learned dynamics rather than multi-step propagation. This leads to an online mini-batch policy gradient framework that enables policy improvement from streamed interaction data. The proposed framework is evaluated on several simulated nonlinear control benchmarks and two real-world hardware platforms, including a Kinova Gen3 robotic arm and a Unitree Go1 quadruped. Experimental results demonstrate improved sample efficiency over model-free RL baselines, superior control performance relative to model-based RL baselines, and control performance comparable to classical model-based methods that rely on exact system dynamics.
Wenjian Hao, Yuxuan Fang, Zehui Lu +1
School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN 47907, USA · independent researcher
This paper presents an efficient model predictive path integral (MPPI) control framework for systems with complex nonlinear dynamics. To improve the computational efficiency of classic MPPI while preserving control performance, we replace the nonlinear dynamics used for trajectory propagation with a learned linear deep Koopman operator (DKO) model, enabling faster rollout and more efficient trajectory sampling. The DKO dynamics are learned directly from interaction data, eliminating the need for analytical system models. The resulting controller, termed MPPI-DK, is evaluated in simulation on pendulum balancing and surface vehicle navigation tasks, and validated on hardware through reference-tracking experiments on a quadruped robot. Experimental results demonstrate that MPPI-DK achieves control performance close to MPPI with true dynamics while substantially reducing computational cost, enabling efficient real-time control on robotic platforms.
Reinforcement learning (RL) algorithms classically suffer from poor sample efficiency. In robotics, a recent line of work has emerged addressing this problem by encoding physics priors in the learning process. However, most of these approaches are validated on well-defined, low-dimensional benchmark systems rather than high-dimensional robots with complex nonlinear dynamics. In this paper, we introduce SKooP (Symmetric Koopman Predictions), an approach combining the advantages of morphological symmetries with those of a Koopman model learned via autoencoder to enhance policy learning. SKooP learns a Koopman model of the system dynamics alongside the policy. The resulting Koopman predictions are used as privileged observations for the critic, allowing the agent to learn based on smoother, more informative features. We also incorporate group symmetries into the actor, critic, encoder and decoder networks to produce a highly equivariant policy. The SKooP approach is validated via in-depth analysis of the learned Koopman models and symmetric policies to showcase how each of these influences the agent's performance. We also show that the learned policies are transferable to different simulation environments. Our results show that SKooP consistently reduces convergence time and increases the learned reward for multiple challenging bipedal locomotion tasks on a quadruped robot. Project page: https://evelyd.github.io/SymmetricKoopmanPredictions
Evelyn D'Elia, Weishu Zhan, Giulio Turrisi +5
Machine Learning and Optimisation, University of Manchester, M13 9PL Manchester, U.K. · Dynamic Legged Systems Lab, IIT, 16163 Genoa, Italy · Generative Bionics S.R.L, 16163 Genoa, Italy +3