We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.
Figures & tables
Figure 1: Bipedal walking mode of a quadruped robot on outdoor asphalt pavement supported by the proposed method.
Figure 2: A diagram showing the proposed K-MBRL algorithm workflow. The red blocks highlight the approximate policy objective mechanism. Red and blue arrows show the policy learning flow, and black arrows show the policy deployment flow.
Figure 3: Torque-constrained pendulum swing-up.
Figure 4: Ablations in Dynamics (top) and critic (bottom).
Figure 5: Comparison of diminishing and constant step sizes (top) and performance of K-MBRL with different buffer sizes (bottom).
Figure 6: Lifting dimensions (top) and number of neurons (bottom) ablations.
Figure 7: Performance comparisons in MuJoCo. Solid lines and shaded regions denote the mean and standard deviation, respectively, across 5 algorithm runs.
Figure 8: Dynamics learning ablation.
Figure 9: Go1 quadruped robot performing bipedal walking using the proposed K-MBRL in outdoor experiments.
Figure 10: Training curves of the rising, bipedal walking, and landing phases using K-MBRL on Go1 over 3 algorithm runs. Both the straight-walking and turning tasks share the same rising maneuver and feedback landing policy. From-scratch training denotes that each phase runs K-MBRL independently, whereas full-cycle training denotes subsequent sequential refinement, where each phase is initialized from the terminal state of the preceding phase. For the rising phase, the full-cycle curve reports the search fitness of the open-loop trajectory optimization, while the turning policy is initialized from the trained bipedal straight walking policy.
Figure 11: Bipedal straight walking and turning on a marble floor using K-MBRL. Notably, during a bipedal right turn, Go1’s left hind foot steps onto a raised floor socket cover not encountered during training, yet the robot maintains balance and completes the turn.
Parameter
Value
Optimizer
Adam
Critic step size
3×10−4
Lifting/dynamics step size
3×10−4
Discount factor γ
0.99
Data buffer capacity
160,000 data triplets
Data batch size
256
Table 2: Shared K-MBRL training hyperparameters.
Parameter
Policy
Critic
Lifting network
Input dimension
48
60
48
Hidden widths
[256,256]
[256,256]
[400,400]
Hidden activation
ReLU
ReLU
ReLU
Output dimension
12
1
96
Output activation
tanh
Linear
Linear
Table 3: Shared network architectures used by K-MBRL.
Parameter
FS-rising
FS-walking
FS-landing
Bipedal left/right turns
Bipedal straight walking
Shared landing
Policy step size
10−4
10−4
10−4
2×10−6
10−6
10−6
Initial exploration std.
0.20
0.20
0.20
0.05
0.05
0.05
Final exploration std.
0.01
0.05
0.01
0.02
0.02
0.02
Reward scale
0.001
1
0.01
1
1
0.01
Table 4: K-MBRL training settings.
Parameter
Value
Optimized variables
Launch/catch keyframes and timing
Number of generations
20
Candidates per generation
24
Elite candidates
4
Training trials per candidate
12
Held-out seed-set size
24
Table 5: Open-loop rising hyperparameters for the elite-based Gaussian search.
Component
FS-rising
FS-walking
FS-landing
Upright orientation reward
1
–
–
Normalized height reward
0.8
–
–
Valid reared-posture bonus
2
–
–
Rise reference-pose reward
2
–
–
Handoff velocity cost
−0.25
–
–
Handoff pose cost
−1
–
–
Table 6: Reward coefficients for from-scratch training.
Component
Bipedal left/right turns
Bipedal straight walking
Shared landing
Squared action-change cost
−0.02
−0.02
−0.001
Loaded-foot slip cost
−10
−1
−10
Heading-hold cost
−4
−16
–
Trunk/feet twist cost
−3
−6
–
Rear-calf torque cost
–
−1
–
Rear-action change cost
–
−3
–
Table 7: Reward coefficients for full-cycle training.