A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic

Chengyang Gu; Yuxin Pan; Hui Xiong; Yize Chen

eess.SYMar 30, 2026

A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic

Authors: Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen

Organizations: Information Hub, HKUST (Guangzhou), Guangzhou, China · the Department of Computer Science, City University of Hong Kong, Hong Kong, China · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada

Abstract

Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned dynamics models for policy optimization. However, the effectiveness of methods such as actor-critic is often limited by compounding model errors, which degrade long-horizon value estimation. Existing approaches, such as Model-Based Value Expansion (MVE), partially mitigate this issue through multi-step rollouts, but remain sensitive to rollout horizon selection and residual model bias. Motivated by the Pontryagin Maximum Principle (PMP), we propose Hamiltonian Actor-Critic (HAC), a model-based approach that eliminates explicit value function learning by directly optimizing a Hamiltonian defined over the learned dynamics and reward for deterministic systems. By avoiding value approximation, HAC reduces sensitivity to model errors while admitting convergence guarantees. Extensive experiments on continuous control benchmarks, in both online and offline RL settings, demonstrate that HAC outperforms model-free and MVE-based baselines in control performance, convergence speed, and robustness to distributional shift, including out-of-distribution (OOD) scenarios. In offline settings with limited data, HAC matches or exceeds state-of-the-art methods, highlighting its strong sample efficiency.

Figures & tables

Panel of Figure 1 — Fig. 1 : Learning Curves Comparison (Online RL): We compare our HAC with DDPG, SAC and MVE-DDPG on Pendulum , MountainCar , Swimmer and Hopper . For MVE-DDPG and HAC which require multi-step imaginary rollouts, we set the rollout horizon to $K=10$ for Pendulum to cover all control horizon, $K=5$ for MountainCar and Swimmer , and $K=3$ for Hopper . All results are averaged over 5 random seeds.

Env	IQL	SAC-Off	MOPO	HAC
LQR	-16.5 $\pm$ 0.2	-19.7 $\pm$ 1.3	-19.0 $\pm$ 1.1	-15.0 $\pm$ 0.0
Pendulum	-999.8 $\pm$ 6.4	-1021.0 $\pm$ 9.6	-1017.1 $\pm$ 1.3	-803.0 $\pm$ 0.1
MountainCar	-97.1 $\pm$ 1.4	-98.8 $\pm$ 0.7	-94.3 $\pm$ 0.3	-83.5 $\pm$ 10.9
Swimmer	32.6 $\pm$ 11.1	31.7 $\pm$ 4.7	36.7 $\pm$ 8.3	51.9 $\pm$ 7.4
Hopper	47.4 $\pm$ 0.9	160.9 $\pm$ 118.1	210.3 $\pm$ 202.9	167.8 $\pm$ 7.3

TABLE I : Offline RL performance (rewards, averaged over 10 trials).

Panel of Figure 3 — Fig. 2 : Learning Curves Comparison (Offline RL) . We compare our HAC with IQL, SAC-Off and MOPO on LQR , Pendulum , and MountainCar , Swimmer . For MOPO and HAC which require multi-step imaginary rollouts, rollout horizon is to $K=10$ for LQR and Pendulum to cover all control horizon, $K=5$ for MountainCar and Swimmer . Results are averaged over 5 random seeds.

Panel of Figure 4 — Fig. 3 : Robustness comparison between MVE-DDPG and proposed HAC against initial state shift . The experiment is conducted on a 10-step LQR task with state dimension 5 and action dimension 3. Plots (a) and (b) illustrate optimal trajectories generated by HAC and MVE-DDPG in dimension 0 (averaged on 10 random seeds) under non-shifted and shifted initial states.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Fig. 4 : Critic Estimation Comparison: Plots comparing normalized estimated Q-values (in MVE-DDPG) and Hamiltonian (in our HAC) against ground-truth sampled from 10 random test episodes in LQR. An ideal critic should concentrates on the red unity line.

Ensemble number	Performance		Computation time per gradient step (s)
	MVE-DDPG	HAC	MVE-DDPG	HAC
None	37.491	44.366	0.011	0.017
3	41.076		0.021
5	45.923		0.030

Appendix

TABLE II : Performance and computation time of MVE-DDPG with varying ensemble sizes on the Swimmer environment, averaged over 5 random seeds. An ensemble of 5 critic networks enables MVE-DDPG to achieve performance comparable to HAC.

Model-Based RL

Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.

See all 245 papers

Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting
Mar 9, 2026Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu +3Offline RL Model-Based RL
CityUHK · Tencent · UChicago +1
Post-Hoc Robustness for Model-Based Reinforcement Learning
Jun 2, 2026Siemen Herremans, Ali Anwar, Siegfried MercelisAdversarial Robustness Robust RL
1IDLab - University of Antwerp, imec
Offline Reinforcement Learning with Universal Horizon Models
May 15, 2026Hojun Chung, Junseo Lee, Songhwai OhOffline RL Model-Based RL
Interdisciplinary Program in Artificial Intelligence and ASRI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University

Mar 9, 2026J/K move · Enter open · S save

Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting

Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu, Yao Shu+2

CityUHK · Tencent · UChicago+1

Offline RL Model-Based RL Robust RL

Open paper AskPDF arXiv

Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, model exploitation could occur due to inevitable model errors, degrading algorithm performance. Adversarial model learning offers a theoretical framework to mitigate model exploitation by solving a maximin formulation. Within such a paradigm, RAMBO~\citep{rigter2022rambo} has emerged as a representative and most popular method that provides a practical implementation with model gradient. However, we empirically reveal that severe Q-value underestimation and gradient explosion can occur in RAMBO with only slight hyperparameter tuning, suggesting that it tends to be overly conservative and suffers from unstable model updates. To address these issues, we propose RObust value-aware Model learning with Implicitly differentiable adaptive weighting (ROMI). Instead of updating the dynamics model with model gradient, ROMI introduces a novel robust value-aware model learning approach. This approach requires the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, enabling controllable conservatism and stable model updates. To further improve out-of-distribution (OOD) generalization during multi-step rollouts, we propose implicitly differentiable adaptive weighting, a bi-level optimization scheme that adaptively achieves dynamics- and value-aware model learning. Empirical results on D4RL and NeoRL datasets show that ROMI significantly outperforms RAMBO and achieves competitive or superior performance compared to other state-of-the-art methods on datasets where RAMBO typically underperforms. Code is available at https://github.com/zq2r/ROMI.git.

Abstract

Model-Based RL

Explore similar work

Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting

Post-Hoc Robustness for Model-Based Reinforcement Learning

Offline Reinforcement Learning with Universal Horizon Models