Organizations: Information Hub, HKUST (Guangzhou), Guangzhou, China · the Department of Computer Science, City University of Hong Kong, Hong Kong, China · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada
Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned dynamics models for policy optimization. However, the effectiveness of methods such as actor-critic is often limited by compounding model errors, which degrade long-horizon value estimation. Existing approaches, such as Model-Based Value Expansion (MVE), partially mitigate this issue through multi-step rollouts, but remain sensitive to rollout horizon selection and residual model bias. Motivated by the Pontryagin Maximum Principle (PMP), we propose Hamiltonian Actor-Critic (HAC), a model-based approach that eliminates explicit value function learning by directly optimizing a Hamiltonian defined over the learned dynamics and reward for deterministic systems. By avoiding value approximation, HAC reduces sensitivity to model errors while admitting convergence guarantees. Extensive experiments on continuous control benchmarks, in both online and offline RL settings, demonstrate that HAC outperforms model-free and MVE-based baselines in control performance, convergence speed, and robustness to distributional shift, including out-of-distribution (OOD) scenarios. In offline settings with limited data, HAC matches or exceeds state-of-the-art methods, highlighting its strong sample efficiency.
Figures & tables
Fig. 1 : Learning Curves Comparison (Online RL): We compare our HAC with DDPG, SAC and MVE-DDPG on Pendulum , MountainCar , Swimmer and Hopper . For MVE-DDPG and HAC which require multi-step imaginary rollouts, we set the rollout horizon to K=10 for Pendulum to cover all control horizon, K=5 for MountainCar and Swimmer , and K=3 for Hopper . All results are averaged over 5 random seeds.
Env
IQL
SAC-Off
MOPO
HAC
LQR
-16.5 ± 0.2
-19.7 ± 1.3
-19.0 ± 1.1
-15.0 ± 0.0
Pendulum
-999.8 ± 6.4
-1021.0 ± 9.6
-1017.1 ± 1.3
-803.0 ± 0.1
MountainCar
-97.1 ± 1.4
-98.8 ± 0.7
-94.3 ± 0.3
-83.5 ± 10.9
Swimmer
32.6 ± 11.1
31.7 ± 4.7
36.7 ± 8.3
51.9 ± 7.4
Hopper
47.4 ± 0.9
160.9 ± 118.1
210.3 ± 202.9
167.8 ± 7.3
TABLE I : Offline RL performance (rewards, averaged over 10 trials).
Fig. 2 : Learning Curves Comparison (Offline RL) . We compare our HAC with IQL, SAC-Off and MOPO on LQR , Pendulum , and MountainCar , Swimmer . For MOPO and HAC which require multi-step imaginary rollouts, rollout horizon is to K=10 for LQR and Pendulum to cover all control horizon, K=5 for MountainCar and Swimmer . Results are averaged over 5 random seeds.
Fig. 3 : Robustness comparison between MVE-DDPG and proposed HAC against initial state shift . The experiment is conducted on a 10-step LQR task with state dimension 5 and action dimension 3. Plots (a) and (b) illustrate optimal trajectories generated by HAC and MVE-DDPG in dimension 0 (averaged on 10 random seeds) under non-shifted and shifted initial states.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 4 : Critic Estimation Comparison: Plots comparing normalized estimated Q-values (in MVE-DDPG) and Hamiltonian (in our HAC) against ground-truth sampled from 10 random test episodes in LQR. An ideal critic should concentrates on the red unity line.
Ensemble number
Performance
Computation time per gradient step (s)
MVE-DDPG
HAC
MVE-DDPG
HAC
None
37.491
44.366
0.011
0.017
3
41.076
0.021
5
45.923
0.030
Appendix
TABLE II : Performance and computation time of MVE-DDPG with varying ensemble sizes on the Swimmer environment, averaged over 5 random seeds. An ensemble of 5 critic networks enables MVE-DDPG to achieve performance comparable to HAC.
Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, model exploitation could occur due to inevitable model errors, degrading algorithm performance. Adversarial model learning offers a theoretical framework to mitigate model exploitation by solving a maximin formulation. Within such a paradigm, RAMBO~\citep{rigter2022rambo} has emerged as a representative and most popular method that provides a practical implementation with model gradient. However, we empirically reveal that severe Q-value underestimation and gradient explosion can occur in RAMBO with only slight hyperparameter tuning, suggesting that it tends to be overly conservative and suffers from unstable model updates. To address these issues, we propose RObust value-aware Model learning with Implicitly differentiable adaptive weighting (ROMI). Instead of updating the dynamics model with model gradient, ROMI introduces a novel robust value-aware model learning approach. This approach requires the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, enabling controllable conservatism and stable model updates. To further improve out-of-distribution (OOD) generalization during multi-step rollouts, we propose implicitly differentiable adaptive weighting, a bi-level optimization scheme that adaptively achieves dynamics- and value-aware model learning. Empirical results on D4RL and NeoRL datasets show that ROMI significantly outperforms RAMBO and achieves competitive or superior performance compared to other state-of-the-art methods on datasets where RAMBO typically underperforms. Code is available at https://github.com/zq2r/ROMI.git.
To improve the real-world applicability of reinforcement learning (RL), the field of adversarially robust RL studies how to train agents under adversarial environment perturbations. In this setting, a protagonist agent optimizes a policy under environmental perturbations from an adversary, resulting in a zero-sum Markov game. When adversarially robust RL is combined with model-based RL, the adversary can target a learned transition model instead of the training environment. Extending this idea, this work introduces post-hoc robustification of deep RL agents at inference time. By using the learned model in combination with a trained nominal policy, our approach performs a robust policy improvement step. The goal is to improve robustness without any additional training of neural networks. Specifically, we utilize model-predictive control under adversarial rollouts, which are approximated via projected gradient descent within a bounded uncertainty set. Furthermore, these offline rollouts are performed while considering and mitigating out-of-distribution issues. The proposed methodology is validated by demonstrating significant improvements in robustness when the algorithm is evaluated in perturbed Gymnasium MuJoCo environments, while considering the computational limitations of the post-hoc inference setting.
Model-based reinforcement learning (RL) offers a compelling approach to offline RL by enabling value learning on imagined on-policy trajectories. However, it often suffers from compounding errors due to repeated model inference on self-generated states. While geometric horizon models (GHM) alleviate this issue through direct prediction over a discounted infinite-horizon future, they remain challenged in accurately modeling distant future states. To this end, we introduce universal horizon models (UHM), a generalization of GHM that directly predicts future states under arbitrary horizons. Leveraging this flexibility, we propose a scalable value learning method that employs a winsorized horizon distribution to stabilize training by capping excessively large horizons. Experimental results on 100 challenging OGBench tasks demonstrate that the proposed method outperforms competitive baselines, particularly on tasks with highly suboptimal datasets and those requiring long-horizon reasoning. Project page: https://rllab-snu.github.io/projects/UHM/
Hojun Chung, Junseo Lee, Songhwai Oh
Interdisciplinary Program in Artificial Intelligence and ASRI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University