cs.LGJun 4, 2025

PPO in the Fisher-Rao geometry

Authors: Razvan-Andrei Lascu, David Šiška, Łukasz Szpruch

Abstract

Proximal Policy Optimization (PPO) is widely used in reinforcement learning due to its strong empirical performance, yet it lacks formal guarantees for policy improvement and convergence. PPO's clipped surrogate objective is motivated by a lower bound on linearization of the value function in flat geometry setting. We derive a tighter surrogate objective and introduce Fisher-Rao PPO (FR-PPO) by leveraging the Fisher-Rao (FR) geometry. Our scheme provides strong theoretical guarantees, including monotonic policy improvement. In the direct parametrization setting, we show that FR-PPO achieves sub-linear convergence, and for parametrized policies we further obtain sub-linear convergence up to the compatible function approximation error. Finally, although our primary focus is theoretical, we also demonstrate empirically that FR-PPO can be implemented as a drop-in replacement for PPO and achieves performance comparable to PPO across a range of standard reinforcement learning tasks.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

    Oct 9, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +5RL for Language ModelsRL for Language Model Reasoning

  2. Bounded Ratio Reinforcement Learning

    Apr 20, 2026Yunke Ao, Le Chen, Bruce D. Lee +5Reinforcement LearningGroup Relative Policy Optimization

  3. Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

    Mar 13, 2026Zelal Su, Mustafaoglu, Sungyoung Lee +3Policy OptimizationProximal Policy Optimization