Policy Iteration

Follow this topic Ask about this topic

Policy iteration alternates between estimating how well a policy performs and changing it to choose better actions, making it a basic approach to optimal control and reinforcement learning. Methods range from exact Bellman updates and Monte Carlo policy iteration to least-squares or neural function approximation, entropy-regularized updates, and natural policy gradients; applications include Markov decision processes, stochastic control, and Hamilton–Jacobi–Bellman equations. Researchers test these methods in settings such as continuous-control and Atari benchmarks, as well as planning and multi-agent problems. A central challenge is obtaining reliable convergence and efficiency guarantees: outcomes can depend on sampling schedules, approximation error, and update rules, while some classical variants have exponential worst-case iteration counts.

Latest papers

20 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Conservative Policy Iteration For Zero-Sum POSGs
Oct 27, 2026Erwan C. Escudie, Matthia Sabatelli, Jilles Steeve DibangoyePartially Observable Stochastic Games Zero-Sum Games
University of Groningen
A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
Oct 20, 2026Chaorui Yao, Yuming Paul Zhang, Lin Yang +1Continuous-Time RL Model-Free RL
Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.
Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems
Oct 3, 2026Qi Feng, Gu WangStochastic Optimal Control Policy Iteration
Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609
Policy Iteration Is Not Strongly Polynomial for Deterministic Markov Decision Processes: The Price of Algorithmic Anarchy
Sep 30, 2026Han Zhong, Yinyu YeMarkov Decision Processes Policy Iteration
Shanghai Jiao Tong University. · Shanghai Jiao Tong University, SIMIS, and Stanford University.
Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
Sep 14, 2026Jin Ma, Gaozhan Wang, Jianfeng Zhang +1Reinforcement Learning Stochastic Optimal Control
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027
Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement
Sep 11, 2026Hongyao Tang, Yi Ma, Pengyi Li +1Self-Improving Agents Recursive Self-Improvement
Tianjin University · Shanxi University
Exact Model-Free Policy Iteration for Co-safe LTL Planning
Aug 5, 2026Zetong Xuan, Yu WangReinforcement Learning Policy Iteration
Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL 32611 USA
Scalar-Stepsize Nonuniform Monte Carlo Optimistic Policy Iteration: A Certified Counterexample
Jun 14, 2026Yuanlong ChenMarkov Decision Processes Policy Optimization
Convergence of Monte Carlo Optimistic Policy Iteration: Beyond Uniform State-Action Updates
Jun 9, 2026Octave Oliviers, Glenn VinnicombeMarkov Decision Processes Reinforcement Learning
Department of Engineering University of Cambridge Cambridge, UK
Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework
May 11, 2026Phalguni Nanda, Zaiwei ChenPolicy Gradient Reinforcement Learning
Edwardson School of Industrial Engineering, Purdue University
Monotone Neural Policy Iteration for High-Dimensional First-Order Hamilton--Jacobi--Bellman Equations
May 8, 2026Minseok Kim, Yeongjong Kim, Namkyeong Cho +1Neural PDE Solvers Stochastic Optimal Control
Department of Applied Artificial Intelligence, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Department of Mathematics, Pohang University of Science and Technology, Pohang, Republic of Korea. · Department of Mathematical Finance and Big Data, Gachon University, Seongnam, Republic of Korea +1
Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL
May 6, 2026Dillon Sandhu, Ronald ParrPolicy Optimization Policy Iteration
Department of Computer Science, Duke University, USA
Risk-seeking conservative policy iteration with agent-state based policies for Dec-POMDPs with guaranteed convergence
Apr 10, 2026Amit Sinha, Matthieu Geist, Aditya MahajanPartially Observable Markov Decision Processes Policy Iteration
Department of Electrical and Computer Engineering, McGill University, Montreal, Quebec · Earth Species Project
Automatic feature identification in least-squares policy iteration using the Koopman operator framework
Mar 27, 2026Christian Mugisho Zagabe, Sebastian PeitzRL Control Policy Iteration
Department of Computer Science & Lamarr Institute, TU Dortmund University, Germany
Algorithms for Deciding the Safety of States in Fully Observable Non-deterministic Problems: Technical Report
Mar 16, 2026Johannes Schmalz, Chaahat JainPolicy Iteration Model-Based Planning
Saarland University, Germany
Frozen Policy Iteration: Computationally Efficient RL under Linear $Q^π$ Realizability for Deterministic Dynamics
Feb 28, 2026Yijing Ke, Zihan Zhang, Ruosong WangRegret Minimization in RL Policy Iteration
School of EECS, Peking University · Department of Computer Science and Engineering, HKUST · CFCS and School of Computer Science, Peking University
Iterative Policy Optimization: Batch-Online Multi-Turn RL via Single-Turn RL
Nov 26, 2025Daniel R. Jiang, Ankur Samanta, Yukai Yang +3Reinforcement Learning Policy Iteration
Meta · Columbia University · FAIR at Meta
Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence
Jun 9, 2025Qi Feng, Gu WangStochastic Optimal Control Policy Iteration
Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609
Gaussian process policy iteration with additive Schwarz acceleration for forward and inverse HJB and mean field game problems
May 1, 2025Xianjin Yang, Jingguo ZhangMean Field Games Stochastic Optimal Control
Department of Computing and Mathematical Sciences, California Institute of Technology, CA, USA. · Department of Mathematics and Risk Management Institute, National University of Singapore, Singapore.
Bridging the Gap between Newton-Raphson Method and Regularized Policy Iteration
Oct 11, 2023Zeyang Li, Chuxiong Hu, Yunan Wang +4Markov Decision Processes Reinforcement Learning
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA 02139, United States · Department of Mechanical Engineering, Tsinghua University, Beijing 100084, China · College of Artificial Intelligence and the School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China

Oct 27, 2026J/K move · Enter open · S save

Conservative Policy Iteration For Zero-Sum POSGs

Erwan C. Escudie, Matthia Sabatelli, Jilles Steeve Dibangoye

University of Groningen

Partially Observable Stochastic Games Zero-Sum Games Policy Iteration Game-Theoretic RL Policy Optimization

Open paper AskPDF arXiv

Occupancy-state reductions make dynamic programming possible for zero-sum partially observable stochastic games (zs-POSGs). Existing point-based solvers usually optimise a central value function. This restores Bellman recursion, but it requires point-based updates over high-dimensional occupancy spaces. We introduce an approximate policy-iteration algorithm for finite-horizon zs-POSGs. The method maintains one policy per player and updates both jointly along occupancy trajectories induced by conservative policy updates. Instead of approximating a full central value function, each update uses a single supporting hyperplane induced by the current joint policy. The algorithm alternates local policy improvements, conservative updates, and persistent exploration. Computation is therefore concentrated on occupancy regions where the current joint policy is exploitable. On finite-horizon zero-sum POSG benchmarks, the method is competitive at small horizons and often improves the runtime-exploitability trade-off at larger horizons compared with PBVI, HSVI, and CFR+.