Trust Region Policy Optimization

Follow this topic Ask about this topic

Trust Region Policy Optimization studies how to improve a reinforcement-learning policy while limiting how far it moves from the current policy, often using KL-divergence constraints and surrogate objectives. Natural policy gradients and TRPO provide principled updates, while PPO-style clipping and newer divergence-based methods offer more practical approximations. These approaches are tested in continuous control such as MuJoCo, Atari games, safety-constrained and multi-agent tasks, and language-model reasoning. A central tension is how to retain useful stability or safety guarantees while scaling to long sequences, asynchronous or off-policy data, and high-dimensional policies where clipping may not accurately enforce a true trust region.

Latest papers

22 papers3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
Oct 21, 2026Chengxi Li, Hanlin Wu, Zhicheng Cai +4RL for Language Model Reasoning Proximal Policy Optimization
Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
LOCU: Lowdin-Orthogonalized Constraint Updates for Multi-Constraint Policy Optimization
Oct 19, 2026Joonyoung Lim, Younghwan YooConstrained RL Trust Region Policy Optimization
Department of Information Convergence Engineering, Pusan National University · School of Computer Science and Engineering, Pusan National University
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
Oct 9, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +5RL for Language Models RL for Language Model Reasoning
Institute for AI Industry Research (AIR), Tsinghua University SIA-Lab of Tsinghua AIR and ByteDance Seed
Group Adaptive Clipping Policy Optimization
Aug 31, 2026Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan +1Group Relative Policy Optimization Policy Optimization
University of Toronto · Amazon
Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning
Jul 24, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +5Asynchronous RL RL for Language Model Reasoning
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +2
Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
Jul 21, 2026Junyao Yang, Yucheng Shi, Zongxia Li +6Asynchronous RL Proximal Policy Optimization
1Tencent Hy LLM Frontier · 2National University of Singapore · University of Maryland, College Park +3
Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning
Jun 24, 2026Bang Giang Le, Viet Cuong TaSequential MARL Trust Region Policy Optimization
Human Machine Interaction Laboratory, VNU University of Engineering and Technology, 144 Xuan Thuy, Cau Giay, 100000, Hanoi, Vietnam.
TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning
Jun 16, 2026Zijie Meng, Ziwei Li, Yufei Liu +5Cyber-Physical Systems Reinforcement Learning
Peking University · National Taiwan University · WHU +1
$α$ -fair heterogeneous agent reinforcement learning
Jun 11, 2026Yao-hua Franck Xu, Tayeb Lemlouma, Jean-Marie Bonnin +1Algorithmic Fairness Policy Optimization
Orange Innov Lannion, France · IRISA Université de Rennes Lannion, France · IRISA IMT Atlantique Rennes, France
Rethinking the Divergence Regularization in LLM RL
Jun 8, 2026Jiarui Yao, Xiangxin Zhou, Penghui Qi +3RL for Language Models Policy Optimization
1Tencent Hunyuan · 2UIUC · 3NUS
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Jun 2, 2026Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5Non-Stationary RL Reinforcement Learning
Hong Kong University of Science and Technology · Fudan University · Mila - Québec AI Institute +2
Trust Region Q Adjoint Matching
May 26, 2026Yonghoon Dong, Kyungmin Lee, Changyeon Kim +2Reinforcement Learning KL-Regularized RL
KAIST AI · Seoul National University · RLWRLD
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement Learning Policy Optimization
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3
Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
May 9, 2026Chulabhaya Wijesundara, Andrea Baisero, Zhongheng Li +3Multi-Agent System Optimization Policy Optimization
Northeastern University · STR · University of California, Irvine
ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields
May 4, 2026Yiheng Zhang, Yiming Wang, Kaiyan Zhao +3Policy Gradient Methods Proximal Policy Optimization
University of Macau · Wuhan University · National University of Singapore +1
Bounded Ratio Reinforcement Learning
Apr 20, 2026Yunke Ao, Le Chen, Bruce D. Lee +5Reinforcement Learning Group Relative Policy Optimization
ETH Zurich · MPI for Intelligent Systems · University of Alberta +2
Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
Mar 13, 2026Zelal Su, Mustafaoglu, Sungyoung Lee +3Policy Optimization Proximal Policy Optimization
Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin
Fibration Policy Optimization
Mar 9, 2026Chang Li, Tshihao Tsu, Yaren Zhang +2Policy Optimization RL for Language Models
JD Explore Academy · Carleton University
BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
Mar 5, 2026Yuan Li, Bo Wang, Yufei Gao +4RL for Language Models Proximal Policy Optimization
Fudan University · Shanghai Innovation Institute
QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
Feb 4, 2026Doyeon Lee, Eunyi Lyou, Hyunsoo Cho +3Trust-Region Optimization Policy Learning
Seoul National University · Department of Artificial Intelligence / Institute for Multiscale Matter and Systems, Ewha Woman University · Geogia Institute of Technology
SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
Dec 29, 2025Dominik Wagner, Ankit Kanwar, Luke OngSafety-Critical Control Policy Optimization
NTU Singapore · Sony Corporation
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Dec 28, 2025Yingru Li, Jiacai Liu, Jiawei Xu +4RL for Language Models Reinforcement Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen

Oct 21, 2026J/K move · Enter open · S save

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao+3

Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed

RL for Language Model Reasoning Proximal Policy Optimization Trust Region Policy Optimization Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate

r-1-\log r

, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at https://github.com/LiChengxi666/EPIC.