Policy Optimization

Follow this topic Ask about this topic

Policy optimization studies how to update a decision-making rule so it earns higher rewards while meeting constraints such as safety, fairness, or limited data. Common approaches include policy gradients, actor-critic methods, trust regions and mirror descent, as well as policy iteration and newer generative-policy methods. Work spans continuous-control benchmarks such as MuJoCo, offline and multi-agent settings, and language-model training for mathematical reasoning, coding, and tool use, where token-, segment-, or trajectory-level rewards guide updates. A central challenge is balancing effective exploration and sample reuse with stable updates, useful credit assignment, and improvements that carry over from training to deployment.

Latest papers

224 papers32 papers in the last four weeks, up 52% on the four weeks before. 0.2% of all new papers.

DDO-RM: Distribution-Level Policy Improvement after Reward Learning
Apr 13, 2026Tiantian Zhang, Jierui Zuo, Michael Chen +1Policy Optimization KL-Regularized RL
Department of Computer Science Columbia University, New York, NY 10027, USA · Department of Management Science and Engineering Tsinghua University, Beijing, China · Department of Computer Science Carnegie Mellon University, Pittsburgh, PA 15213, USA
Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
Apr 11, 2026Yiming Huang, Zhenbo Shi, Shuzheng Gao +3RL for Language Model Reasoning Policy Optimization
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory
Adaptive Simulation Experiment for LLM Policy Optimization
Apr 9, 2026Mingjie Hu, Siyang Gao, Jian-qiang Hu +1Active Experimental Design Policy Optimization
School of Management, Fudan University · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · Department of Systems Engineering, City University of Hong Kong
Target Policy Optimization
Apr 7, 2026Jean KaddourPolicy Optimization RL Post-Training
Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning
Apr 2, 2026Yuhui Chen, Haoran Li, Zhennan Jiang +4Policy Optimization Robotic RL
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · CFCS, School of Computer Science, Peking University, Beijing 100871
Policy Improvement Reinforcement Learning
Apr 1, 2026Huaiyang Wang, Xiaojie Li, Xiaohan Wang +10RL for Language Models RL for Language Model Reasoning
Beihang University · Meituan · Peking University
Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schrödinger Bridge
Mar 23, 2026Yuehu Gong, Zeyuan Wang, Yulin Chen +3Policy Optimization Continuous Control
School of Data Science, Fudan University · Laboratory for Big Data and Decision, National University of Defense Technology · ShanghaiTech University +2
PolicySim: An LLM-Based Agent Social Simulation Sandbox for Proactive Policy Optimization
Mar 20, 2026Renhong Huang, Ning Tang, Jiarong Xu +6Large Language Model-Based Social Simulation Policy Optimization
Zhejiang University, Hangzhou, China · Fudan University, Shanghai, China · HKUST, Hong Kong, China +3
Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL
Mar 19, 2026Chenlu Ye, Xuanchang Zhang, Yifan Hao +6Importance Sampling for RL RL for Language Models
University of Illinois Urbana-Champaign · Amazon
Evaluating Model-Free Policy Optimization in Masked-Action Environments via an Exact Blackjack Oracle
Mar 19, 2026Kevin SongPolicy Optimization Reinforcement Learning
Department of Biomedical Engineering The University of Alabama at Birmingham
Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
Mar 18, 2026Ting Gao, Stavros Orfanoudakis, Nan Lin +3Policy Optimization Maximum Entropy RL
AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization
Mar 18, 2026Dailan He, Guanlin Feng, Xingtong Ge +5RL for Video Generation Streaming Video Generation
CUHK MMLab, China · Vivix Group Limited, China · HKUST, China +1
Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
Mar 13, 2026Zelal Su, Mustafaoglu, Sungyoung Lee +3Policy Optimization Proximal Policy Optimization
Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin
Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces
Mar 10, 2026Ji Gao, Caleb Ju, Guanghui Lan +1Reinforcement Learning Policy Optimization
School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, GA, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, GA, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, GA, USA
Fibration Policy Optimization
Mar 9, 2026Chang Li, Tshihao Tsu, Yaren Zhang +2Policy Optimization RL for Language Models
JD Explore Academy · Carleton University
Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning
Mar 6, 2026Xuan Li, Zhanke Zhou, Zongze Li +4Molecular Optimization Large Language Model-Guided Optimization
Hong Kong Baptist University · CMIC, Shanghai Jiao Tong University · DAMO Academy, Alibaba Group +1
EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection
Mar 5, 2026Shuo Yang, Soyeon Caren Han, Xueqi Ma +3Policy Optimization LLM Agent Self-Improvement
School of Computing and Information Systems, The University of Melbourne
HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization
Mar 4, 2026Hao Zhang, Yaru Niu, Yikai Wang +2Decentralized MARL Heterogeneous MARL
ETAIC Lab, Department of Electrical Engineering, University of Texas at Arlington, USA · Safe AI Lab, Department of Mechanical Engineering, Carnegie Mellon University, USA.
Wasserstein Proximal Policy Gradient
Mar 3, 2026Zhaoyu Zhu, Shuhan Zhang, Rui Gao +1Policy Optimization Policy Gradient Methods
Zhiyuan College, Shanghai Jiao Tong University, Shanghai 200240, China · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong, China · McCombs School of Business, The University of Texas at Austin, Austin, TX, USA
Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment
Mar 2, 2026Chenxing Wei, Jiazhen Kang, Hong Wang +8RL for Diffusion Models Policy Optimization
Shenzhen University, China · Guangming Laboratory, China · ByteDance, China +5
Stabilizing Policy Optimization via Logits Convexity
Mar 1, 2026Hongzhan Chen, Tao Yang, Yuhua Zhu +3Policy Optimization Reinforcement Learning
School of Computer Science and Engineering, Sun Yat-sen University, China · Shanghai Innovation Institute · Wechat Search, Tencent Inc, China +1
MemPO: Self-Memory Policy Optimization for Long-Horizon Agents
Feb 28, 2026Ruoran Li, Xinghua Zhang, Haiyang Yu +7Agent Memory Management Memory-Augmented Agents
Tsinghua University · Tongyi Lab, Alibaba Group
Smooth Gate Functions for Soft Advantage Policy Optimization
Feb 22, 2026Egor Denisov, Svetlana Glazyrina, Maksim Kryzhanovskiy +1Policy Optimization RL for Language Model Reasoning
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
MePoly: Max Entropy Polynomial Policy Optimization
Feb 19, 2026Hang Liu, Sangli Teng, Maani GhaffariPolicy Optimization Maximum Entropy RL
University of Michigan, Ann Arbor · University of California, Berkeley
Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models
Feb 16, 2026Mufan Xu, Kehai Chen, Xuefeng Bai +4RL for Language Model Reasoning Policy Optimization
School of Computer Science and Technology, Harbin Institute of Technology, China · Baidu Inc., Beijing, China
Difficulty-Estimated Policy Optimization
Feb 6, 2026Yu Zhao, Fan Jiang, Tianle Liu +5RL for Language Model Reasoning Policy Optimization
Alibaba International Digital Commerce · School of Software Technology, Dalian University of Technology Dalian, China
QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
Feb 4, 2026Doyeon Lee, Eunyi Lyou, Hyunsoo Cho +3Trust-Region Optimization Policy Learning
Seoul National University · Department of Artificial Intelligence / Institute for Multiscale Matter and Systems, Ewha Woman University · Geogia Institute of Technology
Data- and Variance-dependent Regret Bounds for Online Tabular MDPs
Feb 2, 2026Mingyi Li, Taira Tsuchiya, Kenji YamanishiRegret Minimization in RL Markov Decision Processes
Department of Mathematical Informatics, The University of Tokyo, Tokyo, Japan · RIKEN AIP, Tokyo, Japan
RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning
Jan 30, 2026Yuexin Bian, Jie Feng, Tao Wang +3Policy Optimization Continuous Control
University of California San Diego, La Jolla, USA
Cantelli Constrained Policy Optimization
Jan 30, 2026Rohan Tangri, Jan-Peter CalliessPolicy Optimization Constrained RL
Machine Learning Research Group, University of Oxford, Oxford, United Kingdom

Apr 13, 2026J/K move · Enter open · S save

DDO-RM: Distribution-Level Policy Improvement after Reward Learning

Tiantian Zhang, Jierui Zuo, Michael Chen, Wenping Wang

Department of Computer Science Columbia University, New York, NY 10027, USA · Department of Management Science and Engineering Tsinghua University, Beijing, China · Department of Computer Science Carnegie Mellon University, Pittsburgh, PA 15213, USA

Policy Optimization KL-Regularized RL Reward Modeling

Open paper AskPDF arXiv

Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement.