RL Post-Training

RL: Reinforcement Learning

Reinforcement learning post-training studies how to adapt pretrained language, multimodal, and action models using rewards from people, verifiers, evaluators, or environments. Common approaches include reinforcement learning from human feedback and verifiable rewards, with policy-gradient methods such as PPO and GRPO, alongside preference optimization, self-distillation, offline replay, and adaptive curricula. Work also develops critics and step-level credit assignment for long-horizon agents, as well as systems that make rollout generation and training more efficient. A central tension is improving task performance while preserving exploration, concise outputs, prior capabilities, and reliable generalization when feedback is sparse or imperfect.

Latest papers

426 papers89 papers in the last four weeks, up 134% on the four weeks before. 0.5% of all new papers.

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex
May 7, 2026Yun Qu, Qi Wang, Yixiu Mao +11RL for Language Model Reasoning Group Relative Policy Optimization
Department of Automation, Tsinghua University · 2LLM Department, Tencent
ASTOR: Multi-Task Code Reinforcement Learning via Utility-Driven Coordination
May 7, 2026Yujia Chen, Yang Ye, Xiao Chu +2RL for Code Generation Reinforcement Learning
Harbin Institute of Technology, Shenzhen · Huawei Cloud Computing Technologies Co., Ltd.
CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies
May 6, 2026Keyu Chen, Nanfei Ye, Yida Wang +4Reinforcement Learning Autonomous Driving Planning
School of Vehicle and Mobility, Tsinghua University · Li Auto Inc
Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
May 4, 2026Inoussa Mouiche, Abderaouf BahiLLM Alignment Direct Preference Optimization
WASP Lab, School of Computer Science University of Windsor, ON N9B 3P4
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Tensor Parallelism LLM Inference Acceleration
Anhui University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · University of Science and Technology of China +2
Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
May 1, 2026Yi Wang, Xinchen Li, Pengwei Xie +13Robot Policy Learning Robotic Manipulation
Shanghai Innovation Institute. · AGIBOT Finch. · Columbia University.
Exploration Hacking: Can LLMs Learn to Resist RL Training?
Apr 30, 2026Eyon Jang, Damon Falck, Joschka Braun +6RL Exploration LLM Safety
MATS · UC San Diego · Anthropic +1
Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training
Apr 30, 2026Jasmine Brazilek, Juliana SeawellMoral Reasoning in Language Models RL Post-Training
†Compassion Aligned Machine Learning (CaML)
Co-Evolving Policy Distillation
Apr 29, 2026Naibin Gu, Chenxu Yang, Qingyi Si +7Reinforcement Learning with Verifiable Rewards Multi-Teacher Knowledge Distillation
Institute of Information Engineering, CAS, 2School of Cyber Security, UCAS, 3JD.COM
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Apr 29, 2026Hayate Iso, Tiyasa Mitra, Sudipta Mondal +15Reinforcement Learning Speculative Decoding
NVIDIA
Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
Apr 29, 2026Bolian Li, Yifan Wang, Yi Ding +3RL for Language Models Reinforcement Learning
Purdue University, West Lafayette, IN, USA
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
Apr 29, 2026Tianhao Hu, Xiangcheng Liu, Youshao Xiao +15RL for Language Models Asynchronous RL
Meituan, China
Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
Apr 27, 2026Dan Shi, Zhuowen Han, Simon Ostermann +3Reinforcement Learning RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarbrücken, Germany
Compute Aligned Training: Optimizing for Test Time Inference
Apr 27, 2026Adam Ousherovitch, Ambuj TewariTest-Time Optimization Inference-Time Scaling
Department of Statistics University of Michigan
EPM-RL: Reinforcement Learning for On-Premise Product Mapping in E-Commerce
Apr 27, 2026Minhyeong Yu, Wonduk SeoE-Commerce RL Post-Training
AI Research, Enhans Seoul, South Korea
JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training
Apr 26, 2026Zhengding Hu, Hehua Ouyang, Chang Chen +6RL Post-Training Pipeline Parallelism
University of California, San Diego
Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
Apr 24, 2026Alex NikulkovReward Modeling Value Function Estimation
AI at Meta
Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
Apr 23, 2026Qinan Yu, Alexa Tartaglini, Peter Hase +2CoT Faithfulness Reinforcement Learning with Verifiable Rewards
1Stanford University · 2Schmidt Sciences
When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning
Apr 23, 2026Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay +2Offline RL RL Post-Training
College of Engineering and Computer Science University of Central Florida, Orlando, FL, USA
ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
Apr 22, 2026Shelly Golan, Michael Finkelson, Ariel Bereslavsky +2Multi-Objective Reinforcement Learning RL for Image Generation
Tel Aviv University · Lightricks · Adobe Research
EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
Apr 21, 2026Chengjun Pan, Shichun Liu, Jiahang Lin +10RL for Language Models Policy Gradient Methods
Peking University · Fudan University · Shanghai AI Lab +1
Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
Apr 20, 2026Zhenwen Liang, Yujun Zhou, Sidi Lu +3Reinforcement Learning RL for Language Model Reasoning
Tencent AI Lab · University of Notre Dame
Poly-EPO: Training Exploratory Reasoning Models
Apr 19, 2026Ifdita Hasan Orney, Jubayer Ibn Hamid, Shreya S Ramanujam +5Reinforcement Learning RL for Language Model Reasoning
Stanford University
Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs
Apr 19, 2026Carissa Cullen, Harry Garland, Alexander Roman +3Agent Reliability AI Agent Safety
University of Oxford · University College London · New College of Florida +3
A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
Apr 19, 2026Zhiyin Yu, Yuchen Mou, Juncheng Yan +17RL for Language Models RL Post-Training
1Peking University · 2Shanghai Artificial Intelligence Laboratory · 3National University of Singapore +6
SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
Apr 18, 2026Yifu Huo, Chenglong Wang, Ziming Zhu +9Reinforcement Learning RL for Language Model Reasoning
1Northeastern University, Shenyang, China · 2CAS Key Laboratory of Behavioral Science, Beijing, China · 3Independent Researcher, Beijing, China +1
Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
Apr 18, 2026Weiyu Ma, Yongcheng Zeng, Yan Song +4Reinforcement Learning RL Post-Training
King Abdullah University of Science and Technology (KAUST) · Chinese Academy of Sciences, Institute of Automation (CASIA) · AI Centre, Department of Computer Science, University College London +1
Beyond Distribution Sharpening: The Importance of Task Rewards
Apr 17, 2026Sarthak Mittal, Leo Gagnon, Guillaume LajoieReinforcement Learning RL for Language Model Reasoning
1Mila · 2Universit´e de Montr´eal
S-GRPO: Unified Post-Training for Large Vision-Language Models
Apr 17, 2026Yuming Yan, Kai Tang, Sihong Chen +4VLM Adaptation Group Relative Policy Optimization
Tencent Shenzhen, Guangdong, China

May 7, 2026J/K move · Enter open · S save

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Yun Qu, Qi Wang, Yixiu Mao, Heming Zou+10

Department of Automation, Tsinghua University · 2LLM Department, Tencent

RL for Language Model Reasoning Group Relative Policy Optimization Policy Optimization Reinforcement Learning with Verifiable Rewards RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-based policy gradient is prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines a target distribution on the response simplex and projects toward it via first-order approximation. Building on this insight, we propose Listwise Policy Optimization (LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting the proximal RL objective to the response simplex, and then projects the policy via exact divergence minimization. This framework provides (i) monotonic improvement on the listwise objective with bounded, zero-sum, and self-correcting projection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typical policy gradient baselines under matched targets, while intrinsically preserving optimization stability and response diversity.