RL Post-Training

RL: Reinforcement Learning

Reinforcement learning post-training studies how to adapt pretrained language, multimodal, and action models using rewards from people, verifiers, evaluators, or environments. Common approaches include reinforcement learning from human feedback and verifiable rewards, with policy-gradient methods such as PPO and GRPO, alongside preference optimization, self-distillation, offline replay, and adaptive curricula. Work also develops critics and step-level credit assignment for long-horizon agents, as well as systems that make rollout generation and training more efficient. A central tension is improving task performance while preserving exploration, concise outputs, prior capabilities, and reliable generalization when feedback is sparse or imperfect.

Latest papers

426 papers89 papers in the last four weeks, up 134% on the four weeks before. 0.5% of all new papers.

From $P(y|x)$ to $P(y)$ : Investigating Reinforcement Learning in Pre-train Space
Apr 15, 2026Yuqiao Tan, Minzheng Wang, Bo Liu +5Reinforcement Learning RL for Language Model Reasoning
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore +1
GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification
Apr 15, 2026Wangjie Gan, Miao Pan, Linbo Xi +4RL Post-Training RL for Language Models
School of Software Technolog,Zhejiang University
SCORP: Scene-Consistent Multi-agent Diffusion Planning with Stable Online Reinforcement Post-Training for Cooperative Driving
Apr 13, 2026Haojie Bai, Aimin Li, Ruoyu Yao +5Autonomous Driving Planning Autonomous Driving
the School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518071, China · the Middle East Technology University (METU), Ankara, 06800, Turkiye · the Robotics and Autonomous Systems Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China +1
Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale
Apr 13, 2026Liujie Zhang, Benzhe Ning, Rui Yang +8Reinforcement Learning RL Post-Training
AI Platform, Xiaohongshu Inc · The University of Hong Kong · University of Science and Technology of China
Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration
Apr 13, 2026Zhipeng Chen, Tao Qian, Wayne Xin Zhao +1RL Post-Training Reinforcement Learning with Verifiable Rewards
Gaoling School of Artificial Intelligence, Renmin University of China. · China University of Mining and Technology (Beijing).
Efficient RL Training for LLMs with Experience Replay
Apr 9, 2026Charles Arnal, Vivien Cabannes, Taco Cohen +2Experience Replay RL Post-Training
FAIR at Meta · NYU Courant Institute and CDS
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China
SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility
Apr 9, 2026Xuyang Zhi, Peilun zhou, Chengqiang Lu +10RL Post-Training Multi-Objective Language Model Alignment
University of Science and Technology of China · Xiaohongshu Inc.
An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeRL for Code Generation Reinforcement Learning
Handshake AI · ETH Zurich
Target Policy Optimization
Apr 7, 2026Jean KaddourPolicy Optimization RL Post-Training
The Role of Generator Access in Autoregressive Post-Training
Apr 6, 2026Amit Kiran RegeLanguage Model Post-Training Autoregressive Generation
Department of Computer Science, University of Colorado Boulder, Boulder, Colorado 80309
A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
Apr 2, 2026Ming Lei, Christophe BaehrEntropy Regularization RL for Language Model Reasoning
Shanghai Jiao Tong University, Shanghai · M´et´eo France/CNRS, Universit´e de Toulouse, CNRM UMR 3589, Toulouse
Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning
Apr 2, 2026Rafael Pardinas, Ehsan Kamalloo, David Vazquez +1LLM Inference Efficiency RL for Language Model Reasoning
ServiceNow AI Research
Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
Apr 2, 2026Dong Shu, Denghui Zhang, Jessica HullmanRL Post-Training RL for Language Model Reasoning
Northwestern University · Stevens Institute of Technology
Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training
Apr 2, 2026William Hoy, Binxu Wang, Xu PanEvolution Strategies RL Post-Training
Department of Industrial and Systems Engineering, University of Miami · Kempner Institute, Harvard University · Harvard University
Policy Improvement Reinforcement Learning
Apr 1, 2026Huaiyang Wang, Xiaojie Li, Xiaohan Wang +10RL for Language Models RL for Language Model Reasoning
Beihang University · Meituan · Peking University
ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training
Mar 31, 2026Rui Ai, Yu Pan, David Simchi-Levi +1RL Post-Training Reward Design for RL
MIT · University of Sydney · University of Texas at Dallas
SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
Mar 24, 2026Yiqi Zhang, Huiqiang Jiang, Xufang Luo +7RL for Language Models RL for Language Model Reasoning
National University of Singapore · Microsoft Research Asia · The Chinese University of Hong Kong
On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Mar 23, 2026Kexin Huang, Haoming Meng, Junkang Wu +10RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Alibaba Group
PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
Mar 22, 2026Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang +9RL Post-Training RL for LLM Agents
NVIDIA · UC Berkeley
Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts
Mar 22, 2026Andrei Baroian, Rutger BergerPrompt Selection for RL RL Post-Training
Leiden Institute of Advanced Computer Science Leiden University
Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States
Mar 20, 2026Yurun Yuan, Tengyang XieRL for Language Model Reasoning RL Post-Training
UW-Madison
Are complicated loss functions necessary for teaching LLMs to reason?
Mar 19, 2026Gabriele Carrino, Andrea Sassella, Nicolo Brunello +2RL for Language Model Reasoning Group Relative Policy Optimization
DEIB, Politecnico di Milano Via Ponzio 34/5, 20133, Milano (MI), Italy
Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning
Mar 19, 2026Yinan Xia, Haotian Zhang, Huiming WangRL for Language Model Reasoning Efficient Language Model Reasoning
Tianjin University · Kuaishou Technology
Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting
Mar 17, 2026Wanpeng Zhang, Hao Luo, Sipeng Zheng +6Instance Reweighting Robot Policy Adaptation
Peking University · BeingBeyond · Tsinghua University
Learning from Mistakes: Post-Training for Driving VLA with Takeover Data
Mar 16, 2026Yinfeng Gao, Deqing Liu, Qichao Zhang +7Autonomous Driving VLMs for Autonomous Driving
University of Science and Technology Beijing · Xiaomi EV · Institute of Automation, Chinese Academy of Sciences
From $\log π$ to $π$ : Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
Mar 15, 2026Xiaoliang Fu, Jiaye Lin, Yangyi Fang +6RL for Language Model Reasoning Policy Gradient Methods
Meituan · Fudan University · Tsinghua University +2
GRPO and Reflection Reward for Mathematical Reasoning in Large Language Models
Mar 14, 2026Zhijie WangSelf-Reflection in Language Model Agents LLM Reasoning
School of Computer Science and Engineering, Sun Yat-sen University(SYSU), Guangzhou, China
Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
Mar 14, 2026Haitao Jiang, Wenbo Zhang, Jiarui Yao +3LLM Post-Training Supervised Fine-Tuning
North Carolina State University · UC Irvine · University of Illinois Urbana-Champaign +2
Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
Mar 13, 2026David McAllister, Miika Aittala, Tero Karras +4RL for Image Generation RL for Diffusion Models
UC Berkeley · NVIDIA

Apr 15, 2026J/K move · Enter open · S save

From $P(y|x)$ to $P(y)$ : Investigating Reinforcement Learning in Pre-train Space

Yuqiao Tan, Minzheng Wang, Bo Liu, Zichen Liu+4

Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore+1

Reinforcement Learning RL for Language Model Reasoning RL Post-Training

Open paper AskPDF arXiv

While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model's existing output distribution. Optimizing the marginal distribution P(y) in the Pre-train Space addresses this bottleneck by encoding reasoning ability and preserving broad exploration capacity. Yet, conventional pre-training relies on static corpora for passive learning, leading to a distribution shift that hinders targeted reasoning enhancement. In this paper, we introduce PreRL (Pre-train Space RL), which applies reward-driven online updates directly to P(y). We theoretically and empirically validate the strong gradient alignment between log P(y) and log P(y|x), establishing PreRL as a viable surrogate for standard RL. Furthermore, we uncover a critical mechanism: Negative Sample Reinforcement (NSR) within PreRL serves as an exceptionally effective driver for reasoning. NSR-PreRL rapidly prunes incorrect reasoning spaces while stimulating endogenous reflective behaviors, increasing transition and reflection thoughts by 14.89x and 6.54x, respectively. Leveraging these insights, we propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard RL for fine-grained optimization. Extensive experiments demonstrate that DSRL consistently outperforms strong baselines, proving that pre-train space pruning effectively steers the policy toward a refined correct reasoning subspace.