RL Post-Training

RL: Reinforcement Learning

Reinforcement learning post-training studies how to adapt pretrained language, multimodal, and action models using rewards from people, verifiers, evaluators, or environments. Common approaches include reinforcement learning from human feedback and verifiable rewards, with policy-gradient methods such as PPO and GRPO, alongside preference optimization, self-distillation, offline replay, and adaptive curricula. Work also develops critics and step-level credit assignment for long-horizon agents, as well as systems that make rollout generation and training more efficient. A central tension is improving task performance while preserving exploration, concise outputs, prior capabilities, and reliable generalization when feedback is sparse or imperfect.

Latest papers

426 papers89 papers in the last four weeks, up 134% on the four weeks before. 0.5% of all new papers.

Bayes-Adaptive MDPs for Rollout Allocation to Enhance GRPO Training
Oct 27, 2026Faeze Moradi Kalarde, Armin Toroghi, Nam Ho Koh +4RL Post-Training Group Relative Policy Optimization
University of Toronto · Vector Institute of AI · Korea Advanced Institute of Science & Technology
SetGRPO: Calibrating Grader-Induced Policy Optimization
Oct 27, 2026Jiale Huang, Zixu Li, Zhiwei Chen +3Reinforcement Learning Group Relative Policy Optimization
Shandong University · National University of Singapore
Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization
Oct 26, 2026Xiangchi Yuan, Xingjian Diao, Zheyuan Liu +7Multi-Objective Reinforcement Learning RL Post-Training
Georgia Institute of Technology · Dartmouth College · University of Notre Dame
DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training
Oct 25, 2026Yingping Wang, Min Cai, Chenyu Yuan +3Multi-Teacher Knowledge Distillation RL Post-Training
The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta
Intervention-Certified Amplification: A Validity Gate for $σ$ -Weighted Post-Training
Oct 24, 2026Leiyue Chen, Meng Wei, Xuefeng Hu +1RL Post-Training RL for Language Model Reasoning
ByteDance
How Does Post-Training Shape LLM Value Preferences?
Oct 23, 2026Yang Liu, Shaoyang Xu, Masahiro Kaneko +3LLM Post-Training Social Bias in Language Models
Kyoto University · Singapore University of Technology and Design · Mohamed bin Zayed University of Artificial Intelligence
Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
Oct 22, 2026Barış Alparslan, Sandy Tanwisuth, Caleb PettersonRL Post-Training Reinforcement Learning with Verifiable Rewards
ASELSAN · BlueDot Impact · University of Minnesota - Twin Cities
RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
Oct 22, 2026Jingxuan Xu, Zihang Xu, Yuechen Luo +9Code Generation RL for LLM Agents
National University of Singapore, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Tsinghua University, Beijing, China +1
Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs
Oct 21, 2026Yeonbum Kim, Hyun Kyu Lee, Sung Whan YoonRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea · Department of Electrical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, Republic of Korea.
FlashNext-RL: A System Optimization Study for Long-Horizon Agentic RL of Qwen3.8-Flash-Next
Oct 20, 2026Zhewei Yao, Yuxiong HeReinforcement Learning RL for LLM Agents
Snowflake AI Research
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Oct 20, 2026Yi Wen, Hao Chen, Wanyu Wang +9RL for Language Model Reasoning RL Post-Training
City University of Hong Kong · Tencent · Nanjing University
From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks
Oct 20, 2026Taiheng PanRL Post-Training RL for Language Models
The University of Melbourne
Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
Oct 19, 2026Haoqun Cao, Yurun Yuan, Tengyang XiePolicy Gradient Methods Off-Policy RL
University of Wisconsin–Madison
Predictive Bandit Curriculum Reinforcement Learning via Gradient-Based Sample Group Discovery
Oct 19, 2026Liang Xu, Shufan Shen, Zhongni Hou +5Curriculum RL Curriculum Learning
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan
OPDArena: Understanding Learning and Forgetting in On-Policy Distillation
Oct 19, 2026Yuanze Xu, Tianzhu Ye, Hui Sun +3On-Policy Distillation Catastrophic Forgetting
Microsoft Research · Nankai University
Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training
Oct 18, 2026Jonah Ellison, Mateo Ruiz, Owen Pell +4RL for Language Model Reasoning RL Post-Training
University of Michigan · University of Edinburgh
Value as Memory: Online Value-Guided Post-training for General Robot Policies
Oct 15, 2026Wenye Yu, Yijin Chen, Kaiyi Xiong +7Robotic RL RL Post-Training
Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.
EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search
Oct 15, 2026Hanane Nour Moussa, Doeun Lee, Sachin KumarLanguage Model Post-Training RL Post-Training
The Ohio State University
Baselines and Score Centering in Policy Gradient Estimation
Oct 15, 2026Shuo Han, Ziqing HuPolicy Gradient Policy Gradient Methods
University of Illinois Chicago · Perplexity AI
Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine
Oct 15, 2026Junyao Yang, Zongxia Li, Yucheng Shi +6Terminal Agents Recursive Self-Improvement
ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
Oct 15, 2026Yukun Tian, Junwei Su, Haofeng Xu +6RL Post-Training RL for Language Model Reasoning
The University of Hong Kong · Alibaba Group · University of Science and Technology of China +1
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Oct 15, 2026Bingxi Hou, Guochao Jiang, Guofeng Quan +4Policy Distillation On-Policy Distillation
Alibaba Cloud Computing
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Oct 15, 2026Yushu Li, Jiin Woo, Xiaoxiao Li +1Reward Hacking RL for Language Models
University of British Columbia · Amazon Annapurna Labs
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Oct 14, 2026Sheldon Yu, Tong Yu, Nikki Lijing Kuang +6RL for Language Model Reasoning RL Post-Training
University of California, San Diego · Adobe Research · University of Virginia +2
When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
Oct 14, 2026Kaifei Wang, Zikang Shan, Liwei Wang +2Bandit Learning Momentum Methods
Peking University · Shanghai Jiao Tong University, SIMIS, and Stanford University · Shanghai Jiao Tong University
Training Language Models to Fail Gracefully
Oct 14, 2026Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh +6RL for Language Models LLM Abstention
Carnegie Mellon University · UC Berkeley
Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
Oct 14, 2026Min-Young YuAI Agent Safety Reinforcement Learning with Verifiable Rewards
Corners Co., Ltd.
Contextual Credit Policy Optimization
Oct 13, 2026Huiwen Han, Ziyao Zeng, Zihao Zhu +6RL for LLM Agents Step-Level Credit Assignment in RL
Stanford University · Texas A&M University · Yale University +3
Backpropagating Verifier Rewards Through Continuous Flow Language Models
Oct 13, 2026Xu Han, Jiajing Hu, Li-Ping LiuRL for Language Model Reasoning Language Modeling
Tufts University
Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
Oct 13, 2026Yuante Li, Junwei Quan, Kai Qiu +2Reinforcement Learning Reinforcement Learning with Verifiable Rewards
Carnegie Mellon University · University of Toronto

Oct 27, 2026J/K move · Enter open · S save

Bayes-Adaptive MDPs for Rollout Allocation to Enhance GRPO Training

Faeze Moradi Kalarde, Armin Toroghi, Nam Ho Koh, Doo Hwan Hwang+3

University of Toronto · Vector Institute of AI · Korea Advanced Institute of Science & Technology

RL Post-Training Group Relative Policy Optimization RL for Language Model Reasoning

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) has demonstrated strong performance in LLM post-training. However, it allocates a uniform rollout budget to every training question, ignoring variations in question difficulty. As a result, highly difficult questions may receive too few rollouts to generate informative positive reward signals, limiting their contribution to training, while moderately difficult questions often receive more rollouts than necessary. In this work, we formulate the problem of difficulty-aware rollout budget allocation to maximize training utility under the same total rollout budget as standard GRPO. We propose AdaRoll-GRPO, which reformulates rollout allocation as a sequential decision-making problem and leverages a Bayes-adaptive Markov decision process (MDP) to learn an adaptive allocation policy. The learned policy is computed offline and applied during GRPO training without introducing additional online optimization overhead. Extensive experiments on multiple reasoning datasets and LLM architectures demonstrate that AdaRoll-GRPO consistently improves GRPO training performance under a fixed rollout budget, highlighting the effectiveness of adaptive rollout allocation for GRPO in LLM post-training.