RL Post-Training

RL: Reinforcement Learning

Reinforcement learning post-training studies how to adapt pretrained language, multimodal, and action models using rewards from people, verifiers, evaluators, or environments. Common approaches include reinforcement learning from human feedback and verifiable rewards, with policy-gradient methods such as PPO and GRPO, alongside preference optimization, self-distillation, offline replay, and adaptive curricula. Work also develops critics and step-level credit assignment for long-horizon agents, as well as systems that make rollout generation and training more efficient. A central tension is improving task performance while preserving exploration, concise outputs, prior capabilities, and reliable generalization when feedback is sparse or imperfect.

Latest papers

426 papers89 papers in the last four weeks, up 134% on the four weeks before. 0.5% of all new papers.

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL
Mar 12, 2026Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12Reinforcement Learning RL Post-Training
UC San Diego · MBZUAI-IFM · Carnegie Mellon University
FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning
Mar 12, 2026Yijun Pan, Weikang Qiu, Qiyao Ma +4RL for Recommendation Recommender Systems
Yale University · University of California, Davis · Snap Inc.
Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
Mar 11, 2026Zichao Li, Jie Lou, Fangchen Dong +8RL for Language Models RL Post-Training
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.
Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHF
Mar 10, 2026Keertana Chidambaram, Sanath Kumar Krishnamurthy, Qiuling Xu +2Recommender Systems Generative Recommendation
Department of Management Science & Engineering, Stanford University, CA, USA · Meta, CA, USA · Netflix Research, CA, USA
GIPO: Gaussian Importance Sampling Policy Optimization
Mar 4, 2026Chengxuan Lu, Zhenquan Zhang, Shukuan Wang +3Reinforcement Learning Importance Sampling for RL
1IROOTECH TECHNOLOGY · 2Wolf 1069 b Lab, Sany Group
Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models
Mar 1, 2026Adel Javanmard, Baharan Mirzasoleiman, Vahab MirrokniData Selection for Language Model Fine-Tuning LLM Fine-Tuning
University of Southern California · Google Research · University of California Los Angeles
Towards Better RL Training Data Utilization via Second-Order Rollout
Feb 26, 2026Zhe Yang, Yudong Wang, Rang Li +1RL for Language Model Reasoning RL Post-Training
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · ByteDance BandAI
Rethinking Floorplan Generation with Space Syntax: From Design Theory to Model Improvement
Feb 26, 2026Zhuoyang Jiang, Dongqing Zhang, Qingxiang Liu +1Floor Plan Generation RL Post-Training
College of Architecture and Urban Planning, Tongji University, Shanghai, China · Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Feb 24, 2026Xu Wan, Yansheng Wang, Wenqi Huang +1RL for Language Model Reasoning RL Post-Training
Zhejiang University · Bytedance Seed Robotics · Peking University +1
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
Feb 24, 2026Zhengyao Gu, Jonathan Light, Raul Astudillo +7Curriculum RL Curriculum Learning
University of Illinois Chicago · Caltech · RPI +2
LAD: Learning Advantage Distribution for Reasoning
Feb 23, 2026Wendi Li, Sharon LiRL for Language Model Reasoning RL Post-Training
Department of Computer Sciences, University of Wisconsin–Madison
LLMs Can Learn to Reason Via Off-Policy RL
Feb 22, 2026Daniel Ritter, Owen Oertell, Bradley Guo +3RL for Language Model Reasoning RL Post-Training
Cornell University · Databricks · Harvard University
Soft Sequence Policy Optimization
Feb 22, 2026Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman IschenkoLLM Alignment RL Post-Training
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Feb 19, 2026Luke J. Huang, Zhuoyang Zhang, Qinghao Hu +2RL for Language Models Asynchronous RL
MIT · NVIDIA
VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Study
Feb 18, 2026Zhicheng Zhang, Ziyan Wang, Yali Du +1RL for Language Models RL Exploration
Carnegie Mellon University · King’s College London
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
Feb 17, 2026Jialu Wang, Heinrich Peters, Asad A. Butt +6LLM Alignment Group Relative Policy Optimization
1Apple Inc., California, United States.
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
Feb 6, 2026Yingru Li, Jiawei Xu, Ziniu Li +10RL for Language Model Reasoning Policy Gradient Methods
The Chinese University of Hong Kong, Shenzhen · Fudan University · Hong Kong University of Science and Technology +4
HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning
Jan 30, 2026Weiqi Wang, Xin Liu, Binxuan Huang +13RL for Language Models RL for Language Model Reasoning
Stores Foundational AI, Amazon Inc., Palo Alto, CA, USA · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States
Jan 29, 2026Lei Yang, Wei Bi, Chenxi Sun +2RL for Language Model Reasoning RL Exploration
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Kuaishou Technology
R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Jan 17, 2026Jingchu Wang, Bingbing Xu, Yige Yuan +4RL for Language Model Reasoning Policy Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore
How Does RL Post-training Induce Skill Composition? A Case Study on Countdown
Dec 1, 2025Simon Park, Simran Kaur, Sanjeev AroraCompositional Generalization RL Post-Training
Princeton Language and Intelligence (PLI), Princeton University
Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Nov 25, 2025Chenliang Li, Adel Elmahdy, Alex Boyd +7RL for Language Model Reasoning Importance Sampling
Texas A&M University · GE HealthCare · Independent Researcher +1
Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
Oct 24, 2025Hoang Phan, Xianjun Yang, Yuanshun Yao +6RL Post-Training Catastrophic Forgetting
Meta Superintelligence Labs · New York University · Johns Hopkins University
Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
Oct 21, 2025Howard Chen, Noam Razin, Karthik Narasimhan +1Continual Learning for LLMs RL Post-Training
Princeton Language and Intelligence, Princeton University.
Representation-Based Exploration for Language Models: From Test-Time to Post-Training
Oct 13, 2025Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy +1RL for Language Model Reasoning Test-Time Scaling
Princeton University · Microsoft Research NYC
BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewards
Oct 10, 2025Sangyun Lee, Brandon Amos, Giulia FantiGenerative Modeling RL for Generative Models
Carnegie Mellon University
Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
Oct 10, 2025Xinyi Wang, Jinyi Han, Zishang Jiang +7Reinforcement Learning RL for Language Model Reasoning
School of Data Science, Fudan University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
Oct 10, 2025Yincen Qu, Huan Xiao, Feng Li +5Reward Modeling LLM Planning
Trip.com Group · Chongqing University · Stanford University
SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
Oct 9, 2025Chen Wang, Zhaochun Li, Jionghao Bai +4Reinforcement Learning RL Post-Training
College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +2
Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
Oct 2, 2025Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad +3RL for Language Model Reasoning LLM Inference Acceleration
UNC Chapel Hill · The University of Texas at Austin · Microsoft Research

Mar 12, 2026J/K move · Enter open · S save

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

Zhoujun Cheng, Yutao Xie, Yuxiao Qu, Amrith Setlur+11

UC San Diego · MBZUAI-IFM · Carnegie Mellon University

Reinforcement Learning RL Post-Training Language Model Scaling Laws

Open paper AskPDF arXiv

While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.