RL Post-Training

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 206% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
Jul 15, 2026Patrick Wilhelm, Odej KaoGroup Relative Policy Optimization RL Post-Training
BIFOLD Technische Universität Berlin Germany
Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
Jul 14, 2026Xinyu Tang, Qianggang Cao, Yurou Liu +13RL for Language Models RL for Language Model Reasoning
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Ant Group · 3Tsinghua University +1
Learning Residual Kinematic Corrections for Continuous Neural Decoding via Reinforcement Learning
Jul 13, 2026Jiamian Li, Niall McShane, Attila Korik +6Brain-Computer Interfaces Neural Decoding
Department of Computer Science, University of Bath, Bath, BA2 7AY · Intelligent Systems Research Centre, University of Ulster, BT48 7JL · Bath Institute for the Augmented Human, University of Bath, Bath, BA2 7AY +1
Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals
Jul 13, 2026Daocheng Fu, Rong Wu, Yu Yang +7Large Language Model-Guided Optimization RL Post-Training
Shanghai AI Laboratory · Fudan University · Zhejiang University +2
Predictive Divergence Masks for LLM RL
Jul 12, 2026Xiangxin Zhou, Jiarui Yao, Penghui Qi +4RL for Language Models Reinforcement Learning
1Tencent Hunyuan · 2UIUC · 3NUS
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
Jul 11, 2026Kexin Huang, Junkang Wu, Jinda Lu +7Reinforcement Learning KL-Regularized RL
University of Science and Technology of China · Peking University · Dartmouth College
Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards
Jul 11, 2026Pengfei Cai, Utkarsh Utkarsh, Alan Edelman +2RL for Code Generation Scientific Code Generation
Massachusetts Institute of Technology
Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
Jul 11, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +3Reinforcement Learning RL for Language Model Reasoning
Institute for AI Industry Research (AIR), Tsinghua University · ByteDance Seed · SIA-Lab of Tsinghua AIR and ByteDance Seed.
REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation
Jul 11, 2026Li Guo, Anas M. Tahir, Z. Jane WangRadiology Report Generation Radiology VLMs
University of British Columbia, Vancouver, BC, Canada
PAC-ACT: Post-training Actor-Critic for Action Chunking Transformers
Jul 10, 2026Yujie Pang, Zudong LiReinforcement Learning Contact-Rich Robotic Manipulation
Dept. of Mechanical and Energy Engineering · Southern University of Science and Technology · Shenzhen, China
Mach-Mind-4-Flash Technical Report
Jul 10, 2026Foundation Model TeamAgentic RL Token Efficiency
Foundation Model, Li Auto Inc.
Post-Training in End-to-End Autonomous Driving
Jul 9, 2026Ruining Yang, Muxing Wang, Yixiao Chen +8End-to-End Autonomous Driving Autonomous Driving
Northeastern University, Boston, USA · Purdue University, West Lafayette, USA
RL Post-Training Builds Compositional Reasoning Strategies
Jul 8, 2026Azwar Abdulsalam, Nishil Patel, Andrew SaxeCompositional Reasoning RL for Language Model Reasoning
Gatsby Computational Neuroscience Unit, UCL, London, United Kingdom.
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic RL Asynchronous RL
Tsinghua University · Work done while ZH and YL interned at Z.AI.
UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma
Jul 8, 2026Chongyu Fan, Pengfei Liu, Jingjia Huang +2RL for Language Model Reasoning Policy Optimization
ByteDance Seed · Michigan State University
Optimal Transport Q-Learning for Flow Policy Steering and Acceleration
Jul 7, 2026Andreas Sochopoulos, Esmeralda S. Whitammer, Nikolaos Tsagkas +3Robot Policy Adaptation Q-Learning
University of Edinburgh · 2Honda Research Institute Europe
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Jul 6, 2026Yu Li, Xiuyu Li, Mingyang Yi +4LLM Alignment Importance Sampling
Renmin University of China
RL Forgets! Towards Continual Policy Optimization
Jul 5, 2026Mao-Lin Luo, Zhe-Xu Wang, Zi-Hao Zhou +4RL Post-Training Continual Learning for VLMs
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1
Spectral Rewiring for Exploration, Purification, and Model Merging
Jul 3, 2026Zhilong Zhang, Hongli Yu, Huan-ang Gao +5RL for Language Model Reasoning RL Post-Training
SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Jul 2, 2026Meng Wang, Haohan Zhao, Wenzhuo Liu +7Continual Learning for LLMs On-Policy Self-Distillation
Centre for Artificial Intelligence and Robotics, HKISI, CAS · Institute of Automation, CAS · University of Chinese Academy of Sciences +1
Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL
Jul 1, 2026Juliette Decugis, Sean O'Brien, Francis Bach +2Policy Gradient RL for Language Model Reasoning
FAIR at Meta · Work done at Meta, now working at the University of California, San Diego · Inria, Ecole Normale Supérieure
Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Jul 1, 2026Ye Liu, Srijan Bansal, Bo Pang +6Language Model Distillation Reinforcement Learning with Verifiable Rewards
Salesforce AI Research
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Jul 1, 2026Zijian Zhang, Rizhen Hu, Athanasios Glentis +4RL for Language Models Fine-Tuning
University of Minnesota · Peking University · Amazon
Scaling Laws for Collapse in Asynchronous GRPO
Jul 1, 2026Jingwei Song, Haofeng Xu, Jie Xiao +7Asynchronous RL Reinforcement Learning
The University of Hong Kong · Shanghai Jiao Tong University · Gradient +2
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
Jun 29, 2026Wenhan Ma, Jianyu Wei, Liang Zhao +10Multi-Teacher Knowledge Distillation On-Policy Distillation
Peking University · Xiaomi · LLM Core, Xiaomi +2
The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Jun 28, 2026Jing Liang, Hongyao Tang, Yi Ma +9Reinforcement Learning RL for Language Model Reasoning
1Tianjin · 2Alibaba
ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents
Jun 26, 2026Qitai Tan, Zefang Zong, Yang Li +1Agentic RL On-Policy Distillation
Tencent Inc. · Tsinghua University
NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning
Jun 26, 2026Tianlin Pan, Lianyu Pang, Cheng Da +4Flow Matching RL for Image Generation
The Hong Kong University of Science and Technology · Kuaishou Technology · University of Chinese Academy of Sciences
Reinforcement Learning without Ground-Truth Solutions can Improve LLMs
Jun 25, 2026Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang +6RL for Code Generation Reinforcement Learning
University of California, San Diego · Snowflake AI Research
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
Jun 25, 2026Rongjian Chen, Jianmin Hu, Kejiang Ye +1Reinforcement Learning RL for Language Model Reasoning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · University of Chinese Academy of Sciences, Beijing, China · Southern University of Science and Technology, Shenzhen, China

Jul 15, 2026, cs.LGJ/K move · Enter open · S save

Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback

Patrick Wilhelm, Odej Kao

BIFOLD Technische Universität Berlin Germany

Group Relative Policy Optimization RL Post-Training

Open paper AskPDF arXiv

Reinforcement Learning (RL) post-training is increasingly used to adapt foundation models for reasoning, planning, and feedback-driven robot-learning pipelines, but constrained post-training resources are often summarized by a single total FLOP budget. We study the fixed-budget decision problem behind this practice: under the same post-training budget, should one use a larger policy, train a smaller policy longer, generate more rollout search, or spend compute on stronger reward feedback? We introduce a FLOP-accounting framework for GRPO post-training that decomposes compute into rollout/search, policy-update/learning, and reward- or feedback-model evaluation. Across LoRA-adapted Qwen2.5 policies, we find conditional allocation frontiers: the best observed allocation changes with model size, compute budget, reward system, and evaluation target. Same-FLOP model-size comparisons show that model choice and training allocation are coupled because larger policies consume more per-token compute and therefore buy fewer updates or rollouts under the same budget. Reward systems also change the accounting: rule-based rewards spend nearly all non-update compute on policy rollouts, while PRM-style feedback allocates a visible part of the budget to reward-model inference. We present RACE as a diagnostic pilot-grid protocol, not a guarantee of held-out improvement, for identifying allocation regimes before expensive validation runs; our results suggest that RL post-training papers should report total FLOPs together with how compute is divided among model size, search, learning, and feedback.

Momentum

Related topics

Latest papers 309