RL Post-Training

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 206% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think
May 27, 2026Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali +1Reinforcement Learning RL for Language Model Reasoning
Criteo AI Lab · Criteo AI Lab, CREST IP Paris
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
May 26, 2026Shijin Gong, Erhan Xu, Kai Ye +3RL for Language Model Reasoning Critic-Free RL
School of Management, University of Science and Technology of China · Department of Statistics, London School of Economics and Political Science · Department of Engineering Science, University of Oxford
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RL Step-Level Credit Assignment in RL
Independent Researcher · Tencent · DeepWisdom
Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Tasks
May 26, 2026Yihua Zhu, Qianying Liu, Fei Cheng +4Logical Reasoning Abductive Reasoning
Kyoto University · NII LLMC · University of Tokyo +1
Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
May 26, 2026Woojeong Kim, Ziyi Yang, Jing Nathan Yan +1Reinforcement Learning RL for Language Model Reasoning
Databricks · Cornell University
GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
May 25, 2026Yuelin Hu, Zhenbo Yu, Zhengxue Cheng +2Supervised Fine-Tuning Adaptive Loss Weighting
Shanghai Jiao Tong University · Shanghai Maritime University
Quantifying Empirical Compute-Supervision Tradeoffs in RLVR
May 24, 2026Ryo Mitsuhashi, Patrick Chen, Isabelle Tseng +2Reinforcement Learning RL for Language Model Reasoning
1Princeton University
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
May 21, 2026Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld +6Multi-Objective Reinforcement Learning Reinforcement Learning
1MIT · 2Improbable AI Lab · 4Sakana AI +1
F-TIS: Harnessing Diverse Models in Collaborative GRPO
May 21, 2026Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer +1RL for Language Models Communication-Efficient Distributed Training
Gensyn · University of Neuchatel · TU Delft
Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization
May 20, 2026Zheyuan Zhang, Kaiwen Shi, Han Bao +3RL for Language Model Reasoning Policy Optimization
University of Notre Dame · †Corresponding Author
Value-Gradient Hypothesis of RL for LLMs
May 20, 2026Arip Asadulaev, Daniil Ognev, Karim Salta +1RL for Language Models Policy Gradient
MBZUAI · Independent
You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
May 20, 2026Zhepei Wei, Xinyu Zhu, Wei-Lin Chen +3RL for Language Model Reasoning Low-Rank Approximation
University of Virginia · Washington University in St. Louis
Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor
May 19, 2026Xiaocan Li, Shiliang Wu, Zheng ShenLLM Quantization 4-Bit Quantization
Huawei Canada
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
May 19, 2026Chongyu Fan, Gaowen Liu, Mingyi Hong +2Muon Optimizer Vision-Language-Action Models
Michigan State University · Cisco · University of Minnesota +1
FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning
May 18, 2026Xikai Zhang, Yongzhi Li, Likang Xiao +6RL for Language Model Reasoning RL Exploration
Hangzhou International Innovation Institute, Beihang University · Kuaishou Technology · School of Artificial Intelligence, Beihang University
SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning
May 18, 2026Yufei Ma, Zihan Liang, Ben Chen +6On-Policy Self-Distillation RL for Language Model Reasoning
Kuaishou Technology
How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
May 17, 2026Minghao Tian, Yunfei Xie, Chen WeiRL for Language Model Reasoning Group Relative Policy Optimization
Rice University
Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training
May 16, 2026Peng Cui, Boyao Yang, Jun ZhuRL for Language Model Reasoning RL Post-Training
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing 100084, China · Dept. of Automation, Tsinghua University, Beijing 100084, China
Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning
May 16, 2026Peng Cui, Boyao Yang, Jun ZhuMultimodal Robustness VLM Reasoning
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing 100084, China · Dept. of Automation, Tsinghua University, Beijing 100084, China
GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero
May 14, 2026Shangjian Yin, Yu Fu, Yue Dong +1Reinforcement Learning RL for Language Model Reasoning
University of California, Riverside
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2GPU Acceleration Self-Attention
1Amazon Web Services · 2Google · University of Minnesota
Dynamic Latent Routing
May 14, 2026Fangyuan Yu, Xin Su, Amir AbdullahLatent Action Learning RL Post-Training
Thoughtworks AI Labs (TAILS)
Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference
May 14, 2026Tianle Zhong, Neiwen Ling, Yifan Pi +5Reinforcement Learning LLM Inference
ByteDance · The University of Virginia
Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
May 13, 2026Yang Bai, Kaiyuan Liu, Ziyuan Zhuang +5Multi-Objective Reinforcement Learning RL for Language Models
Meituan, China
Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
May 13, 2026Feng Zhang, Xinhong Ma, Ziqiang Dong +5RL for Language Model Reasoning Group Relative Policy Optimization
Beijing Institute of Technology · Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen +1
Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation
May 12, 2026Yuwei Zhang, Sha Li, Changlong Yu +9Continual Learning for LLMs Self-Distillation
UC San Diego1 · Amazon2 · Georgia Institute of Technology3
Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
May 12, 2026Rasool Fakoor, Murdock Aubry, Nicholas Stranges +1Importance Sampling Policy Optimization
Boson AI
Variance-aware Reward Modeling with Anchor Guidance
May 12, 2026Shuxing Fang, Ruijian Han, Liangyu Zhang +1Pairwise Preference Learning Reward Modeling
Department of Data Science and Artificial Intelligence, Hong Kong Polytechnic University · School of Statistics and Data Science, Shanghai University of Finance and Economics
Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping
May 11, 2026Haoyuan Sun, Jing Wang, Yuxin Song +9Reward Hacking Preference Optimization
Nanyang Technological University · Baidu Inc. · Zhejiang University +3
RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
May 11, 2026Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang +9Reinforcement Learning Rubric-Based RL
University of Illinois Urbana-Champaign · Google Cloud AI Research

May 27, 2026, cs.LGJ/K move · Enter open · S save

Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali, Flavian Vasile

Criteo AI Lab · Criteo AI Lab, CREST IP Paris

Reinforcement Learning RL for Language Model Reasoning RL Post-Training Policy Learning Off-Policy RL

Open paper AskPDF arXiv

Large scale reinforcement learning has become a central tool for improving reasoning in large language models. At this scale, generation is often lagged or asynchronous, so updates are performed on data collected by older policies. This makes learning inherently off-policy. Most existing approaches nevertheless remain rooted in PPO-style trust-region objectives, treating training as approximately on-policy and using importance weights to correct distribution mismatch. These corrections can introduce high variance, destabilize optimization, and accelerate entropy collapse. Recent work suggests an alternative: rather than correcting the mismatch, one can embrace off-policy data and remove importance weights, often yielding stronger algorithms. In this paper, we provide an intuitive construction of off-policy objectives that include successful off-policy objectives and show that their effectiveness can be understood through implicit pessimism: they optimize toward target policies that are more conservative than their nominal objectives suggest. This perspective explains why some particular implementation choices improve stability: they implicitly control the effective target distribution. We then propose a principled modification that stabilize this induced distribution and improve off-policy learning.

Momentum

Related topics

Latest papers 309