RL for Language Model Reasoning

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

124 papers in the last four weeks, up 75% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 780

All topics

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning
May 31, 2026Yixiu Mao, Yun Qu, Qi Wang +2Reinforcement Learning RL for Language Model Reasoning
Department of Automation, Tsinghua University
Deep Research as Rubric for Reinforcement Learning
May 31, 2026Wangyi Mei, Zhouhong Gu, Zhenhan Bai +9Open-Ended Generation Reinforcement Learning
Fudan University · Xiaohongshu Inc. · Beijing University of Posts and Telecommunications
Enhancing LLM Metacognition via Cognitive Pairwise Training
May 30, 2026Weitao Li, Hao Zhou, Xuanyu Lei +11Pairwise Preference Learning RL for Language Model Reasoning
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · WeChat AI, Tencent Inc., China
Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
May 30, 2026Xuewei Yang, Jiachen Yu, Jie Wu +3On-Policy Self-Distillation RL for Language Model Reasoning
Tsinghua University
The Paradox of Outcome Optimization: A Causal Information-Theoretic Bound on Reasoning Shortcuts in LLMs
May 30, 2026Zihan Chen, Yiming Zhang, Wenxiang Geng +2Process Reward Models RL for Language Model Reasoning
1HFIPS, Chinese Academy of Sciences · University of Science and Technology of China
PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning
May 29, 2026Daize Dong, Junlin Chen, Haolong Jia +9RL for Language Model Reasoning Expert Routing
♣Rutgers University · ♦AMD · ⋆MBZUAI
ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
May 29, 2026Rodney Lafuente-MercadoToken-Level Credit Assignment RL for Language Model Reasoning
Scale AI
LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards
May 29, 2026Nianyi Lin, Jiajie Zhang, Lei Hou +1LLM Grounding RL for Language Model Reasoning
Tsinghua University
Are Full Rollouts Necessary for On-Policy Distillation?
May 29, 2026Yaocheng Zhang, Jiajun Chai, Yuqian Fu +7RL for Language Model Reasoning Language Model Distillation
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Meituan +1
CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO
May 29, 2026Yang Li, Gongle Xue, Yijia Guo +3RL for Language Model Reasoning Group Relative Policy Optimization
School of Software and Microelectronics, Peking University, Beijing · School of Artificial Intelligence, Peking University, Beijing · School of Computer Science, Peking University, Beijing +1
EchoRL: Reinforcement Learning via Rollout Echoing
May 29, 2026Jinhe Bi, Aniri, Minglai Yang +9Reinforcement Learning RL for Language Model Reasoning
Huawei Heisenberg Research Center · LMU Munich · Munich Center for Machine Learning +5
AdaptR1: Reinforcement Learning Based Adaptive Interleaved Thinking in Multi-hop Question Answering
May 29, 2026Yuxin Wang, Jiahao Lu, Qifeng Wu +5Multi-Hop QA RL for Language Model Reasoning
1Computer Science, Fudan University · Institute of Modern Languages and Linguistics, Fudan University · 3Shanghai Innovation Institute +1
Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoning
May 29, 2026Yuhuan Yuan, Zhouliang Yu, Minghao Liu +2Physical Reasoning Visual Spatial Reasoning
HKUST(GZ) · CUHK · ZODA
Distilling LLM Feedback for Lean Theorem Proving
May 29, 2026Gaetan Narozniak, Gérard Biau, Rémi Munos +2RL for Language Model Reasoning Mathematical Reasoning
FAIR at Meta, Inria · Sorbonne Université, Institut universitaire de France · FAIR at Meta +2
SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoning
May 29, 2026Jian Yao, Xiongcai Luo, Ran Cheng +1RL for Language Model Reasoning Efficient Language Model Inference
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Bytedance · The Hong Kong Polytechnic University-Daya Bay Technology and Innovation Research Institute, Huizhou,Guangdong Province, China.
Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
May 29, 2026Yiming Ren, Yiran Xu, Zicheng Lin +8Reinforcement Learning RL for Language Model Reasoning
Tsinghua University · Shanghai AI Laboratory · The Chinese University of Hong Kong +1
VeriGate: Verifier-Gated Step-Level Supervision for GRPO
May 28, 2026Aakriti Agrawal, Minghui Liu, Furong HuangProcess Reward Models RL for Language Model Reasoning
University of Maryland, College Park
ESPO: Early-Stopping Proximal Policy Optimization
May 28, 2026Zihang Li, Rui Zhou, Yingcheng Shi +8Early Stopping Reinforcement Learning
1Tongyi Lab , Alibaba Group · 2Peking University
Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering
May 28, 2026Shicheng Fan, Haochang Hao, Dehai Min +3Process Reward Models RL for Language Model Reasoning
University of Illinois Chicago
DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learning
May 28, 2026Yang He, Xiao Ding, Bibo Cai +5Reinforcement Learning RL for Language Model Reasoning
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
Rubric-Guided Process Reward for Stepwise Model Routing
May 28, 2026Shenghao Ye, Yu Guo, Zhengheng Li +2Process Reward Models RL for Language Model Reasoning
University of Science and Technology of China · Southeast University
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
May 28, 2026Shufan Li, Konstantinos Kallidromitis, Akash Gokul +2Token-Level Credit Assignment RL for Language Model Reasoning
UCLA · Panasonic AI Research · NVIDIA
When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer
May 28, 2026Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski +1RL for Language Models RL for Language Model Reasoning
Fin AI Research · Salk Institute for Biological Studies
OISD: On-Policy Internal Self-Distillation of Language Models
May 27, 2026Xinyu Liu, Darryl Cherian Jacob, Yang Zhou +2RL for Language Model Reasoning Self-Distillation
Auburn University · William & Mary
DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution
May 27, 2026Yunhai Hu, Zining Liu, Xiangyang Yin +5Efficient Multimodal Inference RL for Language Model Reasoning
New York University · University of Pennsylvania · Cerebras Systems
DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes
May 27, 2026Caijun Xu, Changyi Xiao, Zhongyuan Peng +1LLM Self-Correction RL for Language Model Reasoning
1Fudan University · 2Shanghai Innovation Institute
FABSVer: Faster Training and Better Self-Verification for LLM Mathematical Reasoning
May 27, 2026Haihui Pan, Junwei Bao, Hongfei Jiang +1LLM Answer Verification RL for Language Model Reasoning
Zuoyebang Education Technology
Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
May 27, 2026Yue Cheng, Jiajun Zhang, Xiaohui Gao +3Reinforcement Learning RL for Language Model Reasoning
Beijing Jiaotong University · AntGroup · Northwestern Polytechnical University +2
Plan Before Search: Search Agents Need Plan
May 27, 2026Zhipeng Qian, Zihan Liang, Yufei Ma +7Multi-Hop QA Tool-Augmented Language Model Agents
1Kuaishou Technology · 2Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · 3VCIP, CS, Nankai University
Diversifying RLVR Rollouts via First-Token Exploration
May 27, 2026Soeun Kim, Albert NoRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Artificial Intelligence, Yonsei University

May 31, 2026, cs.LGJ/K move · Enter open · S save

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

Yixiu Mao, Yun Qu, Qi Wang, Heming Zou+1

Department of Automation, Tsinghua University

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Sample-Efficient RL Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). However, its effectiveness is substantially hindered by the prevalence of ineffective training data: many sampled prompts yield response groups that are either entirely correct or entirely incorrect, resulting in zero-variance rewards and limited learning signals. Recent state-of-the-art methods address this issue through extensive LLM rollouts to filter ineffective samples, but at the cost of considerable computational overhead. Alternative approaches, including predictive sampling and trajectory replay, aim to improve data efficiency but often remain insufficient and may introduce additional issues such as systematic bias or suboptimal constraints. To address these limitations, we propose Group Prioritized Off-Policy Optimization (POPO), a simple yet effective framework that fully exploits effective training batches without additional rollout overhead. POPO comprises two key components: prioritized group replay and decoupled off-policy optimization. The former replaces ineffective on-policy groups with effective off-policy groups via a recency-based replay mechanism that jointly considers sample quality and the degree of off-policiness. To further mitigate the off-policy gap, POPO employs decoupled importance sampling to correct off-policy bias while maintaining stable policy updates under consistent trust-region constraints. Empirical evaluations across diverse reasoning tasks, including mathematics, planning, and visual geometry, demonstrate that POPO substantially accelerates RL finetuning and achieves strong reasoning performance with significantly fewer rollouts.

Momentum

Related topics

Latest papers 780