RL for Language Model Reasoning

RL: Reinforcement Learning

This field studies how reward feedback can teach language models to solve multi-step problems, use tools, and decide how much reasoning a task needs. A common approach is reinforcement learning from verifiable rewards (RLVR), often using group-relative policy optimization, where sampled solutions are scored for correctness; other methods add step-level rewards, process verifiers, search, or teacher guidance to address sparse feedback. Work also explores curricula, adaptive rollout budgets, self-distillation, and controls for reasoning length and diversity. Key open tensions include balancing exploration with stable learning, ensuring rewards and verifiers do not encourage shortcuts, and improving generalization without excessive computation or overthinking.

Latest papers

1,105 papers170 papers in the last four weeks, up 124% on the four weeks before. 1.0% of all new papers.

SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
Apr 18, 2026Yifu Huo, Chenglong Wang, Ziming Zhu +9Reinforcement Learning RL for Language Model Reasoning
1Northeastern University, Shenyang, China · 2CAS Key Laboratory of Behavioral Science, Beijing, China · 3Independent Researcher, Beijing, China +1
MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
Apr 18, 2026Zhaokang Liao, Yingguo Gao, Yi Yang +2RL for Language Model Reasoning Group Relative Policy Optimization
Ant Group · Zhejiang University
Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning
Apr 18, 2026Benteng Chen, Weida Wang, Shufei Zhang +2RL for Language Model Reasoning LLM Inference Acceleration
East China Normal University · Shanghai AI Laboratory · Fudan University +1
Learning to Reason with Insight for Informal Theorem Proving
Apr 17, 2026Yunhe Li, Hao Shi, Bowen Deng +8Automated Theorem Proving RL for Language Model Reasoning
City University of Hong Kong · Tsinghua University · Ke Holdings Inc. +2
Beyond Distribution Sharpening: The Importance of Task Rewards
Apr 17, 2026Sarthak Mittal, Leo Gagnon, Guillaume LajoieReinforcement Learning RL for Language Model Reasoning
1Mila · 2Universit´e de Montr´eal
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
Apr 17, 2026Max Henning Höth, Kristian Kersting, Björn Deiseroth +1CoT Faithfulness Feature Attribution
Aleph Alpha Research Lab1141 · TU Darmstadt Hessian.AI Lab1141
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
Apr 17, 2026Jiazheng Zhang, Ziche Fu, Zhiheng Xi +13Reward Modeling RL for Language Model Reasoning
College of Computer Science and Artificial Intelligence, Fudan University · Huazhong University of Science and Technology · The University of Hong Kong +3
Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning
Apr 17, 2026Yangyi Fang, Jiaye Lin, Xiaoliang Fu +2RL for Language Model Reasoning LLM Mathematical Reasoning
Tsinghua University · Fudan University · Peking University
LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
Apr 16, 2026Lukas Helff, Quentin Delfosse, David Steinmann +6Reward Hacking Logical Reasoning
TU Darmstadt · DFKI · Intrinsic +4
IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
Apr 16, 2026Zihan Liang, Yufei Ma, Ben Chen +6Retrieval-Augmented Generation RL for Language Model Reasoning
Kuaishou Technology
LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Reinforcement Learning RL for Language Model Reasoning
1Peking University · 2Shanghai Jiao Tong University · 3Beijing University of Posts and Telecommunications
Targeted Exploration via Unified Entropy Control for Reinforcement Learning
Apr 16, 2026Chen Wang, Lai Wei, Yanzhi Zhang +5Reinforcement Learning RL for Language Model Reasoning
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3
From $P(y|x)$ to $P(y)$ : Investigating Reinforcement Learning in Pre-train Space
Apr 15, 2026Yuqiao Tan, Minzheng Wang, Bo Liu +5Reinforcement Learning RL for Language Model Reasoning
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore +1
DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off
Apr 15, 2026Xiaofan Li, Ming Yang, Zhiyuan Ma +9Reinforcement Learning RL for Language Model Reasoning
East China Normal University · Ant Group · Xiamen University +1
Foresight Optimization for Strategic Reasoning in Large Language Models
Apr 15, 2026Jiashuo Wang, Jiawen Duan, Jian Wang +6Multi-Agent Reasoning RL for Language Model Reasoning
Department of Computing, Hong Kong Polytechnic University · Department of Language Science and Technology, Hong Kong Polytechnic University · Apple +3
SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization
Apr 15, 2026Xiaole Su, Kasey Zhang, Andy LyuLanguage Model Post-Training LLM Post-Training
Osmosis AI
From Prediction to Justification: Aligning Sentiment Reasoning with Human Rationale via Reinforcement Learning
Apr 15, 2026Shihao Zhang, Ziwei Wang, Jie Zhou +6Aspect-Based Sentiment Analysis RL for Language Model Reasoning
School of Computer Science and Technology, East China Normal University · Shanghai Qiji Zhifeng Co., Ltd. · Ocean University of China
Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization
Apr 14, 2026Shiping Gao, Hongzhan Chen, Xiaojun Quan +2Process Reward Models RL for Language Model Reasoning
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, Guangdong, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · Meta AI, Menlo Park, California, USA +1
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
Apr 14, 2026Xingyu Lin, Yilin Wen, Du Su +5RL for Language Model Reasoning Policy Optimization
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Calibration-Aware Policy Optimization for Reasoning LLMs
Apr 14, 2026Ziqi Wang, Xingzhou Lou, Meiqi Wu +2Language Model Calibration RL for Language Model Reasoning
National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing National Research Center for Information Science and Technology, Tsinghua University
KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
Apr 14, 2026Linhao Yu, Tianmeng Yang, Siyu Ding +8RL for Language Model Reasoning LLM Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Baidu Inc. · Institute of Information Engineering, Chinese Academy of Sciences.
Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
Apr 14, 2026Zhiyuan Zeng, Jiameng Huang, Zhangyue Yin +8Reinforcement Learning RL for Language Model Reasoning
Fudan University · M-A-P · Peking University +3
KG-Reasoner: A Reinforced Model for End-to-End Multi-Hop Knowledge Graph Reasoning
Apr 14, 2026Shuai Wang, Yinan YuLLM Reasoning with Graphs Graph Reasoning
Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, SE-41296 Gothenburg, Sweden
Solving Physics Olympiad via Reinforcement Learning on Physics Simulators
Apr 13, 2026Mihir Prabhudesai, Aryan Satpathy, Yangmin Li +6Scientific Reasoning in Language Models Synthetic Data Generation for Language Models
Carnegie Mellon University
Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games
Apr 13, 2026Keyang Zhong, Junlin Xie, Hefeng Wu +2VLM Reasoning Multimodal Reasoning
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen
Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach
Apr 13, 2026Haolin Li, Shuyang Jiang, Ruipeng Zhang +3Medical LLMs RL for Language Model Reasoning
College of Computer Science and Artificial Intelligence, Fudan University · Shanghai AI Laboratory · Department of Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine +3
Triviality Corrected Endogenous Reward
Apr 13, 2026Xinda Wang, Zhengxu Hou, Yangshijie Zhang +6RL for Language Models Reward Design for RL
Peking University · Alibaba Group · Lanzhou University
Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR
Apr 13, 2026Yuhang He, Haodong Wu, Siyi Liu +7RL for Language Model Reasoning Token-Level Credit Assignment
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.
SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
Apr 12, 2026Binbin Zheng, Xing Ma, Yiheng Liang +6On-Policy Distillation RL for Language Model Reasoning
University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University +2
Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
Apr 11, 2026Yiming Huang, Zhenbo Shi, Shuzheng Gao +3RL for Language Model Reasoning Policy Optimization
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory

Apr 18, 2026J/K move · Enter open · S save

SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models

Yifu Huo, Chenglong Wang, Ziming Zhu, Shunjie Xing+8

1Northeastern University, Shenyang, China · 2CAS Key Laboratory of Behavioral Science, Beijing, China · 3Independent Researcher, Beijing, China+1

Reinforcement Learning RL for Language Model Reasoning RL Exploration RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has emerged as a promising paradigm for training reasoning-oriented models by leveraging rule-based reward signals. However, RL training typically tends to improve single-sample success rates (i.e., Pass@1) while offering limited exploration of diverse reasoning trajectories, which is crucial for multi-sample performance (i.e., Pass@k). Our preliminary analysis reveals that this limitation stems from a fundamental squeezing effect, whereby probability mass is excessively concentrated on a narrow subset of high-reward trajectories, restricting genuine exploration and constraining attainable performance under RL training. To address this issue, in this work, we propose Steering Probability Squeezing (SPS), a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL). SPS treats on-policy rollouts as demonstrations and employs IRL to explicitly reshape the induced trajectory distribution, thereby enhancing exploration without introducing external supervision. Experiments on five commonly used reasoning benchmarks demonstrate that SPS can enable better exploration and improve Pass@k. Beyond algorithmic contributions, we provide an analysis of RL learning dynamics and identify an empirical upper bound on Pass@k, shedding light on intrinsic exploration limits in RL-based reasoning models. Our findings suggest that alternating between RL and IRL offers an effective pathway toward extending the exploration capacity of reasoning-oriented large language models.