RL for Language Model Reasoning

RL: Reinforcement Learning

This field studies how reward feedback can teach language models to solve multi-step problems, use tools, and decide how much reasoning a task needs. A common approach is reinforcement learning from verifiable rewards (RLVR), often using group-relative policy optimization, where sampled solutions are scored for correctness; other methods add step-level rewards, process verifiers, search, or teacher guidance to address sparse feedback. Work also explores curricula, adaptive rollout budgets, self-distillation, and controls for reasoning length and diversity. Key open tensions include balancing exploration with stable learning, ensuring rewards and verifiers do not encourage shortcuts, and improving generalization without excessive computation or overthinking.

Latest papers

1,103 papers170 papers in the last four weeks, up 124% on the four weeks before. 1.0% of all new papers.

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Mar 23, 2026Kexin Huang, Haoming Meng, Junkang Wu +10RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Alibaba Group
P^2O: Joint Policy and Prompt Optimization
Mar 23, 2026Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2
Adaptive Robust Estimator for Multi-Agent Reinforcement Learning
Mar 23, 2026Zhongyi Li, Wan Tian, Jingyu Chen +8Multi-Agent Reinforcement Learning RL for Language Model Reasoning
Beihang University · Peking University · Chinese Academy of Sciences +1
Counterfactual Credit Policy Optimization for Multi-Agent Collaboration
Mar 23, 2026Zhongyi Li, Wan Tian, Jinju Chen +4Counterfactual Credit Assignment RL for Language Model Reasoning
Beihang University · Peking University · Beijing University of Posts and Telecommunications
KG-Hopper: Empowering Compact Open LLMs with Knowledge Graph Reasoning via Reinforcement Learning
Mar 22, 2026Shuai Wang, Yinan YuLLM Reasoning with Graphs Knowledge Graph Question Answering
Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, SE-41296 Gothenburg, Sweden
When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning
Mar 22, 2026Zhengxian Wu, Kai Shi, Chuanrui Zhang +10Multimodal Reasoning Self-Training for Language Models
OPPO AI Center · Tsinghua University · Nanyang Technological University +1
WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement
Mar 22, 2026Fangyuan Li, Pengfei Li, Shijie Wang +4Language Model Self-Improvement RL for Language Model Reasoning
Harbin Institute of Technology · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory
LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning
Mar 22, 2026Jianing Wang, Jianfei Zhang, Qi Guo +24Large Language Model-Guided Theorem Proving Automated Theorem Proving
Meituan LongCat Team
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
Mar 21, 2026Kaiyuan Li, Jing-Cheng Pang, Yang YuLanguage Model Reasoning Evaluation RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China
Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
Mar 20, 2026Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2Hindsight Experience Replay RL for Language Model Reasoning
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences
Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States
Mar 20, 2026Yurun Yuan, Tengyang XieRL for Language Model Reasoning RL Post-Training
UW-Madison
What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
Mar 20, 2026Dong Yan, Jian Liang, Yanbo Wang +3RL for Language Model Reasoning Test-Time RL
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
Mar 20, 2026Chiyu Ma, Shuo Yang, Kexin Huang +7RL for Language Model Reasoning RL for Language Models
Qwen Pilot Team, Alibaba Group
EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models
Mar 20, 2026J. Ben Tamo, Yuxing Lu, Benoit L. Marteau +2LLM Hallucination Mitigation LLM Grounding
Georgia Institute of Technology · Pekin University
Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL
Mar 19, 2026Chenlu Ye, Xuanchang Zhang, Yifan Hao +6Importance Sampling for RL RL for Language Models
University of Illinois Urbana-Champaign · Amazon
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation
Mar 19, 2026Zhuolin Yang, Zihan Liu, Yang Chen +14RL for Language Model Reasoning RL for LLM Agents
MoRI: Learning Motivation-Grounded Reasoning for Scientific Ideation in Large Language Models
Mar 19, 2026Chenyang Gu, Jiahao Cheng, Meicong Zhang +3Scientific Reasoning in Language Models Large Language Model-Guided Scientific Discovery
School of Economics and Management, East China Normal University
Context Bootstrapped Reinforcement Learning
Mar 19, 2026Saaket Agashe, Jayanth Srinivasa, Gaowen Liu +2RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of California, Santa Barbara · Cisco Research
Are complicated loss functions necessary for teaching LLMs to reason?
Mar 19, 2026Gabriele Carrino, Andrea Sassella, Nicolo Brunello +2RL for Language Model Reasoning Group Relative Policy Optimization
DEIB, Politecnico di Milano Via Ponzio 34/5, 20133, Milano (MI), Italy
Thinking with Constructions: A Benchmark and Policy Optimization for Visual-Text Interleaved Geometric Reasoning
Mar 19, 2026Haokun Zhao, Wanshi Xu, Haidong Yuan +3Visual Spatial Reasoning Mathematical Reasoning
College of Computer Science and Artificial Intelligence, Fudan University · School of ECE, Peking University · School of Software and Microelectronics, Peking University +1
Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning
Mar 19, 2026Yinan Xia, Haotian Zhang, Huiming WangRL for Language Model Reasoning Efficient Language Model Reasoning
Tianjin University · Kuaishou Technology
Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
Mar 19, 2026Haechan Kim, Soohyun Ryu, Gyouk Chu +2Reinforcement Learning RL for Language Model Reasoning
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea · KRAFTON, Seoul, South Korea · AITRICS, Seoul, South Korea
PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching
Mar 19, 2026Ruishuo Chen, Yu Chen, Zhuoran Li +1RL for Language Models Reinforcement Learning
Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China
Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum
Mar 18, 2026Nived Rajaraman, Audrey Huang, Miro Dudik +3Efficient Language Model Training Curriculum RL
Microsoft Research · University of Illinois Urbana Champaign
MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models
Mar 18, 2026Philippe Formont, Maxime Darrin, Ismail Ben Ayed +1Molecular Generation Molecule Design
Universit´e Paris-Saclay · ´ETS Montr´eal · ILLS – International Laboratory on Learning Systems +4
CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution
Mar 18, 2026Teng Pan, Yuchen Yan, Zixuan Wang +6Language Model Self-Verification RL for Language Model Reasoning
Zhejiang University · Baidu Inc.
TRiMS: Real-Time Tracking of Minimal Sufficient Length for Efficient Reasoning via RL
Mar 18, 2026Tingcheng Bian, Jinchang Luo, Mingquan Cheng +5Efficient Language Model Reasoning CoT Compression
Baidu Inc. · Shenzhen University · Harbin Institute of Technology
InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning
Mar 18, 2026Chengwei Wei, Jung-jae Kim, Longyin Zhang +2Efficient Language Model Reasoning RL for Language Model Reasoning
Institute for Infocomm Research (I2R), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore
Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations
Mar 18, 2026Haozheng Luo, Yimin Wang, Jiahao Yu +2LLM Safety Alignment Jailbreak Defense
Northwestern University · University of Michigan · Illinois Institute of Technology
PRISM: Demystifying Retention and Interaction in Mid-Training
Mar 17, 2026Bharat Runwal, Ashish Agrawal, Anurag Roy +1LLM Training Scientific Reasoning in Language Models
IBM Research, MIT-IBM Watson AI Lab

Mar 23, 2026J/K move · Enter open · S save

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Kexin Huang, Haoming Meng, Junkang Wu, Jinda Lu+9

Alibaba Group

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference

Δ\log p

between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that

Δ\log p

more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a test-time extrapolation method that amplifies the policy along the learned

Δ\log p

direction to improve reasoning accuracy without further training; (2) a training-time reweighting method that focuses learning on low-probability (corresponding to higher

Δ\log p

) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.