RL for Language Model Reasoning

RL: Reinforcement Learning

This field studies how reward feedback can teach language models to solve multi-step problems, use tools, and decide how much reasoning a task needs. A common approach is reinforcement learning from verifiable rewards (RLVR), often using group-relative policy optimization, where sampled solutions are scored for correctness; other methods add step-level rewards, process verifiers, search, or teacher guidance to address sparse feedback. Work also explores curricula, adaptive rollout budgets, self-distillation, and controls for reasoning length and diversity. Key open tensions include balancing exploration with stable learning, ensuring rewards and verifiers do not encourage shortcuts, and improving generalization without excessive computation or overthinking.

Latest papers

1,103 papers170 papers in the last four weeks, up 124% on the four weeks before. 1.0% of all new papers.

When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective
Mar 17, 2026Zelin Zhang, Fei Cheng, Chenhui ChuUnsupervised RL Reinforcement Learning
Kyoto University
Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism
Mar 17, 2026Kaixuan Du, Meng Cao, Hang Zhang +3RL for Language Model Reasoning Self-Training for Language Models
School of Automation Science and Electrical Engineering, Beihang University
DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
Mar 17, 2026Long Li, Zhijian Zhou, Tianyi Wang +7Behavioral Diversity in RL Reinforcement Learning
Griffith University, Australia · Fudan University, China · Shanghai Innovation Institute, China +3
ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning
Mar 17, 2026Yu Li, Rui Miao, Zhengling Qi +1RL for Language Model Reasoning LLM Mathematical Reasoning
Department of Electrical and Computer Engineering, George Washington University · Department of Mathematical Sciences, University of Texas at Dallas · School of Business, George Washington University
SAGE: Multi-Agent Self-Evolution for LLM Reasoning
Mar 16, 2026Yulin Peng, Xinxin Zhu, Chenxing Wei +4LLM Reasoning Self-Training for Language Models
College of Computer Science and Software Engineering, Shenzhen University, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China · School of Information Technology, Carleton University, Canada
PA3: Policy-Aware Agent Alignment through Chain-of-Thought
Mar 15, 2026Shubhashis Roy Dipta, Daniel Bis, Kun Zhou +4LLM Alignment RL for Language Model Reasoning
University of Maryland, Baltimore County · Amazon Alexa AI
An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs
Mar 15, 2026Qian Zhu, Xinnan Guo, Jingjing Huo +5Domain Adaptation for LLMs LLM Alignment
Ant Group
From $\log π$ to $π$ : Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
Mar 15, 2026Xiaoliang Fu, Jiaye Lin, Yangyi Fang +6RL for Language Model Reasoning Policy Gradient Methods
Meituan · Fudan University · Tsinghua University +2
GRPO and Reflection Reward for Mathematical Reasoning in Large Language Models
Mar 14, 2026Zhijie WangSelf-Reflection in Language Model Agents LLM Reasoning
School of Computer Science and Engineering, Sun Yat-sen University(SYSU), Guangzhou, China
MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning
Mar 14, 2026Hongjun Wang, Wei Liu, Weibo Gu +2Policy Gradient Methods RL for Language Model Reasoning
The University of Hong Kong · Tencent Youtu Lab
MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
Mar 13, 2026Tianyang Luo, Tao Feng, Zhigang Hua +4RL for Language Model Reasoning RL Fine-Tuning
University of Illinois Urbana-Champaign · Meta
When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
Mar 13, 2026Yu Li, Tian Lan, Zhengling QiRL for Language Model Reasoning Group Relative Policy Optimization
Department of Electrical and Computer Engineering, George Washington University · School of Business, George Washington University
Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation
Mar 13, 2026Wayner Barrios, SouYoung JinReasoning Evaluation MLLM Reasoning
Dartmouth College, Hanover, NH, USA
Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks
Mar 13, 2026Kun Wang, Reinhard HeckelLLM Evaluation Inference-Time Language Model Alignment
School of Computation, Information and Technology, Technical University of Munich · Munich Center for Machine Learning
Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design
Mar 13, 2026Xu Guo, Qiming Ge, Jian Tong +8RL for Language Model Reasoning Multiple-Choice Question Answering
Shanghai AI Laboratory · Shanghai Innovation Institute · Fudan University
Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
Mar 13, 2026Vishnu Teja Kunde, Fatemeh Doudi, Mahdi Farahbakhsh +3RL for Diffusion Models Diffusion Language Model Training
Department of Electrical & Computer Engineering, Texas A&M University, College Station, TX 77843
LatentGeo: Learnable Auxiliary Constructions in Latent Space for Multimodal Geometric Reasoning
Mar 12, 2026Haiying Xu, Zihan Wang, Song Dai +3Multimodal Large Language Models Multimodal Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Nankai University · Communication University of China
Entropy-Preserving Reinforcement Learning
Mar 12, 2026Aleksei Petrenko, Ben Lipkin, Kevin Chen +4Policy Gradient Methods Behavioral Diversity in RL
Apple · MIT
Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating
Mar 11, 2026Yuning Wu, Ke Wang, Haoran Liu +3Reinforcement Learning RL for Language Model Reasoning
Amazon
DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
Mar 11, 2026Hanxu Hu, Yuxuan Wang, Maggie Huan +4RL for Language Model Reasoning RL Fine-Tuning
University of Zurich · University of Pennsylvania · ETH Zurich +2
TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs
Mar 11, 2026Yutao Xie, Nathaniel Thomas, Nicklas Hansen +3RL for Language Model Reasoning Reward Shaping
UC San Diego · AWS AI
Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
Mar 11, 2026Jongwoo Ko, Sara Abdali, Young Jin Kim +2On-Policy Distillation RL for Language Model Reasoning
Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
Mar 11, 2026Yixiu Mao, Yun Qu, Qi Wang +2Prompt Selection for RL RL for Language Model Reasoning
Department of Automation, Tsinghua University
mAceReason-Math: A Dataset of High-Quality Multilingual Math Problems Ready For RLVR
Mar 11, 2026Konstantin Dobler, Simon Lehnerer, Federico Scozzafava +2LLM Mathematical Reasoning Mathematical Reasoning
Apple · Hasso Plattner Institute & ELLIS Unit Potsdam
Reinforcement Learning with Conditional Expectation Reward
Mar 11, 2026Changyi Xiao, Caijun Xu, Yixin CaoRL for Language Model Reasoning LLM Answer Verification
Fudan University
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Mar 10, 2026Sijia Cui, Pengyu Cheng, Jiajun Song +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences
Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning
Mar 10, 2026Tiehua Mei, Minxuan Lv, Leiyu Pan +5RL for Language Model Reasoning LLM Reasoning
School of Data Science, Fudan University · University of Chinese Academy of Sciences · College of Intelligence and Computing, Tianjin University
C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis
Mar 10, 2026Miaosen Luo, Zhenhao Yang, Jieshen Long +3Sentiment Analysis Multimodal Sentiment Analysis
School of Computer Science, South China Normal University Guangzhou, Guangdong, China · School of Electronic and Information Engineering, South China University of Technology Guangzhou, Guangdong, China
Social-R1: Towards Human-like Social Reasoning in LLMs
Mar 10, 2026Jincenzi Wu, Yuxuan Lei, Jianxun Lian +5Social Reasoning RL for Language Model Reasoning
The Chinese University of Hong Kong · Microsoft Research Asia · Princeton University
Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
Mar 10, 2026Zhengzhao Ma, Xueru Wen, Boxi Cao +6Reinforcement Learning RL for Language Model Reasoning
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +2

Mar 17, 2026J/K move · Enter open · S save

When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective

Zelin Zhang, Fei Cheng, Chenhui Chu

Kyoto University

Unsupervised RL Reinforcement Learning RL for Language Model Reasoning LLM Reasoning Intrinsic Reward Methods

Open paper AskPDF arXiv

Although outcome-based reinforcement learning (RL) significantly advances the mathematical reasoning capabilities of Large Language Models (LLMs), its reliance on computationally expensive ground-truth annotations imposes a severe scalability bottleneck. Unsupervised RL guided by intrinsic rewards offers a scalable alternative, yet it suffers from opaque training dynamics and catastrophic instability, such as policy collapse and reward hacking. In this paper, we first design and evaluate a suite of intrinsic rewards that explicitly enforce concise and certain generation. Second, to discover the boundaries of this approach, we test base models across a spectrum of intrinsic reasoning capabilities, revealing how a model's foundational logical prior dictates its success or failure. Finally, to demystify why certain configurations stabilize while others collapse, we introduce a novel geometric diagnostic lens, showing that successful cases are enveloped by manifolds. Ultimately, our work goes beyond merely demonstrating that enforcing concise and certain responses successfully boosts mathematical reasoning; we reveal when this unsupervised approach breaks down and geometrically diagnose why.