RL for Language Model Reasoning

RL: Reinforcement Learning

This field studies how reward feedback can teach language models to solve multi-step problems, use tools, and decide how much reasoning a task needs. A common approach is reinforcement learning from verifiable rewards (RLVR), often using group-relative policy optimization, where sampled solutions are scored for correctness; other methods add step-level rewards, process verifiers, search, or teacher guidance to address sparse feedback. Work also explores curricula, adaptive rollout budgets, self-distillation, and controls for reasoning length and diversity. Key open tensions include balancing exploration with stable learning, ensuring rewards and verifiers do not encourage shortcuts, and improving generalization without excessive computation or overthinking.

Latest papers

1,103 papers170 papers in the last four weeks, up 124% on the four weeks before. 1.0% of all new papers.

Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge
Apr 3, 2026Yiyang Shen, Lifu Tu, Weiran WangRL for Language Model Reasoning Knowledge Distillation
University of Iowa · Oracle AI
A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
Apr 2, 2026Ming Lei, Christophe BaehrEntropy Regularization RL for Language Model Reasoning
Shanghai Jiao Tong University, Shanghai · M´et´eo France/CNRS, Universit´e de Toulouse, CNRM UMR 3589, Toulouse
Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
Apr 2, 2026Bangji Yang, Hongbo Ma, Jiajun Fan +1Efficient Language Model Reasoning Efficient Language Model Inference
University of Illinois Urbana-Champaign · Tsinghua University
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Apr 2, 2026Gengsheng Li, Tianyu Yang, Junfeng Fang +6RL for Language Model Reasoning On-Policy Self-Distillation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore +2
Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning
Apr 2, 2026Rafael Pardinas, Ehsan Kamalloo, David Vazquez +1LLM Inference Efficiency RL for Language Model Reasoning
ServiceNow AI Research
ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning
Apr 2, 2026Jingyue Gao, Yanjiang Guo, Xiaoshuai Chen +1Agentic RL RL for Language Model Reasoning
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute · Independent Researcher
Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
Apr 2, 2026Dong Shu, Denghui Zhang, Jessica HullmanRL Post-Training RL for Language Model Reasoning
Northwestern University · Stevens Institute of Technology
ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
Apr 2, 2026Difan Jiao, Qianfeng Wen, Blair Yang +2RL for Language Model Reasoning LLM Reasoning
Department of Computer Science, University of Toronto · Coolwei AI Lab
Improving Latent Generalization Using Test-time Compute
Apr 1, 2026Arslan Chaudhry, Sridhar Thiagarajan, Andrew LampinenRL for Language Model Reasoning
Google DeepMind · Work done while at Google DeepMind
Policy Improvement Reinforcement Learning
Apr 1, 2026Huaiyang Wang, Xiaojie Li, Xiaohan Wang +10RL for Language Models RL for Language Model Reasoning
Beihang University · Meituan · Peking University
RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning
Apr 1, 2026Shaopeng Fu, Xingxing Zhang, Li Dong +1RL for Code Generation LLM Self-Refinement
Work done during an internship at Microsoft Research · KAUST · Microsoft Research
Learning to Hint for Reinforcement Learning
Apr 1, 2026Yu Xia, Canwen Xu, Zhewei Yao +2Reinforcement Learning RL for Language Model Reasoning
University of California, San Diego · Snowflake AI Research
TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning
Apr 1, 2026Wenxuan Jiang, Yuxin Zuo, Zijian Zhang +8In-Context RL RL for Language Model Reasoning
The Hong Kong Polytechnic University · Meituan-M17 · Institute of Computing Technology, Chinese Academy of Sciences +3
Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?
Mar 31, 2026Max Kaufmann, David Lindner, Roland S. Zimmermann +1CoT Monitoring CoT Supervision
Google DeepMind
Learning Diagnostic Reasoning for Decision Support in Toxicology
Mar 31, 2026Nico Oberländer, David Bani-Harouni, Tobias Zellner +3Clinical Decision Support RL for Language Model Reasoning
Computer Aided Medical Procedures, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Germany · Department of Clinical Toxicology and Poison Control Center Munich, TUM Klinikum rechts der Isar, Germany
ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
Mar 30, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning RL for Language Models
School of Computer and Information Science, Southwest University, Chongqing, 400715, China
SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
Mar 30, 2026Yifan Wang, Bolian Li, David Cho +3RL for Language Model Reasoning Reinforcement Learning
Purdue University · Texas Instruments
Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMs
Mar 29, 2026Amirul Rahman, Aisha Karim, Kenji Nakamura +1Efficient Language Model Reasoning Inference-Time Compute Allocation
University of Malaya
PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization
Mar 27, 2026Zelin Tan, Zhouliang Yu, Bohan Lin +9RL for Language Model Reasoning Process Reward Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +2
R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning
Mar 26, 2026Zirui Zhang, Haoyu Dong, Kexin Pei +1Multimodal Reasoning Cross-Modal Learning
Rutgers University · Columbia University · University of Chicago
TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning
Mar 26, 2026Xu Huang, Zhejian Lai, Zixian Huang +2Cross-Lingual Reasoning in Language Models RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology, Nanjing University · Shanghai Artificial Intelligence Laboratory
Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models
Mar 25, 2026Isha Puri, Mehul Damani, Idan Shenfeld +3RL for Language Model Reasoning Efficient Language Model Reasoning
Massachusetts Institute of Technology
Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR
Mar 25, 2026Haobo Xu, Sirui Chen, Ruizhong Qiu +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Illinois at Urbana-Champaign · Amazon
Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization
Mar 25, 2026Fei Bai, Zhipeng Chen, Chuan Hao +6RL for Language Model Reasoning Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University
From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents
Mar 25, 2026Sirui Xia, Yikai Zhang, Aili Chen +3Automated Algorithm Discovery RL for Language Model Reasoning
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University · School of Data Science, Fudan University
HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
Mar 25, 2026Ken DingRL for Language Model Reasoning LLM Reasoning
NVIDIA
SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
Mar 24, 2026Yiqi Zhang, Huiqiang Jiang, Xufang Luo +7RL for Language Models RL for Language Model Reasoning
National University of Singapore · Microsoft Research Asia · The Chinese University of Hong Kong
Off-Policy Value-Based Reinforcement Learning for Large Language Models
Mar 24, 2026Peng-Yuan Wang, Ziniu Li, Tian Xu +8Reinforcement Learning RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1
Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness
Mar 24, 2026Abhinaba Basu, Pavan ChakrabortyLanguage Model Reasoning Evaluation Structured Reasoning in Language Models
Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Mar 23, 2026Haoming Meng, Kexin Huang, Shaohang Wei +6RL for Language Model Reasoning RL Fine-Tuning
Alibaba Group

Apr 3, 2026J/K move · Enter open · S save

Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge

Yiyang Shen, Lifu Tu, Weiran Wang

University of Iowa · Oracle AI

RL for Language Model Reasoning Knowledge Distillation Numerical Reasoning in Language Models LLM-as-a-Judge Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning (RL) substantially improves the reasoning capabilities of language models, but most existing RL fine-tuning approaches rely entirely on ground-truth verifiable rewards and thus labeled datasets with verifiable answers. To overcome this, we propose a RL framework for reasoning distillation that leverages continuous, LLM-based rewards. Our method employs an efficient mechanism that computes a continuous CoT reward (CCR) directly from a single-token logit of a judge LLM, evaluating the student model's reasoning trajectory. This formulation provides an effective and scalable online training signal that can be applied to large volumes of unlabeled data. We demonstrate that, when paired with a strong judge, simply using CCR achieves performance comparable to that of ground-truth or pseudo-label verifiable rewards, and even surpasses them as the amount of unlabeled data increases. Furthermore, we find that combining them in a semi-supervised setup is highly synergistic: verifiable rewards help stabilize the CCR, while CCR improves the generalizability of verifiable rewards to related tasks. We also provide a comprehensive empirical comparison of various reward sources across multiple model architectures and dataset sizes. Our results show that this semi-supervised approach consistently enhances mathematical reasoning, yielding an absolute improvement of 5-10% on multiple reasoning tasks.