RL for Language Model Reasoning

RL: Reinforcement Learning

This field studies how reward feedback can teach language models to solve multi-step problems, use tools, and decide how much reasoning a task needs. A common approach is reinforcement learning from verifiable rewards (RLVR), often using group-relative policy optimization, where sampled solutions are scored for correctness; other methods add step-level rewards, process verifiers, search, or teacher guidance to address sparse feedback. Work also explores curricula, adaptive rollout budgets, self-distillation, and controls for reasoning length and diversity. Key open tensions include balancing exploration with stable learning, ensuring rewards and verifiers do not encourage shortcuts, and improving generalization without excessive computation or overthinking.

Latest papers

1,103 papers170 papers in the last four weeks, up 124% on the four weeks before. 1.0% of all new papers.

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
Jan 29, 2026Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1Efficient Language Model Inference RL for Language Model Reasoning
Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence
Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States
Jan 29, 2026Lei Yang, Wei Bi, Chenxi Sun +2RL for Language Model Reasoning RL Exploration
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Kuaishou Technology
Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
Jan 28, 2026Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5RL for Language Model Reasoning Test-Time Training
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Jan 26, 2026Shobhita Sundaram, John Quan, Ariel Kwiatkowski +3Self-Play RL RL for Language Model Reasoning
1MIT · 2Meta FAIR · 3New York University
GraphDancer: Training LLMs to Explore and Reason over Graphs via Curriculum Reinforcement Learning
Jan 24, 2026Yuyang Bai, Zhuofeng Li, Ping Nie +3LLM Reasoning with Graphs RL for Language Model Reasoning
Texas A&M University · University of Waterloo · Lambda +1
Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study
Jan 22, 2026Zachary BurtonInference-Time Search RL for Language Model Reasoning
Massachusetts Institute of Technology, Cambridge, MA, USA.
CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning
Jan 21, 2026Tianshi Xu, Yuteng Chen, Meng LiAgentic RL Reinforcement Learning
Peking University · NTU, Singapore
R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Jan 17, 2026Jingchu Wang, Bingbing Xu, Yige Yuan +4RL for Language Model Reasoning Policy Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Jan 16, 2026Lecheng Yan, Ruizhe Li, Guanhua Chen +5Reward Hacking Memorization in Language Models
Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen +5
ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition
Jan 7, 2026Muyang Zhao, Chunhao Tian, Qi Qi +1Cost-Aware Inference Metacognition in Language Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models
Jan 7, 2026Yuxuan Jiang, Francis FerraroProcess Reward Models RL for Language Model Reasoning
University of Maryland, Baltimore County
Dementia-R1: Reinforced Pretraining and Reasoning from Unstructured Clinical Notes for Real-World Dementia Prognosis
Jan 6, 2026Choonghan Kim, Hyunmin Hwang, Hangeol Chang +4Clinical Outcome Prediction Alzheimer's Disease Progression Prediction
Graduate School of AI, KAIST, Republic of Korea · Haeundae Paik Hospital, Inje University, Republic of Korea · Asan Medical Center, University of Ulsan College of Medicine, Republic of Korea
Dual-Evolving Reinforcement Learning for Automated Reward Design
Dec 15, 2025Sitao Cheng, Tianle Li, Xuhan Huang +2Reward Design for RL Reinforcement Learning
University of Waterloo · The University of Hong Kong · The Chinese University of Hong Kong, Shenzhen +1
KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering
Dec 10, 2025Xin Sun, Zhongqi Chen, Xing Zheng +6LLM Grounding RL for Language Model Reasoning
University of Science and Technology of China, Hefei, China · Independent · Institute of Automation, Chinese Academy of Sciences, Beijing, China.
Training Language Models to Use Prolog as a Tool
Dec 8, 2025Niklas Mellgren, Peter Schneider-Kamp, Lukas Galke PoechReward Hacking Logical Reasoning
University of Southern Denmark
Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Nov 25, 2025Chenliang Li, Adel Elmahdy, Alex Boyd +7RL for Language Model Reasoning Importance Sampling
Texas A&M University · GE HealthCare · Independent Researcher +1
ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
Nov 24, 2025Long Lian, Sida Wang, Felix Juefei-Xu +7Parallel Reasoning in Language Models RL for Language Model Reasoning
Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States
Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
Nov 8, 2025Renfei Zhang, Manasa Kaniselvan, Rylan Schaeffer abd Niloofar MireshghallahRL for Language Model Reasoning Factual Knowledge in Language Models
Carnegie Mellon University · MIT · Stanford University
OpenSIR: Open-Ended Self-Improving Reasoner
Nov 1, 2025Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3Self-Play RL RL for Language Model Reasoning
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
e1: Learning Adaptive Control of Reasoning Effort
Oct 30, 2025Michael Kleinman, Matthew Trager, Alessandro Achille +2Efficient Language Model Reasoning Cost-Aware Inference
AWS Agentic AI
Learning to Reason Efficiently with Discounted Reinforcement Learning
Oct 27, 2025Alex Ayoub, Kavosh Asadi, Dale Schuurmans +2Reinforcement Learning RL for Language Model Reasoning
Amazon, University of Alberta · Amazon · University of Alberta
Representation-Based Exploration for Language Models: From Test-Time to Post-Training
Oct 13, 2025Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy +1RL for Language Model Reasoning Test-Time Scaling
Princeton University · Microsoft Research NYC
Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
Oct 10, 2025Xinyi Wang, Jinyi Han, Zishang Jiang +7Reinforcement Learning RL for Language Model Reasoning
School of Data Science, Fudan University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts
Oct 10, 2025Jiuheng Lin, Cong Jiang, Zirui Wu +2RL for Language Models RL for Language Model Reasoning
Wangxuan Institute of Computer Technology, Peking University
Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
Oct 9, 2025Yuchen Zhu, Wei Guo, Jaemoo Choi +4RL for Language Model Reasoning Policy Optimization
Georgia Institute of Technology.
SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
Oct 9, 2025Chen Wang, Zhaochun Li, Jionghao Bai +4Reinforcement Learning RL Post-Training
College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +2
Group Contrastive Policy Optimization: Learning Beyond the Self-Exploration Limit with Golden Answers
Oct 9, 2025Hao Wu, Wei Liu, Shizhe Xiang +3RL for Language Model Reasoning LLM Reasoning
Base Models Know How to Reason, Thinking Models Learn When
Oct 8, 2025Constantin Venhoff, Iván Arcuschin, Philip Torr +2LLM Interpretability RL for Language Model Reasoning
University of Oxford, UK · MATS · Poseidon Research.
The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction
Oct 7, 2025Xinyu Guo, Zhengliang Shi, Minglai Yang +2Relation Extraction Structured Reasoning in Language Models
University of Arizona, Tucson, AZ, United States · Shandong University, Qingdao, China.
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1

Jan 29, 2026J/K move · Enter open · S save

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, Haitham Bou Ammar

Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence

Efficient Language Model Inference RL for Language Model Reasoning

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a token-level decomposition of the autoregressive power distribution, showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over

10\times

compared to MCMC-based sampling.