RL for Language Model Reasoning

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

91 papers in the last four weeks, up 176% on the four weeks before. 0.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 707

All topics

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
Feb 4, 2026Doyeon Lee, Eunyi Lyou, Hyunsoo Cho +3RL for Language Model Reasoning Policy Optimization
Seoul National University · Department of Artificial Intelligence / Institute for Multiscale Matter and Systems, Ewha Woman University · Geogia Institute of Technology
Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning
Feb 1, 2026Yu Li, Mingyang Yi, Xiuyu Li +6RL for Language Model Reasoning Gradient Interference
School of Information, Renmin University of China, Beijing, China · Bytedance Inc., Beijing, China · Bytedance Inc., San Jose, USA
Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients
Jan 30, 2026Cheng Ge, Caitlyn Heqi Yin, Hao Liang +1RL for Language Model Reasoning Group Relative Policy Optimization
Department of Aeronautics and Astronautics, MIT · Department of Statistics, University of Wisconsin–Madison · Department of Informatics, King’s College London +1
Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning
Jan 30, 2026Arvind Mahankali, Kaiyue Wen, Tengyu MaMulti-Agent LLM Systems CoT Reasoning
Stanford University
HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning
Jan 30, 2026Weiqi Wang, Xin Liu, Binxuan Huang +13RL for Language Models RL for Language Model Reasoning
Stores Foundational AI, Amazon Inc., Palo Alto, CA, USA · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs
Jan 29, 2026Lei Yang, Wei Bi, Chenxi Sun +2RL for Language Model Reasoning RL Exploration
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Kuaishou Technology
Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
Jan 28, 2026Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5RL for Language Model Reasoning Test-Time Training
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Jan 26, 2026Shobhita Sundaram, John Quan, Ariel Kwiatkowski +3Self-Play RL RL for Language Model Reasoning
1MIT · 2Meta FAIR · 3New York University
Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study
Jan 22, 2026Zachary BurtonInference-Time Search RL for Language Model Reasoning
Massachusetts Institute of Technology, Cambridge, MA, USA.
CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning
Jan 21, 2026Tianshi Xu, Yuteng Chen, Meng LiAgentic RL Reinforcement Learning
Peking University · NTU, Singapore
R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Jan 17, 2026Jingchu Wang, Bingbing Xu, Yige Yuan +4RL for Language Model Reasoning Policy Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Jan 16, 2026Lecheng Yan, Ruizhe Li, Guanhua Chen +5Reward Hacking Memorization in Language Models
Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen +5
SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models
Jan 7, 2026Yuxuan Jiang, Francis FerraroProcess Reward Models RL for Language Model Reasoning
University of Maryland, Baltimore County
KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering
Dec 10, 2025Xin Sun, Zhongqi Chen, Xing Zheng +6LLM Grounding RL for Language Model Reasoning
University of Science and Technology of China, Hefei, China · Independent · Institute of Automation, Chinese Academy of Sciences, Beijing, China.
Training Language Models to Use Prolog as a Tool
Dec 8, 2025Niklas Mellgren, Peter Schneider-Kamp, Lukas Galke PoechReward Hacking Logical Reasoning
University of Southern Denmark
Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Nov 25, 2025Chenliang Li, Adel Elmahdy, Alex Boyd +7RL for Language Model Reasoning Importance Sampling
Texas A&M University · GE HealthCare · Independent Researcher +1
ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
Nov 24, 2025Long Lian, Sida Wang, Felix Juefei-Xu +7RL for Language Model Reasoning Inference-Time Scaling
Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States
Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
Nov 8, 2025Renfei Zhang, Manasa Kaniselvan, Rylan Schaeffer abd Niloofar MireshghallahRL for Language Model Reasoning Factual Knowledge in Language Models
Carnegie Mellon University · MIT · Stanford University
OpenSIR: Open-Ended Self-Improving Reasoner
Nov 1, 2025Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3Self-Play RL RL for Language Model Reasoning
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
Learning to Reason Efficiently with Discounted Reinforcement Learning
Oct 27, 2025Alex Ayoub, Kavosh Asadi, Dale Schuurmans +2Reinforcement Learning RL for Language Model Reasoning
Amazon, University of Alberta · Amazon · University of Alberta
Representation-Based Exploration for Language Models: From Test-Time to Post-Training
Oct 13, 2025Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy +1RL for Language Model Reasoning Test-Time Scaling
Princeton University · Microsoft Research NYC
Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
Oct 10, 2025Xinyi Wang, Jinyi Han, Zishang Jiang +7Reinforcement Learning RL for Language Model Reasoning
School of Data Science, Fudan University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts
Oct 10, 2025Jiuheng Lin, Cong Jiang, Zirui Wu +2RL for Language Models RL for Language Model Reasoning
Wangxuan Institute of Computer Technology, Peking University
Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
Oct 9, 2025Yuchen Zhu, Wei Guo, Jaemoo Choi +4RL for Language Model Reasoning Policy Optimization
Georgia Institute of Technology.
Base Models Know How to Reason, Thinking Models Learn When
Oct 8, 2025Constantin Venhoff, Iván Arcuschin, Philip Torr +2LLM Interpretability RL for Language Model Reasoning
University of Oxford, UK · MATS · Poseidon Research.
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
Oct 2, 2025Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad +3RL for Language Model Reasoning LLM Inference Acceleration
UNC Chapel Hill · The University of Texas at Austin · Microsoft Research
Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
Oct 1, 2025Xin-Qiang Cai, Wei Wang, Feng Liu +3Reinforcement Learning RL for Language Model Reasoning
RIKEN AIP, Tokyo, Japan · The University of Tokyo, Tokyo, Japan · The University of Melbourne, Melbourne, Australia
Interactive Learning for LLM Reasoning
Sep 30, 2025Hehai Lin, Shilei Cao, Sudong Wang +5RL for Language Model Reasoning LLM Agent Training
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University · 4National University of Singapore +2
TagPR: Tag-Guided Process Supervision for Personalization Reasoning in Large Language Models
Sep 27, 2025Song Jin, Juntian Zhang, Ruyu Lyu +7LLM Alignment LLM Personalization
Gaoling School of Artificial Intelligence, Renmin University of China · Meituan · Wuhan University

Feb 4, 2026, cs.LGJ/K move · Enter open · S save

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim+2

Seoul National University · Department of Artificial Intelligence / Institute for Multiscale Matter and Systems, Ewha Woman University · Geogia Institute of Technology

RL for Language Model Reasoning Policy Optimization Trust-Region Optimization Trust Region Policy Optimization RL Fine-Tuning Policy Learning

Open paper AskPDF arXiv

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.

Momentum

Related topics

Latest papers 707