RL for Language Model Reasoning

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

58 papers in the last four weeks, up 49% on the four weeks before. 0.7% of all new papers.

Jul 6Week of Sep 21

Latest papers 707

All topics

Distilling LLM Feedback for Lean Theorem Proving
May 29, 2026Gaetan Narozniak, Gérard Biau, Rémi Munos +2RL for Language Model Reasoning Mathematical Reasoning
FAIR at Meta, Inria · Sorbonne Université, Institut universitaire de France · FAIR at Meta +2
SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoning
May 29, 2026Jian Yao, Xiongcai Luo, Ran Cheng +1RL for Language Model Reasoning Efficient Language Model Inference
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Bytedance · The Hong Kong Polytechnic University-Daya Bay Technology and Innovation Research Institute, Huizhou,Guangdong Province, China.
Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
May 29, 2026Yiming Ren, Yiran Xu, Zicheng Lin +8Reinforcement Learning RL for Language Model Reasoning
Tsinghua University · Shanghai AI Laboratory · The Chinese University of Hong Kong +1
VeriGate: Verifier-Gated Step-Level Supervision for GRPO
May 28, 2026Aakriti Agrawal, Minghui Liu, Furong HuangProcess Reward Models RL for Language Model Reasoning
University of Maryland, College Park
ESPO: Early-Stopping Proximal Policy Optimization
May 28, 2026Zihang Li, Rui Zhou, Yingcheng Shi +8Early Stopping Reinforcement Learning
1Tongyi Lab , Alibaba Group · 2Peking University
Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering
May 28, 2026Shicheng Fan, Haochang Hao, Dehai Min +3Process Reward Models RL for Language Model Reasoning
University of Illinois Chicago
DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learning
May 28, 2026Yang He, Xiao Ding, Bibo Cai +5Reinforcement Learning RL for Language Model Reasoning
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
Rubric-Guided Process Reward for Stepwise Model Routing
May 28, 2026Shenghao Ye, Yu Guo, Zhengheng Li +2Process Reward Models RL for Language Model Reasoning
University of Science and Technology of China · Southeast University
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
May 28, 2026Shufan Li, Konstantinos Kallidromitis, Akash Gokul +2Token-Level Credit Assignment RL for Language Model Reasoning
UCLA · Panasonic AI Research · NVIDIA
When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer
May 28, 2026Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski +1RL for Language Models RL for Language Model Reasoning
Fin AI Research · Salk Institute for Biological Studies
OISD: On-Policy Internal Self-Distillation of Language Models
May 27, 2026Xinyu Liu, Darryl Cherian Jacob, Yang Zhou +2RL for Language Model Reasoning Self-Distillation
Auburn University · William & Mary
DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution
May 27, 2026Yunhai Hu, Zining Liu, Xiangyang Yin +5Efficient Multimodal Inference RL for Language Model Reasoning
New York University · University of Pennsylvania · Cerebras Systems
DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes
May 27, 2026Caijun Xu, Changyi Xiao, Zhongyuan Peng +1LLM Self-Correction RL for Language Model Reasoning
1Fudan University · 2Shanghai Innovation Institute
FABSVer: Faster Training and Better Self-Verification for LLM Mathematical Reasoning
May 27, 2026Haihui Pan, Junwei Bao, Hongfei Jiang +1LLM Answer Verification RL for Language Model Reasoning
Zuoyebang Education Technology
Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
May 27, 2026Yue Cheng, Jiajun Zhang, Xiaohui Gao +3Reinforcement Learning RL for Language Model Reasoning
Beijing Jiaotong University · AntGroup · Northwestern Polytechnical University +2
Plan Before Search: Search Agents Need Plan
May 27, 2026Zhipeng Qian, Zihan Liang, Yufei Ma +7Multi-Hop QA Tool-Augmented Language Model Agents
1Kuaishou Technology · 2Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · 3VCIP, CS, Nankai University
Diversifying RLVR Rollouts via First-Token Exploration
May 27, 2026Soeun Kim, Albert NoRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Artificial Intelligence, Yonsei University
IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage
May 27, 2026Yuhan Li, Mingxu Zhang, Dazhong Shen +1Reinforcement Learning RL for Language Model Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Nanjing University of Aeronautics and Astronautics · The 63rd Research Institute, National University of Defense Technology, Nanjing
Joint Training of Multi-Token Prediction in Reinforcement Learning via Optimal Coefficient Calibration
May 27, 2026Zili Wang, Jiajun Chai, Lin Chen +3Multi-Token Prediction RL for Language Model Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China · Meituan
Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think
May 27, 2026Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali +1Reinforcement Learning RL for Language Model Reasoning
Criteo AI Lab · Criteo AI Lab, CREST IP Paris
Deconstructing Spatial Complexity: Hierarchical Decomposition for LLM Spatial Reasoning
May 27, 2026Yi Wang, Haojie Lu, Zhaofan Zhang +2Hierarchical Planning RL for Language Model Reasoning
The Hong Kong University of Science and Technology (Guangzhou)
Long Live The Balance: Information Bottleneck Driven Tree-based Policy Optimization
May 27, 2026Hao Jiang, Shurui Li, Tianpeng Bu +7RL for Language Models Exploration-Exploitation Tradeoff
Alibaba Cloud Computing, Alibaba Group.
BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
May 27, 2026Qingfei Zhao, Huan Song, Shuyu Tian +2RL for Language Model Reasoning Group Relative Policy Optimization
1TeleAI · 2Shanghai Jiao Tong University
Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback
May 27, 2026Bowen Wei, Nan Wang, Yuqing Zhou +2Self-Training for Language Models RL for Language Model Reasoning
George Mason University · Amazon
GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization
May 27, 2026Shengmin Piao, Sanghyun ParkToken-Level Credit Assignment RL for Language Model Reasoning
Yonsei University Seoul, South Korea
C-MIG: Multi-view Information Gain-based Retrieval-Augmented Generation for Clinical Diagnosis Reasoning
May 27, 2026Yuwei Miao, Gen Li, Yunsheng Zeng +8Retrieval-Augmented Generation Healthcare
1Baidu Inc.
DecomposeRL: Learning to Ask Useful, Informative, and Diverse Questions for Semi-Supervised, Traceable Claim Verification
May 27, 2026Shubhashis Roy Dipta, Ankur Padia, Francis FerraroClaim Verification RL for Language Model Reasoning
Department of Computer Science and Electrical Engineering · University of Maryland Baltimore County · Baltimore, MD 21250 USA
Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
May 26, 2026Zehao Liu, Yuanpu Cao, Jinghui Chen +1Reinforcement Learning On-Policy Self-Distillation
College of Information Sciences and Technology Pennsylvania State University
Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization
May 26, 2026Cihan Xiao, Yiwen Shao, Chenxing Li +5RL for Language Model Reasoning Audio Reasoning
Johns Hopkins University · Tencent Hunyuan
Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
May 26, 2026Yi Jing, Zao Dai, Jinwu Hu +4RL for Language Model Reasoning LLM Post-Training
Tsinghua University

May 29, 2026, cs.AIJ/K move · Enter open · S save

Distilling LLM Feedback for Lean Theorem Proving

Gaetan Narozniak, Gérard Biau, Rémi Munos, Ahmad Rammal+1

FAIR at Meta, Inria · Sorbonne Université, Institut universitaire de France · FAIR at Meta+2

RL for Language Model Reasoning Mathematical Reasoning Language Model Distillation Self-Distillation RL Post-Training

Open paper AskPDF arXiv

Post-training for reasoning models typically combines supervised fine-tuning with reinforcement learning from verifiable rewards, most commonly with GRPO. However, this algorithm suffers from sparse rewards, limited exploration, and mode collapse. Building upon recent works on self-distillation, we propose Feedback Distillation, a training method where the model is trained to match, at the token level, its own distribution conditioned on privileged feedback produced by a language model. Feedback Distillation offers token-level supervision and can inject external knowledge. Evaluating our method for Lean4 theorem-proving, we find that Feedback Distillation maintains greater diversity in generated trajectories than GRPO, yielding higher policy entropy and better pass@k scaling. The two methods are complementary: initializing GRPO from a Feedback Distillation checkpoint outperforms either method alone. All in all, our results suggest a promising avenue to improve post-training for complex reasoning.

Momentum

Related topics

Latest papers 707