RL for Language Models

RL: Reinforcement Learning

This field studies how feedback can shape a language model’s behavior after pretraining, including how it answers, reasons, uses tools, and handles uncertainty. Common approaches include preference-based reinforcement learning from human or AI feedback, reinforcement learning with verifiable rewards for tasks such as math and coding, and policy methods such as PPO and GRPO; rubrics and step- or turn-level rewards provide finer guidance. Researchers evaluate these methods on reasoning benchmarks as well as dialogue, tool use, multilingual tasks, and domain-specific applications. A central tension is that sparse or imperfect rewards can be costly to obtain and easy to exploit, while reliable credit assignment, exploration, and generalization beyond the training setting remain difficult.

Latest papers

441 papers79 papers in the last four weeks, up 126% on the four weeks before. 0.5% of all new papers.

What Should LLMs Forget: Reinforcement Unlearning of Knowledge Associations
Oct 30, 2026Wutong Xu, Yun Qu, Haiyan Feng +2Machine Unlearning LLM Unlearning
Department of Automation, Tsinghua University, Beijing, China
RTL-Interact: Training EDA-Interactive Models for Multi-Task RTL Design
Oct 30, 2026Wenxuan Shi, Yaoyu Zhu, Di Huang +13LLM Tool Use Electronic Design Automation
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Oct 29, 2026Yu Li, Yufa Zhou, Tian LanTraining-Inference Mismatch RL for Language Models
George Washington University · Duke University
Training Foundation Models with a Single Algorithm
Oct 29, 2026Yuda Song, Alvin Zhang, Fahim Tajwar +7LLM Training RL for Language Models
Carnegie Mellon University · Johns Hopkins University · Aurora Innovation
Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training
Oct 28, 2026Albert Bou, Javier Valdes-Aleman, Jon M. Laurent +4RL for Language Model Reasoning LLM Post-Training
Edison Scientific Inc., San Francisco, CA, USA
Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation
Oct 28, 2026Chengsheng Zhang, Wenshen Xu, Yuchuan Wu +6RL for Language Models Rubric-Based RL
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · Nanjing University +1
Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
Oct 27, 2026Peize DingRL for Language Models Reinforcement Learning
Columbia University, New York, NY, USA
When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning
Oct 27, 2026Zehao Wang, Hanzhu Chen, Runquan Gui +2RL for Language Models RL for Language Model Reasoning
University of Science and Technology of China · Tencent · Shanghai Artificial Intelligence Laboratory
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Oct 27, 2026Jixuan Leng, Si Si, Fei Wang +4Reinforcement Learning RL for Language Models
CMU · Google DeepMind · Google +1
Integrate the Reward, Don't Sample It: Window Likelihood for Interruptions and Backchannels in Full-Duplex Speech Models
Oct 26, 2026Guoqiang Hu, Lichao Zhang, Yizhou Peng +5Full-Duplex Spoken Dialogue Systems Spoken Dialogue Systems
Nanyang Technological University, Singapore · Tencent
S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
Oct 26, 2026Shuo Shen, Guangran Cheng, Chengqi Lyu +3RL for Language Model Reasoning RL for Language Models
Shanghai AI Laboratory · Independent Researcher
Reinforcement Learning Done Right for Explicit Confidence Targets
Oct 26, 2026Zefeng He, Yichen He, Yuan Lin +2LLM Abstention Selective Prediction
ByteDance Seed · Shanghai Jiao Tong University · NTU
Correlation-Aware Exploration for Long-Trace Reasoning
Oct 26, 2026Junxing Hu, Zicheng Zhang, Zhiqian Zhang +4RL for Language Model Reasoning RL for Language Models
Microsoft
Uncovering the Turing Test's Latent Signal: Indistinguishability as a Reward
Oct 26, 2026Emanuele Rimoldi, Riccardo Neumarker, Tomaso Poggio +1Adversarial Imitation Learning RL for Language Models
MIT · EPFL · ETH Zurich +1
Audio-R1: Audio Reasoning through Reinforcement Learning on Self-Supervised Tasks
Oct 25, 2026Jihoon Jeong, Mirco Ravanelli, Zhepei Wang +1Audio Reasoning Audio-Language Models
Mila-Québec AI Institute · Université Laval · Concordia University +1
ScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped Transformers
Oct 25, 2026Dezhen Lu, Zixiang Ni, Runguo LiRL for Language Models Efficient Language Model Inference
University of Tübingen, Tübingen, Germany. · Xi’an Jiaotong University, Xi’an, China. · University of Illinois Urbana-Champaign, Urbana, IL, USA.
A Distribution-Matching Perspective on LLM Training Algorithms
Oct 24, 2026Yanxing Huang, Bingzhang Yan, Zihan Tang +3LLM Training Language Model Distillation
Qiuzhen College, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Weiyang College, Tsinghua University, Beijing, China +2
Implicit Beliefs as Scalable Process Rewards for Strategic Persuasion
Oct 24, 2026Minqian Liu, Yuexi Shen, Heajun An +5Persuasion in Language Models Process Reward Models
Virginia Tech · UC Santa Barbara · STIR Lab +1
Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning
Oct 23, 2026Yijing Ke, Tianle Cai, Ruosong WangRL for Language Models RL Stability
ByteDance Seed · Peking University
KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning
Oct 23, 2026Bangjun Xiao, Chiheng Lou, Xiaodong Ji +4RL for Language Models Reinforcement Learning
School of Computer Science, Peking University · LLM-Core Xiaomi
Free Lunch for LLMs Confidence: The RL Critic Already Knows
Oct 22, 2026Shujin Wu, Heng JiConfidence Estimation in Language Models LLM Reliability
University of Illinois Urbana-Champaign
Scaling Laws for Generative Reward Models
Oct 22, 2026Angel Raychev, Nathan Lile, Rafael RafailovReward Modeling RL for Language Models
Stanford University · Thinking Machines
Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable
Oct 21, 2026Marius De Kuthy Meurers, Aaron Parisi, Bernd BohnetMoral Reasoning in Language Models Reasoning Consistency in Language Models
Google DeepMind
RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Oct 21, 2026Mingkang Liu, Weiming Meng, Peipei Jiao +4Speculative Decoding LLM Inference Acceleration
LIFT: Scoped Constraints for Evaluating and Training Instruction Following
Oct 20, 2026Eran Goldstein, Yuval Belfer, Yonatan Belinkov +1Instruction Following Generative AI Evaluation
AI21 Labs
Measure First, Then Allocate: Calibrated Rollout Budgets for MCP Agents in Reinforcement Learning
Oct 20, 2026Hieu Tran, Zonghai Yao, Sunjae Kwon +1RL for LLM Agents RL for Language Models
University of Massachusetts Amherst · University of Massachusetts Lowell
SimHack: Predicting the onset of detectable reward hacking before RL training
Oct 20, 2026Junwei Deng, Alex Bie, Natalia PonomarevaReward Hacking RL for Language Models
University of Illinois Urbana-Champaign · Google Research
Reward Misspecification Degrades Chain-of-Thought Monitorability
Oct 20, 2026Tianyi Lorena Yan, John HewittLLM Safety Monitoring CoT Monitoring
Columbia University
From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks
Oct 20, 2026Taiheng PanRL Post-Training RL for Language Models
The University of Melbourne
Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties
Oct 19, 2026Afonso Baldo, António Farinhas, Marcos Treviso +2Reward Hacking RL for Language Models
Sword Health · Instituto Superior Técnico, Universidade de Lisboa · Gandara AI +2

Oct 30, 2026J/K move · Enter open · S save

What Should LLMs Forget: Reinforcement Unlearning of Knowledge Associations

Wutong Xu, Yun Qu, Haiyan Feng, Qi Wang+1

Department of Automation, Tsinghua University, Beijing, China

Machine Unlearning LLM Unlearning RL for Language Models

Open paper AskPDF arXiv

Large language models (LLMs) acquire extensive factual knowledge during pre-training, which may include sensitive, private, or copyrighted information, motivating the development of unlearning methods that selectively remove specific knowledge while preserving general capabilities. However, existing approaches typically optimize entire responses without explicitly identifying the knowledge to be forgotten, resulting in coarse and uncontrolled forgetting. To address this, we revisit the forgetting target of LLM unlearning and introduce Knowledge Association (KA), which characterizes the association between a query context and its target knowledge and is instantiated by the corresponding conditional predictive distribution, as the fundamental target of forgetting. Based on KA, we propose Knowledge Association Reinforcement Unlearning (KARU), a unified reinforcement learning framework that suppresses forgotten knowledge associations while preserving retained ones. Specifically, KARU derives complementary mechanisms from the KA-based formulation: token-level reward construction for finer-grained forgetting supervision, distribution-level regularization for precise knowledge suppression, and proximity-aware retain sampling to mitigate collateral degradation. Experiments across diverse LLMs on representative benchmarks demonstrate that KARU consistently improves forgetting effectiveness over existing baselines while maintaining comparable retention performance and general utility.