RL for Language Models

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

36 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 271

All topics

Weight-Space Geometry of Offline Reasoning Training
Jun 21, 2026Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov +1RL for Language Models Direct Preference Optimization
1Keenable AI · 2Accenture · 3Yandex
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Jun 20, 2026Zhao Yang, Yuxuan Jiang, Ting-Chih Chen +18RL for Language Models Credit Assignment in RL
VU Amsterdam, The Netherlands · University of Maryland, Baltimore County, USA · Leiden University, The Netherlands +4
Post-Training Speech Enhancement Language Models with Perceptual Rewards
Jun 19, 2026Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis +1Multi-Objective Reinforcement Learning RL for Language Models
ETH Zürich, Switzerland
Learning User Simulators with Turing Rewards
Jun 17, 2026Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu +5Reward Modeling RL for Language Models
Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Jun 17, 2026Minseo Kim, Minjae Lee, Seunghyuk Oh +7RL for Language Models Reinforcement Learning
FuriosaAI · University of California, Berkeley
Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization
Jun 17, 2026Lanqing Li, Shentong Mo, Yang Yu +1RL for Language Models Reinforcement Learning
The Chinese University of Hong Kong · MBZUAI · Hong Kong University of Science and Technology
Self-CTRL: Self-Consistency Training with Reinforcement Learning
Jun 16, 2026Itamar Pres, Laura Ruis, Melat Ghebreselassie +2RL for Language Models Faithfulness of Language Model Explanations
MIT CSAIL
SuCo: Sufficiency-guided Continuous Adaptive Reasoning
Jun 16, 2026Jiahao Wang, Bingyu Liang, Chenhao Hu +5RL for Language Models Large Reasoning Models
Harbin Institute of Technology (Shenzhen), Guangdong, China. · TeleAI of China Telecom, China.
Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games
Jun 16, 2026Zirui Cheng, Zeyu Shen, Thomas L. Griffiths +1RL for Language Models Cognitive Modeling
1Princeton University
GD $^2$ PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
Jun 15, 2026Haotian Liu, Yihao Liu, Jingwei Ni +11Multi-Objective Reinforcement Learning RL for Language Models
Alibaba · Renmin University of China · Peking University +3
A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
Jun 15, 2026Jianghan Shen, Siqi Luo, Yue Li +9RL for Language Models Group Relative Policy Optimization
1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +4
Teaching agentic AI to generalize expert diagnostic reasoning in rare diseases
Jun 15, 2026Minh-Ha Nguyen, Erica Gray, Bryce A. Schuler +16RL for Language Models Rare Disease Diagnosis
Department of Epidemiology, Vanderbilt University, Nashville, TN, USA · Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA · Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA +4
Reinforcement Learning for LLM-based Event Forecasting
Jun 14, 2026Amit Arnold LevyEvent Time Prediction RL for Language Models
University of Oxford
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
Jun 14, 2026Ning Li, Zixuan Guo, Yan Xu +7RL for Language Models LLM Hallucination Mitigation
1Shanghai Jiao Tong University · 2Tencent · 3The
AP-GRPO: Anchor-Gated Phonetic Alignment with Policy Optimization for Pathological Speech Reconstruction
Jun 14, 2026Pengfei Zhang, Hoang H Nguyen, Yutong Song +6RL for Language Models Speech Processing
University of California Irvine · University of Illinois Chicago · Kennesaw State University
EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management
Jun 14, 2026Rongzhi Zhu, Xiang Huang, Yuchuan Wu +8RL for Language Models LLM Evaluation
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Qwen-Character Team , Alibaba Group
Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization
Jun 10, 2026Frank Xiao, Mary PhuongReward Hacking RL for Language Models
California Institute of Technology
ReCal: Reward Calibration for RL-based LLM Routing
Jun 10, 2026Qihang Yu, Hanwen Tong, Zhengqi Zhang +5Multi-Objective Reinforcement Learning RL for Language Models
Zhejiang University · Ant Group · Shanghai AI Laboratory
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
Jun 9, 2026Guozheng Li, Xiyan Fu, Yiwen GuoRL for Language Models Sample-Efficient RL
Southeast University · Nanyang Technological University · Independent Researcher
Rethinking the Divergence Regularization in LLM RL
Jun 8, 2026Jiarui Yao, Xiangxin Zhou, Penghui Qi +3RL for Language Models Policy Optimization
1Tencent Hunyuan · 2UIUC · 3NUS
One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems
Jun 8, 2026Mingzhe Li, Jing Xiang, Enguo Zhou +5Multi-Objective Reinforcement Learning RL for Language Models
Baichuan-M4: A Clinical-Grade Medical Agent System for Continuous Care
Jun 8, 2026Aiyuan Yang, Canbin Piao, Chengfeng Dou +25RL for Language Models Healthcare
Baichuan AI and THUBPM Group, Tsinghua University
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
Jun 7, 2026Jiashun Liu, Runze Liu, Xu Wan +3RL for Language Models Constrained RL
1Hong Kong University of Science and Technology · 2Zhejiang University · 3Tianjin University
Towards Long-Horizon Vessel Trajectory and Destination Forecasting with Reasoning Large Language Models
Jun 7, 2026Hongwei Wang, Miao Zhou, Fengde Wang +10RL for Language Models Vessel Trajectory Prediction
Institute of High Performance Computing (IHPC), A*STAR, Singapore 138632, Singapore. · The Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, Shanghai 201804, China. · Meituan Inc., Shenzhen, China. +3
CATPO: Critique-Augmented Tree Policy Optimization
Jun 6, 2026Ayush Singh, Umang Goyal, Ankur DahiyaRL for Language Models Reinforcement Learning
Vision and Language Group Indian Institute of Technology Roorkee Roorkee, Uttarakhand, India
LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection
Jun 6, 2026David Eje, Tanmay Sharma, Khush Patel +2RL for Language Models Online Anomaly Detection
Department of Computer Science, Innopolis University, Innopolis, Russia
Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation
Jun 6, 2026Boxuan Lyu, Haiyue Song, Zhi Qu +3RL for Language Models Machine Translation
Institute of Science Tokyo · Preferred Networks, Inc. · Nara Institute of Science and Technology
SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models
Jun 5, 2026Yuchen He, Baolong Bi, Shenghua Liu +7Multi-Objective Reinforcement Learning RL for Language Models
Institute of Computing Technology, Chinese Academy of Sciences · University of Electronic Science and Technology of China
Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning
Jun 5, 2026Pratik Jayarao, Chaitanya Dwivedi, Himanshu Gupta +5RL for Language Models LLM Tool Use
Amazon Stores Foundation AI
Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards
Jun 5, 2026Shihao Zhang, Xiaoman Wang, Yuan Liu +2RL for Language Models Reinforcement Learning
East China Normal University

Jun 21, 2026, cs.LGJ/K move · Enter open · S save

Weight-Space Geometry of Offline Reasoning Training

Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov, Karina Romanova

1Keenable AI · 2Accenture · 3Yandex

RL for Language Models Direct Preference Optimization Offline RL Weight-Space Analysis Language Model Distillation

Open paper AskPDF arXiv

Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone. We ask whether they are mechanistically distinct or converge to a similar weight update. Training six methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) on identical math rollouts from a single base model (Qwen3-4B) with attention-only LoRA, we analyze the resulting deltas via cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA. We observe: (i) SFT, RFT, and RIFT have nearly colinear weight deltas (cosine >= 0.97, top-1 principal angle ~7 deg median over 144 modules) and comparable GSM8K accuracy (87-88%, n=1319; pairwise McNemar p >= 0.15); (ii) DFT diverges further in direction than any reward-weighted method despite using the same data; (iii) Offline GRPO adds a substantial component orthogonal to the SFT direction (~67% globally, up to ~86% in late layers) while staying in the SFT loss basin; (iv) DPO sits in a near-orthogonal subspace, shows a mode-connectivity barrier, and collapses late-layer CKA to ~0.46. DPO also reaches the highest accuracy in our protocol on both GSM8K (93.5%, McNemar p < 10^-9 vs. each other method) and AIME26 (30.0% vs. 3.3-10.0%); its training uses a 10x smaller learning rate than the others (the standard convention), so the update-norm and accuracy gaps reflect loss-function and optimizer choices jointly, and a learning-rate-matched DPO comparison is left for future work.

Momentum

Related topics

Latest papers 271